{"id":"3f2235e8-f0ac-47e6-b249-c066b28612d2","arxiv_id":"2412.16410","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A fine-tuned CogVLM2 with chain-of-thought is applied to autonomous driving tasks, but its claimed performance gains are not supported by the reported quantitative results.","lead":"This paper tests whether a multimodal large language model, fine-tuned on a small question-answer dataset, can understand road scenes, predict events, and make driving decisions. It reports qualitative demos and a highway-env simulation comparing the approach with reinforcement learning and model predictive control, but the evidence is thin and partly contradictory.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I's highway row contradicts the central claim: the proposed method is worse than the prior CogVLM agent on inefficiency and average time, with no error bars or statistical support.","rationale":"The reader's REJECT verdict is justified, but the most decisive flaw is not only possible train/test overlap; it is that the only quantitative table directly contradicts the paper's strongest claim. The proposed method is beaten by its own baseline on Highway, and the metric granularity is internally inconsistent. Even if every qualitative demo used held-out images, the headline claim of significant performance improvements is not supported by the numbers. This is an internal inconsistency, not a disagreement with current consensus. The reader's memorization concern is plausible and also load-bearing for the qualitative results, but the Table I contradiction is more immediate and would remain even with a perfect train/test split. I therefore leave the REJECT verdict unchanged. The paper deserves credit for clearly describing the VQA dataset construction and the CoT decomposition, but those strengths do not repair the central quantitative evidence. The proposed test—re-running the 50-scenario evaluation with exact integer counts, error bars, and the action-decoder mapping—would settle whether the claimed improvements actually exist.","tokens_in":8734,"tokens_out":2591,"duration_ms":24495,"concrete_test":"Re-run Table I's evaluation with a fixed seed over exactly 50 scenarios, recording per-scenario integer counts of failures and ineffective decisions plus the full distribution of completion times. Check whether Ours outperforms [10] on the Highway row. If the highway comparison remains reversed, the 'significant performance improvements' claim is false. As part of the same check, require the action-decoder code (or a precise mapping from the third-part answer text to simulator actions) to be released so the Table I results can be reproduced from the described pipeline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of 'significant performance improvements' rests entirely on Table I, but the table's Highway row undermines it: Ours reports Fail 0%, Inefficiency 5%, Average Time 23.6s, while the earlier CogVLM agent [10] reports 0%, 2%, 22.3s. On the highway scenario the proposed method is strictly worse on two of three metrics and equal on the third. The paper provides no error bars, no number of repetitions, and no statistical test. Moreover, with '50 different traffic scenarios', the reported 2.3% inefficiency for the roundabout implies 1.15 ineffective events, which cannot arise from integer counts unless the metric is defined differently than stated. Separately, the simulation pipeline relies on an 'action decoder' mentioned only in Sec. III-A, with no implementation details or validation; without a demonstrated mapping from language outputs to simulator actions, the reported driving decisions are not reproducible. These are internal inconsistencies in the paper's quantitative evidence, not mere disagreements with prior benchmarks.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes fine-tuning the CogVLM2 multimodal large language model on a 100-image VQA dataset for autonomous driving, using a three-stage chain-of-thought (scene understanding, prediction, decision-making), and evaluates the resulting agent in Highway-env against RL, MPC, and the earlier CogVLM agent of reference [10]. The central claim is that this MLLM-based framework effectively executes AD tasks with limited resources and few-shot capability, and that MLLM-driven AD systems provide significant performance improvements. The evidence consists of qualitative examples for scene understanding, prediction, and decision-making, plus the simulation metrics in Table I.","tokens_in":8932,"tokens_out":2996,"duration_ms":27988,"significance":"If the central claim were established, a fine-tuned open VLM with chain-of-thought prompting could be a practically relevant interpretable AD decision agent. The construction of an AD-specific VQA dataset and the decomposition of decisions into three CoT stages are sensible ideas. However, the paper provides no held-out evaluation, no quantitative metrics for the understanding/prediction/decision tasks, no statistical support for the simulation results, and no code or data release. The simulation table is internally inconsistent and, on the highway scenario, the proposed method is worse than the prior method on two of three metrics. As presented, the evidence does not support the claimed significant improvements, and the contribution is not yet verifiable.","major_comments":[{"comment":"The highway row of Table I directly contradicts the paper's central claim of \"significant performance improvements\": for Highway, the proposed method reports Fail 0%, Inefficiency 5%, Average Time 23.6s, while the prior CogVLM agent [10] reports 0%, 2%, and 22.3s. The proposed method is strictly worse on Inefficiency and Average Time and equal on Fail. The paper gives no error bars, no number of repetitions, and no statistical test, so the quantitative superiority claim is unsupported. This is load-bearing for the first and third contribution bullets.","section":"Table I, Section IV-F"},{"comment":"The reported metric values are not consistent with the stated experimental setup. The paper says 50 traffic scenarios were tested, yet the roundabout row reports Inefficiency 2.3% and the merge row reports 4.7%, which cannot arise from integer counts of ineffective actions out of 50 trials unless the metric is defined differently than stated. The authors must define each metric precisely, report raw counts and denominators, and clarify how \"Inefficiency\" and \"Average Time\" are computed.","section":"Table I, Section IV-F"},{"comment":"The scene understanding, prediction, and decision-making results are supported only by qualitative example images. The paper never reports a train/test split for the 100-image VQA dataset, which was built with ChatGPT-4 and Gemini annotations and used to fine-tune the model. The demonstration images in Figures 4, 6, and 7 appear to come from the same BDD100K/KITTI source domains as the fine-tuning data. Without a held-out split, the qualitative successes may reflect memorization rather than driving competence. A quantitative evaluation on a disjoint test set, with per-task accuracy or agreement metrics, is needed.","section":"Sections IV-C, IV-D, IV-E, Figures 4, 6, 7"},{"comment":"The action decoder is mentioned only in Section III-A as a mechanism for converting the model's third-stage answer into simulator actions, but no implementation details, action-space definition, or validation are provided. Without a reproducible mapping from language outputs to Highway-env actions, the simulation results in Table I cannot be independently verified or compared with the prior CogVLM agent.","section":"Section III-A"}],"minor_comments":[{"comment":"The term \"Virtual Question Answering\" in the abstract and Section III-A should be \"Visual Question Answering,\" and the word \"wippers\" in Section IV-E should be \"wipers.\"","section":"Abstract, Section IV-E"},{"comment":"Section IV-E refers to \"the thought process outlined in Sec III-B,\" but the chain-of-thought construction is described in Section III-A; the cross-reference is inconsistent.","section":"Section IV-E versus Section III-A"},{"comment":"There is a citation formatting error: \"GPT-4 [ [31]\" should be \"GPT-4 [31].\"","section":"Section II-C"},{"comment":"The Highway-env environment is cited as reference [23] in Section IV-F but as reference [14] in the introduction; the numbering should be made consistent.","section":"Section IV-F"},{"comment":"The hardware description lists eight NVIDIA A100 GPUs but no training details such as fine-tuning epochs, learning rate, batch size, or use of parameter-efficient methods; this makes the fine-tuning procedure difficult to reproduce or assess.","section":"Section IV-A"}],"recommendation":"reject","confidential_remarks":"The paper is a preliminary workshop-style report rather than a journal-strength contribution. The central claim of significant performance improvement is not supported by the evidence as submitted: the simulation table is internally inconsistent, the highway comparison is unfavorable to the proposed method, and the qualitative demonstrations lack any held-out test. These problems are not merely presentational; they concern the validity of the core evaluation. Even with substantial revision the authors would need to add a proper train/test split, quantitative metrics, statistical support, and a reproducible action-decoder description, which goes well beyond a minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small extension of Chen and Lu's CogVLM driving agent [10], and the evidence does not back the 'significant performance improvements' claim. Worth a glance for the three-stage CoT framing, but not for the numbers.\n\nWhat's actually new: the swap from CogVLM to CogVLM2, a 100-image VQA dataset annotated with ChatGPT-4/Gemini over four iterations, and a chain-of-thought split into scene understanding, prediction, and decision. They also ran highway-env simulations against RL and MPC. That is a legitimate small extension of an existing pipeline, and the paper is honest about building on [10].\n\nThe paper's own table undermines it. On the highway scenario, the proposed method reports 5% inefficiency and 23.6s average time versus 2% and 22.3s for [10]. That is worse on both counts, yet the abstract claims 'significant performance improvements.' No error bars, no repetition counts, no statistical test anywhere. With 50 scenarios, the reported 2.3% inefficiency for the roundabout implies 1.15 ineffective events, which cannot come from integer counts unless the metric is defined differently than stated — and the paper does not say how.\n\nThe qualitative demos are nice pictures but not evidence. The 100-image pool from BDD100K/KITTI was used to fine-tune the model, and the demo images appear to come from the same domains, with no train/test split reported. So the 'understanding' results might reflect memorization rather than driving competence. Figure 7's with/without CoT comparison is anecdotal.\n\nThe action decoder is mentioned once in Section III-A with no implementation detail. Without knowing how language outputs become simulator actions, the driving decisions are not reproducible. No code or data is released either.\n\nThe related work is fine, and the limitations section is reasonable. But the load-bearing results are missing.\n\nThis paper is for readers who want a quick example of MLLM+CoT applied to AD tasks. It is not for someone who needs reliable numbers. I would not cite it, and I would not send it to peer review in this form. A serious referee would need held-out splits, error bars, a clear action-decoder description, and a corrected table before the claims could be evaluated. If the authors fix those, a much shorter empirical note could be useful.","headline":"Thin extension of an existing CogVLM driving agent; the quantitative evidence contradicts the central claim and lacks any statistical support.","tokens_in":9465,"tokens_out":2412,"would_cite":false,"duration_ms":20623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned multimodal language model can act as an autonomous-driving decision agent.","keywords":["autonomous driving","multimodal large language model","chain-of-thought","visual question answering","fine-tuning","CogVLM2","highway-env","scene understanding"],"falsifier":"Build a strict evaluation where the 100 VQA samples are split into train and test sets, hold out the demo images, and measure scene-understanding accuracy and simulation failure probability; if performance collapses on held-out images from the same domains, the claim of generalizable driving competence is falsified.","tokens_in":8527,"feed_emoji":"🚗","tokens_out":5802,"duration_ms":45284,"temperature":0.7,"pith_summary":"This paper argues that a multimodal large language model, fine-tuned on just 100 visual question-answer examples drawn from BDD100K and KITTI, can serve as a practical autonomous-driving decision agent. The key move is to decompose driving decisions into three chain-of-thought stages—scene understanding, prediction, and decision—so the model produces interpretable reasoning before choosing an action. The paper reports that this fine-tuned CogVLM2 agent achieves near-zero failure rates and lower inefficiency than reinforcement learning and model-predictive-control baselines in highway-env simulations, while also handling corner cases with limited compute. A sympathetic reader would care because this points toward local, interpretable, few-shot deployment of language-model agents in vehicles rather than cloud-scale black boxes.","feed_headline":"Fine-tuned multimodal LLM outperforms RL and MPC in driving sims","feed_subtitle":"Three-stage chain-of-thought gives interpretable predictions and decisions from a model fine-tuned on only 100 images.","key_machinery":"The central machinery is the three-stage chain-of-thought that splits driving decisions into scene understanding, prediction, and decision, paired with a 100-image Visual Question Answering dataset built from BDD100K and KITTI to fine-tune CogVLM2. The CoT is what produces the step-by-step interpretable output that the paper credits with reducing model illusions and improving attention; the VQA dataset is what adapts a general-purpose MLLM to the driving domain. The action decoder reads the final stage of the answer to translate the model's textual output into driving actions.","core_discovery":"On its own terms, the paper's central claim is that an MLLM-driven framework effectively executes AD tasks with limited computer resources, few shots, multi-modality, and complex scenarios. Specifically, a CogVLM2 model fine-tuned on a hand-annotated and LLM-refined VQA dataset, guided by a three-stage chain-of-thought, can understand road scenes, predict other agents' behavior, and choose driving actions. The evidence is qualitative demonstrations on real road photos plus quantitative comparisons in three highway-env scenarios (intersection, roundabout, highway), where the proposed agent shows lower failure likelihood than RL, MPC, and the earlier [10] CogVLM agent in most settings.","pith_inferences":["Scaling the VQA dataset beyond 100 images would test whether the reported gains come from fine-tuning itself or from the CoT structure; the paper leaves this open.","The paper's demonstrations are still-image only, so a natural extension is to test whether the same CoT decomposition transfers to video-based or LiDAR-fused scene understanding.","Because the VQA dataset is English-only, the framework's claims are tied to English-language instruction following, a point the paper acknowledges in its limitations."],"forward_implications":["If the central claim holds, a small fine-tuned MLLM with a three-stage CoT can be deployed on an onboard GPU-like workstation rather than only in the cloud.","The three-stage decomposition makes decisions auditable: a driver or inspector can read the model's scene understanding, prediction, and final choice step by step.","In the reported simulations, the MLLM agent achieves a 0% failure rate in intersection and roundabout scenarios, compared with 4–10% for the RL and MPC baselines.","The paper positions this as an initial step toward safe, few-shot, local, and interpretable MLLM-driven AV systems."],"supporting_citations":[{"why":"Supplies the earlier CogVLM driving agent that this paper updates and compares against.","marker":"[10]"},{"why":"One of the two source datasets for the images used to build the VQA fine-tuning set.","marker":"[35]"},{"why":"The other source dataset for the VQA fine-tuning images.","marker":"[36]"},{"why":"The highway-env simulation environment used for the RL/MPC/ours comparisons.","marker":"[14]"},{"why":"Motivated the hand-annotation approach used to bootstrap the VQA dataset.","marker":"[38]"}],"fun_headline_variants":["MLLM agent beats RL, MPC in driving with chain-of-thought","Fine-tuned multimodal LLM steers safer than RL and MPC","Chain-of-thought MLLM drives with fewer failures than baselines","100 images fine-tune LLM to outperform RL and MPC in sims","Multimodal LLM uses CoT to outdrive RL and MPC in tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the qualitative demonstrations in Figures 4, 6, and 7 generalize beyond the specific 100-image training set, since the paper reports no train/test split and the demo images come from the same source datasets as fine-tuning.","fun_headline_variants_meta":{"raw":{"variants":["MLLM agent beats RL, MPC in driving with chain-of-thought","Fine-tuned multimodal LLM steers safer than RL and MPC","Chain-of-thought MLLM drives with fewer failures than baselines","100 images fine-tune LLM to outperform RL and MPC in sims","Multimodal LLM uses CoT to outdrive RL and MPC in tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000711,"raw_usage":{"total_tokens":3132,"prompt_tokens":811,"completion_tokens":2321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":427,"completion_tokens_details":{"reasoning_tokens":2222}},"tokens_in":427,"tokens_out":2321,"duration_ms":14373,"temperature":1.0,"reasoning_tokens":2222,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:36:43.245456+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a strict evaluation where the 100 VQA samples are split into train and test sets, hold out the demo images, and measure scene-understanding accuracy and simulation failure probability; if performance collapses on held-out images from the same domains, the claim of generalizable driving competence is falsified.","supporting_citations":[{"cited_title":"An advanced driving agent with the multimodal large language model for autonomous vehicles,","cited_arxiv_id":null,"evidence_quote":"Supplies the earlier CogVLM driving agent that this paper updates and compares against."}],"review_version":1}