{"id":"c71c1191-0644-44c4-9ee9-dfd3868db934","arxiv_id":"2412.09951","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Joint training on driving-knowledge QA (LingoQA, DRAMA) and CARLA trajectory data yields a VLM (WiseAD) that improves closed-loop driving score by 11.9% over trajectory-only training.","lead":"WiseAD is a vision-language model that combines driving-related question answering with trajectory planning, trained on LingoQA, DRAMA, and CARLA data. It reports large gains in CARLA closed-loop driving metrics, raising driving score from 62.46 to 69.88 and route completion from 83.47 to 93.79.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Knowledge gain in Table 2 may be largely an inference-prompt effect: Table 6 shows the attention prefix alone moves DS by +3.0 and RC by +8.44, yet the Carla-only baseline's prompt condition is not reported.","rationale":"The reader's weakest_assumption correctly identifies the attention-prompt confound as the most load-bearing issue. My analysis of the paper's own numbers strengthens this: Table 6 quantifies a prompt-induced swing of 8.44 RC and 3.0 DS on the full model, which is a large fraction of the headline gains in Table 2 (10.32 RC and 7.42 DS). The manuscript does not clarify the prompt condition of the Carla-only baseline, and the prompt is described in Sec. 3.2 as a mechanism to 'leverage learned knowledge,' which assumes the causal effect it aims to demonstrate. A second confound, data diversity versus semantic knowledge, is also present but secondary; the prompt issue is sharper and directly testable from the paper's own tables. I therefore agree with the reader's CONDITIONAL verdict: the claim is plausible but requires controlled re-evaluation. I recommend keeping the verdict UNCHANGED (still CONDITIONAL), pending the concrete test I propose, because the paper does provide independent evidence on knowledge QA benchmarks and some driving gains even without the prompt, so rejecting outright would be too strong.","tokens_in":12456,"tokens_out":4107,"duration_ms":47870,"concrete_test":"Run two controlled evaluations on the same Town05 routes: (1) Evaluate the Carla-only baseline from Table 2 both with the attention-prefix prompt ('Pay attention to your surroundings and do not break traffic rules...') and without it, reporting DS and RC for each. (2) Train a control model with the same joint recipe and inference prompt, but replace LingoQA+DRAMA with an equal-size non-driving or generic visual QA set (e.g., VQAv2 or COCO captions) while keeping the Carla trajectory data. If the Carla-only baseline with the prompt scores near 66.89 DS / 85.35 RC (the full model without prompt), then the knowledge data contribute little beyond the prompt, and the 11.9%/12.4% claim would need to be revised to separate prompt and knowledge effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that joint training with driving knowledge improves closed-loop driving by 11.9% DS and 12.4% RC (abstract, Sec. 4.4). The comparison is between a Carla-only baseline (DS 62.46, RC 83.47) and the full knowledge-augmented model (DS 69.88, RC 93.79). However, Table 6 ablates the attention-prefix prompt on the full model and reports a drop from 69.88 to 66.89 DS (−3.0) and from 93.79 to 85.35 RC (−8.44) when the prompt is removed. Thus, on the full model, the prompt alone explains 8.44 of the 10.32-point RC gain and 3.0 of the 7.42-point DS gain relative to the Carla baseline. The paper never states whether the Carla-only baseline in Table 2 was evaluated with or without the attention prefix. If it was evaluated without, then the headline gain conflates the training-data intervention (knowledge) with an inference-time intervention (prompt), and the residual knowledge-only contribution shrinks to +4.43 DS and +1.88 RC (full model without prompt vs. baseline). Even if the baseline used the prompt, the ablation shows the prompt is a large, uncontrolled factor; and the design still cannot separate semantic driving knowledge from generic instruction-following or auxiliary-task regularization, since no control training set (e.g., non-driving QA) is included. The paper's own Sec. 3.2 describes the prefix as a 'trigger to leverage learned knowledge,' which presupposes the very causal role that needs to be established. This ambiguity is load-bearing because the paper's title, abstract, and Sec. 4.4 make a knowledge-specific causal claim, not merely a recipe-level improvement claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes WiseAD, a vision-language model for end-to-end autonomous driving that is jointly trained on CARLA trajectory data and driving-knowledge QA datasets (LingoQA and DRAMA). The model outputs textual waypoints for closed-loop control and is evaluated on the CARLA Town05 benchmark plus in-domain and out-of-domain knowledge QA sets (BDDX, DriveLM, HAD). The central claims are that expanding driving knowledge improves closed-loop driving, with an 11.9% driving-score gain and a 12.4% route-completion gain over a Carla-only baseline, while also reducing critical accidents, and that the model achieves state-of-the-art zero-shot closed-loop performance.","tokens_in":12872,"tokens_out":3234,"duration_ms":37379,"significance":"The question investigated is meaningful: whether explicit driving-knowledge training can improve closed-loop trajectory planning in a vision-language model. The paper has real strengths: it uses external benchmarks (BDDX, DriveLM, HAD) for knowledge evaluation, which reduces circularity; it includes closed-loop evaluation in CARLA with standard metrics; and it reports a practical joint-training recipe that avoids the catastrophic forgetting observed in sequential fine-tuning. The main trend in Table 2—that adding DRAMA suggestion/risk data improves driving and reduces accidents—is plausible and worth pursuing. However, the headline causal claim is not yet established because the reported comparisons conflate the knowledge intervention with an inference-time prompt and with increased auxiliary training data.","major_comments":[{"comment":"The central causal claim is confounded by the inference-time attention prefix. The headline comparison is between a Carla-only baseline (DS 62.46, RC 83.47) and the full knowledge-augmented model (DS 69.88, RC 93.79), but the paper never states whether the baseline was evaluated with or without the prefix introduced in Sec. 3.2. Table 6 shows that removing the prefix from the full model degrades DS from 69.88 to 66.89 and RC from 93.79 to 85.35. If the baseline was evaluated without the prefix, the residual knowledge-only contribution is +4.43 DS and +1.88 RC, not the advertised 11.9% and 12.4%. The authors must report the Carla-only baseline under the same prompt condition and, ideally, add a full-model row without the prefix to Table 2.","section":"Sec. 4.4, Table 2 and Sec. 4.6, Table 6"},{"comment":"The joint-training design changes multiple factors at once: the addition of LingoQA and DRAMA QA data, the data-mixing ratio, and the auxiliary task losses. There is no control condition that isolates semantic driving knowledge from generic multi-task regularization or instruction following. A control trained on an equal-size non-driving QA dataset, or on LingoQA/DRAMA with shuffled answers, is needed to support the abstract's attribution of driving improvements to 'fundamental driving knowledge' rather than to increased data diversity or auxiliary-task regularization.","section":"Sec. 4.4, Table 2"},{"comment":"The accident counts are non-monotonic and no uncertainty estimates are given. Adding LingoQA alone increases red-light infractions from 2.60 to 3.79 and vehicle collisions from 2.35 to 5.60; the authors attribute this to extended driving distances, but no per-route exposure measure or variance is reported. Without multiple random seeds or confidence intervals, the claim of a 'significant drop in all critical driving accidents' in the final row is not statistically supported.","section":"Sec. 4.4, Table 2"},{"comment":"The closed-loop comparison against TransFuser, NEAT, Roach, ST-P3, and VAD is uncontrolled: these methods use different input views (multiview or BEV), different training data regimes, and different model architectures. The 'state-of-the-art' and 'best zero-shot' claims therefore cannot be established from Table 5. A matched comparison—same routes, same sensor input configuration, and comparable training-data budget—or an explicit statement of the evaluation protocol and its differences is required.","section":"Sec. 4.5, Table 5"}],"minor_comments":[{"comment":"The heading 'Methodolgy' in Section 3 is a typo and should read 'Methodology'.","section":"Section heading"},{"comment":"Column headers such as 'Route compl' and 'Infrac.' should be spelled out or defined in the caption, and the units of infraction counts should be clarified.","section":"Table 4 and Table 5"},{"comment":"The dataset name is written inconsistently as 'BDDX' and 'BDD-X'; please choose one form and use it throughout.","section":"Sec. 3.2"},{"comment":"References [21] and [22] appear to cite the same arXiv paper; please check and remove the duplicate.","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid empirical recipe paper whose headline causal claim—knowledge improves closed-loop driving—is confounded by an inference-time prompt. The trend is visible and the paper does several things well, but the central attribution needs rework.\n\nWhat's actually new: WiseAD's joint training of driving QA (LingoQA, DRAMA) with textual trajectory planning, plus the incremental study of knowledge depth and breadth on closed-loop metrics. That integration is not in prior work. The paper also evaluates zero-shot on BDDX, DriveLM, and HAD, which is honest external evaluation, and the incremental DRAMA risk/suggestion addition reducing collisions is the most convincing result.\n\nSoft spots: the attention-prefix prompt is load-bearing. Table 6 shows removing it drops DS by 3.0 and RC by 8.44 on the full model. The paper never states whether the Carla-only baseline in Table 2 was run with or without that prompt. If without, a large chunk of the headline 11.9% DS and 12.4% RC gains could be prompt-driven, not knowledge-driven; the residual knowledge contribution shrinks to roughly +4.4 DS and +1.9 RC. Even if the baseline used the prompt, the design cannot separate semantic driving knowledge from generic instruction-following or auxiliary-task regularization, since there is no control training set (e.g., non-driving QA). No error bars or multiple seeds either, and the SOTA comparison in Table 5 mixes models with different input views, data, and compute, so the \"best\" claim is weak.\n\nWhat's good: the paper is clearly written, the incremental ablations are a sensible way to probe data mixing, and the external knowledge benchmarks reduce circularity. The authors are transparent about two-stage training causing catastrophic forgetting.\n\nWho it's for: people working on VLM-based driving or closed-loop evaluation. It is a useful recipe and an honest baseline for future work, but read the causal claim skeptically.\n\nRecommendation: deserves a serious referee, not a desk reject, but it needs major revision. Ask for the baseline with and without the prompt, a non-driving QA control, multiple seeds with error bars, and a tempered SOTA claim. Code and data release would also help. I would accept it for review and request those experiments.","headline":"Plausible recipe paper, but the knowledge-augmentation claim is confounded by an unaccounted inference-time prompt; needs controlled ablations and data release.","tokens_in":13378,"tokens_out":3219,"would_cite":true,"duration_ms":32783,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding driving knowledge to a vision-language model cuts critical accidents in closed-loop driving, raising the driving score by 11.9%.","keywords":["vision-language model","autonomous driving","closed-loop evaluation","driving knowledge","trajectory planning","CARLA simulator","joint training","zero-shot driving"],"falsifier":"A controlled experiment that fixes the inference prompt, the total training data volume, and the dataset composition, while varying only whether the QA pairs contain semantically meaningful driving knowledge versus shuffled or templated non-driving text, would settle whether the gains are knowledge-driven. If a model trained on non-driving QA pairs plus the same prompt achieves the same driving score and accident reduction, the knowledge claim fails.","tokens_in":12297,"feed_emoji":"🚗","tokens_out":1549,"duration_ms":18062,"temperature":0.7,"pith_summary":"WiseAD is a vision-language model for autonomous driving that is jointly trained on both driving knowledge question-answer data and trajectory planning data. The paper claims that expanding the depth and breadth of this fundamental driving knowledge—covering scene understanding, risk analysis, action justification, and suggestions—consistently improves closed-loop driving performance in the CARLA simulator. Compared with training only on the CARLA planning data, the knowledge-augmented joint training improves the driving score by 11.9% and route completion by 12.4%, with notably fewer critical accidents such as collisions and red-light violations. The authors also report state-of-the-art zero-shot driving performance and superior knowledge evaluation results over several larger VLMs.","feed_headline":"Driving knowledge in a VLM cuts accidents by 11.9%","feed_subtitle":"Training a vision-language model on driving QA and planning data raises the CARLA driving score from 62.5 to 69.9.","key_machinery":"The key machinery is joint learning on a mixed dataset of driving knowledge QA pairs (LingoQA, DRAMA) and trajectory planning data from CARLA, in roughly equal proportions, on top of a pre-trained MobileVLM backbone with a frozen CLIP ViT encoder and a learnable projector. The trajectory is represented textually as a sequence of waypoints, and an attention-prefix prompt at inference explicitly cues the model to apply its learned knowledge. This mechanism allows the model to learn both 'how' to plan (imitation of the autopilot) and 'why' (the reasoning behind driving decisions), and the authors argue that this rationale-aligned training is what reduces accidents.","core_discovery":"The central claim is that injecting explicit, structured driving knowledge into a vision-language model through joint training with planning data makes the model a safer and more competent end-to-end driver. The discovery is not just that a VLM can answer driving questions, but that the learned knowledge transfers to the trajectory planning head: as the authors add knowledge domains (LingoQA for reasoning and perception, then DRAMA suggestions, then DRAMA risk analysis), the CARLA closed-loop driving score and route completion rise monotonically, while critical accident counts drop. This is evidenced by Table 2, where the final configuration reaches a 69.88 driving score and 93.79 route completion, up from 62.46 and 83.47 for the Carla-only baseline. The paper further shows that a simple attention-prefix prompt, 'Pay attention to your surroundings and do not break traffic rules,' is a key trigger for leveraging this knowledge, contributing a 3-point driving-score gain (Table 6).","pith_inferences":["The paper's causal attribution is correlational: joint training changes several factors at once (data diversity, instruction-following, and the inference prompt), so the observed gains might partly come from better instruction following or regularization, not from the semantic content of driving knowledge. A controlled experiment that holds the prompt constant and varies only the knowledge content","A testable extension is to measure whether the knowledge transfer holds under domain shift beyond CARLA, e.g., in a different simulator or in real-world closed-loop settings, where the gap between simulation and reality could be large.","The attention-prefix ablation suggests that the prompt itself contributes a large portion of the driving-score gain; an analysis of what the prompt changes internally (e.g., attention maps or hidden states) could reveal whether it actually retrieves risk knowledge or merely biases output formatting."],"forward_implications":["If the claim holds, knowledge-augmented joint training becomes a simple, scalable recipe for improving closed-loop driving safety in VLM-based planners, without changing the network architecture.","The monotonic improvement with added knowledge breadth suggests a positive scaling law: adding more diverse driving knowledge domains should further reduce accidents and improve route completion.","The attention-prefix prompt result implies that textual prompting can act as a switch to activate learned knowledge, which could be a general mechanism for steering VLM behavior in deployment.","The zero-shot generalization to BDD-X, DriveLM, and HAD indicates that the learned knowledge is transferable across datasets and task formats, supporting the idea that the driver is not overfitting to a single simulator."],"supporting_citations":[{"why":"LingoQA provides the primary driving knowledge QA data (reasoning, object recognition, action justification, scene description) and the Lingo-Judge evaluation metric.","marker":"[28]"},{"why":"DRAMA supplies the risk analysis and driving suggestion QA pairs that the paper shows reduce critical accidents when added.","marker":"[27]"},{"why":"LMDrive sets the closed-loop evaluation protocol in CARLA and the data collection configuration with a rule-based agent, plus the two-PID controller for waypoint tracking.","marker":"[36]"},{"why":"MobileVLM is the base vision-language model and training framework that WiseAD builds upon, providing the pretrained weights and architecture.","marker":"[8]"},{"why":"The CARLA simulator provides the closed-loop evaluation environment and the scoring metrics (driving score, route completion, infraction score).","marker":"[11]"},{"why":"BDD-X is used as a zero-shot evaluation set for action justification, testing whether the learned knowledge transfers to unseen action reasoning.","marker":"[17]"},{"why":"DriveLM supplies the zero-shot object recognition evaluation, with a graph-structured QA format that differs from the training distributions.","marker":"[37]"},{"why":"HAD provides the zero-shot driving attention evaluation, grounding the 'What should the driver pay attention to?' question type.","marker":"[18]"}],"fun_headline_variants":["Knowledge-aware VLM boosts CARLA driving score 11.9%","WiseAD: VLM driving knowledge lifts CARLA score 11.9%","Adding driving QA to VLM raises route completion 12.4%","VLM learned road rules boost end-to-end driving by 11.9%","Knowledge injection into VLM improves CARLA by 11.9%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That the observed closed-loop driving improvements come from the semantic content of the added driving knowledge, rather than from confounds such as the inference-time attention prompt, increased data diversity, or the regularization effect of joint training.","fun_headline_variants_meta":{"raw":{"variants":["Knowledge-aware VLM boosts CARLA driving score 11.9%","WiseAD: VLM driving knowledge lifts CARLA score 11.9%","Adding driving QA to VLM raises route completion 12.4%","VLM learned road rules boost end-to-end driving by 11.9%","Knowledge injection into VLM improves CARLA by 11.9%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000901,"raw_usage":{"total_tokens":3884,"prompt_tokens":953,"completion_tokens":2931,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":2830}},"tokens_in":569,"tokens_out":2931,"duration_ms":23447,"temperature":1.0,"reasoning_tokens":2830,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:31:57.221197+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment that fixes the inference prompt, the total training data volume, and the dataset composition, while varying only whether the QA pairs contain semantically meaningful driving knowledge versus shuffled or templated non-driving text, would settle whether the gains are knowledge-driven. If a model trained on non-driving QA pairs plus the same prompt achieves the same driving score and accident reduction, the knowledge claim fails.","supporting_citations":[{"cited_title":"Drama: Joint risk localization and captioning in driving","cited_arxiv_id":null,"evidence_quote":"DRAMA supplies the risk analysis and driving suggestion QA pairs that the paper shows reduce critical accidents when added."},{"cited_title":"Lmdrive: Closed-loop end-to-end driving with large language models","cited_arxiv_id":null,"evidence_quote":"LMDrive sets the closed-loop evaluation protocol in CARLA and the data collection configuration with a rule-based agent, plus the two-PID controller for waypoint tracking."},{"cited_title":"Carla: An open urban driving simulator","cited_arxiv_id":null,"evidence_quote":"The CARLA simulator provides the closed-loop evaluation environment and the scoring metrics (driving score, route completion, infraction score)."},{"cited_title":"Textual explanations for self-driving vehicles","cited_arxiv_id":null,"evidence_quote":"BDD-X is used as a zero-shot evaluation set for action justification, testing whether the learned knowledge transfers to unseen action reasoning."},{"cited_title":"Grounding human-to-vehicle advice for self- driving vehicles","cited_arxiv_id":null,"evidence_quote":"HAD provides the zero-shot driving attention evaluation, grounding the 'What should the driver pay attention to?' question type."}],"review_version":1}