{"id":"6f3df11b-4f48-481b-9ffb-c198c39c4e5a","arxiv_id":"2506.05883","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A vision-language driving planner with three-stage chain-of-thought reasoning and post-hoc trajectory smoothing reaches 7.74 RFS, second in the 2025 Waymo E2E challenge.","lead":"HMVLM is a driving system built on a vision-language model that reasons in three steps, understands the scene, chooses a maneuver, and predicts a trajectory, and it achieved second place in the 2025 Waymo end-to-end driving challenge. It combines known ideas, multi-view prompts, chain-of-thought reasoning, and trajectory smoothing, but without ablations or code, the size of each contribution is unclear.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablations or reported baseline support the causal claim that the three upgrades drive the 2.77% RFS improvement; the central claim is unverifiable from the paper.","rationale":"I agree with the reader's REJECT verdict, though my primary objection differs slightly from the stated weakest assumption. The reader flagged the unsupervised auto-labeled VQA data as the weakest link; I view the absence of ablations and any reported baseline as more directly fatal to the abstract's causal claim. The paper explicitly states that the three upgrades \"enable\" the 7.7367 RFS and the 2.77% margin, but Section 3 contains no experiment that varies any of these components. In the absence of a control condition, the score is compatible with many alternative explanations: the fine-tuned Qwen2.5-VL-3B could alone achieve the result, the auto-labeled data could be doing the work, or the gain could reflect leaderboard-specific metric weighting. This is not an internal inconsistency—the system may perform exactly as claimed—but it is a missing-evidence problem for the central attribution. The data-quality issue in Section 2.1 is real but secondary: even perfect annotations would not establish the causal role of the three upgrades. I also note the paper does report concrete implementation details (3000 iterations, LR 2e-5, Qwen2.5-VL-3B, Savitzky-Golay with a 25-degree keypoint threshold), which is useful for reproducibility; however, no code, model weights, or evaluation harness are released, so independent verification is impossible. The honest disposition is REJECT/UNCHANGED: the current document is a competition report rather than a self-contained scientific paper. A revision that adds (i) the public baseline score and per-category comparison, (ii) one ablation per proposed upgrade, and (iii) a human-validated sample of the VQA annotations would be sufficient to move toward acceptance. My proposed concrete test targets exactly this missing evidence.","tokens_in":4863,"tokens_out":4110,"duration_ms":34772,"concrete_test":"Request the authors' submission to the 2025 Waymo challenge (or rerun on the Waymo E2E validation set) and run three ablations: (a) feed all six surround views instead of the selective five, (b) replace the three-stage <DESC>/<DECI>/<TRAJ> prompt with a single flat trajectory prompt, (c) disable the Savitzky-Golay/spline refinement, keeping all other training settings fixed. Report RFS and ADE for each variant, plus the public baseline RFS and its per-category breakdown. If removing any one component leaves RFS unchanged within the leaderboard's reported precision, the causal claim in the abstract fails; if the baseline score is released and the 2.77% delta holds, the headline result is at least arithmetically correct.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that selective five-view prompting, multi-stage CoT, and spline smoothing enable HMVLM to surpass the public baseline by 2.77% (RFS 7.7367, 2nd place)—is untestable because the paper reports no controlled experiments isolating any of the three components. Section 3.2 (Table 1) lists only HMVLM's own scores; the public baseline score is never stated, and no per-category baseline comparison is given, so the \"2.77%\" cannot be checked even arithmetically. Section 2.1 mentions a \"systematic comparison\" of open-source VLMs for baseline selection but reports no results, so the choice of Qwen2.5-VL-3B as base is also unexamined. More importantly, no ablation removes the selective five-view prompting (e.g., using all six views), no variant strips the <DESC>/<DECI>/<TRAJ> special tokens to a single-stage prompt, and no post-processing is run with spline smoothing disabled. Without these three ablations, the paper's own attributive language (\"these upgrades enable HMVLM to achieve...\") is unsupported: the RFS could be driven entirely by fine-tuning the base VLM on Waymo data, by the auto-labeled VQA corpus, or by the public leaderboard's choice of metric. The paper is therefore a competition score report, not a verifiable scientific claim about the described mechanism. Secondary but related: the auto-labeled VQA training data (Section 2.1) come from an unnamed pre-trained LLM with no human verification, annotation statistics, or inter-annotator agreement; if trajectories or navigation instructions leak into the captions, the \"scene understanding\" stage could be trivial. This reinforces, rather than replaces, the need for ablations and a reported baseline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes HMVLM, a vision-language-model-based slow planner for end-to-end driving, built by fine-tuning Qwen2.5-VL-3B on part of the Waymo Open Dataset. Three contributions are claimed: selective five-view prompting with embedded ego-kinematics history, multi-stage chain-of-thought prompting using special tokens, and spline-based trajectory post-processing. The paper reports a Rater Feedback Score (RFS) of 7.7367, second place in the 2025 Waymo Vision-based End-to-End Driving Challenge, and an unspecified 2.77% improvement over the public baseline. The manuscript is a short competition-style report; it provides no ablations, no baseline scores, no error bars, and no annotation-quality statistics.","tokens_in":5196,"tokens_out":5115,"duration_ms":51319,"significance":"If the reported leaderboard result is taken at face value, the paper demonstrates that a 3B-parameter VLM fine-tuned solely on Waymo data can rank competitively on an external autonomous-driving benchmark. The use of an external benchmark is a genuine strength and avoids any circularity in evaluation. The structured-reasoning prompting scheme and the trajectory-refinement idea are plausible and potentially useful to practitioners. However, the scientific contribution is substantially weakened by the absence of controlled experiments: the paper's central claim that the three named upgrades 'enable' the reported performance is not testable from the material presented. No code, model checkpoints, or data are released, so the empirical result itself is not independently reproducible within the manuscript. The paper is best read as a challenge report rather than a mechanistic study.","major_comments":[{"comment":"The public baseline score is never stated, and no competitor scores are listed. Consequently, the claimed 2.77% improvement over the baseline and the second-place claim cannot be checked even arithmetically from the manuscript. The authors should report the baseline RFS (overall and per category) alongside the HMVLM values, together with the scores of the other ranked systems or at least the first- and third-place entries.","section":"Section 3.2, Table 1"},{"comment":"The central causal claim that selective five-view prompting, multi-stage CoT, and spline smoothing 'enable' the RFS result is unsupported by any ablation. No experiment removes the selective view selection, no variant strips the <DESC>/<DECI>/<TRAJ> structure to a single-stage prompt, and no post-processing run disables the smoothing. Without these variants, the reported RFS could be driven by fine-tuning the base VLM on Waymo data, by the auto-labeled VQA corpus, or by the constant-velocity completion step described in Section 3.1. The authors should add ablations, including a base Qwen2.5-VL-3B model fine-tuned without the proposed upgrades, and report RFS for each variant.","section":"Abstract and Sections 2.2-2.3"},{"comment":"The auto-labeled VQA training data are generated by an unspecified pre-trained LLM, and no human verification, annotation statistics, or quality-control measures are reported. Because these annotations are the training signal for the scene-understanding and driving-decision stages, their quality is load-bearing for the claimed reasoning behavior. The authors should identify the labeling model, report the number of samples, and provide a verification protocol or a sample of human-checked annotations.","section":"Section 2.1"},{"comment":"The inference procedure is stochastic (temperature 0.01, top-p 0.7, top-k 50), and the trajectories are post-processed with a constant-velocity completion step. Only a single leaderboard submission is reported, so the statement that HMVLM 'consistently maintains a leading position' is unsupported. The authors should report repeated inference runs with variance statistics, or at minimum state explicitly that the leaderboard score is a single run and avoid the word 'consistently'.","section":"Section 3.1 and Section 3.2"}],"minor_comments":[{"comment":"The abstract contains typographical spacing errors: '4 shistory' should be '4s history', and 'aRater Feedback Score' should be 'a Rater Feedback Score'.","section":"Abstract"},{"comment":"There are formatting and wording issues: 'V AD[4]' should be 'VAD[4]', and the sentence 'while Senna pairs a Senna[5] planner' repeats 'Senna'.","section":"Section 1"},{"comment":"The phrase 'fewer than < 0.003%' is redundant; 'occur in fewer than 0.003% of daily driving' would be sufficient.","section":"Section 1"},{"comment":"The title and abstract call the post-processing 'spline-based', but the method description in Section 2.3 says 'adaptive Savitzky-Golay filtering', which is polynomial smoothing rather than spline fitting. The terminology should be aligned.","section":"Section 2.3"},{"comment":"The 'systematic comparison' of open-source VLMs that is said to justify the choice of Qwen2.5-VL-3B is mentioned but no results are reported. A small table or a citation with the comparison results would make the selection verifiable.","section":"Section 2.1"},{"comment":"The phrase 'Although its ADE scores are slightly higher compared to some competitors' refers to competitor ADE values that are not reported in the table; either add those values or remove the comparison.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has the flavor of a workshop/competition report and would need substantial additional experimental evidence to meet the bar of a full journal paper. The missing ablations and baseline reporting are, in principle, obtainable, so I do not recommend outright rejection; however, the authors must be required to provide controlled experiments and annotation-quality details before the central claim can be evaluated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague – here's my read on 2506.05883.\n\nThe honest summary: this is a competition report, not a complete scientific paper. The headline claim — HMVLM gets RFS 7.74, 2nd place in the Waymo E2E challenge, 2.77% above the public baseline — is the one thing people will quote, and it's unverifiable from the text. The baseline score is never given, no competitor results appear anywhere, and there are no ablations isolating any of the three 'upgrades.' So the causal story ('these upgrades enable...') is not supported by evidence in the paper.\n\nWhat's genuine: the writing is clear, the fast–slow framing is honest (this is the slow branch only), and the three components — five-view prompting with 4s kinematics, multi-stage CoT with special tokens, and Savitzky-Golay-based trajectory smoothing — are all sensible engineering choices. The paper also reports per-category RFS numbers, which at least let you see where the model is strong (construction, single-lane) and weak (spotlight, cyclists). If the leaderboard result is real, it's a useful data point for the VLM-driving community. The model itself may be fine; the paper just doesn't give us the means to check.\n\nThe soft spots are serious. Missing baseline and competitor numbers make the 2.77% arithmetic impossible to verify. No ablation means we can't know whether the improvement comes from the three proposed techniques, from fine-tuning Qwen2.5-VL on Waymo data at all, or from the auto-labeled VQA corpus described in Sec 2.1. That auto-labeling pipeline is itself a concern: an unnamed pre-trained LLM generates the training annotations, with no human verification, no statistics, no quality control. If those labels leak trajectory or navigation information into the captions, the 'scene understanding' stage could be trivial. And the constant-velocity completion for short trajectories (Sec 3.1) is the kind of post-processing that can materially change ADE/RFS; its impact is unreported. No code, no data, no error bars. Single run on a leaderboard.\n\nWho is this for? Someone tracking VLM-based driving systems who wants a quick snapshot of a strong competition result. It doesn't deserve acceptance as a self-contained contribution, but it does deserve a serious referee: a top-2 leaderboard result in this area should get review, with the clear expectation that the authors add the baseline number, ablations, and annotation quality analysis. I'd send it out with those demands.","headline":"A top-2 Waymo leaderboard score that is unverifiable from the paper: no baseline, no ablations, and a causal claim that needs experimental support.","tokens_in":5808,"tokens_out":2696,"would_cite":false,"duration_ms":25377,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 3B vision-language model with staged reasoning and trajectory smoothing reaches a 7.7367 Rater Feedback Score, taking 2nd place in the 2025 Waymo E2E Driving Challenge.","keywords":["HMVLM","vision-language model","end-to-end autonomous driving","chain-of-thought prompting","Waymo Open Dataset","trajectory smoothing","fast-slow architecture","long-tailed driving scenarios"],"falsifier":"Audit a random sample of the auto-generated VQA pairs from Section 2.1 by comparing each stated driving decision with the ground-truth trajectory and navigation input that prompted it, or re-run the trained model on the Waymo validation split with the three-stage CoT tokens replaced by a flat prompt; a substantial mismatch rate or a flat RFS would indicate the claimed reasoning-driven gain is not coming from the stated mechanism.","tokens_in":4610,"feed_emoji":"🚗","tokens_out":11737,"duration_ms":94032,"temperature":0.7,"pith_summary":"This paper argues that a relatively small vision-language model can act as the deliberative planner in an autonomous-driving stack and handle long-tail scenes if its output is organized as structured reasoning. HMVLM feeds five surround-view images and a four-second history of ego velocity and acceleration into Qwen2.5-VL-3B, then forces the model to answer in the fixed order: scene understanding, driving decision, trajectory inference. A trajectory refiner based on Savitzky-Golay filtering removes jitter and sharp turns while preserving key points and endpoints. On the Waymo Open Dataset, the method scores 7.7367 on the Rater Feedback Score, taking second place in the 2025 Vision-based End-to-End Driving Challenge and beating the public baseline by 2.77%. The paper's point is that prompt structure and output post-processing can improve a VLM planner's generalization to rare, safety-critical events without a larger model.","feed_headline":"VLM planner with staged reasoning takes 2nd in Waymo driving challenge","feed_subtitle":"A three-stage reasoning chain maps scenes to decisions to waypoints; it beats the baseline by 2.77%.","key_machinery":"The central mechanism is the multi-stage chain-of-thought format: a fixed, special-token-delimited answer structure, <DESC START> ... <DESC END> <DECI START> ... <DECI END> <TRAJ START> ... <TRAJ END>, which compels the VLM to reason in the order Scene Understanding → Driving Decision → Trajectory Inference. This turns token generation into an interpretable chain and aligns training labels with the same structure. It is supported by two secondary mechanisms: selective five-view prompting that injects four seconds of ego velocity and acceleration into the text prompt, and an adaptive Savitzky-Golay trajectory refiner that removes late-stage jitter and sharp turns while preserving endpoints and key geometric features.","core_discovery":"On the paper's own terms, the discovery is that a vision-language trajectory planner can be made competitive on long-tail driving by tightening the prompt-design and output-format loop rather than by scaling the model. The core is a three-stage chain-of-thought format with dedicated special tokens that forces the model to emit a scene description, then a natural-language driving decision, then a sequence of BEV waypoints for the trajectory. Around that core, the authors add selective five-view prompting with an embedded four-second ego-kinematics history, and a trajectory-refinement stage that applies adaptive Savitzky-Golay filtering with z-score outlier removal, key-point preservation, and strict endpoint maintenance. The measured outcome is an RFS of 7.7367 on the Waymo E2E benchmark, with the highest category scores in construction (8.6663) and single-lane maneuvers (8.3563) and lower scores in spotlight (6.7269) and cyclist (7.3925) scenes.","pith_inferences":["The paper does not ablate the three upgrades separately, so an untested possibility is that the trajectory refiner, which directly modifies the scored waypoints, accounts for a large share of the 2.77% gain rather than the chain-of-thought format.","Because the auto-labeling pipeline gives the labeler the ground-truth trajectory and navigation input before asking for a driving decision, the VLM may learn to rationalize given trajectories instead of generating new plans; a random human audit of the VQA labels would test this.","A cheap external test of the recipe is to swap Qwen2.5-VL-3B for another open vision-language model and rerun the same five-view, three-stage, smoothing pipeline on the Waymo validation split.","The gap between the RFS ranking and the reported ADE values suggests that human raters may reward interpretable, human-aligned behavior more than raw trajectory accuracy, a trade-off the paper leaves unquantified."],"forward_implications":["HMVLM's ranking shows that a 3B-parameter vision-language planner can reach second place on long-tail driving through imitation learning alone, without reinforcement learning or a modular perception stack.","The three-stage reasoning output makes the planner's decisions inspectable, letting an operator read the scene summary and driving decision before accepting the trajectory.","The trajectory refiner is reported to remove late-stage oscillations and reduce collision events, which is the safety-relevant payoff of the smoothing stage.","Because HMVLM is designed as the slow branch of a fast-slow architecture, its natural deployment path is to pair it with a millisecond-latency fast controller, which the paper names as future work.","The category breakdown indicates that structured reasoning helps most in clearly defined contexts such as construction and single-lane maneuvers, while dynamic interactions like spotlight and cyclist scenes remain the harder tail."],"supporting_citations":[{"why":"Supplies the base vision-language model that HMVLM fine-tunes and extends.","marker":"[1]"},{"why":"Defines the slow-only VLM planning paradigm whose high compute cost HMVLM positions against.","marker":"[3]"},{"why":"Defines the fast controller side of the fast-slow split that HMVLM adopts.","marker":"[4]"},{"why":"Motivates the use of dedicated special tokens to structure the three-stage reasoning output.","marker":"[6]"},{"why":"Supplies the adaptive Savitzky-Golay filtering technique used in trajectory refinement.","marker":"[11]"},{"why":"Motivates the dual fast-slow architecture by attaching a language planner to a real-time stack.","marker":"[12]"},{"why":"Establishes chain-of-thought prompting as the reasoning technique behind the three-stage format.","marker":"[13]"},{"why":"Demonstrates step-by-step reasoning in vision-language models, which the authors apply to driving.","marker":"[14]"}],"fun_headline_variants":["HMVLM: multistage reasoning takes 2nd in Waymo driving","Scene-decision-trajectory VLM beats baseline by 2.77%","HMVLM's 3-stage CoT wins 2nd in Waymo E2E challenge","Multistage reasoning VLM: 2nd in Waymo, +2.77% over baseline","Slow-planner VLM with 3-stage CoT takes Waymo 2nd"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the auto-labeled VQA training examples, generated by an unspecified pre-trained LLM on the Waymo data without human verification or reported quality statistics, are accurate enough to teach the model the mapping from visual scenes to human-like driving decisions.","fun_headline_variants_meta":{"raw":{"variants":["HMVLM: multistage reasoning takes 2nd in Waymo driving","Scene-decision-trajectory VLM beats baseline by 2.77%","HMVLM's 3-stage CoT wins 2nd in Waymo E2E challenge","Multistage reasoning VLM: 2nd in Waymo, +2.77% over baseline","Slow-planner VLM with 3-stage CoT takes Waymo 2nd"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000858,"raw_usage":{"total_tokens":3731,"prompt_tokens":955,"completion_tokens":2776,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":2660}},"tokens_in":571,"tokens_out":2776,"duration_ms":19968,"temperature":1.0,"reasoning_tokens":2660,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:12:51.023923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit a random sample of the auto-generated VQA pairs from Section 2.1 by comparing each stated driving decision with the ground-truth trajectory and navigation input that prompted it, or re-run the trained model on the Waymo validation split with the three-stage CoT tokens replaced by a flat prompt; a substantial mismatch rate or a flat RFS would indicate the claimed reasoning-driven gain is not coming from the stated mechanism.","supporting_citations":[{"cited_title":"What is a savitzky-golay filter?[lecture notes]","cited_arxiv_id":null,"evidence_quote":"Supplies the adaptive Savitzky-Golay filtering technique used in trajectory refinement."}],"review_version":1}