{"id":"f8f10697-5d53-49be-916a-d750e45c9e0f","arxiv_id":"2607.23537","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Real adverse-weather camera–LiDAR–radar MCQs expose VLM failures from observability estimation through spatial grounding to trajectory safety, partially mitigated by SFT+RL.","lead":"The paper releases ObsDriveBench, a real multi-sensor VQA benchmark that tests driving VLMs under fog, rain, snow, and night along observability, spatial reliability, and risk-aware decisions. It shows current models degrade systematically and that normal-weather SFT plus adverse-weather RL partly closes the gap.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The headline diagnosis (observability awareness is the primary bottleneck; models \"know to be cautious but not what is risky\") rests on ground truth generated by the same VLM class being ranked — with the top-scoring model (Gemini-3.1-Pro) also serving as the label refiner.","rationale":"The reader's weakest_assumption identifies exactly this load-bearing point: label independence from the evaluated teacher models. My pass confirms and sharpens it with specifics the reader only gestured at — the refiner (Gemini-3.1-Pro) is also the leaderboard winner, the verification protocol is confirmation-structured rather than generative, and the fallback-action \"paradox\" has a plausible template-prior explanation. I also considered the open-loop MCQ proxy for trajectory safety and the promised-but-unshipped artifacts; both are real but secondary — the paper explicitly disclaims closed-loop claims (§2.2 discussion), and artifact release is a process condition the reader already flagged. The circularity concern is the one that, if it lands, moves the paper's central diagnosis from \"finding\" to \"artifact of the annotation pipeline.\" However, the reader's CONDITIONAL verdict already prices this in (\"accept-shaped contribution if ... clearer independence of labels from evaluated teachers [is] secured\"), and the cross-model consistency plus cross-dataset transfer results provide genuine independent support that the degradation phenomenon is real even if magnitudes are biased. So the correct action is to hold CONDITIONAL with the concrete re-annotation test as the gating check, not to escalate to REJECT or downgrade confidence.","tokens_in":17006,"tokens_out":1705,"duration_ms":146969,"concrete_test":"Independently re-derive the ground truth for a stratified ~800-MCQ test subset without any VLM in the loop: (a) compute LiDAR/radar observability labels programmatically from K-Radar raw data (point/return counts within each 3D box vs. a clear-weather baseline), and (b) have PhD annotators answer the spatial and risk MCQs from scratch, blind to the generated options' provenance. Then recompute Table 1 on this subset. If the observability-AVG column remains the lowest for most models and Gemini/Qwen3VL retain their ranking within ~5 points, the concern does not land; if the bottleneck ordering or the top-model ranking flips, the headline claim weakens materially. As a cheap companion check, run a text-only (no-image) baseline on fallback-action items; scores >85% would confirm those items measure option priors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — a systematic observability→spatial→decision capability gap, with observability awareness as the primary bottleneck (Table 1; §5.2.1) — is only as strong as the MCQ answer key. Per §3.2, observability labels and MCQ answers were drafted by InternVL3-38B and Qwen3VL-32B, refined by Gemini-3.1-Pro, then human-verified. Two specific risks follow. (1) Rank circularity: Gemini-3.1-Pro is the best off-the-shelf model in Table 1 (80.22 risk AVG, 88.04 spatial AVG) and Qwen3VL-32B is the best open model (79.71 risk AVG). A model that refines the answer key will trivially align with its residual priors; models with different annotation priors (e.g., Alpamayo-1.5 at 14.25 observability AVG) will look broken even where their answers are defensible. The \"bottleneck\" ordering could partly measure stylistic distance from the annotator ensemble rather than capability. (2) Confirmation-biased verification: annotators were shown the generated MCQs and asked to correct errors; reported pass rates (99.96% / 97.20% / 99.96%) measure agreement with VLM-drafted content, not independent derivation. A secondary concern reinforcing this: fallback-action scores >90% for nearly all models likely reflect that \"slow down and proceed with caution\" is the textually obvious option — a template prior, not a capability — which inflates the \"paradox\" contrast with risk identification. ObsDrive's jump to 99.0 on trajectory evaluation (vs ~50–66 for others) is also consistent with fitting the annotation policy of its own training pipeline rather than generalizable risk judgment. None of this shows the claims are false — degradation patterns are corroborated across 14 models and cross-dataset transfer (Tables 4–5) is real — but the quantitative ranking and the bottleneck diagnosis are not yet independently anchored.","agreement_with_reader":"agree"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper introduces ObsDriveBench, a multiple-choice benchmark for vision-language models in autonomous driving under real adverse weather, built on K-Radar sequences with synchronized camera, LiDAR, and radar inputs (LiDAR/radar rendered as BEV images). Tasks are organized along three capability dimensions — observability awareness, spatial reliability, and risk-aware decision-making — with ground truth produced by a pipeline in which InternVL3-38B and Qwen3VL-32B draft labels/questions, Gemini-3.1-Pro refines them, and PhD-level annotators verify them (reported pass rates 99.96%/97.20%/99.96%). Evaluation of 14 off-the-shelf models plus two trained variants (ObsDrive-SFT, ObsDrive-RL) shows: (i) observability awareness is the weakest dimension, (ii) spatial reliability is limited, (iii) models select conservative fallback actions (>90%) while remaining near-chance on risk identification and trajectory evaluation, and (iv) the authors' normal-weather SFT + adverse-weather GRPO training improves all three dimensions, with limited cross-dataset transfer checks on DriveBench and DriveLMM-o1.","tokens_in":17447,"tokens_out":3200,"duration_ms":61600,"significance":"If the ground truth holds up, this is a useful and well-scoped contribution: it is the first driving VLM benchmark built on real adverse-weather camera+LiDAR+radar data (K-Radar), it formalizes an object-level, per-modality observability annotation (O_m(o_i) in {0,1,2}) that other benchmarks lack, it evaluates 14 off-the-shelf systems with modality- and weather-level breakdowns (Tables 1–3), and it ships a train/test split that respects the official K-Radar sequence split rather than random frame splitting. The cross-dataset checks on DriveBench and DriveLMM-o1 (Tables 4–5) and the input-modality ablation (Table 3) are welcome, if modest. The observability-aware framing — diagnosing where along the perception→spatial→decision chain failures originate — is genuinely more informative than trajectory-only metrics, and the authors appropriately disclaim closed-loop validity (§2.2). The main risk to significance is not the design but the provenance of the answer key, which is addressable.","major_comments":[{"comment":"The headline claims (observability awareness is the primary bottleneck; the overall model ranking in Table 1) rest on an answer key drafted by InternVL3-38B and Qwen3VL-32B and refined by Gemini-3.1-Pro (§3.2, Stage 1/Stage 3) — the same model family being ranked. Gemini-3.1-Pro is the top off-the-shelf model in Table 1 (88.04 spatial AVG, 80.22 risk AVG) and Qwen3VL-32B is the top open model (79.71 risk AVG). Human verification mitigates this but does not eliminate it, because annotators were shown the generated MCQs and asked to correct errors: the reported pass rates (99.96%/97.20%/99.96%) measure agreement with VLM-drafted content, not independent derivation, and correction-on-review systematically under-detects plausible-but-wrong keys. A concrete, bounded fix: (i) have annotators independently answer a random subset (e.g., 300–500 test MCQs, stratified across the nine sub-tasks) wi","section":"§3.2 (Data Preparation); Table 1; §5.2.1"},{"comment":"The 'apparent paradox' (models adopt conservative fallback actions yet fail risk identification and trajectory evaluation) is partly an artifact of task design. The fallback-action template (Fig. 2) offers options of the form 'slow down and proceed with caution' vs. 'maintain speed' vs. 'speed up to pass quickly' — the safe option is identifiable from the question text alone, without any scene understanding. Scores >90% for nearly all models (Table 1) are therefore a template prior, not measured capability, and the contrast with risk identification (~50–68%) conflates task difficulty asymmetry with a capability dissociation. The claim 'models know they should be cautious, but do not fully understand what exactly is risky' would be much better supported if the fallback task included plausible conservative distractors (e.g., 'change lanes to overtake the slowing vehicle' vs. 'brake to a st","section":"§5.2.1, finding (3); §3.3 task family (3b)"},{"comment":"Trajectory evaluation is a binary yes/no task (chance = 50%). Most off-the-shelf models score 49–66%, i.e., near or modestly above chance, while ObsDrive-SFT/RL reach 98.09/99.00. A near-perfect score on a task where all 14 evaluated models sit near chance suggests the feasible/counterfactual trajectory pairs may carry systematic surface cues (e.g., counterfactuals drawn from a different distribution of curvature, length, or endpoint offset) that RL on the training split can exploit without genuine scene reasoning. The authors should (i) report the base rate of safe vs. unsafe labels in train and test, (ii) run a text-only or geometry-only control on the trajectory strings to measure how much of the gap is visual, and (iii) describe the counterfactual generation procedure (currently deferred to the supplement). Without this, the largest single-cell improvement in Table 1 is also the leas","section":"Table 1, Traj. Eval. column; §3.3 (3c)"},{"comment":"The transfer claims are stated strongly ('improvements transfer across scenes and corruption patterns', §5.4) but the effect sizes are small: Table 4 shows gains of roughly 0.1–1.5 points across 20 settings, and Table 5 gains of 0.93–2.62 points, with no variance estimates, no significance testing, and no reported number of evaluation samples. Uniform-sign improvements of this magnitude on multiple-choice accuracy are consistent with noise unless per-condition sample sizes are large. Please report per-setting sample counts and either confidence intervals or a paired significance test, and temper the language if the margins are not significant. This matters because §5.4 is the only evidence that ObsDrive's gains are not K-Radar-specific.","section":"§5.4, Tables 4 and 5"}],"minor_comments":[{"comment":"Table 2 omits several models present in Table 1 (InternVL3-8B, InternVL3-38B, Qwen3VL-8B, Qwen3VL-32B). Since Qwen3VL-32B is a label seed model, its per-weather and per-modality breakdown is of particular interest and should be included for completeness.","section":"Table 2"},{"comment":"Table 2 appears internally inconsistent for some rows: e.g., Perception-R1-7B scores 13.91/7.27/11.70/11.76/11.71 across weather conditions yet 47.43/46.07/45.75 across CAM/LIDAR/RADAR, and Alpamayo-1.5 shows the identical weather row (13.91/7.27/11.70/11.76/11.71) as Perception-R1-7B. Identical rows for two different models strongly suggest a copy error; please audit this table.","section":"Table 2"},{"comment":"The observability meta-annotation relies on VLM judgment of 3D boxes projected into BEV renderings, but the rendering details (color coding, point density thresholds, occlusion handling) are deferred to the supplement. Given that observability labels anchor the entire benchmark, a brief summary of the rendering convention and its sensitivity belongs in the main text of §3.2.","section":"§3.2, Stage 1"},{"comment":"Evaluation uses LLM-as-Judge [15] to select the chosen option (§5.1). For multiple-choice answers, exact-match parsing would be more reproducible; if the judge is needed for formatting failures, report the judge's agreement with exact-match on a sample and the judge model used.","section":"§5.1"},{"comment":"Fig. 3 shows the weather distribution of train vs. test frames/samples, but the proportions visibly differ between splits (e.g., normal-weather share). Please discuss whether per-weather results in Table 2 are confounded by differing condition mixtures, or confirm conditioning is per-weather as the column structure implies.","section":"§3.4, Fig. 3"},{"comment":"Notation: the object token format <CAM, x, y, BEV, X, Y> is used throughout (Fig. 1–2) but never formally defined; state the coordinate frames and units once in §3.3.","section":"§3.3"},{"comment":"Typos: 'muti-modal' (Fig. 2 caption), 'inherit split of source sequence' (§3.2, grammar), 'Will it possible intersect' (Fig. 2), spacing around bolded terms in the abstract and contribution list.","section":"Various"},{"comment":"The claim in §5.1 that 'all 14 off-the-shelf models in Table 1 are evaluated without access to the training split' is good practice, but Table 1 lists 15 rows including ObsDrive variants; please count and phrase precisely.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and the benchmark addresses a real gap. My main editorial concern is a mild conflict-of-interest structure in the evaluation: the strongest proprietary system evaluated (Gemini-3.1-Pro) is also part of the annotation pipeline. The authors do disclose this in §3.2, and human verification is performed, so this is not hidden — but it is unusual enough that I would insist on the rank-stability experiment in Major Comment 1 before publication. The cross-dataset transfer claims (Tables 4–5) are, in my reading, currently too weak to carry the language used about them, and the authors should either strengthen or soften them."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a solid benchmark paper for driving VLMs under real fog/rain/snow/night with camera–LiDAR–radar, not another synthetic-corruption suite. The distinctive move is object-level observability labels plus MCQs that walk the chain from “what can I see?” to spatial consistency to risk/trajectory judgment.\n\nWhat is new and done well: they build on K-Radar with synchronized three-sensor inputs, ~14k/13k capability-oriented questions, and a clean three-axis report. Table 1 across many proprietary, general, and driving models is the real payload. The weather/modality breakdown (Table 2), input ablations (Table 3), and limited transfer to DriveBench/DriveLMM-o1 (Tables 4–5) are the right extras. They also state plainly that MCQs diagnose intermediate failure, not closed-loop planning. The SFT-on-normal + GRPO-on-adverse baseline is a reasonable reference, not the main claim.\n\nSoft spots, in proportion: the stress-test on annotation circularity is real but not fatal. Labels and questions are drafted by InternVL/Qwen and refined by Gemini—the same class that tops Table 1—then human-verified. Pass rates mostly measure agreement with VLM drafts, so absolute ranks and the “observability is the bottleneck” ordering could partly reflect distance from the annotator ensemble. Fallback scores >90% for almost everyone also look template-easy and inflate the “cautious but not precise” paradox. ObsDrive’s huge jump on trajectory eval is consistent with fitting its own pipeline. None of that erases the multi-model degradation pattern or the external transfer bits. Artifacts are promised, not yet in hand; LLM-as-judge scoring is minor.\n\nWho it’s for: people building or evaluating driving VLMs/VLAs under weather and multi-sensor inputs. Citation pattern and related-work framing are fair relative to DriveLM, DriveBench, RoboDriveVLM. Math is light (O∈{0,1,2}, simple GRPO reward)—fine for this genre.\n\nI’d send it to peer review. Ask referees to push on label independence, a non-teacher re-annotation sample, and whether fallback/trajectory templates are too easy. Engage if you work this area; release of data/code is the practical hinge.","headline":"Useful real multi-modal adverse-weather VLM benchmark with a clear observability→spatial→decision frame; ranking strength is tempered by teacher-model labels, but the core degradation story still holds.","tokens_in":18512,"tokens_out":594,"would_cite":true,"duration_ms":19119,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Under real fog, rain, snow, and night, driving VLMs fail first at knowing what they can see, then at space and safe paths.","keywords":["adverse weather","observability awareness","multimodal VLMs","autonomous driving benchmark","spatial reliability","risk-aware decision-making","camera-LiDAR-radar","reinforcement learning"],"falsifier":"If strong models scored high and evenly on modality-specific observability, cross-modal comparison, 2D–3D grounding, risk identification, and trajectory safety on the held-out adverse-weather split—or if normal-weather SFT plus adverse-weather RL did not improve those scores relative to the untuned baseline—the claimed bottleneck chain and training fix would not hold.","tokens_in":18092,"feed_emoji":"🌫️","tokens_out":984,"duration_ms":23381,"temperature":0.7,"pith_summary":"This paper argues that the hard problem for vision-language models in bad weather is not just noisy sensors, but degraded environmental observability: camera, LiDAR, and radar become unreliable and disagree with one another. Existing driving benchmarks mostly stay in clear weather, use synthetic corruptions, or skip the intermediate steps between perception and action, so they cannot show where understanding breaks. ObsDriveBench is a real multi-modal benchmark built on synchronized camera, LiDAR, and radar with object-level observability labels and more than 14k training and 13k test multiple-choice questions along three linked capabilities: knowing what is observable, keeping spatial understanding reliable, and making risk-aware decisions. Across many models the same pattern appears—observability is the weakest link, spatial grounding stays limited, and models often pick cautious fallbacks while still misjudging where risk is and which trajectories are safe. A two-stage model trained with normal-weather supervised fine-tuning then adverse-weather reinforcement learning raises scores on all three dimensions, supporting the claim that explicit observability-aware training helps.","feed_headline":"Driving VLMs fail first at knowing what they can see","feed_subtitle":"A real fog-rain-snow benchmark ties broken observability to weak space sense and unsafe path checks","key_machinery":"ObsDriveBench’s three-dimension task chain—observability awareness, spatial reliability, and risk-aware decision-making—built from object-level observability meta-annotations (fully / partially / unobservable per camera, LiDAR, radar), scene descriptions, and capability-oriented multiple-choice questions over synchronized multi-modal inputs.","core_discovery":"Under real adverse-weather multi-modal inputs, current vision-language models show a systematic gap along an observability → spatial reliability → risk-aware decision chain: they struggle most to estimate what each sensor can reliably see and when sensors agree, maintain only limited spatial consistency under partial views, and often choose conservative actions while still failing risk localization and trajectory safety checks. ObsDriveBench makes that chain measurable, and normal-weather SFT plus adverse-weather RL improves all three capabilities.","pith_inferences":["Regulators and fleet tests that only score end trajectories may certify systems that still cannot say which sensor to trust in fog or snow.","The same observability-first chain could transfer to other multi-sensor robots (drones, marine, warehouse) where weather or dust breaks cross-modal agreement.","Closed-loop simulators that inject the benchmark’s observability labels into planning rewards would test whether MCQ gains become safer control.","If label generation still leans on the same VLM family being ranked, a fully human-only observability re-annotation subset would be the cleanest stress test of ranking validity."],"forward_implications":["Benchmarking only final trajectories will miss whether failure started in observability, spatial grounding, or risk assessment.","Training that never forces models to state what is unobservable will leave the primary bottleneck intact.","Conservative fallback behavior can look safe while risk localization and trajectory evaluation remain weak.","Multi-modal BEV inputs help some spatial and decision scores but do not by themselves fix observability estimation.","Released training annotations enable future driving VLMs to be supervised explicitly for observability under real weather."],"fun_headline_variants":["VLMs break first on what sensors can still see in weather","ObsDriveBench maps VLM failures from observability to risk","Adverse weather hits VLM observability harder than spatial sense","Fog-rain-snow inputs expose VLM sensor-agreement blind spots","ObsDrive lifts all three links in the weather failure chain"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The object-level observability labels and multiple-choice answers, first drafted by large models then human-checked, are accurate and independent enough to fairly rank the same class of models and to stand in for how real failures would propagate into driving decisions.","fun_headline_variants_meta":{"raw":{"variants":["VLMs break first on what sensors can still see in weather","ObsDriveBench maps VLM failures from observability to risk","Adverse weather hits VLM observability harder than spatial sense","Fog-rain-snow inputs expose VLM sensor-agreement blind spots","ObsDrive lifts all three links in the weather failure chain"]},"model":"grok-4.5","effort":"low","cost_usd":0.00202,"raw_usage":{"total_tokens":937,"prompt_tokens":846,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":20204000,"prompt_tokens_details":{"text_tokens":846,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":20,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":846,"tokens_out":71,"duration_ms":2405,"temperature":1.0,"reasoning_tokens":20,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T19:45:54.783908+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"If strong models scored high and evenly on modality-specific observability, cross-modal comparison, 2D–3D grounding, risk identification, and trajectory safety on the held-out adverse-weather split—or if normal-weather SFT plus adverse-weather RL did not improve those scores relative to the untuned baseline—the claimed bottleneck chain and training fix would not hold.","supporting_citations":[],"review_version":1}