{"id":"f062176c-908a-4ea1-a88d-eb421b801f5f","arxiv_id":"2506.09557","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AD^2-Bench is a new adverse-weather driving benchmark with hierarchical chain-of-thought annotations and LLM-based quality metrics; 12 MLLMs all scored below 60%.","lead":"The authors built a new benchmark, AD^2-Bench, that tests how well multimodal AI models reason step by step about driving scenes in rain, fog, snow, sandstorms, and at night. It has over 5,400 human-annotated reasoning chains on 10,000 real road images, and all tested models score below 60%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT-4o-based CoT metrics (SACS/SLPS/ORCS/DJSS) are never validated against human judgment, so the headline 'below 60%' and 'CoT improves' findings may be judge artifacts.","rationale":"The reader's weakest assumption is precisely the load-bearing gap: no evidence that GPT-4o's 1-10 judgments match human ratings of reasoning correctness, logical progression, coherence, or justification. I agree, and I would keep the verdict CONDITIONAL rather than escalate because the underlying asset (adverse-condition CoT data with multi-level prompts) is plausibly useful and the claims are testable. The concern is not that the benchmark is worthless; it is that the quantitative conclusions rest on an unvalidated judge. The annotation pipeline's use of LLMs to validate/aggregate ground truth makes this a loop, not merely a missing correlation table, and the paper's own inconsistent sample counts (5,406 vs. 5,046) and 'purely manual' vs. 'post-processed by Gemini' statements (Appendix A.13 vs. Section 3.1) indicate that the reported numbers need audit. A human-rater agreement study on a modest sample would settle the issue directly. If the study shows high agreement and stable rankings, the central claims stand; if not, the paper needs to either recalibrate metrics or soften the claims. No change to the reader's CONDITIONAL verdict is needed, but the condition should explicitly include metric validation, not just code/data release.","tokens_in":31108,"tokens_out":5463,"duration_ms":59437,"concrete_test":"Draw a stratified sample of 200 CoT responses (covering all 12 models and all four metrics) and have three domain-expert annotators score each step/chain with the same rubrics and 1-10 scale defined in Section 3.3. Compute per-metric Spearman correlations and exact-agreement rates between GPT-4o scores and the human-expert consensus, and recompute Table 3's model ordering and CoT-gain column using only human scores. If correlation is below roughly 0.7, or if the ordering/CoT-gain conclusions change materially, the headline findings are judge artifacts and the metrics require recalibration. A weaker but useful secondary check is to re-score the same outputs with a different model judge (e.g., GPT-4.1 or Claude); ranking stability across judges is necessary but not sufficient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 defines SACS, SLPS, ORCS, and DJSS (Eqs. 1-4) entirely through LLM evaluation functions Evalacc, Evalprog, Evalcoh, Evaljust, and Appendix A.4 states that GPT-4o is used as the evaluator for similarity between predictions and ground truth. The paper provides no human-agreement study, no score distributions, no alternative judge, and no error bars for these metrics. This is load-bearing because Table 3 uses these scores to support the two main empirical claims: models fall below 60% on AD2-Bench, and hierarchical CoT substantially improves performance (e.g., InternVL3 +20.2). If GPT-4o's 1-10 scores track output length, formatting adherence, or lexical overlap with the annotation style rather than factual reasoning correctness, the rankings and CoT-gain conclusions are not established. The risk is amplified by the annotation loop: atomic CoT ground truth is itself validated and aggregated by Gemini 2.5-Pro/GPT-4o/Qwen 2.5-Max (Section 3.1 and Appendix A.2), so a style-matching judge can reward models that imitate the annotation style. Secondary inconsistencies (5,406 vs. 5,046 CoT samples; 'purely manual' vs. LLM post-processing) reinforce the need for an external validation step, but the metric-validation gap is the central threat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"AD^2-Bench is a proposed benchmark for evaluating chain-of-thought (CoT) reasoning in multimodal large language models (MLLMs) for autonomous driving under adverse weather and complex traffic conditions. The paper describes a dataset of roughly 10k real-world images, 70k QA pairs, and over 5.4k CoT instances with atomic step-level ground truth; introduces region-, point-, and text-level visual prompts; proposes four CoT quality metrics (SACS, SLPS, ORCS, DJSS) based on an LLM evaluator; and reports an evaluation of 12 open-source MLLMs, concluding that models generally score below 60% and that hierarchical CoT prompting improves performance (e.g., +20.2 for InternVL3).","tokens_in":31372,"tokens_out":6655,"duration_ms":64840,"significance":"If the annotation quality and evaluation metrics are validated, AD^2-Bench would fill a genuine gap in the autonomous-driving vision-language benchmark landscape: it combines adverse-condition imagery, hierarchical CoT annotations, and multi-level visual prompts in a single resource. The atomic annotation design, the breadth of subtasks (33), the detailed qualitative failure analysis, and the systematic comparison across 12 models are clear strengths. However, the paper's central quantitative claims currently rest on an unvalidated LLM-as-judge protocol and on internally inconsistent dataset counts and annotation descriptions. These issues are load-bearing for the benchmark's value, so the significance is not yet fully established.","major_comments":[{"comment":"The four proposed metrics are all defined through an LLM evaluation function Eval_*, and Appendix A.4 states that GPT-4o is used as the evaluator for similarity between predictions and ground truth. The paper provides no human-agreement study, no score distributions, no alternative judge, and no error bars for these 1-10 scores. Because Table 3 and Section 4.2 use these scores to support the headline claims (models below 60%, CoT gains such as +20.2 for InternVL3), the results may reflect GPT-4o's rating preferences rather than reasoning quality. Please validate the metrics on a sample against human judgments, report Pearson or Kendall correlations with human scores, provide per-step score distributions, and show that model rankings are stable when a second LLM judge (e.g., Gemini or a Qwen model) is used.","section":"§3.3, Eqs. (1)-(4)"},{"comment":"The number of full CoT instances is inconsistent: the abstract and Introduction say \"over 5.4k\", Section 1 gives \"5,406\", Section 3.2 says a \"dedicated 5,046-image CoT test suite\", and Appendix A.13 repeats \"over 5,400\". The paper also reports 70K QA pairs and 10K images without defining how these counts decompose into the CoT test suite, validation sets, and perception/relation splits. Please reconcile these numbers and provide an exact split table; this is necessary for reproducibility and for interpreting the per-model scores.","section":"Abstract, §1 vs. §3.2, Appendix A.13"},{"comment":"The claims of \"purely manual\" CoT annotations and \"All VQA and CoT steps are manually annotated by domain experts\" conflict with the annotation pipeline described in Section 3.1, where Gemini 2.5-Pro, GPT-4o, and Qwen 2.5-Max score the atomic annotations and provide detailed evaluations, and Section 3.2, where Gemini 2.5-Pro post-processes manual VQA responses and generates multiple-choice counterparts. Moreover, despite claims of cross-validation by three or more experts, no inter-annotator agreement is reported. Please quantify the human versus LLM contribution to each annotation step and report agreement statistics (e.g., Cohen's kappa or Fleiss' kappa) for a sample of the CoT instances.","section":"§3.1, §3.2, Appendix A.13"},{"comment":"The headline claim that models score \"below 60% accuracy\" is ambiguous because Table 3 reports w/ CoT scores above 60 for several models (e.g., InternVL3 63.04, InternVL2.5 61.64), while Table 2's Avg-S column is below 60. Section 4.2's statement that direct reasoning scores are \"typically below 40% (Tab. 2)\" is also inconsistent with Table 3's w/o CoT values (e.g., InternVL3 42.21, Qwen2.5-VL 41.73). Please define the exact aggregate metric and the experimental setting behind each headline number, and clarify whether Table 2's \"Event Res.\" column is the same as Table 3's \"w/ CoT\" column.","section":"Abstract, §1, Table 2, Table 3"}],"minor_comments":[{"comment":"There are several typos and naming inconsistencies: \"dementions\" in Figure 2, \"Pesestrain2\" in Figure 3, \"Intervl2.5\" and \"GroudTruth\" in Figure 5, and inconsistent use of \"ADBench\" versus \"AD^2-Bench\" across the text and figure captions.","section":"Figures 2, 3, 5; throughout"},{"comment":"Table 6 uses the header \"OCR-SER\" while the text and Appendix A.5 define Character Error Rate (CER); please align the notation to avoid confusion.","section":"Table 6, Appendix A.5"},{"comment":"The paper calls the benchmark bilingual (Section 3) but reports only English-language evaluation results; please specify the language of the evaluated prompts or clarify how the bilingual release is realized.","section":"§3, §4"},{"comment":"The checklist states that anonymous URLs are provided, but no URLs appear in the main text; please add a data/code availability statement with concrete links and license information so that reproducibility claims can be checked.","section":"NeurIPS Checklist, §5"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision rather than rejection: the benchmark is potentially valuable and the core design is defensible, but the unvalidated GPT-4o-based metrics and the dataset-count and annotation-pipeline inconsistencies are load-bearing for the paper's central claims. These issues can be addressed with additional analysis (human agreement study, alternate judge, precise dataset statistics, clarified annotation protocol) rather than requiring a fundamentally new submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nBottom line: AD2-Bench fills a real gap—no prior benchmark targets CoT reasoning for driving under adverse weather with atomic step-level ground truth—and the point-prompt idea for occluded objects is genuinely useful. The dataset construction is described in unusual detail, the 12-model sweep is broad, and the bad-case analysis is readable. But the headline claims (below 60%, CoT adds up to +20 points) rest on GPT-4o-as-judge with no human validation, and the paper has a few internal contradictions that need ironing out.\n\nWhat is actually good: The benchmark covers rain, snow, fog, sand, night, with 10k real images and 70k QA pairs. The hierarchical CoT decomposition into perception/understanding/reasoning steps with atomic ground truth goes beyond DriveLM's graph QA. The multi-level prompts—image, region, point, text—are a thoughtful answer to the vision encoder's weakness in bad weather; point prompts in particular are a nice trick for occluded objects. They evaluated 12 open-source models, which is a useful reference point. The qualitative failure cases are informative.\n\nSoft spots, in order of severity. First, the SACS/SLPS/ORCS/DJSS metrics are defined as LLM evaluation functions with no human-agreement study, no alternative judge, no score distributions, no error bars. Worse, the w/o-CoT vs w/CoT comparison in Table 3 uses the same GPT-4o similarity judgment, so the '+20.2' for InternVL3 and the 'CoT helps' conclusion may be a stylistic artifact of the judge rather than a reasoning gain. That is a load-bearing issue: if GPT-4o rewards verbose, well-formatted chains, the rankings and the empowerment claim don't hold. This needs a human-validation experiment, even a small one.\n\nSecond, there are concrete inconsistencies. The abstract and intro say 5,406 full CoT instances; Section 3.2 says a 5,046-image CoT test suite. That's a 360-image gap with no explanation. Third, the paper claims 'purely manual' annotations, but the appendix describes Gemini 2.5-Pro and GPT-4o post-processing and aggregation. That mismatch feeds the circularity concern. Fourth, no inter-annotator agreement is reported, and no data or code is available at this version.\n\nI want to be clear: the central artifact—the benchmark itself—looks valuable, and most of these issues are fixable. The metric-validation gap is the one that matters. If a small human study shows the LLM judge tracks human scores, the paper should be accepted; if not, the main claims need to be softened to 'GPT-4o-judged CoT quality' rather than 'CoT improves performance.'\n\nFor a reading group: maybe, if someone is working on MLLM evaluation for driving. I wouldn't cite it until the data and code are out. But I would send it to peer review—a serious referee can force the validation and the consistency fixes, and the field needs this benchmark. Send it.\n\nBest,\n[Your name]","headline":"Valuable adverse-weather CoT benchmark with a novel point-prompt design, but the GPT-4o-judged metrics need human validation before the headline claims can be trusted.","tokens_in":31928,"tokens_out":3958,"would_cite":false,"duration_ms":40155,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new driving benchmark grades models on step-by-step reasoning and finds today's best score below 60 percent.","keywords":["chain-of-thought reasoning","multimodal large language models","autonomous driving","adverse weather","visual question answering","reasoning benchmark","atomic annotation"],"falsifier":"Re-score a random subset of the CoT outputs with independent human experts using the same 1–10 rubrics and compare the per-model rankings to the GPT-4o-based ones; if the rankings diverge on model ordering or on the CoT-vs-no-CoT gap, the headline conclusions are an artifact of the chosen judge rather than a property of the models.","tokens_in":30899,"feed_emoji":"🚗","tokens_out":5712,"duration_ms":57613,"temperature":0.7,"pith_summary":"AD2-Bench is introduced as the first benchmark for evaluating chain-of-thought (CoT) reasoning by multimodal large language models in autonomous driving under adverse weather and complex scenes. The paper argues that existing benchmarks mostly cover benign conditions and grade only final answers, so they miss where multi-step driving reasoning fails. To close that gap, the authors built roughly 10,000 real-world images with 70,000 question-answer pairs, of which 5,406 carry complete, manually annotated CoT chains in which each reasoning step is an atomic unit with its own ground truth. On this benchmark, twelve state-of-the-art models all score below 60 percent average accuracy, which the authors read as evidence that safer, interpretable end-to-end driving still needs substantial progress.","feed_headline":"New driving AI benchmark: top models score below 60 percent","feed_subtitle":"5,400 hand-annotated chain-of-thought driving scenes test reasoning under rain, fog, and night.","key_machinery":"The load-bearing mechanism is the hierarchical CoT pipeline, a fixed five-stage decomposition from weather and road conditions, to key vehicles, to vulnerable road users, to traffic rules, to a final driving suggestion, with each stage annotated as an atomic ground-truth unit. Around this pipeline the benchmark supplies three visual-prompt levels (region bounding boxes, point cues, and text-level references) to reduce vision-encoder failure in adverse conditions, and four CoT quality metrics—SACS, SLPS, ORCS, and DJSS—that use an LLM evaluator to score step accuracy, logical progression, overall coherence, and decision justification on a 1–10 scale. These metrics are what turn the dataset from a VQA collection into a reasoning-process benchmark.","core_discovery":"The paper's central claim is that fine-grained CoT evaluation under adverse conditions exposes systematic weaknesses in current multimodal driving models that final-answer benchmarks hide. Treating each intermediate reasoning step as an atomic labeled unit lets the benchmark measure not just whether a model reaches the right answer but whether it perceives, understands, and reasons correctly along the way. The reported results show that no evaluated model surpasses 60 percent accuracy, that some models hallucinate hazards or miss key vehicles under rain and light fog, and that hierarchical CoT prompting lifts scores substantially for models trained with rationale-augmented data. The authors conclude that AD2-Bench provides a standardized platform for improving reliable, interpretable autonomous driving systems.","pith_inferences":["If the four CoT metrics are re-scored by human experts on a random sample, the paper's model rankings and its claim that CoT improves quality might shift; a public human-agreement study would make the benchmark's conclusions independent of a single judge.","The benchmark's atomic-step design could transfer to other safety-critical vision-language domains such as robot navigation, medical imaging, or industrial inspection, where reasoning process matters as much as final output.","The observed gap between individual step accuracy and overall coherence suggests a testable hypothesis: models trained with explicit intermediate-supervision losses should close that gap, and AD2-Bench could serve as the evaluation instrument.","Extending the benchmark to video or temporal chains would test whether models can maintain coherent reasoning over time, not just within a single frame."],"forward_implications":["Any multimodal model proposed for autonomous driving should be tested on adverse-weather and complex-scene reasoning, because benign-condition benchmarks overstate readiness.","The sub-60 percent ceiling on current models implies that perception, relation understanding, and decision reasoning under rain, fog, snow, and night remain open problems.","Hierarchical CoT prompting can add roughly 15–20 points for some models, so training with intermediate supervision and rationale-augmented data is a promising axis for improvement.","Since average step correctness tends to exceed whole-chain coherence, developers can localize failures to specific reasoning stages rather than treating a wrong final answer as a single error.","Point-level and region-level visual prompts help models notice occluded or distant hazards, offering a practical interface for steering attention during evaluation or deployment."],"supporting_citations":[{"why":"It introduces chain-of-thought prompting, the reasoning paradigm the benchmark evaluates.","marker":"[49]"},{"why":"It supplies graph-structured driving dialogue data that AD2-Bench extends with adverse-condition CoT annotations.","marker":"[42]"},{"why":"It is the main real-world corner-case image source used to curate adverse-condition driving scenes.","marker":"[29]"},{"why":"It contributes adverse-condition images such as rain, fog, and night scenes that the benchmark draws on.","marker":"[41]"},{"why":"It provides adverse-weather vehicle detection images used during data collection.","marker":"[25]"},{"why":"It is a large-scale driving QA benchmark whose format and baseline performance frame the comparison.","marker":"[40]"},{"why":"It is the closest existing multimodal driving QA benchmark with reasoning annotations, the baseline AD2-Bench distinguishes itself from.","marker":"[10]"},{"why":"It motivates the need for reliable annotations and metrics for vision-language models in driving.","marker":"[52]"}],"fun_headline_variants":["New benchmark: AI driving models fail CoT test under rain, fog","Driving AI benchmark exposes reasoning gaps: top scores under 60%","AD^2-Bench: CoT benchmark for driving under weather extremes","First CoT driving benchmark shows models score below 60% accuracy","Why AI drivers fail: new benchmark tests reasoning step-by-step"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's quality scores, including rankings and the conclusion that CoT helps, depend on GPT-4o's judgments of reasoning correctness, progression, coherence, and justification, and the paper does not show that these judgments agree with human experts.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark: AI driving models fail CoT test under rain, fog","Driving AI benchmark exposes reasoning gaps: top scores under 60%","AD^2-Bench: CoT benchmark for driving under weather extremes","First CoT driving benchmark shows models score below 60% accuracy","Why AI drivers fail: new benchmark tests reasoning step-by-step"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000724,"raw_usage":{"total_tokens":3254,"prompt_tokens":960,"completion_tokens":2294,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":2201}},"tokens_in":576,"tokens_out":2294,"duration_ms":15428,"temperature":1.0,"reasoning_tokens":2201,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:44:58.999394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score a random subset of the CoT outputs with independent human experts using the same 1–10 rubrics and compare the per-model rankings to the GPT-4o-based ones; if the rankings diverge on model ordering or on the CoT-vs-no-CoT gap, the headline conclusions are an artifact of the chosen judge rather than a property of the models.","supporting_citations":[{"cited_title":"Drivelm: Driving with graph visual question answering","cited_arxiv_id":null,"evidence_quote":"It supplies graph-structured driving dialogue data that AD2-Bench extends with adverse-condition CoT annotations."},{"cited_title":"Coda: A real-world road corner case dataset for object detection in autonomous driving","cited_arxiv_id":null,"evidence_quote":"It is the main real-world corner-case image source used to curate adverse-condition driving scenes."},{"cited_title":"Acdc: The adverse conditions dataset with corre- spondences for semantic driving scene understanding","cited_arxiv_id":null,"evidence_quote":"It contributes adverse-condition images such as rain, fog, and night scenes that the benchmark draws on."}],"review_version":1}