Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Current vision-language models can describe road scenes but largely fail to turn that perception into safe driving decisions — a 6,650-question benchmark exposes the gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 09:05 UTC pith:NUQUHDCE

load-bearing objection A useful progressive benchmark for VLM driving evaluation, but the paper ships under the wrong title, over-generalizes a two-model correlation result, and needs reproducibility fixes before it can be cited. the 4 major comments →

arxiv 2601.14702 v2 pith:NUQUHDCE submitted 2026-01-21 cs.AI cs.CVcs.RO

Drive-P2D: A Progressive Perception-to-Decision Benchmark for VLMs in Autonomous Driving

Zecong Tang , Zixu Wang , Yifei Wang , Weitong Lian , Tianjian Gao , Haoran Li , Tengju Ru , Lingyi Meng
show 5 more authors
Zhejun Cui Yichen Zhu Qi Kang Kaixuan Wang Yu Zhang
This is my paper
classification cs.AI cs.CVcs.RO
keywords vision-language modelsautonomous drivingbenchmarkperception-to-decision gapdriving decision-makingchain-of-thought reasoningsimilar-scene robustnesserror taxonomy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Drive-P2D is a benchmark for testing whether vision-language models (VLMs) can do what an autonomous driver must: move from seeing an object to choosing a safe action. Built on 1,295 front-view road images and 6,650 multiple-choice questions, it chains three levels — object identification, scene understanding, and decision-making — so a model's answer at each stage can be checked against the others. The paper's central finding is that current VLMs, open-weight and commercial alike, perform better at object and scene perception than at decisions, and that the two barely correlate: knowing what is in the image does not predict whether the model picks the right action. On pairs of near-identical scenes, models also fail both images more often than chance would predict, which the authors read as evidence that decisions ride on superficial cues rather than causal reasoning. The paper argues that combining objective scoring with chain-of-thought reasoning analysis is necessary to expose this gap, and that ordinary scale alone will not close it.

Core claim

The paper's claim, stated on its own terms, is that "current VLMs struggle to translate perceptual information into decisions." The evidence comes in three forms. First, pairwise Pearson correlations among the six task scores (object, scene, decision) run mostly between −0.2 and 0.2; decision tasks correlate only mildly with object recognition and not with scene understanding, so a model's perception scores say little about its decision scores. Second, on 60 pairs of visually similar scenes, the joint accuracy — the chance a model answers both images in a pair correctly — falls below the squared baseline of the single-image accuracies, significantly so for several small models; the authors r

What carries the argument

The load-bearing tool is the progressive three-level protocol itself: six tasks arranged in a chain — Object-1 (key object), Object-2 (object state), Scene-1 (weather/illumination), Scene-2 (special scene factors), Decision-1 (optimal action), Decision-2 (risk of a suboptimal action) — with two metrics per level and the same image reused across tasks, so a model's failure can be localized to a stage. Around this core sit three instruments. A separated reasoning-and-answer protocol asks each model to emit a chain-of-thought rationale in <think> tags and a final answer in <answer> tags, so answers are scored objectively while reasoning is analyzed separately. A correlation analysis computes Pe

Load-bearing premise

The load-bearing premise is that every question has one correct, unambiguous answer, even though each item is judged from a single front-view image by two annotators (with a third arbitrator breaking ties) and the paper does not report how often annotators disagreed.

What would settle it

Re-annotate a random subsample of the 6,650 questions — especially the Decision and Scene-2 items and the 60 similar-scene pairs — using new drivers, and measure inter-annotator agreement. If agreement is low, or if the newly agreed labels reproduce the models' 'errors' as plausible alternatives, the weak perception–decision correlations would reflect question ambiguity rather than a capability gap. A second check: for each similar-scene pair, ask human drivers to state which visual difference justifies different answers; pairs where drivers cannot articulate such a difference are weak evidenc

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the weak coupling is real, a VLM's object and scene perception scores cannot be used as evidence of driving readiness; decision accuracy must be evaluated directly.
  • Scaling alone will not close the gap: larger models get more stable and less negatively coupled, but not more coupled, so integrating perception and decision requires architectural or training changes beyond model size.
  • Choice-only benchmarks overstate capability: correct final answers frequently come with flawed reasoning, so reasoning-trace analysis is needed to expose actual failure modes.
  • Few-shot prompting is an unreliable lever for decision tasks — it can bias the answer distribution or degrade performance — so example selection needs its own care.
  • Performance does not scale monotonically: one intermediate 38-billion-parameter model in a scaled family shows a pronounced drop caused by overly constrained reasoning, so evaluations should sample multiple sizes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • My inference: the similar-scene protocol could be turned into a direct causal test — alter one decision-relevant cue (brake light on or off, pedestrian present or not) between paired images and measure whether the decision flips; the paper does not run this intervention, but its design invites it.
  • My inference: the weak-coupling result, if it holds for single images, may not survive the introduction of temporal information; a fourth video-based level would test whether motion cues — the missing modality in this benchmark — are what connect perception to decision.
  • My inference: the paper never reports inter-annotator agreement for its two-expert-plus-arbitrator labeling, so a re-annotation study reporting agreement rates would establish how much of the measured perception–decision gap is model failure versus answer ambiguity.
  • My note: the manuscript's title, abstract, and running text use different names for the benchmark (Drive-P2D versus AutoDriDM); readers should confirm which name the released artifacts use.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents Drive-P2D (referred to as AutoDriDM in the body), a benchmark for vision-language models in autonomous driving comprising 6,650 QA items over Object, Scene, and Decision levels built from nuScenes, KITTI, and BDD100K. All items are multiple-choice with objectively scored answers, and models are prompted to produce structured chain-of-thought reasoning. The authors evaluate a range of open- and closed-source VLMs under zero- and few-shot settings, report risk-stratified results, analyze Pearson correlations between task accuracies, test robustness on 60 visually similar scene pairs, study InternVL scaling, and train a 7B analyzer to tag reasoning error modes. The central claim is that current VLMs perform acceptably on object perception but fail to translate this into reliable decision-making, as evidenced by weak perception-decision correlations and drops in joint accuracy on similar-scene pairs, and that reasoning traces reveal systematic logical errors.

Significance. If the evidence were complete, the benchmark would fill a real gap: it couples perception and decision in a single progressive protocol, uses objective scoring instead of LLM-judged free-form outputs, and adds risk-stratified and similar-scene analyses. The objective answer format and the attempt to use reasoning traces for failure-mode analysis are useful methodological contributions for the driving-VLM community. However, the headline generalization about 'current VLMs' is currently supported by only a subset of the evaluated models, and the benchmark's validity depends on annotation quality that is not yet reported. The paper's contributions are promising and within scope for a computer-vision/robotics venue, but the empirical backbone needs substantial strengthening.

major comments (4)
  1. [§4.4, Fig. 4, Fig. 15, §3.2] The central claim that 'current VLMs struggle to translate perceptual information into decisions' rests on the correlation analysis, but this analysis is shown only for Qwen2.5-VL-7B and 72B. Section 3.2 says correlations are computed 'for each model,' yet no matrices or summary statistics are reported for GPT-4.1, Gemini-2.5-Pro, Llama-90B, Llava-72B, Phi-6B, or the InternVL family. Please report per-model correlations (or at least the Object–Decision and Scene–Decision coefficients for all models) and qualify the conclusion accordingly, or the headline statement is a two-model observation. Also, Figure 15 states each cell is computed over the images common to the two tasks, but the number of common images is not reported; since Scene tasks exist only in BDD100K while Object/Decision tasks exist in all three datasets, different cells are based on different subsets, which complicates com
  2. [§3.2, Limitations, Tables 1 and 6] No inter-annotator agreement is reported despite the protocol in §3.2 of two independent annotators plus a third arbitrator. Because the benchmark uses only single front-facing images and the Limitations acknowledge the absence of temporal/multi-sensor context, some Scene-2 and Decision items may have no well-defined answer. Low model accuracies on Scene-2 (e.g., 0.89–46.44% in Table 1) could reflect question ambiguity rather than model deficiency. Report per-task agreement (Cohen's kappa or Fleiss' kappa), and preferably analyze robustness of the main results when contested items are excluded.
  3. [§4.7–§4.8, Fig. 6, Table 2] The explainability finding that logical reasoning errors, semantic feature omissions, and hallucinations dominate is based on Qwen 7B/72B only, and it is unclear whether Figure 6 was produced by the fine-tuned analyzer of §4.8 or by manual annotation. If the analyzer was used, the analysis is partly circular: the 7B analyzer was trained on 1,500 traces labeled by the authors with the same nine-category taxonomy, and its Exact Match is 65.36% with 23.13% Mismatch on only 180 evaluation traces. State explicitly how Figure 6 labels were obtained, provide a manual validation sample for a subset of traces, and restrict the conclusion to the models actually analyzed.
  4. [§4.5, Table 7, Appendix D.5] The similar-scene robustness test is the only multi-model evidence for the perception-to-decision link, but it has 60 pairs, uses a squared single-image-accuracy baseline that assumes equal per-image difficulty, and reports only star thresholds without confidence intervals. It should be clarified that the Decision-1 values in Table 7 are measured on the 60 similar pairs, not on the full benchmark (they are 10–15 points lower than the full-benchmark Decision-1). Please provide bootstrap confidence intervals or per-pair results, and state how the 60 pairs were selected from the similarity filtering.
minor comments (6)
  1. [Title and Abstract] The abstract and title use 'Drive-P2D' while the body uses 'AutoDriDM'. This inconsistency should be resolved throughout the manuscript.
  2. [Fig. 4] The figure contains a stray caption line 'Figure 1: Correlation matrix for Qwen2.5-VL-Instruct' inside the main Figure 4. Remove this artifact.
  3. [§C.3, Tables 1, 6, 7] The repetition protocol averages two runs, but no variance or confidence intervals are reported. For binary accuracies, especially with the small 60-pair test, reporting standard deviations or CIs would support comparisons.
  4. [Fig. 15 caption] The caption calls the values 'Pearson ϕ correlation'; for binary indicators, the phi coefficient and Pearson correlation coincide, but the wording is confusing. Use one term consistently and define it.
  5. [Table 2] Specify the exact base model and fine-tuning setup for 'Ours 7B' (e.g., Qwen2.5-VL-7B-Instruct), and state the training/evaluation split for the 1,500/180 traces.
  6. [Appendix D.5] The text mentions 'Blue superscript stars' but the table does not show colored stars; align the table formatting with the description.

Circularity Check

0 steps flagged

No significant circularity: central results are externally grounded measurements; the analyzer-based explainability component is a supervised tool with held-out validation, not a by-construction reduction.

full rationale

The paper's central claims rest on an externally constructed benchmark: images from nuScenes, KITTI, and BDD100K, with dual human annotation plus third-arbiter adjudication. The headline results (task accuracies, high-risk splits, few-shot trends, similar-scene joint-accuracy tests, InternVL scaling behavior) are objective measurements on that externally grounded data, not derivations from the paper's own assumptions. The perception–decision correlation analysis is a Pearson/phi computation over binary correctness indicators; it is reported only for Qwen 7B/72B and is statistically attenuated by the marginal distributions of task accuracies, but this is an evidentiary/validity limitation, not a circular reduction. The only plausible circularity concern is the explainability component: the authors define a nine-category taxonomy and fine-tune a 7B analyzer on 1,500 traces labeled with that taxonomy. However, the paper's conclusion that logical reasoning errors, semantic feature omissions, and hallucinations dominate is drawn from the human-labeled explainability analysis in Section 4.7, which appears before the analyzer is introduced in Section 4.8; the analyzer is then validated on 180 held-out traces (Table 2). Its outputs are therefore not equivalent to its training labels by construction. No load-bearing step reduces to its own input, and no self-citation chain is invoked as evidence. The per-model correlation gap noted by the skeptic is a reporting/completeness issue, not circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The central claims rest on the validity of the human annotations as ground truth, the sufficiency of single front-facing images for decision labeling, and the meaningfulness of the error taxonomy; the benchmark design thresholds (similarity 0.9, risk 4.0) are hand-chosen and modulate the high-risk and similar-scene analyses.

free parameters (2)
  • high-risk threshold = 4.0 (mean of two annotators' 1-5 risk scores)
    Hand-chosen threshold defining the high-risk split; affects Section 4.2 and the conclusion that VLMs do better on high-risk scenes.
  • similarity threshold for filtering = 0.9 cosine (ResNet-50 embeddings)
    Hand-chosen threshold for near-duplicate removal and similar-scene pair construction; determines the 60-pair robustness test.
axioms (6)
  • domain assumption Two experienced drivers plus a third arbitrator produce correct ground-truth answers for all 6,650 questions
    Section 3.2; no inter-annotator agreement is reported, so the reliability of the labels is assumed.
  • domain assumption A single front-facing image contains sufficient information to determine the optimal driving action and risk level
    Section 3.1 and Limitations; no temporal or multi-sensor data, so decisions that require motion cues are underdetermined.
  • domain assumption ResNet-50 cosine similarity identifies semantic near-duplicates and visually similar driving scenes
    Sections 3.1 and 3.2; used for deduplication and construction of the 60 similar-scene pairs.
  • domain assumption Chain-of-thought text inside <think>...</think> reflects the model's internal reasoning
    Section 3.3; CoT may be unfaithful to model decision processes.
  • domain assumption The nine error categories are exhaustive, mutually exclusive, and applicable across models
    Section 3.3 and Appendix B.1; categories overlap (e.g., E1 vs E6) and are defined by the authors.
  • domain assumption Joint-accuracy below the squared single-image accuracy implies reliance on superficial cues rather than task ambiguity
    Section 4.5; the independence assumption across similar images is questionable.

pith-pipeline@v1.3.0-alltime-deepseek · 32596 in / 12639 out tokens · 133253 ms · 2026-08-03T09:05:12.865447+00:00 · methodology

0 comments
read the original abstract

Autonomous driving requires reliable perception and safe decision-making in complex scenarios. Recent vision-language models (VLMs) demonstrate reasoning and generalization abilities, opening new possibilities for autonomous driving; however, existing benchmarks often evaluate perception and decision-making separately, limit failure analysis with choice-only formats, or introduce evaluation bias through LLM-scored long-form outputs. To address these issues, we present Drive-P2D, a progressive perception-to-decision benchmark with 6,650 questions across Object, Scene, and Decision levels. Drive-P2D adopts a separated reasoning-and-answer protocol: final answers are scored objectively, while reasoning is analyzed to identify error modes exposed along the progressive perception-to-decision chain. We evaluate mainstream VLMs across all and high-risk scenarios, and further characterize the perception-to-decision capability boundary through correlation analysis and similar-scene robustness testing. Reasoning further exposes failure modes such as logical reasoning errors and semantic feature omissions, and we train a lightweight analyzer model to automate large-scale error-mode annotation of reasoning. Together, these designs provide practical insights for building safer and more reliable VLMs for real-world autonomous driving.

Figures

Figures reproduced from arXiv: 2601.14702 by Haoran Li, Kaixuan Wang, Lingyi Meng, Qi Kang, Tengju Ru, Tianjian Gao, Weitong Lian, Yichen Zhu, Yifei Wang, Yu Zhang, Zecong Tang, Zhejun Cui, Zixu Wang.

Figure 1
Figure 1. Figure 1: Overview of AutoDriDM. The framework is organized into three stages: (Left) Data Source & Filtering, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The word cloud about autonomous driving collected from our QA pairs in AutoDriDM. OCR/text reasoning, and hallucination robustness (Fu et al., 2024a; Liu et al., 2024a; Yu et al., 2024; Yue et al., 2024, 2025; Guan et al., 2024; Liu et al., 2024b; Tang et al., 2025; Mathew et al., 2021; Zhang et al., 2025). In autonomous driving, recent VLM-oriented benchmarks provide driving scenes and tasks for perceptio… view at source ↗
Figure 3
Figure 3. Figure 3: GPT-4.1 Failure Cases. The six subfigures illustrate GPT-4.1 failure cases across different tasks. Each [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 1
Figure 1. Figure 1: Correlation matrix for Qwen2.5-VL-Instruct Figure 4: Correlation Matrices of Qwen (7B) and Qwen (72B) [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 5
Figure 5. Figure 5: ”Scaling behavior” of Internvl models most all models, the joint correct rate is lower than the expected squared baseline. Models such as Llama (11B) and Llava (7B) show significant drops (p < 0.05 or p < 0.10). This pattern confirms that small-scale models tend to recombine percep￾tual elements without truly assessing which visual components are decision-relevant. Larger models demonstrate limited robustn… view at source ↗
Figure 6
Figure 6. Figure 6: Results of Explainability Analysis for Qwen (7B) and Qwen (72B) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Examples of similar-scene robustness test. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 11
Figure 11. Figure 11: Example image used for the six-task annota [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗
Figure 8
Figure 8. Figure 8: Representative Error Examples in Explainability Analysis [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Model performance on all-scenarios under 0/1/2/5-shot prompting [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Model performance on high-risk-scenarios under 0/1/2/5-shot prompting [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: Annotation templates instantiated on an example image. Underlined fields ( [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: System instruction and user input templates for the single-choice and multiple-choice Chain-of-Thought [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: System instruction and user input template for the explainability error-tagging prompt. [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Correlation matrices across the six evaluation tasks—for Qwen (7B) and Qwen (72B). For each model, [PITH_FULL_IMAGE:figures/full_fig_p021_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Driving

    cs.RO 2026-02 conditional novelty 6.0

    A VLM-based and a vision-only end-to-end planner are behaviorally complementary in a long tail of driving scenarios; selecting the better trajectory lifts NAVSIM PDMS from 90.80 to 92.10 at modest compute.

Reference graph

Works this paper leans on

11 extracted references · 2 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Logical Reasoning Error: Perception may be correct, but the subsequent reasoning is logically invalid or contradicts traffic rules or causality

  2. [2]

    Semantic Feature Omission: The model overlooks or misjudges safety-critical seman- tic/visual cues (e.g., brake lights, turn signals, pedestrian gestures) or other discriminative at- tributes necessary for correct recognition and categorization

  3. [3]

    Model Hallucination: The model produces information inconsistent with the actual input, e.g., inventing non-existent objects, attributes, or relations

  4. [4]

    Modality Imbalance: The model over-relies on one modality while neglecting information from another (e.g., text vs. image)

  5. [5]

    Spatial Relation Misjudgment: The model makes incorrect judgments about spatial rela- tions (e.g., distance, relative position, depth)

  6. [6]

    Limited Logical Inference: The model draws decisions from insufficient local evidence with- out integrating broader scene context

  7. [7]

    Generalization Deficit: The model generalizes poorly to rare or out-of-distribution scenarios (e.g., construction zones, traffic incidents, tem- porary lane closures)

  8. [8]

    Decision Boundary Instability: Near criti- cal operating points, small input perturbations cause large changes in the chosen action

  9. [9]

    maintaining speed)

    Target Priority Misjudgment: The model fails to correctly prioritize among competing objectives (e.g., yielding to pedestrians vs. maintaining speed). Figure 8 provides representative, annotated ex- amples for some categories. B.2 Explainability Model I/O Schematic The analyzer consumes an image, question, op- tions, the ground-truth answer, and a model’s...

    2023

  10. [2021]

    ArXiv:2107.03374 [cs]

    Evaluating Large Language Models Trained on Code.arXiv preprint. ArXiv:2107.03374 [cs]. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF confe...

    Pith/arXiv arXiv 2023

  11. [2024]

    Tianyu Zhang, Suyuchen Wang, Lu Li, Ge Zhang, Perouz Taslakian, Sai Rajeswar, Jie Fu, Bang Liu, and Yoshua Bengio

    Vision-Language Models for Vision Tasks: A Survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5625–5644. Tianyu Zhang, Suyuchen Wang, Lu Li, Ge Zhang, Perouz Taslakian, Sai Rajeswar, Jie Fu, Bang Liu, and Yoshua Bengio. 2025. VCR: A Task for Pixel- Level Complex Reasoning in Vision Language Mod- els via Restoring Occluded Text.ar...

    Pith/arXiv arXiv 2025