Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

A driving-safety benchmark that fuses external hazards with in-cabin risk finds that current vision-language models fall below the passing bar, while a safety-tuned 7B model scores 68.4 vs 49.5 for the best baseline.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 21:33 UTC pith:A73QYKGA

load-bearing objection A useful safety taxonomy buried under a circular evaluation: the benchmark resource has potential, but the reported numbers are not credible as-is. the 5 major comments →

arxiv 2511.14592 v3 pith:A73QYKGA submitted 2025-11-18 cs.RO cs.AI

DSBench: A Comprehensive Benchmark for Evaluating External and In-Cabin Risks

Xianhui Meng , Yuchen Zhang , Zhijian Huang , Zheng Lu , Ziling Ji , Yandan Lin , Yaoyao Yin , Hongyuan Zhang
show 7 more authors
Wei Zhou Guangfeng Jiang Li Zhang Long Chen Hangjun Ye Jun Liu Xiaoshuai Hao
This is my paper
classification cs.RO cs.AI
keywords autonomous drivingvision-language modelssafety benchmarkin-cabin monitoringdriver behaviorLLM evaluationfine-tuningDSBench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper aims to establish that vision-language models (VLMs) are not yet safety-ready for autonomous driving, because existing evaluations ignore the in-cabin driving context. The authors introduce DSBench, a benchmark that unifies ten categories (traffic rules, obstacles, weather, driver emotion, attention, cockpit environment, etc.) into 28 subcategories and tests a model's safety judgment on 3,000 curated high-risk scenes. Across 16 open, closed, and driving-specialized VLMs, none reaches the passing standard when scored by an LLM-based rubric. The paper further claims that fine-tuning Qwen2.5-VL-7B on 98K DSBench QA pairs yields DSVLM, which surpasses all baselines by 18.88 points on average and by 50.61 points in the cockpit-environment category, while retaining general question-answering ability. The wider point is that external and in-cabin risks must be assessed together for any safety-relevant VLM evaluation.

Core claim

DSBench is presented as the first autonomous-driving benchmark that holistically integrates external environmental risks (traffic rules, lane lines, intersections, weather, static/dynamic obstacles) with in-cabin driving-behavior safety (emotion, attention, driver operations, cockpit environment) in a unified taxonomy of 10 categories and 28 subcategories. The benchmark's 98K QA instances are produced by a hybrid pipeline of expert-designed templates and LLM expansion, with 10% human review and an 80% admission criterion per subcategory. Evaluated on a 3K high-risk test set, all 16 baseline VLMs score below the passing threshold (e.g., best commercial Seed-1.6 averages 49.52/100). Fine-tunin

What carries the argument

The central object is DSBench, a safety-scenario taxonomy and QA dataset built by merging more than ten public driving datasets, with GPT-4o-assisted scene analysis, question generation, and scoring. The load-bearing evaluation mechanism is an LLM-based rubric that scores model outputs on completeness, fluency, conciseness, and safety-judgment accuracy; the paper argues n-gram metrics (BLEU, CIDEr, ROUGE) are uncorrelated with safety correctness. The model-side machinery is DSVLM, a 7B VLM fine-tuned on 98K DSBench QA pairs, which the paper claims becomes a safety-expert.

Load-bearing premise

The central claim depends on the assumption that the model used to generate the questions and answers also produces valid safety scores as judge; the paper asserts close agreement with human evaluators but reports no quantitative correlation, so if that assertion is wrong, every score and conclusion is unsupported.

What would settle it

Take 300 DSBench test scenes, have certified driving instructors independently score the model outputs on safety correctness, and measure the rank correlation between their scores and the LLM judge's scores; if the correlation is weak or the pass/fail labels diverge, the benchmark's conclusion fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark is valid, any VLM intended for assisted or shared control (Level 2/3) should be tested on both external hazards and in-cabin driver state; a score on one axis alone is incomplete.
  • The reported results imply that current open and closed VLMs systematically struggle with fine-grained in-cabin perception (cockpit environment), with all baselines below 30; this names a concrete weakness to fix.
  • The 18.88-point gap between DSVLM and the best commercial baseline suggests that safety-relevant fine-tuning on a well-curated dataset can transfer to a small model, rather than requiring larger parameters.
  • If DSBench's scoring is correlated with human safety judgments, it provides a lightweight 3K-scene test for regression-testing safety during VLM development.
  • The finding that none of the evaluated models passes the safety bar would imply that current VLM-based driving assistants should not be deployed without additional safeguards.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the gap between GPT-4o-generated ground truth and GPT-4o's own scoring introduces a calibration risk; a human-expert validation study on a sample of the 3K test scenes would strengthen the claim that scores reflect safety rather than stylistic preference.
  • The 50.61-point lead in cockpit environment may partly reflect that baselines are not trained on in-cabin data at all; DSBench could serve as a diagnostic to reveal whether models lack perception or safety knowledge, possibly by ablating questions that require only visual recognition versus rule-based reasoning.
  • A testable extension: use DSBench's taxonomy to create a curriculum where models are fine-tuned per subcategory, and measure whether the per-category ordering of difficulty (Dri./Att. easier, Coc./Sta. harder) holds across architectures.
  • The paper's claim that traditional metrics are inappropriate could be made more decisive by reporting the correlation between GPT-4o scores and human rankings, not just the absence of correlation with n-grams.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces DSBench, a vision-language QA benchmark for autonomous driving safety that jointly evaluates external environmental risks (traffic rules, lane lines, intersections, weather, static/dynamic obstacles) and in-cabin risks (emotion, attention, driver operation, cockpit environment), organized into 10 categories and 28 subcategories. It also presents DSVLM, a Qwen2.5-VL-7B model fine-tuned on the 98K DSBench dataset, and reports GPT-4o-scored comparisons across 16 open-source, closed-source, and domain-specific VLMs. The headline empirical claims are that the fine-tuned DSVLM establishes a new state of the art (68.40 average vs. 49.52 for Seed-1.6) and that none of the existing models pass a safety threshold. The central evaluation protocol suffers from a closed loop: GPT-4o generates the benchmark annotations and answers, the same model is used as the judge, and no train/test split is documented between the 98K fine-tuning set and the 3K evaluation set.

Significance. If validated, the taxonomy and the unified external/in-cabin coverage would be a useful contribution to driving-safety evaluation, and the broad baseline coverage plus planned release of code/models would benefit the community. However, as presented, the empirical results cannot support the stated conclusions because the evaluation loop is circular and the state-of-the-art claim is potentially contaminated by training/test overlap. The paper's proposed 'Difficulty Coefficient' is claimed as a contribution but never defined or used. The resource is potentially salvageable, but substantial reworking is needed before the benchmark results can be trusted.

major comments (5)
  1. [§3.4, §4.1, Table 2] The 3,000-item DSBench evaluation set is curated from the same 98K DSBench dataset used to fine-tune DSVLM (§3.4: 'we curate a subset of 3,000 representative high-risk scenes to form DSBench'; §4.1: DSVLM is 'fine-tuned ... using the DSBench-Dataset'). No train/test split is documented. If the 3K evaluation samples are part of the 98K training set, the Table 2 SOTA margin (68.40 vs. 49.52) is a memorization effect. Even if the 3K are disjoint samples, near-duplicate frames from the same source videos (AIDE, BDD-X, etc.) could leak. The paper must report the exact split and ensure no overlap at the video/frame level; otherwise the headline result is unsupported.
  2. [§3.2–3.3, §4.2] GPT-4o generates the scene analyses, expands the QA templates, and defines the ground-truth answers (§3.2–3.3), and then the same model serves as the 'virtual evaluator' scoring all model outputs (§4.2). The introduction calls GPT-4o an 'independent evaluator,' which is inaccurate. No human-agreement statistics are reported; the supplementary only states that scores are 'approximately consistent with human evaluations.' Without a quantitative human correlation or a judge-consistency study, all scores in Table 2 are circular and cannot substantiate the claim that existing models fail or that DSVLM is safer.
  3. [Contribution bullet, §3] The contributions claim 'a novel Difficulty Coefficient metric' that 'quantitatively measures scenario difficulty,' but no definition, formula, or experiment using this metric appears in the main text or supplement. This is a stated contribution that is missing entirely. Either define it and show how it is used, or remove the claim.
  4. [Table 1, Introduction] The paper repeatedly claims DSBench is 'the first' to integrate external risks and in-cabin safety, and that in-cabin assessment is 'a crucial yet overlooked dimension.' However, Table 1 itself lists AIDE with an in-cabin checkmark and 18 categories. The novelty claim is therefore contradicted by the paper's own comparison. The authors need to specify precisely what DSBench adds beyond AIDE (e.g., unified evaluation protocol, category depth) and avoid the 'first' claim unless verified against all listed related benchmarks.
  5. [Table 2, §4.2, Eq. (1)–(3)] All Table 2 scores are single-point GPT-4o judgments with no error bars, no repeated-scoring variance, and no statistical significance tests. The claimed 18.88-point improvement over Seed-1.6 cannot be assessed for significance. Additionally, the 'passing standard' mentioned in the Introduction ('none of the models met the passing standard') is never defined in §3 or §4. Please report judge variance, human correlation, and the explicit passing criterion.
minor comments (5)
  1. [§3.4] Typo: 'qustion-answer pairs' should be 'question-answer pairs.'
  2. [Figure S1 caption] 'Expansion of QA-pais' typo; also the caption does not adequately describe what is shown in the two panels.
  3. [References [34, 43]] Reference [43] ('Pie: A dynamic failure-based technique') is not the correct citation for the PIE pedestrian dataset used in §3.1. Similarly, 'CODA [34]' is cited as Plummer et al., a convergence-diagnosis paper; this is almost certainly the wrong CODA reference for the autonomous-driving dataset.
  4. [Table 2 caption] 'Bold and underline indicate the best and second-best performance, respectively, among specific model groups as shown in the original paper' is unclear because DSVLM is bold across all rows/categories; clarify grouping.
  5. [§4.4, Table 4] The claim that DSVLM 'exhibits a stronger reasoning capability' because Lingo-Judge is higher while BLEU/CIDEr are lower needs statistical support; with a single test set the differences may not be significant.

Circularity Check

2 steps flagged

Closed loop: GPT-4o writes DSBench's reference answers and then grades all models, while DSVLM is fine-tuned on the same 98K pool from which the 3K evaluation set is curated without a stated split.

specific steps
  1. fitted input called prediction [Sec. 3.4 (data construction) and Sec. 4.1 (fine-tuning); scores in Table 2 / Eq. (2)]
    ""In total, we construct 98K QAs covering 28 safety dimensions, combining scale with diversity. For evaluation, we curate a subset of 3,000 representative high-risk scenes to form DSBench." ... "To address these shortcomings, we fine-tune Qwen2.5-VL-7B using the DSBench-Dataset, resulting in a specialized autonomous driving safety model, DSVLM.""

    By the paper's own wording, the 3K DSBench evaluation set is a subset of the same 98K QA pool used as the DSBench-Dataset for fine-tuning. No train/test split is reported anywhere. DSVLM is therefore scored on questions/images that, on the natural reading, are inside its training distribution or are near-duplicates of training items. The headline result (68.40 average vs 49.52 for Seed-1.6) then measures memorization or distribution overlap, not held-out safety generalization. The 'prediction' of DSVLM's superiority is statistically forced by training-set exposure.

  2. self definitional [Sec. 1 (data production and evaluator), Sec. 3.2-3.3 (GPT-4o annotations), Sec. 4.2 and Table 2 caption (GPT-4o scoring)]
    ""With the collaboration of eight expert human annotators and assistance from GPT-4o [19], we produced 98K safety-centric QA annotations and curated a test set of 3K representative scenarios. Leveraging GPT-4o as an independent evaluator, we rigorously assessed ... safety judgment accuracy." ... "All scores are assigned by GPT-4o evaluators equipped with predefined scoring rules.""

    The same model, GPT-4o, generates the scene analyses and expands the expert templates into ground-truth QA pairs, and then acts as the 'virtual evaluator' that assigns every score in Table 2. 'Safety judgment accuracy' is therefore operationalized as agreement with GPT-4o's own generated references as judged by GPT-4o. The 10% human sampling during QA construction is a content filter, not a validation of the scoring function; no quantitative human-evaluator agreement is reported. Models trained on GPT-4o-generated data, especially DSVLM, are rewarded for mimicking GPT-4o's answer style, so the benchmark's safety ranking is a self-agreement loop rather than measurement against an external safety standard.

full rationale

The central SOTA claim is not independently supported. The paper never states a train/test split, and its own text says the 3K DSBench evaluation set is 'a subset' of the 98K QA pool used to fine-tune DSVLM; hence the 68.40 vs 49.52 margin in Table 2 can be explained by training-set exposure. Independently, the evaluation framework is a closed loop: GPT-4o produces the scene descriptions and reference answers (Sec. 3.2-3.3) and then serves as the 'independent' grader (Sec. 4.2). Human expert review samples only 10% of the generated QA pairs during construction and does not quantitatively validate the GPT-4o scoring prompt, so the claimed passing-standard failure and all model rankings are GPT-4o's assessment of agreement with GPT-4o. The LingoQA experiment is external but evaluates general VQA ability, not DSBench safety scores, so it does not break either loop. The taxonomy and curated real-world image pool may still be useful resources, but the paper's headline quantitative results reduce by construction to the benchmark's own generation and evaluation procedure. Score 8 reflects that the central result is forced by the closed evaluation design, while stopping short of 10 because the benchmark artifacts and category structure have independent content.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claims depend on a set of unvalidated domain assumptions about GPT-4o's reliability as annotator and judge, plus hand-chosen thresholds (80% admission, 10% sampling) and an undefined difficulty metric. There are no physics-style fitted constants; the benchmark's design choices themselves act as free parameters.

free parameters (3)
  • Sub-category admission threshold = 80%
    Hand-chosen criterion in Sec. 3.4: a sub-category's QAs are accepted only if the combined accept/revised ratio reaches 80%. This directly shapes the final dataset composition.
  • Human review sampling rate = 10%
    Only 10% of QA pairs are human-reviewed (Sec. 3.4); the other 90% are accepted based on GPT-4o generation plus this small sample.
  • Passing standard = not stated
    Sec. 4.3 claims 'none of the models met the passing standard,' but no numeric passing threshold is ever defined.
axioms (4)
  • domain assumption Text responses to QA pairs are a valid proxy for a VLM's safety awareness in real driving.
    The benchmark scores free-text answers rather than actual driving actions or decisions (Sec. 4.2).
  • domain assumption GPT-4o's scene analysis and QA expansion are factually correct and safety-appropriate.
    GPT-4o is the pre-annotation engine for scene classification, description, and element recognition (Sec. 3.2), and it expands expert templates into QA pairs (Sec. 3.3), with only 10% human verification.
  • domain assumption GPT-4o's holistic scores align with human expert judgments.
    The paper claims the GPT evaluator 'closely aligns' with human evaluation (supplementary) but provides no quantitative agreement (e.g., correlation, kappa).
  • domain assumption The 3,000 curated scenarios are representative of real-world high-risk driving and are disjoint from the training instances.
    Sec. 3.1/4.1 do not demonstrate representativeness and do not explicitly state that the 3K test set is separate from the 98K QA set used for fine-tuning DSVLM.
invented entities (1)
  • Difficulty Coefficient no independent evidence
    purpose: Announced as a new metric to quantitatively measure scenario difficulty for targeted remediation (Sec. 1), but never defined, computed, or used.
    It has no definition, formula, or empirical instantiation anywhere in the main text or supplementary, so it provides no falsifiable handle.

pith-pipeline@v1.3.0-alltime-deepseek · 15551 in / 11391 out tokens · 106147 ms · 2026-08-03T21:33:03.145024+00:00 · methodology

0 comments
read the original abstract

Vision-Language Models (VLMs) show great promise for autonomous driving, but their suitability for safety-critical scenarios is largely unexplored, raising safety concerns. This issue arises from the lack of comprehensive benchmarks that assess both external environmental risks and in-cabin driving behavior safety simultaneously. To bridge this critical gap, we introduce DSBench, the first comprehensive Driving Safety Benchmark designed to assess a VLM's awareness of various safety risks in a unified manner. DSBench encompasses two major categories: external environmental risks and in-cabin driving behavior safety, divided into 10 key categories and a total of 28 sub-categories. This comprehensive evaluation covers a wide range of scenarios, ensuring a thorough assessment of VLMs' performance in safety-critical contexts. Extensive evaluations across various mainstream open-source and closed-source VLMs reveal significant performance degradation under complex safety-critical situations, highlighting urgent safety concerns. To address this, we constructed a large dataset of 98K instances focused on in-cabin and external safety scenarios, showing that fine-tuning on this dataset significantly enhances the safety performance of existing VLMs and paves the way for advancing autonomous driving technology. The benchmark toolkit, code, and model checkpoints will be publicly accessible.

Figures

Figures reproduced from arXiv: 2511.14592 by Guangfeng Jiang, Hangjun Ye, Hongyuan Zhang, Jun Liu, Li Zhang, Long Chen, Wei Zhou, Xianhui Meng, Xiaoshuai Hao, Yandan Lin, Yaoyao Yin, Yuchen Zhang, Zheng Lu, Zhijian Huang, Ziling Ji.

Figure 1
Figure 1. Figure 1: Overview of DSBench. A comprehensive benchmark for evaluating the safety of Vision-Language Models (VLMs) in au￾tonomous driving. The diagram highlights 10 key categories related to in-cabin and external safety scenarios. Each category addresses critical aspects such as driver operation, environmental conditions, and situational awareness, all of which are essential for improving the safety and effectivene… view at source ↗
Figure 2
Figure 2. Figure 2: The Pipeline of Data Extraction, Annotation, and Selection Process. Our annotation process consists of four key components: data extraction and integration (Sec. 3.1), data analysis and preliminary categorization (Sec. 3.2), expansion of question-answer templates (Sec. 3.3), and manual quality control (Sec. 3.4). line competence in understanding driving scenes, we adopt a VLM pre-annotation plus human veri… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the Detailed Categories in DSBench. Our benchmark is organized into 10 major categories and 28 more granular subcategories, encompassing a comprehensive spectrum of potential safety issues encountered in driving scenarios. 3.4. Quality Control To ensure reliability and consistency at scale, we implement a rigorous, iterative quality control process to filter noise from automated generation and … view at source ↗
Figure 4
Figure 4. Figure 4: Performance Comparison in DSBench. Our DSVLM has achieved state-of-the-art performance across all safety cat￾egories and made significant breakthroughs in Coc., a category where existing models have demonstrated limited capabilities. ⋆ Detailed Per-Task Performance Analysis. We conduct a vertical, per-task analysis to deeply investigate the capa￾bility distribution of different models and the inherent chal… view at source ↗
Figure 5
Figure 5. Figure 5: Visualization Results of Existing Models on DSBench. It reveals that, when confronted with both in-cabin and external safety challenges, our fine-tuned model, DSVLM, consistently delivers more comprehensive, reasonable, and factually grounded responses. Additionally, its outputs exhibit greater linguistic fluency and word choices that align more closely with natural human preferences. 5. Conclusion In this… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. GeoDrive-Bench: Benchmarking Region-Specific Multimodal Reasoning in Autonomous Driving

    cs.CV 2026-06 unverdicted novelty 7.0

    GeoDrive-Bench is a new multimodal benchmark and distillation method for testing and improving VLMs on region-specific traffic-rule reasoning in autonomous driving across six countries.

  2. Text-Driven 3D Indoor Scene Synthesis in Non-Manhattan Environments

    cs.AI 2026-07 unverdicted novelty 3.0

    SPG-Layout combines statistical object priors with hierarchical large-object-first placement to produce physically plausible text-driven 3D scenes in non-Manhattan rooms and outperforms baselines on a new 500-scene benchmark.

Reference graph

Works this paper leans on

52 extracted references · 18 linked inside Pith · cited by 2 Pith papers

  1. [1]

    A topology of shared control systems—finding com- mon ground in diversity.IEEE Transactions on Human- Machine Systems, 48(5):509–525, 2018

    David A Abbink, Tom Carlson, Mark Mulder, Joost CF De Winter, Farzad Aminravan, Tricia L Gibo, and Erwin R Boer. A topology of shared control systems—finding com- mon ground in diversity.IEEE Transactions on Human- Machine Systems, 48(5):509–525, 2018. 2

    2018

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  3. [3]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023. 2, 5, 6

    Pith/arXiv arXiv 2023

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 2, 5, 6

    Pith/arXiv arXiv 2025

  5. [5]

    nuscenes: A multi- modal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020. 3

    2020

  6. [6]

    Maplm: A real-world large-scale vision-language benchmark for map and traffic scene un- derstanding

    Xu Cao, Tong Zhou, Yunsheng Ma, Wenqian Ye, Can Cui, Kun Tang, Zhipeng Cao, Kaizhao Liang, Ziran Wang, James M Rehg, et al. Maplm: A real-world large-scale vision-language benchmark for map and traffic scene un- derstanding. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21819– 21830, 2024. 3

    2024

  7. [7]

    Holistic autonomous driving un- derstanding by bird’s-eye-view injected multi-modal large models

    Xinpeng Ding, Jianhua Han, Hang Xu, Xiaodan Liang, Wei Zhang, and Xiaomeng Li. Holistic autonomous driving un- derstanding by bird’s-eye-view injected multi-modal large models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13668– 13677, 2024. 2

    2024

  8. [8]

    Mme-survey: A comprehensive sur- vey on evaluation of multimodal llms.arXiv preprint arXiv:2411.15296, 2024

    Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. Mme-survey: A comprehensive sur- vey on evaluation of multimodal llms.arXiv preprint arXiv:2411.15296, 2024. 3

    Pith/arXiv arXiv 2024

  9. [9]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793, 2024

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793, 2024. 5, 6

    Pith/arXiv arXiv 2024

  10. [10]

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report.arXiv preprint arXiv:2505.07062, 2025. 2, 5, 6

    Pith/arXiv arXiv 2025

  11. [11]

    Mapdistill: Boosting efficient camera-based hd map construction via camera-lidar fusion model distillation

    Xiaoshuai Hao, Ruikai Li, Hui Zhang, Dingzhe Li, Rong Yin, Sangil Jung, Seung-In Park, ByungIn Yoo, Haimei Zhao, and Jing Zhang. Mapdistill: Boosting efficient camera-based hd map construction via camera-lidar fusion model distillation. InEuropean Conference on Computer Vi- sion, pages 166–183. Springer, 2024. 2

    2024

  12. [12]

    Mbfusion: A new multi-modal bev feature fusion method for hd map construc- tion

    Xiaoshuai Hao, Hui Zhang, Yifan Yang, Yi Zhou, Sangil Jung, Seung-In Park, and ByungIn Yoo. Mbfusion: A new multi-modal bev feature fusion method for hd map construc- tion. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 15922–15928. IEEE, 2024

    2024

  13. [13]

    Safemap: Robust hd map construction from incomplete observations.arXiv preprint arXiv:2507.00861, 2025

    Xiaoshuai Hao, Lingdong Kong, Rong Yin, Pengwei Wang, Jing Zhang, Yunfeng Diao, and Shu Zhao. Safemap: Robust hd map construction from incomplete observations.arXiv preprint arXiv:2507.00861, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    What really matters for robust multi-sensor hd map construction? In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

    Xiaoshuai Hao, Yuting Zhao, Yuheng Ji, Luanyuan Dai, Peng Hao, Dingzhe Li, Shuai Cheng, and Rong Yin. What really matters for robust multi-sensor hd map construction? In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2025. 2

    2025

  15. [15]

    Driveaction: A benchmark for exploring human-like driving decisions in vla models.arXiv preprint arXiv:2506.05667, 2025

    Yuhan Hao, Zhengning Li, Lei Sun, Weilong Wang, Naixin Yi, Sheng Song, Caihong Qin, Mofan Zhou, Yifei Zhan, and Xianpeng Lang. Driveaction: A benchmark for exploring human-like driving decisions in vla models.arXiv preprint arXiv:2506.05667, 2025

    arXiv 2025

  16. [16]

    Fuller: Unified multi-modality multi-task 3d perception via multi-level gradient calibration

    Zhijian Huang, Sihao Lin, Guiyu Liu, Mukun Luo, Chao- qiang Ye, Hang Xu, Xiaojun Chang, and Xiaodan Liang. Fuller: Unified multi-modality multi-task 3d perception via multi-level gradient calibration. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3502–3511, 2023

    2023

  17. [17]

    Mak- ing large language models better planners with reasoning- decision alignment

    Zhijian Huang, Tao Tang, Shaoxiang Chen, Sihao Lin, Ze- qun Jie, Lin Ma, Guangrun Wang, and Xiaodan Liang. Mak- ing large language models better planners with reasoning- decision alignment. InEuropean Conference on Computer Vision, pages 73–90. Springer, 2024. 2

    2024

  18. [18]

    Robotron- drive: All-in-one large multimodal model for autonomous driving

    Zhijian Huang, Chengjian Feng, Feng Yan, Baihui Xiao, Ze- qun Jie, Yujie Zhong, Xiaodan Liang, and Lin Ma. Robotron- drive: All-in-one large multimodal model for autonomous driving. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 8011–8021, 2025. 5, 6

    2025

  19. [19]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2, 4, 5, 6

    Pith/arXiv arXiv 2024

  20. [20]

    Drivelmm-o1: A step-by-step reasoning dataset and large multimodal model for driving scenario understanding.arXiv preprint arXiv:2503.10621, 2025

    Ayesha Ishaq, Jean Lahoud, Ketan More, Omkar Thawakar, Ritesh Thawkar, Dinura Dissanayake, Noor Ahsan, Yuhao Li, Fahad Shahbaz Khan, Hisham Cholakkal, et al. Drivelmm-o1: A step-by-step reasoning dataset and large multimodal model for driving scenario understanding.arXiv preprint arXiv:2503.10621, 2025. 5, 6

    Pith/arXiv arXiv 2025

  21. [21]

    Bench2drive: Towards multi-ability bench- marking of closed-loop end-to-end autonomous driving.Ad- vances in Neural Information Processing Systems, 37:819– 844, 2024

    Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability bench- marking of closed-loop end-to-end autonomous driving.Ad- vances in Neural Information Processing Systems, 37:819– 844, 2024. 3

    2024

  22. [22]

    Level-5 autonomous driving—are we there yet? a review of research literature.ACM Computing Surveys (CSUR), 55(2): 1–38, 2022

    Manzoor Ahmed Khan, Hesham El Sayed, Sumbal Malik, Talha Zia, Jalal Khan, Najla Alkaabi, and Henry Ignatious. Level-5 autonomous driving—are we there yet? a review of research literature.ACM Computing Surveys (CSUR), 55(2): 1–38, 2022. 2

    2022

  23. [23]

    Textual explanations for self-driving ve- hicles.Proceedings of the European Conference on Com- puter Vision (ECCV), 2018

    Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving ve- hicles.Proceedings of the European Conference on Com- puter Vision (ECCV), 2018. 3

    2018

  24. [24]

    Joint attention in autonomous driving (jaad).arXiv preprint arXiv:1609.04741, 2016

    Iuliia Kotseruba, Amir Rasouli, and John K Tsotsos. Joint attention in autonomous driving (jaad).arXiv preprint arXiv:1609.04741, 2016. 3

    Pith/arXiv arXiv 2016

  25. [25]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36:28541–28564,

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36:28541–28564,

  26. [26]

    Automated evaluation of large vision- language models on self-driving corner cases.CoRR, 2024

    Yanze Li, Wenhua Zhang, Kai Chen, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated evaluation of large vision- language models on self-driving corner cases.CoRR, 2024. 3

    2024

  27. [27]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vi- sion, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vi- sion, pages 216–233. Springer, 2024. 3

    2024

  28. [28]

    Can lvlms obtain a driver’s license? a benchmark towards reliable agi for autonomous driving

    Yuhang Lu, Yichen Yao, Jiadong Tu, Jiangnan Shao, Yuexin Ma, and Xinge Zhu. Can lvlms obtain a driver’s license? a benchmark towards reliable agi for autonomous driving. InProceedings of the AAAI Conference on Artificial Intelli- gence, pages 5838–5846, 2025. 2, 3

    2025

  29. [29]

    One million scenes for autonomous driving: Once dataset.NeurIPS, 2021

    Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhen- guo Li, Jie Yu, et al. One million scenes for autonomous driving: Once dataset.NeurIPS, 2021. 3

    2021

  30. [30]

    Lingoqa: Visual question answering for autonomous driv- ing

    Ana-Maria Marcu, Long Chen, Jan H ¨unermann, Alice Karn- sund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Visual question answering for autonomous driv- ing. InEuropean Conference on Computer Vision, pages 252–269. Springer, 2024. 3, 7

    2024

  31. [31]

    Exploring category-level articulated object pose tracking on se(3) manifolds, 2025

    Xianhui Meng, Yukang Huo, Li Zhang, Liu Liu, Haonan Jiang, Yan Zhong, Pingrui Zhang, Cewu Lu, and Jun Liu. Exploring category-level articulated object pose tracking on se(3) manifolds, 2025. 3

    2025

  32. [32]

    Reason2drive: Towards interpretable and chain-based reasoning for au- tonomous driving

    Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Reason2drive: Towards interpretable and chain-based reasoning for au- tonomous driving. InEuropean Conference on Computer Vision, pages 292–308. Springer, 2024. 3

    2024

  33. [33]

    Driving behav- ior extraction based on eyes movement patterns

    Alexandru M Pescaru and Mihai V Micea. Driving behav- ior extraction based on eyes movement patterns. In2024 IEEE 18th International Symposium on Applied Computa- tional Intelligence and Informatics (SACI), pages 000139– 000144. IEEE, 2024. 2, 3

    2024

  34. [34]

    Coda: convergence diagnosis and output analysis for mcmc.R news, 6(1):7–11, 2006

    Martyn Plummer, Nicky Best, Kate Cowles, Karen Vines, et al. Coda: convergence diagnosis and output analysis for mcmc.R news, 6(1):7–11, 2006. 3

    2006

  35. [35]

    Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario. InProceedings of the AAAI Conference on Artificial Intelli- gence, pages 4542–4550, 2024. 2, 3

    2024

  36. [36]

    Stability under scrutiny: Benchmarking repre- sentation paradigms for online hd mapping.arXiv preprint arXiv:2510.10660, 2025

    Hao Shan, Ruikai Li, Han Jiang, Yizhe Fan, Ziyang Yan, Bohan Li, Xiaoshuai Hao, Hao Zhao, Zhiyong Cui, Yilong Ren, et al. Stability under scrutiny: Benchmarking repre- sentation paradigms for online hd mapping.arXiv preprint arXiv:2510.10660, 2025. 2

    arXiv 2025

  37. [37]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. InEuropean conference on computer vision, pages 256–274. Springer, 2024. 2, 3

    2024

  38. [38]

    Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent.arXiv preprint arXiv:2411.02265, 2024

    Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, et al. Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent.arXiv preprint arXiv:2411.02265, 2024. 2, 5, 6

    Pith/arXiv arXiv 2024

  39. [39]

    A survey of end-to-end driving: Architectures and training methods.IEEE Trans- actions on Neural Networks and Learning Systems, 33(4): 1364–1384, 2020

    Ardi Tampuu, Tambet Matiisen, Maksym Semikin, Dmytro Fishman, and Naveed Muhammad. A survey of end-to-end driving: Architectures and training methods.IEEE Trans- actions on Neural Networks and Learning Systems, 33(4): 1364–1384, 2020. 3

    2020

  40. [40]

    Mimo-vl technical report, 2025

    Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, Kainan Bao, Hao Tian, Hailin Zhang, Gang Wang, Dawei Zhu, Cici, Chenhong He, Bowen Ye, Bowen Shen, Zihan Zhang, Zihan Jiang, Zhix- ian Zheng, Zhichao Song, Zhenbo Luo, Yue Yu, Yudong Wang, Yuanyuan Tian, Yu Tu, Yihan Yan, Yi Huan...

    2025

  41. [41]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 2

    Pith/arXiv arXiv 2023

  42. [42]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 3

    Pith/arXiv arXiv 2023

  43. [43]

    Jeffrey M. V oas. Pie: A dynamic failure-based tech- nique.IEEE Transactions on software Engineering, 18(8): 717, 1992. 3

    1992

  44. [44]

    X-sam: From segment anything to any segmentation.arXiv preprint arXiv:2508.04655, 2025

    Hao Wang, Limeng Qiao, Zequn Jie, Zhijian Huang, Chengjian Feng, Qingfang Zheng, Lin Ma, Xiangyuan Lan, and Xiaodan Liang. X-sam: From segment anything to any segmentation.arXiv preprint arXiv:2508.04655, 2025. 2

    arXiv 2025

  45. [45]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 3

    Pith/arXiv arXiv 2024

  46. [46]

    Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning.CoRR,

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning.CoRR,

  47. [47]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 5, 6

    Pith/arXiv arXiv 2025

  48. [48]

    Mimo: Unlocking the reasoning potential of language model–from pretraining to posttraining.arXiv preprint arXiv:2505.07608, 2025

    LLM Xiaomi, Bingquan Xia, Bowen Shen, Dawei Zhu, Di Zhang, Gang Wang, Hailin Zhang, Huaqiu Liu, Jiebao Xiao, Jinhao Dong, et al. Mimo: Unlocking the reasoning potential of language model–from pretraining to posttraining.arXiv preprint arXiv:2505.07608, 2025. 2

    Pith/arXiv arXiv 2025

  49. [49]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model.IEEE Robotics and Automation Let- ters, 2024

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model.IEEE Robotics and Automation Let- ters, 2024. 2

    2024

  50. [50]

    Aide: A vision-driven multi-view, multi-modal, multi-tasking dataset for assistive driving per- ception

    Dingkang Yang, Shuai Huang, Zhi Xu, Zhenpeng Li, Shunli Wang, Mingcheng Li, Yuzheng Wang, Yang Liu, Kun Yang, Zhaoyu Chen, Yan Wang, Jing Liu, Peixuan Zhang, Peng Zhai, and Lihua Zhang. Aide: A vision-driven multi-view, multi-modal, multi-tasking dataset for assistive driving per- ception. InProceedings of the IEEE/CVF International Con- ference on Comput...

  51. [51]

    Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?arXiv preprint arXiv:2408.13257, 2024

    Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qing- song Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?arXiv preprint arXiv:2408.13257, 2024. 3

    Pith/arXiv arXiv 2024

  52. [52]

    pedestri- ans

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 3 Is Your VLM for Autonomous Driving Safety-Ready? A Comprehensive Benchmark for Evaluating Exter...

    Pith/arXiv arXiv 2025