Pith. sign in

REVIEW 3 major objections 6 minor 59 references

This paper presents IEDD, a mining and synthesis pipeline that extracts 7.31 million ego-centric interaction segments from five public trajectory datasets, labels each with physics-based intensity and efficiency scores, and turns a subset i

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:16 UTC pith:WKNJ6ES5

load-bearing objection A large, transparently built interaction dataset worth a referee's time, but the efficiency metric in Eq. (8) is dimensionally wrong and every E_Veh label inherits the error. the 3 major comments →

arxiv 2602.20575 v2 pith:WKNJ6ES5 submitted 2026-02-24 cs.CV

An interactive enhanced driving dataset for autonomous driving

Haojie Feng , Xinrui Zhang , Mengjie Tian , Peizhi Zhang , Zhuoren Li , Junpeng Huang , Xiurong Wang , Junfan Zhu
show 3 more authors
Jianzhou Wang Dongxiao Yin Lu Xiong
This is my paper
classification cs.CV
keywords interaction miningautonomous driving datasettrajectory analysisvision-language-action modelsbird's-eye-view videovisual question answeringdriving interaction intensitymulti-agent scenarios
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to solve a data bottleneck for autonomous driving: real-world driving logs are dominated by boring, non-interactive driving, while the rare negotiation-heavy events (merging, crossing, head-on encounters, forced yielding) are exactly what Vision-Language-Action models need. It builds a scalable pipeline that homogenizes five heterogeneous trajectory datasets, detects spatio-temporal overlaps between vehicles, classifies each overlap into four interaction types, and aggregates overlapping pairwise events into multi-agent groups. For each extracted segment it computes a two-axis interaction score: an intensity metric combining pose adjustment, risk variation, and potential-field pressure, and an efficiency metric combining path, time, and smoothness. A further stage renders bird's-eye-view videos from the real trajectories and fills pre-defined language templates with the structured semantics, producing the IEDD-VQA instruction set whose language descriptions are strictly tied to physical ground truth. A sympathetic reader would care because the paper offers a low-cost way to turn existing datasets into a large, reusable interaction corpus for training and evaluating driving VLMs.

Core claim

The central claim is that a trajectory-only pipeline can automatically transform heterogeneous naturalistic driving data into a large-scale, interaction-focused multimodal corpus without new sensor collection or manual annotation. The paper reports 7.31 million extracted ego-centric interaction segments, of which 6.66 million involve more than two agents, with a deliberately balanced distribution across head-on, car-following, merging, and crossing behaviors. It further claims that the trajectory-derived intensity and efficiency metrics provide a defensible physical ground truth for language generation, and that IEDD-VQA's four-level structure (perception, description, quantification, reason

What carries the argument

The load-bearing mechanism is the interaction mining and quantification cascade: trajectory cleaning and resampling, spatio-temporal intersection search with a double-pointer sliding window (Dsearch = 2 m, Tsearch = 3 s), two-stage classification into car-following versus merging/crossing/head-on using heading thresholds (30° and 160°), and recursive multi-agent aggregation around anchor vehicles. Intensity is computed as a weighted sum of pose adjustment, TTC/PET variation, and an artificial potential-field term, with scenario-specific weight triples; efficiency is the product of path-, time-, and smoothness-based scores. The synthesis stage converts continuous trajectories into discrete be

Load-bearing premise

The hand-set thresholds and metric weights in Table 1 determine which trajectory pairs count as interactions and how intense they are, so if those values are arbitrary or miscalibrated, every derived count, type distribution, and VQA ground-truth label inherits that arbitrariness.

What would settle it

Take a random sample of 200 interaction segments from IEDD, show them to human drivers (blinded to the dataset's labels), and ask them to mark whether an interaction occurred, what type it was, and how intense it felt; if inter-rater agreement with IEDD's labels is low, the mining thresholds do not capture human notions of interaction and the quantitative labels would not support the paper's claimed semantic alignment.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the pipeline works as described, any trajectory dataset can be turned into interaction-rich VLA training data without new hardware or manual annotation, making large-scale interaction corpora substantially cheaper to produce.
  • The IEDD-VQA benchmark's four-level structure gives a concrete way to measure where driving VLMs fail: near-perfect language description but severely inaccurate physical quantification, as shown by the large MAE values.
  • The dataset's 91% multi-agent share directly addresses the sparsity of negotiation scenarios in existing trajectory datasets, potentially enabling models to learn group interactions rather than only pairwise ones.
  • Fine-tuning on IEDD-VQA can slash physical-quantification error from 1855.55 to 0.30 while raising logical accuracy from 0.15 to 0.53, demonstrating that trajectory-derived metrics are learnable supervision signals.
  • The observed drop in counterfactual reasoning after fine-tuning warns that domain specialization without task replay trades away general reasoning ability, informing future instruction-tuning designs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The hand-set thresholds and metric weights are never validated against human judgments, so the 7.31M count and every VQA label inherit their arbitrariness; a sensitivity analysis varying Dsearch, Tsearch, Twindow, and the intensity weights would reveal whether the interaction distribution is stable or an artifact of parameter choice.
  • The counterfactual reasoning questions are generated from real trajectories by hypothesizing different ego actions, so they test whether a model can produce linguistically plausible alternatives, not whether those alternatives would physically unfold as described; a causal simulator or trajectory predictor would be needed to make L4 a rigorous physical counterfactual test.
  • The 'open-source models outperform closed-source models' finding is based on only 100 evaluation scenarios and could shift with a larger, more diverse test set; a stratified sample across the four interaction types and five source datasets would make that claim more robust.
  • The intensity-efficiency dual metric could be repurposed as a reward signal for closed-loop driving simulation or as a data filter to select high-value segments for human review, but both uses require the metric to first be calibrated against human risk perception.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents IEDD, a large-scale interaction-oriented driving dataset mined from five public trajectory datasets (Lyft Level 5, Waymo, nuPlan, INTERACTION, and SIND), with 7.31 million ego-centric interaction segments, including 6.66 million multi-agent cases. Each segment is annotated with trajectory-derived interaction intensity Q_i and efficiency E_i metrics, and a subset, IEDD-VQA, provides trajectory-reconstructed BEV videos, structured interaction semantics, and multi-turn question-answer pairs. The authors also report a hierarchical VLM evaluation benchmark (L1–L4) and fine-tuning experiments on Qwen2.5-VL-7B.

Significance. If the metric system and thresholds are accepted, IEDD would be a substantial and reusable resource: it integrates five heterogeneous trajectory sources, provides a transparent mining pipeline, and ships public code and data on GitHub/Zenodo. The internal counts in Table 2 are consistent, and the availability of scripts for trajectory-to-VQA generation is a clear strength. However, the claimed quantitative ground truth rests on two unverified pillars: the efficiency metric in Eq. (8) is dimensionally inconsistent as written, and the interaction thresholds/weights in Table 1 are hand-set with no sensitivity analysis or calibration. Because L3 benchmark answers and the fine-tuning experiments are validated against labels generated by the same formulas, the empirical evidence is self-referential at present. The dataset concept is valuable, but the central quantitative labels need correction and independent validation before the descriptor can be accepted.

major comments (3)
  1. [Methods, Eq. (8)] The path consistency metric is dimensionally inconsistent. Eq. (8) defines E_pi = [(xi0−xiT)^2 + (yi0−yiT)^2] / ∫_0^T vi(t) dt. The numerator has units m^2 and the denominator has units m, so E_pi is not dimensionless and cannot lie in the claimed [0,1] range of Eq. (7). For a straight 100 m trajectory, E_pi = 100. This appears to be a missing square root: the intended ratio d_ref/d_actual would be sqrt((xi0−xiT)^2 + (yi0−yiT)^2) / ∫_0^T vi(t) dt. As written, every E_Veh value in the released dataset and every L3 quantification ground truth derived from Eq. (8) is invalid. The authors must correct the formula and, if E_Veh values were computed with the published formula, regenerate the dataset or explicitly confirm the implemented definition.
  2. [Table 1 and Methods] All interaction labels and quantitative metrics depend on hand-set thresholds and weights: Dsearch=2 m, Tsearch=3 s, Twindow=5 s, θmerge=30°, θcross=160°, dinter=50 m, tinter=3 s, scenario-specific weights (e.g., 0.25/0.35/0.40 for merging), and the intensity/efficiency parameters in Table 1. The paper provides no sensitivity analysis, no comparison against human judgments of interaction, and no calibration against established surrogate safety measures. Consequently, the 7.31M count, the type distribution, and every Q_i and E_i label inherit this arbitrariness. At minimum, the authors should report sensitivity of the counts and metric distributions to plausible variations in the thresholds, and provide some external validation (e.g., human-rated interaction samples or comparison with TTC/PET-based conflict classification) to support the claim that these values are physically meaningful g
  3. [Technical Validation / IEDD-VQA generation] The benchmark and fine-tuning results are self-referential with respect to the dataset's quantitative annotations. L3 MAE is computed against Q_i and E_i values generated by the same formulas (Eqs. 3–10) used to create the training labels, and the L2/L4 evaluation relies on LLM-as-a-Judge (GLM-4.7) with no human validation or inter-annotator agreement reported. The claim that fine-tuning 'learns an internal representation mechanism that maps visual features to physical parameters' is therefore only evidence that the model fits the dataset's internally generated labels. This does not invalidate the dataset's utility, but it does not yet establish transferable physical understanding. The authors should add an independent evaluation component, such as human-annotated held-out samples, external conflict metrics, or a cross-dataset transfer test.
minor comments (6)
  1. [Methods, threshold symbols] Several threshold symbols render as '?search' or '??????' in the text (e.g., in the interaction mining paragraph and the classification description). These should be replaced with the actual symbols defined in Table 1.
  2. [Table 1] The rows for ws, wr, wp list three value sets but do not label which row corresponds to which scenario; add a 'Scenario' column or explicit row labels.
  3. [Data Records / Table 3] The E_Veh field is defined only as 'E values of involved vehicles'. State the formula, units (or normalized range), and which vehicle's efficiency is recorded before the dataset is released.
  4. [Technical Validation, WIS] The MAE-to-normalized-score conversion in L3 uses a tolerance threshold of 0.5, but the relevant MAE values in Table 4 are far larger (e.g., 1855.5); explain how the normalization behaves outside the tolerance and whether the L3 score saturates at zero for most models.
  5. [References] Several references are dated 2026 (e.g., References 7 and 57), which is after the current manuscript year; verify the bibliographic data and publication status.
  6. [Tables 4–6] The column headers are split across lines in a way that makes the table hard to read, and MAE values have inconsistent decimal formatting. A cleaner layout with units and consistent precision would help reproducibility.

Circularity Check

0 steps flagged

No circularity: labels are produced by transparent rule-based formulas from external trajectories; the Eq. (8) normalization issue is a correctness concern, not a circular derivation.

full rationale

The paper's central deliverables—7.31M interaction segments, interaction types, Q_i intensity traces, and E_Veh efficiency scores—are outputs of explicit preprocessing and hand-set rule-based thresholds (Table 1), applied to five external naturalistic trajectory datasets. No parameter is fitted to a target quantity and then renamed as a prediction; the IEDD-VQA question-answer pairs are generated from the same structured semantics and templates that define the dataset, which is a standard benchmark-construction choice rather than a derivation that assumes its own conclusion. The self-citations ([1], [3]) are background support for the importance of interaction scenarios, not load-bearing evidence for the dataset's correctness, and no uniqueness theorem or ansatz is imported from the authors' prior work. The main concern is internal validity rather than circularity: Eq. (8) as printed defines E_pi as squared Euclidean distance divided by path length, which is dimensionally inconsistent and cannot lie in the claimed [0,1] range; also, the Table 1 thresholds are not validated against human labels or external benchmarks. These are correctness/validity limitations and, under the stated circularity criteria, do not make the derivation circular. The finding is therefore no significant circularity, score 0.

Axiom & Free-Parameter Ledger

8 free parameters · 4 axioms · 2 invented entities

The central claims rest less on mathematical assumptions than on domain choices: hand-set mining thresholds, metric weights, and the sufficiency of template-generated language. These choices determine every interaction label and every benchmark answer, and none are validated externally.

free parameters (8)
  • Mining thresholds (Dsearch, Tsearch, Twindow) = 2 m, 3 s, 5 s
    Hand-set thresholds define which trajectory pairs count as interactions; no sensitivity analysis or calibration is reported (Table 1).
  • Type thresholds (θmerge, θcross) = 30°, 160°
    Heading-angle cutoffs determine the merge/cross/head-on classification and thereby the dataset's type distribution (Table 1).
  • State trigger thresholds (dinter, tinter) = 50 m, 3 s
    Spatio-temporal trigger for an 'interactive state' in the stochastic process model (Methods, Table 1).
  • Intensity scenario weights (ws, wr, wp) = Merging 0.25/0.35/0.40; Crossing 0.20/0.55/0.25; Head-on 0.15/0.65/0.20
    Category-adaptive weights are asserted to reflect driver risk perception but are not derived from data or human judgments; they determine every Q_i ground-truth value (Eq. 3, Table 1).
  • Intensity parameters (βQ, vlim, alim, γQ) = 0.4, 20 m/s, 3 m/s², 0.4
    Normalization and sensitivity constants in Eqs. 4-5; chosen by hand and not justified empirically (Table 1).
  • Potential-field parameters (d0, v0, κv) = 8, 5, 1
    Shape and sensitivity constants in Eq. 6; hand-set and no validation is provided (Table 1).
  • Efficiency parameters (αE, anormal, βE) = 0.8, 2 m/s², 1.2
    Sensitivity coefficients in Eqs. 9-10; no external grounding for the claimed efficiency interpretation (Table 1).
  • WIS aggregation weights and L3 MAE tolerance = 0.2/0.2/0.2/0.4; tolerance 0.5
    Benchmark scoring weights are chosen to emphasize L4 reasoning; the MAE normalization threshold is also arbitrary (Eq. 12, Technical Validation).
axioms (4)
  • domain assumption Spatio-temporal proximity plus heading-angle differences are sufficient to identify interaction type and interaction windows.
    The mining pipeline ignores lane topology, road rules, and driver intention, yet these are not needed in the method (Methods, 'Naturalistic driving trajectory preprocessing and scenario slicing').
  • domain assumption TTC, PET, and artificial potential-field formulas are valid surrogate risk measures.
    Eqs. 5-6 rely on standard but approximate surrogate safety measures from the traffic-safety literature; the paper gives no validation for the AD interaction context.
  • ad hoc to paper The multiplicative efficiency score E = Ep * Et * Es is a valid measure of traversal quality.
    The decomposition into path, time, and smoothness is reasonable but is introduced without comparison to human ratings or external safety metrics (Eq. 7-10).
  • ad hoc to paper Template-generated language from structured semantics is 'hallucination-free' and aligned with visual content.
    The paper claims hallucination-free generation and strict alignment, but no human verification or inter-annotator agreement study is reported (Multimodal interaction instruction data generation pipeline).
invented entities (2)
  • Interaction intensity Q_i(t) composite metric no independent evidence
    purpose: Provides a scalar intensity label for every interaction segment and serves as the L3 quantification ground truth in IEDD-VQA.
    Defined by Eqs. 3-6 with hand-picked weights; no external or human validation and no falsifiable prediction outside the dataset.
  • Interaction efficiency E_i composite metric no independent evidence
    purpose: Provides a traversal-quality score for each interaction and is used as part of the VQA ground truth and benchmark.
    Defined by Eqs. 7-10 as a product of three hand-parameterized sub-scores; no independent benchmark or human rating supports its meaning.

pith-pipeline@v1.3.0-alltime-deepseek · 19641 in / 12299 out tokens · 121114 ms · 2026-08-02T21:16:28.484955+00:00 · methodology

0 comments
read the original abstract

Driving interaction data are important for training and evaluating autonomous drivingVision-Language-Action (VLA) models, but existing datasets contain limited denseinteraction samples and weak alignment between trajectories, visual inputs, and languageannotations. This work presents the Interactive Enhanced Driving Dataset (IEDD), alarge-scale interaction-oriented dataset constructed from five naturalistic trajectory datasets:Lyft Level 5, Waymo, nuPlan, INTERACTION, and SIND. IEDD contains 7.31 millionego-centric interaction segments, including 6.66 million multi agents cases, covering head-on,car-following, merging, and crossing interactions. Each segment is associated withtrajectory-derived interaction metrics describing interaction intensity and efficiency. Based onthese annotations, IEDD-VQA further provides trajectory-reconstructed BEV videos,structured interaction semantics, and multi-turn question-answer pairs. The dataset cansupport interaction mining, long-tail scenario analysis, VLA instruction tuning, andhierarchical evaluation of perception, behavior description, physical quantification, andcounterfactual reasoning.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 1 canonical work pages

  1. [1]

    Real-world troublemaker: A 5g cloud-controlled track testing framework for automated driving systems in safety-critical interaction scenarios[J]

    Zhang X, Xiong L, Zhang P, et al. Real-world troublemaker: A 5g cloud-controlled track testing framework for automated driving systems in safety-critical interaction scenarios[J]. IEEEInternetofThingsJournal,2025

    2025

  2. [2]

    A matched case-control analysis of autonomous vs human-driven vehicleaccidents[J].Naturecommunications,2024,15(1):4931

    Abdel-Aty M, Ding S. A matched case-control analysis of autonomous vs human-driven vehicleaccidents[J].Naturecommunications,2024,15(1):4931

    2024

  3. [3]

    Uncertainty-Aware Safety-Critical Decision and Control for Autonomous Vehicles at Unsignalized Intersections[J]

    Yu R, Li Z, Xiong L, et al. Uncertainty-Aware Safety-Critical Decision and Control for Autonomous Vehicles at Unsignalized Intersections[J]. arXiv preprint arXiv:2505.19939, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Survey of General End-to-End Autonomous Driving: A UnifiedPerspective[J].AuthoreaPreprints,2025

    Yang Y, Han C, Mao R, et al. Survey of General End-to-End Autonomous Driving: A UnifiedPerspective[J].AuthoreaPreprints,2025

    2025

  5. [5]

    Vlm-ad: End-to-end autonomous driving through vision-languagemodelsupervision[J].arXivpreprintarXiv:2412.14446,2024

    Xu Y, Hu Y, Zhang Z, et al. Vlm-ad: End-to-end autonomous driving through vision-languagemodelsupervision[J].arXivpreprintarXiv:2412.14446,2024

    Pith/arXiv arXiv 2024

  6. [6]

    Drivevlm: The convergence of autonomous driving and large vision-languagemodels[J].arXivpreprintarXiv:2402.12289,2024

    Tian X, Gu J, Li B, et al. Drivevlm: The convergence of autonomous driving and large vision-languagemodels[J].arXivpreprintarXiv:2402.12289,2024

    Pith/arXiv arXiv 2024

  7. [7]

    LatentVLA: Efficient Vision-Language Models forAutonomous DrivingviaLatentActionPrediction[J].arXivpreprintarXiv:2601.05611,2026

    Xie C, Sun B, Li T, et al. LatentVLA: Efficient Vision-Language Models forAutonomous DrivingviaLatentActionPrediction[J].arXivpreprintarXiv:2601.05611,2026

    arXiv 2026

  8. [8]

    Discrete diffusion for reflective vision-language-action modelsinautonomousdriving[J].arXivpreprintarXiv:2509.20109,2025

    Li P, Zheng Y, Wang Y, et al. Discrete diffusion for reflective vision-language-action modelsinautonomousdriving[J].arXivpreprintarXiv:2509.20109,2025

    arXiv 2025

  9. [9]

    Fastdrivevla: Efficient end-to-end driving via plug-and-play reconstruction-basedtokenpruning[J].arXivpreprintarXiv:2507.23318,2025

    Cao J, Zhang Q, Jia P, et al. Fastdrivevla: Efficient end-to-end driving via plug-and-play reconstruction-basedtokenpruning[J].arXivpreprintarXiv:2507.23318,2025

    arXiv 2025

  10. [10]

    Emma: End-to-end multimodal model for autonomous driving[J].arXivpreprintarXiv:2410.23262,2024

    Hwang J J, Xu R, Lin H, et al. Emma: End-to-end multimodal model for autonomous driving[J].arXivpreprintarXiv:2410.23262,2024

    Pith/arXiv arXiv 2024

  11. [11]

    Xie Y, Xu R, He T, et al. S4-driver: Scalable self-supervised driving multimodal large language model withspatio-temporal visual representation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA: IEEE, 2025:1622-1632

    2025

  12. [12]

    Motionlm: Multi-agent motion forecasting as language modeling[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).Paris,France:IEEE,2023:8579-8590

    Seff A, Cera B, Chen D, et al. Motionlm: Multi-agent motion forecasting as language modeling[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).Paris,France:IEEE,2023:8579-8590

    2023

  13. [13]

    On the assessment of vehicle trajectory data accuracy and application to the Next Generation SIMulation (NGSIM) program data[J]

    Punzo V, Borzacchiello M T, Ciuffo B. On the assessment of vehicle trajectory data accuracy and application to the Next Generation SIMulation (NGSIM) program data[J]. TransportationResearchPartC:EmergingTechnologies,2011,19(6):1243-1262

    2011

  14. [14]

    Caesar H, Bankiti V, Lang A H, et al. nuScenes: A multimodal dataset for autonomous driving[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).Seattle,WA,USA:IEEE,2020:11621-11631

    2020

  15. [15]

    One thousand and one hours: Self-driving motion prediction dataset[C]//Proceedings of the 4th Conference on Robot Learning (CoRL)

    Houston J, Zuidhof G, Bergamini L, et al. One thousand and one hours: Self-driving motion prediction dataset[C]//Proceedings of the 4th Conference on Robot Learning (CoRL). Cambridge,MA,USA:PMLR,2021:409-418

    2021

  16. [16]

    Sun P, Kretzschmar H, Dotiwalla X, et al. Scalability in perception for autonomous driving: Waymo open dataset[C]//Proceedings of the IEEE/CVF Conference on Computer VisionandPatternRecognition(CVPR).Seattle,WA,USA:IEEE,2020:2446-2454

    2020

  17. [17]

    Curse of rarity for autonomous vehicles[J]

    Liu H X, Feng S. Curse of rarity for autonomous vehicles[J]. nature communications, 2024,15(1):4808

    2024

  18. [18]

    Geiger A, Lenz P, Urtasun R. Are we ready for autonomous driving? The KITTI vision benchmark suite[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR).Providence,RI,USA:IEEE,2012:3354-3361

    2012

  19. [19]

    Yu F, Chen H, Wang X, et al. BDD100K: A diverse driving dataset for heterogeneous multitask learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition(CVPR).Seattle,WA,USA:IEEE,2020:2636-2645

    2020

  20. [20]

    arXivpreprintarXiv:2106.11037,2021

    MaoJ,NiuM,JiangC,etal.Onemillionscenesforautonomousdriving:Oncedataset[J]. arXivpreprintarXiv:2106.11037,2021

    Pith/arXiv arXiv 2021

  21. [21]

    DriveLM: Driving with graph visual question answering[C]//Proceedings ofthe European Conferenceon ComputerVision(ECCV)

    Sima C, Renz K, Chitta K, et al. DriveLM: Driving with graph visual question answering[C]//Proceedings ofthe European Conferenceon ComputerVision(ECCV). Milan, Italy:SpringerNatureSwitzerland,2025:256-274

    2025

  22. [22]

    Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving[J]

    Wang W, Xie J, Hu C Y, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving[J]. arXiv preprint arXiv:2312.09245, 2023

    arXiv 2023

  23. [23]

    Gpt-driver: Learning to drive with gpt[J]

    Mao J, Qian Y, Ye J, et al. Gpt-driver: Learning to drive with gpt[J]. arXiv preprint arXiv:2310.01415,2023

    Pith/arXiv arXiv 2023

  24. [24]

    A language agent for autonomous driving[J]

    Mao J, Ye J, Qian Y, et al. A language agent for autonomous driving[J]. arXiv preprint arXiv:2311.10813,2023

    Pith/arXiv arXiv 2023

  25. [25]

    CARLA: An open urban driving simulator[C]//Proceedings ofthe1stConferenceonRobotLearning(CoRL).MountainView, CA,USA:PMLR,2017:1-16

    Dosovitskiy A, Ros G, Codevilla F, et al. CARLA: An open urban driving simulator[C]//Proceedings ofthe1stConferenceonRobotLearning(CoRL).MountainView, CA,USA:PMLR,2017:1-16

    2017

  26. [26]

    Gaidon A, Wang Q, Cabon Y, et al. Virtual worlds as proxy for multi-object tracking analysis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR).LasVegas,NV,USA:IEEE,2016:4340-4349

    2016

  27. [27]

    Jiang, X., Zhao, X., Liu, Y. et al. Anaturalistic trajectory dataset with dense interaction for autonomous driving. Sci Data 12, 1084 ,2025. https://doi.org/10.1038/s41597-025-05344-7

    work page doi:10.1038/s41597-025-05344-7 2025

  28. [28]

    PODAR:Acollision risk model offering valid signals for vehicularinteractions[J].IEEEIntelligentTransportationSystemsMagazine,2025

    Chen C, Liu Z, Liang W, et al. PODAR:Acollision risk model offering valid signals for vehicularinteractions[J].IEEEIntelligentTransportationSystemsMagazine,2025

    2025

  29. [29]

    VistaScenario: Interaction scenario engineering for vehicleswithintelligentsystemsfortransportautomation[J].IEEETransactionsonIntelligent Vehicles,2024

    Chang C, Zhang J, Ge J, et al. VistaScenario: Interaction scenario engineering for vehicleswithintelligentsystemsfortransportautomation[J].IEEETransactionsonIntelligent Vehicles,2024

    2024

  30. [30]

    VanDerHorstR.Time-to-collisionandcollisionavoidancesystems[J].1994

    1994

  31. [31]

    HaywardJC.Nearmissdeterminationthroughuseofascaleofdanger[J].1972

    1972

  32. [32]

    Surrogate safety measures from traffic simulation models[J]

    Gettman D, Head L. Surrogate safety measures from traffic simulation models[J]. Transportationresearchrecord,2003,1840(1):104-115

    2003

  33. [33]

    AllenBL,ShinBT,CooperPJ.Analysisoftrafficconflictsandcollisions[R].1978

    1978

  34. [34]

    Risk assessment in autonomous driving: a comprehensive survey of risk sources, methodologies, and system architectures[J]

    Lu D, Du H, Wu Z, et al. Risk assessment in autonomous driving: a comprehensive survey of risk sources, methodologies, and system architectures[J]. Autonomous Intelligent Systems,2025,5(1):24

    2025

  35. [35]

    Survey on scenario-based safety assessment of automatedvehicles[J].IEEEaccess,2020,8:87456-87477

    Riedmaier S, Ponn T, Ludwig D, et al. Survey on scenario-based safety assessment of automatedvehicles[J].IEEEaccess,2020,8:87456-87477

    2020

  36. [36]

    Caesar H, Kabzan J,TanK S, et al.nuplan:Aclosed-loop ml-based planning benchmark forautonomousvehicles[J].arXivpreprintarXiv:2106.11810,2021

    Pith/arXiv arXiv 2021

  37. [37]

    Chang M F, Lambert J, Sangkloy P, et al. Argoverse: 3D tracking and forecasting with rich maps[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).LongBeach,CA,USA:IEEE,2019:8748-8757

    2019

  38. [38]

    Wilson B, Qi W,AgarwalT, et al.Argoverse 2: Next generation datasets for self-driving perceptionandforecasting[J].arXivpreprintarXiv:2301.00493,2023

    Pith/arXiv arXiv 2023

  39. [39]

    Krajewski R, Bock J, Kloeker L, etal.The highD dataset:Adrone dataset of naturalistic vehicle trajectories on German highways for validation of highly automated driving systems[C]//Proceedings of the 21st International Conference on Intelligent Transportation Systems(ITSC).Maui,HI,USA:IEEE,2018:2118-2125

    2018

  40. [40]

    BockJ,KrajewskiR,MoersT,etal.TheinDdataset:Adronedatasetofnaturalisticroad user trajectories at German intersections[C]//Proceedings of the IEEE Intelligent Vehicles Symposium(IV).LasVegas,NV,USA:IEEE,2020:1929-1934

    2020

  41. [41]

    Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps[J]

    Zhan W, Sun L, Wang D, et al. Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps[J]. arXiv preprintarXiv:1910.03088,2019

    Pith/arXiv arXiv 1910

  42. [42]

    SIND: A drone dataset at signalized intersection in China[J]

    Xu Y, Shao W, Li J, et al. SIND: A drone dataset at signalized intersection in China[J]. arXivpreprintarXiv:2209.02297,2022

    Pith/arXiv arXiv 2022

  43. [43]

    Textual explanations for self-driving vehicles[C]//Proceedings of the European Conference on ComputerVision (ECCV)

    Kim J, Rohrbach A, Darrell T, et al. Textual explanations for self-driving vehicles[C]//Proceedings of the European Conference on ComputerVision (ECCV). Munich, Germany:Springer,2018:563-578

    2018

  44. [44]

    Malla S, Choi C, Dwivedi I, et al. DRAMA: Joint risk localization and captioning in driving[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision(WACV).Waikoloa,HI,USA:IEEE,2023:1043-1052

    2023

  45. [45]

    Qian T, Chen J, Zhuo L, et al. NuScenes-QA: Amulti-modal visual question answering benchmark for autonomous driving scenario[C]//Proceedings of the AAAI Conference on ArtificialIntelligence(AAAI).Vancouver,Canada:AAAIPress,2024,38(5):4542-4550

    2024

  46. [46]

    Nie M, Peng R, Wang C, et al. Reason2Drive: Towards interpretable and chain-based reasoning for autonomous driving[C]//Proceedings of theEuropean Conference on Computer Vision(ECCV).Milan,Italy:SpringerNatureSwitzerland,2025:292-308

    2025

  47. [47]

    Chain-of-thought prompting elicits reasoning in large language models[J]

    Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in neural information processing systems, 2022, 35: 24824-24837

    2022

  48. [48]

    Surds: Benchmarking spatial understanding and reasoning in driving scenarios with vision language models[J]

    Guo X, Zhang R, Duan Y, et al. Surds: Benchmarking spatial understanding and reasoning in driving scenarios with vision language models[J]. arXiv preprint arXiv:2411.13112,2024

    Pith/arXiv arXiv 2024

  49. [49]

    Impromptu vla: Open weights and open data for driving vision-language-actionmodels[J].arXivpreprintarXiv:2505.23757,2025

    Chi H, Gao H, Liu Z, et al. Impromptu vla: Open weights and open data for driving vision-language-actionmodels[J].arXivpreprintarXiv:2505.23757,2025

    Pith/arXiv arXiv 2025

  50. [50]

    Driveaction: A benchmark for exploring human-like driving decisionsinvlamodels[J].arXivpreprintarXiv:2506.05667,2025

    Hao Y, Li Z, Sun L, et al. Driveaction: A benchmark for exploring human-like driving decisionsinvlamodels[J].arXivpreprintarXiv:2506.05667,2025

    arXiv 2025

  51. [51]

    LLM-Driven Kernel Evolution:Automating Driver UpdatesinLinux[J].arXivpreprintarXiv:2511.18924,2025

    KharlamovaA, Liu J, Zhang T, et al. LLM-Driven Kernel Evolution:Automating Driver UpdatesinLinux[J].arXivpreprintarXiv:2511.18924,2025

    arXiv 2025

  52. [52]

    Extending large vision-language model for diverse interactivetasksinautonomousdriving[J].arXivpreprintarXiv:2505.08725,2025

    Zhao Z, Fu H, Liang D, et al. Extending large vision-language model for diverse interactivetasksinautonomousdriving[J].arXivpreprintarXiv:2505.08725,2025

    Pith/arXiv arXiv 2025

  53. [53]

    LingoQA: Visual question answering for autonomous driving[C]//Proceedings of the European Conference on Computer Vision (ECCV).Milan,Italy:SpringerNatureSwitzerland,2025:252-269

    Marcu A M, Chen L, Hünermann J, et al. LingoQA: Visual question answering for autonomous driving[C]//Proceedings of the European Conference on Computer Vision (ECCV).Milan,Italy:SpringerNatureSwitzerland,2025:252-269

    2025

  54. [54]

    MaY,CuiC,CaoX,etal.LaMPilot:Anopenbenchmarkdatasetforautonomousdriving with language model programs[C]//Proceedings of the IEEE/CVF Conference on Computer VisionandPatternRecognition(CVPR).Seattle,WA,USA:IEEE,2024:15141-15151

    2024

  55. [55]

    Driving with LLMs: Fusing object-level vector modality for explainable autonomous driving[C]//Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)

    Chen L, Sinavski O, Hünermann J, et al. Driving with LLMs: Fusing object-level vector modality for explainable autonomous driving[C]//Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Yokohama, Japan: IEEE, 2024: 14093-14100

    2024

  56. [56]

    Fang J, Li L, Zhou J, et al. Abductive ego-view accident video understanding for safe driving perception[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition(CVPR).Seattle,WA,USA:IEEE,2024:22030-22040

    2024

  57. [57]

    Team K, Bai T, Bai Y, et al. Kimi K2. 5: Visual Agentic Intelligence[J]. arXiv preprint arXiv:2602.02276,2026

    Pith/arXiv arXiv 2026

  58. [58]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities[J]

    Comanici G, Bieber E, Schaekermann M, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities[J]. arXivpreprintarXiv:2507.06261,2025

    Pith/arXiv arXiv 2025

  59. [59]

    Lora: Low-rank adaptation of large language models[J]

    Hu E J, Shen Y, Wallis P, et al. Lora: Low-rank adaptation of large language models[J]. Iclr,2022,1(2):3. Funding This work is supported by the National Key Research and Development Program of China underGrants 2024YFB2505704andtheNationalNaturalScienceFoundationofChinaunder Grants52325212. Author contributions Haojie Feng proposed the novel framework, de...

    2022