Pith. sign in

REVIEW 4 major objections 5 minor 43 references

VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a vision-language model's risk-attention dictionary can reconfigure the potential-function cost of a model predictive controller so that urban driving becomes safer and faster.

desk verdict A coherent fast-slow VLM-MPC framework with a real evidence gap: risk-zone gating can silently remove obstacles, and the paper's own intersection run shows it. read the letter →

arxiv 2507.15266 v1 pith:Z4FB3PVP submitted 2025-07-21 cs.RO cs.SYeess.SY

Haichao Liu , Haoren Guo , Pei Liu , Benshan Ma , Yuxiang Zhang , Jun Ma , Tong Heng Lee This is my paper
classification cs.ROcs.SYeess.SY
keywords autonomousdrivingvision-languagemodelpredictivecontrolpotentialfieldretrieval-augmentedgenerationtrajectorypredictionLSTMurban
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that the risk-attention step of human driving can be outsourced to a vision-language model and wired directly into a model predictive controller. The slow VLM/LLM layer reads camera views, retrieves similar past scenes from memory, and emits an attention dictionary; equation (24) turns that dictionary into on/off switches for the repulsive potential functions in the optimal-control cost. In the paper's reported tests, the switched controller achieves zero collisions and zero rule violations across four urban scenarios, completes routes in less time than the optimization baseline it extends, and runs with 9–13 ms solve times, while an ablation shows that each added reasoning component removes collisions and violations. The same reasoning pipeline, with minor prompt changes for six cameras, is demonstrated on a full-size vehicle on a university campus. If the claim holds, it points toward interpretable, context-aware driving that adapts to new scenes without retraining the low-level controller.

What carries the argument

The central objects are the attention dictionary A of Eq. (21) and the potential-field update rule of Eq. (24). The dictionary entries name the scene type, flag which spatial zones are risky, mark left and right lane markings as crossable or not, and set a block-to-wait bit for red lights; Eq. (24) maps those bits directly to the active repulsive potentials for vehicles, vulnerable road users, lane markings, and traffic lights, so the OCP cost is reconfigured without hand-tuning MPC weights. The supporting machinery is the fast-slow architecture (20 Hz MPC with 0.3–1.5 Hz semantic updates), the two-step RAG prompt with CLIP-based memory retrieval, and the multi-kernel moving-average plus LSTM predictor that feeds the OCP predicted states of selected agents.

What would settle it

Compare closed-loop outcomes in the intersection scenario under three conditions: VLM-generated attention, ground-truth attention, and all potential functions always on. A single trial where the always-on or ground-truth version brakes for a crossing vehicle that the VLM had not flagged, while the VLM-gated version collides, would falsify the claim that attention reconfiguration alone improves safety.

Watch

Extended reading notes

Core claim

The paper's central claim is that a small dictionary of semantic attention decisions is enough to make a safe urban controller smarter. The dictionary A = {scene, risk zones, candidate lanes, block-to-wait} is generated by a two-step RAG-augmented reasoning loop: a small vision-language model (SmolVLM in the experiments) converts the camera images into structured answers, and a 4B-parameter language model (Qwen3-4B) converts those answers into the dictionary. That dictionary is not used to generate control actions directly; it chooses which terms of the potential field F(penv, x) enter the OCP cost, so the MPC keeps its stabilizing dynamics while the semantic layer changes which constraints matter. A multi-kernel decomposed LSTM supplies short-horizon trajectory predictions for the agents that the dictionary selects. The paper claims this arrangement beats rule-based, learning-based, reactive, and prior optimization baselines in closed-loop simulation, and that the architecture transfers to a real vehicle with only prompt-level adaptation.

Load-bearing premise

The framework's safety rests on the vision-language model returning a correct attention dictionary almost every cycle, because a missed vehicle or pedestrian simply disappears from the controller's safety field; the paper reports one intersection time-to-collision alarm caused by exactly that kind of miss.

Editorial extensions

If this is right

  • Semantic attention lets the OCP ignore irrelevant traffic participants, cutting solve times to 9–13 ms while retaining safety constraints on the objects that matter.
  • In the four closed-loop scenarios, the reported framework finishes with zero collisions, zero rule violations, and shorter travel times than the UDMC optimization baseline it extends.
  • The ablation attributes the gains to three components: disabling two-step reasoning, risk zones, and memory-based learning yields two collisions and four rule violations in the intersection test, while enabling them in sequence eliminates both.
  • The prediction module reports 27.3% lower MSE and 14.3% lower RMSE than a Transformer baseline at 0.27 ms inference per batch, indicating the trajectory predictor adds accuracy without blocking real-time control.
  • The same reasoning loop transfers from four simulated cameras to six real-vehicle cameras with only prompt-level changes, suggesting the semantic layer is not tied to one sensor layout.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that a wrong dictionary entry is not a comfort issue but a safety-constraint removal: because Eq. (24) deactivates potentials, a missed pedestrian or red light leaves no residual repulsive term except the general tracking cost.
  • An obvious test the authors do not run is to inject bit flips into the attention dictionary during the intersection scenario and compare collision and time-to-collision outcomes against an always-on-potential version; this would quantify how much safety margin the 20 Hz MPC preserves when the slow layer errs.
  • The RAG memory mechanism could be developed into lifelong learning: accumulating retrieved dialogues from new cities or sensor rigs would let the semantic layer adapt without finetuning, which the discussion already names as future work.
  • The same attention-dictionary gating could apply to cooperative or multi-vehicle settings, where each agent's risk zones come from its own VLM and the potential fields are shared across the fleet.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes VLM-UDMC, a hierarchical urban-driving framework combining a slow VLM/LLM subsystem with a fast MPC subsystem. The slow subsystem uses two-step RAG-augmented reasoning over camera images to produce an attention dictionary (scene type, risk zones, candidate lanes, block-to-wait), which reconfigures the potential functions in the OCP cost (Eq. 24). A lightweight multi-kernel decomposed LSTM predicts surrounding-agent trajectories to inform the fast controller. The authors report CARLA simulation comparisons against rule-based, learning-based, and optimization baselines, an ablation study, and a qualitative real-vehicle demonstration, claiming improved safety, efficiency, and interpretability.

Significance. If the central claim were fully validated, the contribution would be significant: it is one of relatively few architectures in which a VLM does not merely retune MPC weights but actually changes the set of active safety-related potential terms in the OCP, and the paper explicitly targets interpretability and long-tail adaptation. The fast-slow formulation in Eqs. (16)-(18) is clearly stated, the potential-field building blocks are standard and coherent, and the authors make an open-source release, which supports reproducibility. The main caveat is that the safety and efficiency claims rest on validation that is currently too thin to establish that the VLM reconfiguration is reliably beneficial rather than occasionally hazardous.

major comments (4)
  1. [Section V-B, Table II] The safety metrics (Col, TRV, IB, TTC Alarm Duration) and travel times are reported as single point values with no repetitions, confidence intervals, or episode counts, despite the nondeterministic VLM inference and stochastic environment. Because VLM-UDMC's behavior depends on stochastic foundation-model outputs, a single zero-collision run cannot be distinguished from a favorable draw. The authors should report multiple episodes per scenario with means and variances, especially for collision counts and TTC alarm durations.
  2. [Eq. (24), Section IV-B3] The active potential field includes only vehicles and VRUs inside the VLM-selected risk zones, and only if the candidate-lane and block-to-wait flags are set correctly. A false negative in the risk-zone or block-to-wait field therefore silently removes the corresponding repulsive potential from the OCP cost, so the safety property of the closed loop depends on VLM recall, not only on the MPC's obstacle-avoidance structure. The paper's own Section V-B reports a TTC alarm caused by an overlooked crossing vehicle, and Section VI-A concedes that performance is highly influenced by prompt engineering. Yet no precision/recall statistics for the attention dictionary, no sensitivity analysis over prompt phrasing or image resolution, and no failure-rate analysis are provided. Without these, the claimed safety advantage over an always-on potential field is not established.
  3. [Section V-B, Table III] The trajectory-prediction comparison omits the dataset used, the training/validation split, the prediction horizon, the input/output sequence lengths, and the units of MSE/RMSE/MAE. This makes it impossible to assess whether the reported improvements (27.3% MSE and 14.3% RMSE over Transformer) reflect a fair, out-of-sample comparison or in-sample evaluation. Because the predictor feeds the fast system in the same CARLA scenarios used for the closed-loop evaluation, the authors should also clarify whether the prediction test set overlaps with the driving-evaluation scenarios.
  4. [Section V-D] The real-vehicle experiments are qualitative only: they show VLM dialogue responses for two campus-driving situations but provide no closed-loop control metrics, no collision or TTC statistics, no quantitative comparison with baselines, and no details on how the VLM outputs were connected to the OCP during the real run. As written, this section demonstrates that the VLM can annotate still images from a six-camera rig, but it does not substantiate the claimed real-world demonstration of the full VLM-UDMC framework.
minor comments (5)
  1. [Fig. 7] The prompt examples contain literal Python string-concatenation artifacts (e.g., 'str(len(image_prompts))' and "''.join(answers_step1)"), which should be replaced with the actual prompt text for readability and reproducibility.
  2. [Section V-A and V-D] The real vehicle uses a different camera configuration than the simulation, and the prompt was modified to accommodate it; the exact modifications should be documented so that the simulation-to-real transfer is reproducible and the claimed adaptability is verifiable.
  3. [Section III-A1] In Eq. (8), the notation pkx and pky is used without explicitly defining the surrounding vehicle's center coordinates in the ego frame; please state the reference frame and define ra and rb relative to the vehicle geometry.
  4. [Table I] The units of the potential-function coefficients (aNR, bNR, aCR, bCR, aV, bV, aTL1, aTL2) are not given; adding units or stating that the quantities are dimensionless would help readers reproduce the OCP.
  5. [Section V-D] The abbreviation 'QWen3-4B' is inconsistent with 'Qwen3-4B' used elsewhere in the paper.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the OCP, potential functions, and VLM attention dictionary are independently specified, and the claimed improvements are checked against external baselines rather than derived from fitted parameters.

full rationale

The central derivation chain is self-contained: the OCP in Eqs. (1)-(2) and the potential functions in Eqs. (4)-(10) are fully specified in the paper with fixed parameter values in Table I, and Eq. (24) is an explicit mapping from the VLM's discrete attention dictionary to active potential terms. The VLM dictionary is an unvalidated perceptual input, not a fitted parameter, and no equation-level reduction makes the predicted safety/efficiency outcome equal to the dictionary by construction. The use of the authors' own UDMC [16] as both a building block and a baseline is not load-bearing: the potential-function and OCP equations are restated in this paper, and the comparison against UDMC, Autopilot, InterFuser, and RSS is an external closed-loop evaluation. The trajectory predictor is benchmarked against SGPR, LSTM, DLinear, and Transformer in Table III, although the paper does not disclose the dataset or train/test split, which is a reproducibility concern rather than circularity. The paper's own admission in Section V-B that a TTC alarm occurred because the VLM overlooked a crossing vehicle, and the Section VI-A concession that performance depends on prompt engineering, identify real safety-validation gaps (no VLM precision/recall statistics, single-episode counts in Table II), but these are robustness and correctness risks: the failure mode is a false negative, not a fitted quantity masquerading as a prediction. The self-citations to [16], [34], and related VLM works are contextual and benchmark-oriented; none is used to import an unverified uniqueness theorem or to define away the target quantity. No circular step meeting the quote-and-reduction standard was found.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are introduced. Risk zones and potential functions are representational devices built from existing concepts, not invented entities with independent empirical handles. The main unquantified inputs are hand-tuned potential coefficients, undisclosed MPC weights, and the trained LSTM weights.

free parameters (5)
  • Lane marking potential coefficients aNR, bNR, aCR, bCR = 100, 2, 10, 0.5 (Table I)
    Hand-tuned strengths and ranges for crossable and non-crossable lane potentials; no fitting procedure or sensitivity study is reported.
  • Surrounding vehicle potential coefficients aV, bV, ra, rb = 500, 1, 2.4, 1.0 (Table I)
    Ellipsoid potential strength and shape for vehicles; hand-selected and untested for robustness across scenarios.
  • Traffic light and VRU potential coefficients = aTL1=200, aTL2=1000; aVRU and bVRU not specified
    Red-light stop potential magnitudes are given, but the VRU potential parameters in Eq. (9) are never assigned numeric values, a gap for reproduction.
  • MPC weight matrices Q, R, Rd = not reported
    The cost function in Eq. (1) depends on these positive semidefinite matrices, but the paper never gives their numeric values; they are implicit tuning parameters.
  • Trained weights of the multi-kernel decomposed LSTM = not released
    The trajectory predictor's accuracy claims rest on weights fit to an undisclosed dataset; no checkpoint or training data are provided.
assumptions (5)
  • domain assumption The discrete-time bicycle dynamics in Eq. (3) accurately model the ego vehicle's motion.
    Taken from [36] and treated as ground truth for MPC; no validation against the real vehicle's dynamics is given.
  • domain assumption SmolVLM and Qwen3 outputs reliably follow the prescribed JSON attention dictionary from camera images.
    The slow system gates safety-relevant potential functions on these outputs; Section VI-A admits sensitivity to prompt engineering, and the intersection scenario records a TTC alarm from an overlooked vehicle.
  • domain assumption CARLA 0.9.15 scenarios are representative of urban driving for evaluating safety and efficiency.
    All quantitative comparisons are in simulation with autopilot-driven traffic; real-world results are qualitative only.
  • domain assumption Trajectory prediction training data, which is unspecified, matches the deployment distribution.
    Section III-B and Table III report predictor performance without naming a dataset or split; if training and test share the same CARLA processes, reported gains may be in-sample.
  • domain assumption Potential functions are sufficient to encode safety-critical interactions.
    Section III-A models lanes, vehicles, VRUs, and red lights as hand-shaped potentials; unmodeled factors such as weather, occlusions, and unusual road furniture are excluded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving." pith.science (2026). https://pith.science/paper/Z4FB3PVP

@misc{pith2026250715266,
  author       = {Pith},
  title        = {Pith review of: VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z4FB3PVP}},
  note         = {Machine review of arXiv:2507.15266}
}
read the original abstract

Scene understanding and risk-aware attentions are crucial for human drivers to make safe and effective driving decisions. To imitate this cognitive ability in urban autonomous driving while ensuring the transparency and interpretability, we propose a vision-language model (VLM)-enhanced unified decision-making and motion control framework, named VLM-UDMC. This framework incorporates scene reasoning and risk-aware insights into an upper-level slow system, which dynamically reconfigures the optimal motion planning for the downstream fast system. The reconfiguration is based on real-time environmental changes, which are encoded through context-aware potential functions. More specifically, the upper-level slow system employs a two-step reasoning policy with Retrieval-Augmented Generation (RAG), leveraging foundation models to process multimodal inputs and retrieve contextual knowledge, thereby generating risk-aware insights. Meanwhile, a lightweight multi-kernel decomposed LSTM provides real-time trajectory predictions for heterogeneous traffic participants by extracting smoother trend representations for short-horizon trajectory prediction. The effectiveness of the proposed VLM-UDMC framework is verified via both simulations and real-world experiments with a full-size autonomous vehicle. It is demonstrated that the presented VLM-UDMC effectively leverages scene understanding and attention decomposition for rational driving decisions, thus improving the overall urban driving performance. Our open-source project is available at https://github.com/henryhcliu/vlmudmc.git.

Figures

Figures reproduced from arXiv: 2507.15266 by the authors.

Figure 1
Figure 1. The ego vehicle is turning left at an unsignalized T-intersection, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The mainstream VLM/VLA utilities for autonomous driving. From top [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Framework of the proposed VLM-enhanced risk-aware urban autonomous driving system is designed to address the complexities of urban driving. This [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The architecture of the proposed multi-kernel prediction approach for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Pipeline of the prompt generation and RAG process for the proposed [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Spatial characterization of risk zones enhances the inference of VLM [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Structure of the prompts to the foundation models. The second stage [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Demonstration of the multimodal dialogues during urban driving. The vision data contains the images from different perspectives, and the key insights [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: The value of the assigned potential functions during different driving [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 11
Figure 11. Figure 11: The perception system of the intelligent vehicle. It contains six RGB [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Multimodal dialogue demonstration during campus driving, with [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 31 canonical work pages

  1. [1]

    Milestones in autonomous driving and intelligent vehicles: Survey of surveys,

    L. Chen, Y . Li, C. Huang, B. Li, Y . Xing, D. Tian, L. Li, Z. Hu, X. Na, Z. Li, et al., “Milestones in autonomous driving and intelligent vehicles: Survey of surveys,” IEEE Transactions on Intelligent Vehicles , vol. 8, no. 2, pp. 1046–1056, 2022

    work page 2022

  2. [2]

    Optimal trajectory planning of connected and automated vehicles at on-ramp merging area,

    Z. Gao, Z. Wu, W. Hao, K. Long, Y .-J. Byon, and K. Long, “Optimal trajectory planning of connected and automated vehicles at on-ramp merging area,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 8, pp. 12675–12687, 2022

    work page 2022

  3. [3]

    Adaptive safe reinforcement learning with full-state constraints and constrained adaptation for autonomous vehicles,

    Y . Zhang, X. Liang, D. Li, S. S. Ge, B. Gao, H. Chen, and T. H. Lee, “Adaptive safe reinforcement learning with full-state constraints and constrained adaptation for autonomous vehicles,” IEEE Transactions on Cybernetics, vol. 54, no. 3, pp. 1907–1920, 2023. 14

    work page 1907

  4. [4]

    Enhance sample efficiency and robustness of End-to-End urban au- tonomous driving via semantic masked world model,

    Z. Gao, Y . Mu, C. Chen, J. Duan, P. Luo, Y . Lu, and S. Eben Li, “Enhance sample efficiency and robustness of End-to-End urban au- tonomous driving via semantic masked world model,” IEEE Transac- tions on Intelligent Transportation Systems , vol. 25, no. 10, pp. 13067– 13079, 2024

    work page 2024

  5. [5]

    Enhanced scene understanding and situation awareness for au- tonomous vehicles based on semantic segmentation,

    Y . Zhao, L. Wang, X. Yun, C. Chai, Z. Liu, W. Fan, X. Luo, Y . Liu, and X. Qu, “Enhanced scene understanding and situation awareness for au- tonomous vehicles based on semantic segmentation,” IEEE Transactions on Systems, Man, and Cybernetics: Systems , vol. 54, no. 11, pp. 6537– 6549, 2024

    work page 2024

  6. [6]

    Interactive decision-making with switchable game modes for automated vehicles at intersections,

    S. Jia, Y . Zhang, X. Li, X. Na, Y . Wang, B. Gao, B. Zhu, and R. Yu, “Interactive decision-making with switchable game modes for automated vehicles at intersections,” IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 11, pp. 11785–11799, 2023

    work page 2023

  7. [7]

    Baidu apollo EM motion planner,

    H. Fan, F. Zhu, C. Liu, L. Zhang, L. Zhuang, D. Li, W. Zhu, J. Hu, H. Li, and Q. Kong, “Baidu apollo EM motion planner,” arXiv preprint arXiv:1807.08048, 2018

    arXiv 2018

  8. [8]

    V AD: Vectorized scene representation for efficient autonomous driving,

    B. Jiang, S. Chen, Q. Xu, B. Liao, J. Chen, H. Zhou, Q. Zhang, W. Liu, C. Huang, and X. Wang, “V AD: Vectorized scene representation for efficient autonomous driving,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 8340–8350, 2023

    work page 2023

Show all 43 references
  1. [9]

    Milestones in autonomous driving and intelligent vehicles—Part I: Control, computing system design, communication, HD map, testing, and human behaviors,

    L. Chen, Y . Li, C. Huang, Y . Xing, D. Tian, L. Li, Z. Hu, S. Teng, C. Lv, J. Wang, D. Cao, N. Zheng, and F.-Y . Wang, “Milestones in autonomous driving and intelligent vehicles—Part I: Control, computing system design, communication, HD map, testing, and human behaviors,” IE...

    2023

  2. [10]

    A finite state machine based automated driving controller and its stochastic opti- mization,

    M. Zhang, N. Li, A. Girard, and I. Kolmanovsky, “A finite state machine based automated driving controller and its stochastic opti- mization,” in Dynamic Systems and Control Conference , vol. 58288, p. V002T07A002, American Society of Mechanical Engineers, 2017

    2017

  3. [11]

    Hierarchical motion plan- ning for autonomous vehicles in unstructured dynamic environments,

    Y . Qi, B. He, R. Wang, L. Wang, and Y . Xu, “Hierarchical motion plan- ning for autonomous vehicles in unstructured dynamic environments,” IEEE Robotics and Automation Letters, vol. 8, no. 2, pp. 496–503, 2022

    2022

  4. [12]

    CARLA: An open urban driving simulator,

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “CARLA: An open urban driving simulator,” in Conference on Robot Learning, pp. 1–16, PMLR, 2017

    2017

  5. [13]

    A safe motion planning and reliable control framework for autonomous vehicles,

    H. Pan, M. Luo, J. Wang, T. Huang, and W. Sun, “A safe motion planning and reliable control framework for autonomous vehicles,”IEEE Transactions on Intelligent Vehicles, vol. 9, no. 4, pp. 4780–4793, 2024

    2024

  6. [14]

    Design, analysis, and experiments of preview path tracking control for autonomous vehicles,

    S. Xu and H. Peng, “Design, analysis, and experiments of preview path tracking control for autonomous vehicles,” IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 1, pp. 48–58, 2019

    2019

  7. [15]

    Stochastic model predictive control with a safety guarantee for automated driving,

    T. Br ¨udigam, M. Olbrich, D. Wollherr, and M. Leibold, “Stochastic model predictive control with a safety guarantee for automated driving,” IEEE Transactions on Intelligent Vehicles, vol. 8, no. 1, pp. 22–36, 2021

    2021

  8. [16]

    UDMC: Unified decision-making and control framework for urban autonomous driving with motion prediction of traffic participants,

    H. Liu, K. Chen, Y . Li, Z. Huang, M. Liu, and J. Ma, “UDMC: Unified decision-making and control framework for urban autonomous driving with motion prediction of traffic participants,” IEEE Transactions on Intelligent Transportation Systems, vol. 26, no. 5, pp. 5856–5871, 2025

    2025

  9. [17]

    Planning-oriented autonomous driving,

    Y . Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, et al. , “Planning-oriented autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17853–17862, 2023

    2023

  10. [18]

    VLM-E2E: Enhancing end-to-end autonomous driving with multimodal driver attention fusion,

    P. Liu, H. Liu, H. Liu, X. Liu, J. Ni, and J. Ma, “VLM-E2E: Enhancing end-to-end autonomous driving with multimodal driver attention fusion,” arXiv preprint arXiv:2502.18042 , 2025

    2025

  11. [19]

    DSDrive: Distilling large language model for lightweight end-to-end autonomous driving with unified reasoning and planning,

    W. Liu, P. Liu, and J. Ma, “DSDrive: Distilling large language model for lightweight end-to-end autonomous driving with unified reasoning and planning,” arXiv preprint arXiv:2505.05360 , 2025

    2025 arXiv

  12. [20]

    DiffusionDrive: Truncated diffusion model for end-to-end autonomous driving,

    B. Liao, S. Chen, H. Yin, B. Jiang, C. Wang, S. Yan, X. Zhang, X. Li, Y . Zhang, Q. Zhang, et al., “DiffusionDrive: Truncated diffusion model for end-to-end autonomous driving,” in Proceedings of the Computer Vision and Pattern Recognition Conference , pp. 12037–12047, 2025

    2025

  13. [21]

    OpenDriveVLA: Towards end-to-end autonomous driving with large vision language action model,

    X. Zhou, X. Han, F. Yang, Y . Ma, and A. C. Knoll, “OpenDriveVLA: Towards end-to-end autonomous driving with large vision language action model,” arXiv preprint arXiv:2503.23463 , 2025

    2025

  14. [22]

    Think2Drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in CARLA-v2),

    Q. Li, X. Jia, S. Wang, and J. Yan, “Think2Drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in CARLA-v2),” in European Conference on Computer Vision , 2024

    2024

  15. [23]

    Explainable ai for safe and trustworthy autonomous driving: A system- atic review,

    A. Kuznietsov, B. Gyevnar, C. Wang, S. Peters, and S. V . Albrecht, “Explainable ai for safe and trustworthy autonomous driving: A system- atic review,” IEEE Transactions on Intelligent Transportation Systems , vol. 25, no. 12, pp. 19342–19364, 2024

    2024

  16. [24]

    VLM-MPC: Vision Language Foundation Model-guided model predictive controller for autonomous driving,

    K. Long, H. Shi, J. Liu, and X. Li, “VLM-MPC: Vision Language Foundation Model-guided model predictive controller for autonomous driving,” arXiv preprint arXiv:2408.04821 , 2024

    2024 arXiv

  17. [25]

    LVLM-MPC: Collabo- ration for autonomous driving: A safety-aware and task-scalable control architecture,

    K. Atsuta, K. Honda, H. Okuda, and T. Suzuki, “LVLM-MPC: Collabo- ration for autonomous driving: A safety-aware and task-scalable control architecture,” arXiv preprint arXiv:2505.04980 , 2025

    2025 arXiv

  18. [26]

    CALMM-Drive: confidence-aware autonomous driving with large mul- timodal model,

    R. Yao, Y . Wang, H. Liu, R. Yang, Z. Peng, L. Zhu, and J. Ma, “CALMM-Drive: confidence-aware autonomous driving with large mul- timodal model,” arXiv preprint arXiv:2412.04209 , 2024

    2024 arXiv

  19. [27]

    Lightweight compressed temporal and compressed spatial attention with augmentation fusion in remaining useful life prediction,

    H. Guo, H. Zhu, J. Wang, V . Prahlad, W. K. Ho, C. W. de Silva, and T. H. Lee, “Lightweight compressed temporal and compressed spatial attention with augmentation fusion in remaining useful life prediction,” in 49th Annual Conference of the IEEE Industrial Electronics Society ...

    2023

  20. [28]

    Transformers in time series: a survey,

    Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, and L. Sun, “Transformers in time series: a survey,” in Proceedings of the Interna- tional Joint Conference on Artificial Intelligence , pp. 6778–6786, 2023

    2023

  21. [29]

    DriveVLM: The convergence of autonomous driving and large vision-language models,

    X. Tian, J. Gu, B. Li, Y . Liu, Y . Wang, Z. Zhao, K. Zhan, P. Jia, X. Lang, and H. Zhao, “DriveVLM: The convergence of autonomous driving and large vision-language models,” in 8th Annual Conference on Robot Learning , 2024

    2024

  22. [30]

    DCoMA: A dynamic coordinative merging assistant strategy for on- ramp vehicles with mixed traffic conditions,

    L. Li, C. Qian, J. Gan, D. Zhang, X. Qu, F. Xiao, and B. Ran, “DCoMA: A dynamic coordinative merging assistant strategy for on- ramp vehicles with mixed traffic conditions,” Transportation Research Part C: Emerging Technologies, vol. 165, p. 104700, 2024

    2024

  23. [31]

    Improved con- sensus admm for cooperative motion planning of large-scale connected autonomous vehicles with limited communication,

    H. Liu, Z. Huang, Z. Zhu, Y . Li, S. Shen, and J. Ma, “Improved con- sensus admm for cooperative motion planning of large-scale connected autonomous vehicles with limited communication,” IEEE Transactions on Intelligent Vehicles, 2024

    2024

  24. [32]

    CoDriveVLM: VLM-enhanced urban cooperative dispatching and motion planning for future autonomous mobility on demand systems,

    H. Liu, R. Yao, W. Liu, Z. Huang, S. Shen, and J. Ma, “CoDriveVLM: VLM-enhanced urban cooperative dispatching and motion planning for future autonomous mobility on demand systems,” arXiv preprint arXiv:2501.06132, 2025

    2025 arXiv

  25. [33]

    Empowering safer socially sensitive autonomous vehicles using human- plausible cognitive encoding,

    H. Lu, M. Zhu, C. Lu, S. Feng, X. Wang, Y . Wang, and H. Yang, “Empowering safer socially sensitive autonomous vehicles using human- plausible cognitive encoding,” Proceedings of the National Academy of Sciences, vol. 122, no. 21, p. e2401626122, 2025

    2025

  26. [34]

    Incremental learning-based real-time trajectory prediction for autonomous driving via sparse Gaussian process regression,

    H. Liu, K. Chen, and J. Ma, “Incremental learning-based real-time trajectory prediction for autonomous driving via sparse Gaussian process regression,” in 2024 IEEE Intelligent Vehicles Symposium, pp. 1–7, 2024

    2024

  27. [35]

    Are Transformers effective for time series forecasting?,

    A. Zeng, M. Chen, L. Zhang, and Q. Xu, “Are Transformers effective for time series forecasting?,” in Proceedings of the AAAI conference on artificial intelligence, pp. 11121–11128, 2023

    2023

  28. [36]

    Numeri- cally stable dynamic bicycle model for discrete-time control,

    Q. Ge, Q. Sun, S. E. Li, S. Zheng, W. Wu, and X. Chen, “Numeri- cally stable dynamic bicycle model for discrete-time control,” in IEEE Intelligent Vehicles Symposium, pp. 128–134, 2021

    2021

  29. [37]

    SmolVLM: Redefining small and efficient multimodal models,

    A. Marafioti, O. Zohar, M. Farr ´e, M. Noyan, E. Bakouch, P. Cuenca, C. Zakka, L. B. Allal, A. Lozhkov, N. Tazi, V . Srivastav, J. Lochner, H. Larcher, M. Morlon, L. Tunstall, L. von Werra, and T. Wolf, “SmolVLM: Redefining small and efficient multimodal models,” arXiv preprin...

    2025 arXiv

  30. [38]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. , “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  31. [39]

    Drive- R1: Bridging reasoning and planning in VLMs for autonomous driving with reinforcement learning,

    Y . Li, M. Tian, D. Zhu, J. Zhu, Z. Lin, Z. Xiong, and X. Zhao, “Drive- R1: Bridging reasoning and planning in VLMs for autonomous driving with reinforcement learning,” arXiv preprint arXiv:2506.18234 , 2025

    2025 arXiv

  32. [40]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning , pp. 8748–8763, 2021

    2021

  33. [41]

    Safety-enhanced autonomous driving using interpretable sensor fusion transformer,

    H. Shao, L. Wang, R. Chen, H. Li, and Y . Liu, “Safety-enhanced autonomous driving using interpretable sensor fusion transformer,” in Conference on Robot Learning , pp. 726–737, 2023

    2023

  34. [42]

    On a formal model of safe and scalable self-driving cars,

    S. Shalev-Shwartz, S. Shammah, and A. Shashua, “On a formal model of safe and scalable self-driving cars,” arXiv preprint arXiv:1708.06374, 2017

    2017 arXiv

  35. [43]

    CasADi – A software framework for nonlinear optimization and opti- mal control,

    J. A. E. Andersson, J. Gillis, G. Horn, J. B. Rawlings, and M. Diehl, “CasADi – A software framework for nonlinear optimization and opti- mal control,” Mathematical Programming Computation , vol. 11, no. 1, pp. 1–36, 2019

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.