Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

LVLM-MPC Collaboration for Autonomous Driving: A Safety-Aware and Task-Scalable Control Architecture

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In highway-traffic simulations, routing a vision-language model's driving commands through MPC Builder with feasibility feedback and an intermediate OCP yields a 100% success rate and 100% safe lane-changing, while baselines that execute…

desk verdict A solid incremental integration of LVLM planning with MPC Builder, but the 'guaranteed safety' language outruns what the heuristic feasibility check and penalty-based MPPI can support; still worth a serious referee. read the letter →

arxiv 2505.04980 v2 pith:AV2G6S2F submitted 2025-05-08 cs.RO cs.SYeess.SY

Kazuki Atsuta , Kohei Honda , Hiroyuki Okuda , Tatsuya Suzuki This is my paper · ORCID
classification cs.ROcs.SYeess.SY
keywords autonomousdrivinglargevision-languagemodelsmodelpredictivecontrolMPCbuilderintermediateOCPtaskswitchingfeasibilityfeedbacksafetyassurance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large vision-language models (LVLMs) can propose driving tasks from images and text, but their proposals are not grounded in what the vehicle can actually do, so executing them directly is unsafe. This paper claims that the gap can be closed by routing every LVLM command through MPC Builder, an automatic generator of optimal control problems, which checks feasibility, feeds a rejection signal back to the LVLM, and when a command is infeasible builds an intermediate OCP that steers the vehicle toward the new task without abandoning the old safe plan. In 30 highway-traffic simulation episodes the proposed system completes every run without collision and makes every lane change within the safe-distance constraint, even when the safety instruction is removed from the LVLM prompt. The point is that a general-purpose language model can act as an open-ended task planner while the control layer, not the model, is what enforces safety.

What carries the argument

The central mechanism is the intermediate OCP (iOCP) working inside the MPC Switcher. An iOCP takes the previous OCP and the newly requested target OCP, forms the Cartesian product of their state spaces and dynamics, keeps the previous OCP's constraints as hard constraints, and adds the target task's constraints as penalty terms in the cost; the solver then drives the vehicle toward the target's feasible region while remaining inside the previous safe set. The MPC Switcher decides between executing the target OCP, running the iOCP, or reverting to the previous OCP and flagging rejection, using a feasibility check based on the previous step's predicted trajectory and an upper limit on consecutive iOCP uses. Because the iOCP retains the previous constraints, the reversion path is claimed to be guaranteed feasible. Also behind the method is MPC Builder's primitive pool, which assembles task-specific OCPs from reusable components so that new tasks do not require hand-designing a new controller.

What would settle it

One concrete test: in the same highway simulator, place the ego vehicle next to a gap that opens and then closes within a single LVLM planning interval, so the previous-step trajectory satisfies Eq. (10) but the sampled MPPI trajectories for the requested lane change do not; if the vehicle violates its safe following distance or collides, the feasibility guarantee is false.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that bidirectional communication between a low-frequency LVLM task planner and a high-frequency MPC Builder makes task switching both safe and scalable. The LVLM emits only a symbolic command drawn from a small task set; MPC Builder selects MPC primitives for that command, composes an optimal control problem, and checks its feasibility by testing the previous predicted trajectory under the previous control sequence. If infeasible, it solves an intermediate OCP that keeps the previous constraints while penalizing violations of the target constraints, guiding the vehicle toward the new task; if too many consecutive iOCP steps are needed, it reverts to the previous OCP and returns a rejection flag to the LVLM. The reported results are a 100% success rate (30/30 episodes) and a 100% safe lane-changing rate, with 9 of the LVLM's 133 lane-change decisions assisted or rejected by the safety layer.

Load-bearing premise

The claimed safety guarantee rests on the assumption that the previous-step feasibility check, which tests only the previously predicted trajectory under the previously optimized controls, catches every constraint violation the sampling-based solver could produce in the next step, and that the solver's penalty weight of 100 makes those constraints effectively hard.

Editorial extensions

If this is right

  • Driving-task commands from an LVLM no longer need to be trustworthy at the control level: the MPC layer is the safety backstop, not the prompt.
  • Infeasible commands do not stall the vehicle: the iOCP either makes the transition possible or the system reverts to the last safe plan and tells the planner the command was rejected.
  • Adding a new task type should require adding an MPC primitive to the pool, not redesigning a controller, which is how the framework claims to scale to open-ended task spaces.
  • The LVLM can plan at a lower frequency than the control loop, since the MPC keeps executing its current task while it waits for the next command.
  • In the reported highway simulations, the safety layer is unchanged by prompt editing: removing the safety-instruction prompt leaves the success and safe-lane-changing rates at 100%.
  • If the same bidirectional feasibility loop were applied to other language-to-control pipelines, one would expect the same pattern: the planner's proposal rate stays high while the control layer filters and reshapes infeasible proposals; this is a testable transfer, not a result of this paper.
  • The feasibility check in Eq. (10) is retrospective, so the reversion could be strengthened by checking a set of sampled successor trajectories rather than only the previous predicted trajectory; doing so would turn a practical safety filter into a formally verified one.
  • Because the iOCP cost weights and penalty coefficients are tuned by hand, a natural stress test is to perturb those values and see whether the 100% safety rates persist; that would reveal how much of the claim rests on parameter tuning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a hierarchical autonomous-driving architecture in which a large vision-language model (LVLM) issues symbolic task commands (LANE_LEFT, IDLE, LANE_RIGHT) and an MPC Builder layer—composed of a primitive assigner, an MPC composer, a feasibility checker, an MPC switcher, and an MPPI solver—generates task-specific optimal control problems (OCPs). The key novel ingredient is an intermediate OCP (iOCP) that, when the target task is deemed infeasible, smoothly transitions the vehicle from the previous OCP toward the target task's feasible region while retaining the previous OCP's constraints, and reverts to the previous OCP after a maximum number of steps. The authors report HighwayEnv simulations over 30 episodes with 100% success and 100% self-defined safe lane-changing rate for the proposed method, versus worse safety metrics for two baselines (LVLM2MPC and LVLM2PID), and an ablation showing robustness when safety instructions are removed from the prompt.

Significance. If the safety and scalability claims were fully supported, the paper would make a useful contribution to LVLM-based autonomous driving by showing a concrete mechanism through which a foundation-model planner can be kept safe by a lower-level control layer without sacrificing task flexibility. The iOCP transition is an elegant idea that addresses a real shortcoming of binary infeasibility flags, and the ablation in Table III is a strong and falsifiable demonstration that the MPC layer, rather than prompt engineering, is responsible for the reported safety. The comparison against unidirectional baselines is appropriate, and the 100% success with a non-trivial LVLM planner is encouraging. However, the manuscript's central 'guaranteed feasibility and safety' statements are not supported by the presented analysis or experiments: the feasibility check is explicitly heuristic, the MPPI solver treats constraints as penalty terms rather than hard constraints, and the empirical evidence is limited to a single 50-second synthetic scenario with 30 episodes and no pointwise constraint-satisfaction data. The task-scalability claim is also asserted from prior work rather than demonstrated here.

major comments (3)
  1. [§IV-D, Eq. (10) and Algorithm 2] The paper repeatedly states that the reversion to Ot,prev is 'guaranteed to be feasible' and that MPC Builder 'guarantees feasibility and safety,' but the feasibility check in Eq. (10) is explicitly acknowledged in the text as 'not a strict condition.' The check only validates the previously optimized input sequence propagated from the current state; it does not check all possible inputs, does not verify terminal feasibility, and can miss constraint violations that arise from the actual chosen control. The guarantee wording in Section IV-D and in the Figure 2 caption should be softened to 'practically feasible' or 'maintained by design,' or the paper should provide a formal certificate that actually holds under the stated assumptions.
  2. [§V-A, Eq. (12)] The MPPI solver treats constraints as soft penalties via indicator functions with coefficient μ=100, not as hard constraints. This undermines any claim that the solved OCP is 'guaranteed feasible,' because the MPPI output is a weighted average of randomly sampled input sequences; even if every sample satisfies a nonconvex constraint (e.g., collision avoidance), the averaged sequence need not satisfy it. The same issue affects the iOCP reversion, since the iOCP's own constraints are also implemented as penalties. The authors should either (i) report pointwise constraint-violation statistics over all episodes to demonstrate that violations are negligible in practice, or (ii) employ a constraint-handling method that provides actual feasibility certificates, and in any case replace 'guaranteed' with an empirically supported statement.
  3. [§V-B and Table II] The empirical evaluation is too narrow to support the claimed 100% safe lane-changing rate and the paper's broader task-scalability conclusion. The results come from a single 50-second scenario, 30 randomly generated initial conditions, no confidence intervals, and no per-episode minimum safety distances or constraint-violation counts; the 'safe lane-changing rate' is a binary per-event measure that can hide near-misses. In addition, the claimed 'task scalability' is not demonstrated: only three task commands and one highway scenario are tested, so the scalability assertion rests on the prior MPC Builder reference [7] rather than on evidence in this manuscript. The authors should add constraint-margin statistics, report variability across episodes, and either add a multi-scenario experiment or explicitly limit the scalability claim to the architecture's design rather than its demonstrated performance.
minor comments (5)
  1. [Algorithm 1, lines 13–14] The comment 'same inequality constraints as OB' is inconsistent with the code, which sets g_AB = g_A (constraints of OA). The text above correctly says the previous OCP's constraints are retained; the comment should be corrected to 'same inequality constraints as OA'.
  2. [§V-A, Eq. (12)] The indicator functions in Eq. (12) use strict comparisons g>0 and h≠0; for numerical equality constraints, please specify the tolerance used to decide satisfaction in the MPPI implementation, since this affects both the cost and the reported feasibility.
  3. [§V-A, Table I] The weight vectors Qlk, Qlc, Qcs, Qacc, the safety distances d_safe^lc, d_safe^acc, d_safe^pv, the MPPI sampling variances, μ=100, nmax=50, and the ACC threshold 2*d_acc are mentioned but their numerical values are not given. Without these values, the experiments cannot be reproduced. Please include a parameter table or release the code.
  4. [§V-B] The sentence 'every time the LVLM performed a single task-planning step, MPC Builder executed an additional 30 control steps beyond the iOCP steps' is ambiguous; clarify whether this means the MPC always runs 30 control steps after each LVLM command, and how the iOCP steps are counted.
  5. [Abstract and §IV-A] The phrase 'ensuring optimality and safety' overstates what is achieved: MPPI is a sampling-based stochastic optimizer that does not guarantee global optimality, and the safety guarantees are at best heuristic. Please rephrase to 'aiming for optimality and safety' or similar.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the safety results are obtained against baselines that share the same LVLM prompts and MPC constraint primitives, so the 100% safe-lane-changing rate is not forced by definition or by a fitted input.

full rationale

The paper's main empirical claim is comparative: LVLM-MPCBuilder is tested against LVLM2MPC, which uses the same MPC Builder primitives and the same LVLM prompts but omits feasibility feedback and iOCP assistance, and still achieves only a 94.8% safe lane-changing rate. This makes the proposed method's 100% result depend on the iOCP/feasibility-check mechanism rather than merely on the presence of safety constraints in the OCP. The feasibility check in Eq. (10) is admittedly 'not a strict condition' (Section IV-D), and the MPPI implementation treats constraints as soft penalties (Eq. (12), µ=100), so the paper's wording that reversion to the previous OCP is 'guaranteed to be feasible' overstates the available certificate; however, this is a support/correctness limitation, not a circular reduction, because the reversion claim is not obtained by defining 'feasible' as the output of Eq. (10) or by fitting a parameter to the reported safety metric. The citations to [7] and [27] are self-citations for the underlying MPC Builder and iOCP constructions, but the present paper spells out the OCP composition (Eqs. (7)-(9)), the iOCP algorithm (Algorithm 1), and the switching logic (Algorithm 2), so the load-bearing argument does not reduce to an unverified self-citation. No prediction in the paper is statistically forced by a fitted input; the ablation studies additionally show that the proposed method maintains 100% success even when the safety instruction is removed from the prompt, further supporting that the result is not an artifact of prompt engineering. Overall, the derivation chain is self-contained for the claims made, with the caveat that the term 'guaranteed' is stronger than what the soft-constraint solver and heuristic check actually establish.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The framework's safety and feasibility claims rest on control-theoretic assumptions (valid dynamics models and constraint enforcement via MPPI penalty) and on the heuristic that previous-step predictive states indicate current feasibility. Most manual parameters are listed as free parameters even though they are not fitted to data in this paper; they are inherited from the MPC primitives and prior work.

free parameters (6)
  • Cost weight vectors Qlk, Qlc, Qcs, Qacc
    Hand-tuned weights in MPC primitives (Table I) that shape the trade-offs between lane keeping, lane changing, speed control, and acceleration effort.
  • Safety distances d_safe^lc, d_safe^acc, d_safe^pv
    Hand-chosen minimum distances in the constraints for lane changes, ACC, and surrounding vehicles (Table I).
  • MPPI penalty coefficient mu = 100 = 100
    Chosen to treat constraints as approximately hard in Eq. (12), but not a true hard constraint.
  • iOCP step limit nmax = 50 = 50
    Upper limit on consecutive iOCP executions in Algorithm 2, set by the authors.
  • ACC selection threshold 2*d_acc
    Rule in Section V-A for choosing between Constant Speed and ACC primitives based on distance to the leading vehicle.
  • MPPI sampling variances (2.0, 0.01) = 2.0, 0.01
    Variances for acceleration and steering sampling in MPPI, chosen by the authors (Section V-A).
assumptions (5)
  • standard math Kinematic bicycle model for ego vehicle dynamics
    Used as the ego dynamics primitive in Table I, a standard control model.
  • domain assumption Constant-velocity model for surrounding vehicles
    Used in the safety primitive in Table I; assumes other vehicles move at constant velocity over the horizon.
  • domain assumption HighwayEnv simulator faithfully represents highway driving
    The experiments and all safety claims rely on the fidelity of this simulator (Section V-A).
  • ad hoc to paper MPPI with penalty coefficient mu=100 yields approximately hard constraint satisfaction
    Eq. (12) and the text state this is a practical approximation, not a hard guarantee.
  • domain assumption Feasibility check based on previous predictive states is a sufficient proxy for current feasibility
    Eq. (10) checks only the previously predicted trajectory; the paper admits it is 'not a strict condition' (Section IV-D).
invented entities (1)
  • Intermediate OCP (iOCP)
    purpose: Smooth task switching by combining constraints and costs of the previous and target OCPs, guiding the vehicle toward the feasible region of the target task.
    The iOCP is a new construct from the authors' prior work and is validated only in this paper's simulation; no separate falsifiable handle is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LVLM-MPC Collaboration for Autonomous Driving: A Safety-Aware and Task-Scalable Control Architecture." pith.science (2026). https://pith.science/paper/AV2G6S2F

@misc{pith2026250504980,
  author       = {Pith},
  title        = {Pith review of: LVLM-MPC Collaboration for Autonomous Driving: A Safety-Aware and Task-Scalable Control Architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AV2G6S2F}},
  note         = {Machine review of arXiv:2505.04980}
}
read the original abstract

This paper proposes a novel Large Vision-Language Model (LVLM) and Model Predictive Control (MPC) integration framework that delivers both task scalability and safety for Autonomous Driving (AD). LVLMs excel at high-level task planning across diverse driving scenarios. However, since these foundation models are not specifically designed for driving and their reasoning is not consistent with the feasibility of low-level motion planning, concerns remain regarding safety and smooth task switching. This paper integrates LVLMs with MPC Builder, which automatically generates MPCs on demand, based on symbolic task commands generated by the LVLM, while ensuring optimality and safety. The generated MPCs can strongly assist the execution or rejection of LVLM-driven task switching by providing feedback on the feasibility of the given tasks and generating task-switching-aware MPCs. Our approach provides a safe, flexible, and adaptable control framework, bridging the gap between cutting-edge foundation models and reliable vehicle operation. We demonstrate the effectiveness of our approach through a simulation experiment, showing that our system can safely and effectively handle highway driving while maintaining the flexibility and adaptability of LVLMs.

Figures

Figures reproduced from arXiv: 2505.04980 by the authors.

Figure 1
Figure 1. Conceptual comparison of hierarchical LVLM-MPC [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. System overview of the proposed method. The orange section denotes LVLM-based task planning (Section IV-B). The [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Transition from OCP(prev) to OCP(target) via the iOCP. If Ot,target is absolutely infeasible, the transition is abandoned, and the iOCP reverts to the safe Ot,prev. will be feasible. If MPC Builder simply rejects an infeasible command, the rejection may hinder the LVLM’s long-term reasoning and potentially lead to a deadlock. To address this issue, we introduce the intermediate OCP (iOCP) [27] to facilitate smooth t… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: In LVLM-MPCBuilder, MPC Builder assists or rejects [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Ablation study for iOCP-based task-switching assis [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 6
Figure 6. Figure 6: The LVLM’s reckless lane-change decision was exe [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: A collision resulting from a dangerous lane-change [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving

    cs.RO 2025-07 conditional novelty 4.0 of 10

    VLM-UDMC uses a vision-language model to switch safety cost functions in a model predictive controller and a multi-kernel LSTM to predict traffic trajectories, reporting improved urban driving metrics in CARLA and cam...

Reference graph

Works this paper leans on

29 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [7]

    Mpc builder for au- tonomous drive: Automatic generation of mpcs for motion planning and control,

    K. Honda, H. Okuda, T. Suzuki, and A. Ito, “Mpc builder for au- tonomous drive: Automatic generation of mpcs for motion planning and control,” in Intelligent V ehicles Symposium. IEEE, 2023, pp. 1–8

    work page 2023

  2. [1]

    GPT-4o: The cutting-edge advancement in multimodal llm,

    R. Islam and O. M. Moushi, “GPT-4o: The cutting-edge advancement in multimodal llm,” Authorea Preprints, 2024

    work page 2024

  3. [2]

    Predictive maneuver planning for an autonomous vehicle in public highway traffic,

    Q. Wang, B. Ayalew, and T. Weiskircher, “Predictive maneuver planning for an autonomous vehicle in public highway traffic,” IEEE Transactions on Intelligent Transportation Systems , vol. 20, no. 4, pp. 1303–1315, 2018

    2018

  4. [3]

    Path planning for autonomous vehicles using model predictive control,

    C. Liu, S. Lee, S. Varnhagen, and H. E. Tseng, “Path planning for autonomous vehicles using model predictive control,” in Intelligent V ehicles Symposium. IEEE, 2017, pp. 174–179

    work page 2017

  5. [4]

    Languagempc: Large language models as decision makers for autonomous driving,

    H. Sha, Y . Mu, Y . Jiang, L. Chen, C. Xu, P. Luo, S. E. Li, M. Tomizuka, W. Zhan, and M. Ding, “Languagempc: Large language models as decision makers for autonomous driving,” 2023. [Online]. Available: https://arxiv.org/abs/2310.03026

    arXiv 2023

  6. [5]

    Bidirectional planning for au- tonomous driving framework with large language model,

    Z. Ma, Q. Sun, and T. Matsumaru, “Bidirectional planning for au- tonomous driving framework with large language model,” Sensors, vol. 24, no. 20, p. 6723, 2024

    work page 2024

  7. [6]

    Empowering autonomous driving with large language models: A safety perspective,

    Y . Wang, R. Jiao, S. S. Zhan, C. Lang, C. Huang, Z. Wang, Z. Yang, and Q. Zhu, “Empowering autonomous driving with large language models: A safety perspective,” in ICLR 2024 Workshop on Large Language Model Agents , 2024

    work page 2024

  8. [8]

    Autonomy 2.0: Why is self-driving always 5 years away?

    A. Jain, L. Del Pero, H. Grimmett, and P. Ondruska, “Autonomy 2.0: Why is self-driving always 5 years away?” arXiv preprint arXiv:2107.08142, 2021

    arXiv 2021

Show all 29 references
  1. [9]

    Modelling driving decisions: a latent plan approach,

    C. F. Choudhury and M. E. Ben-Akiva, “Modelling driving decisions: a latent plan approach,” Transportmetrica A: Transport Science , vol. 9, no. 6, pp. 546–566, 2013

    2013

  2. [10]

    A data-driven lane-changing model based on deep learning,

    D.-F. Xie, Z.-Z. Fang, B. Jia, and Z. He, “A data-driven lane-changing model based on deep learning,” Transportation research part C: emerg- ing technologies , vol. 106, pp. 41–60, 2019

    2019

  3. [11]

    Automated lane change strategy using proximal policy optimization-based deep reinforcement learning,

    F. Ye, X. Cheng, P. Wang, C.-Y . Chan, and J. Zhang, “Automated lane change strategy using proximal policy optimization-based deep reinforcement learning,” in Intelligent V ehicles Symposium. IEEE, 2020, pp. 1746–1752

    2020

  4. [12]

    LLM4Drive: A survey of large language models for autonomous driving,

    Z. Yang, X. Jia, H. Li, and J. Yan, “LLM4Drive: A survey of large language models for autonomous driving,” in NeurIPS 2024 Workshop on Open-World Agents , 2024

    2024

  5. [13]

    A survey on multimodal large language models for autonomous driving,

    C. Cui, Y . Ma, X. Cao, W. Ye, Y . Zhou, K. Liang, J. Chen, J. Lu, Z. Yang, K.-D. Liao et al. , “A survey on multimodal large language models for autonomous driving,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 958–979

    2024

  6. [14]

    Drive like a human: Rethinking autonomous driving with large language models,

    D. Fu, X. Li, L. Wen, M. Dou, P. Cai, B. Shi, and Y . Qiao, “Drive like a human: Rethinking autonomous driving with large language models,” in IEEE/CVF Winter Conference on Applications of Computer Vision Workshops. IEEE, 2024, pp. 910–919

    2024

  7. [15]

    LMDrive: Closed-loop end-to-end driving with large language models,

    H. Shao, Y . Hu, L. Wang, G. Song, S. L. Waslander, Y . Liu, and H. Li, “LMDrive: Closed-loop end-to-end driving with large language models,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 15 120–15 130

    2024

  8. [16]

    GPT-Driver: Learning to drive with gpt,

    J. Mao, Y . Qian, J. Ye, H. Zhao, and Y . Wang, “GPT-Driver: Learning to drive with gpt,” in NeurIPS 2023 F oundation Models for Decision Making Workshop, 2023

    2023

  9. [17]

    DriveGPT4: Interpretable end-to-end autonomous driving via large language model,

    Z. Xu, Y . Zhang, E. Xie, Z. Zhao, Y . Guo, K.-Y . K. Wong, Z. Li, and H. Zhao, “DriveGPT4: Interpretable end-to-end autonomous driving via large language model,” IEEE Robotics and Automation Letters , 2024

    2024

  10. [18]

    DiLu: A knowledge-driven approach to autonomous driving with large language models,

    L. Wen, D. Fu, X. Li, X. Cai, T. MA, P. Cai, M. Dou, B. Shi, L. He, and Y . Qiao, “DiLu: A knowledge-driven approach to autonomous driving with large language models,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  11. [19]

    Receive, reason, and react: Drive as you say, with large language models in autonomous vehicles,

    C. Cui, Y . Ma, X. Cao, W. Ye, and Z. Wang, “Receive, reason, and react: Drive as you say, with large language models in autonomous vehicles,” IEEE Intelligent Transportation Systems Magazine , 2024

    2024

  12. [20]

    VLM-MPC: Vision language foundation model -guided model predictive controller for autonomous driving,

    K. Long, H. Shi, J. Liu, and X. Li, “VLM-MPC: Vision language foundation model -guided model predictive controller for autonomous driving,” arXiv preprint arXiv:2408.04821 , 2024

    2024 arXiv

  13. [21]

    Large language models for robotics: Opportunities, challenges, and perspectives,

    J. Wang, E. Shi, H. Hu, C. Ma, Y . Liu, X. Wang, Y . Yao, X. Liu, B. Ge, and S. Zhang, “Large language models for robotics: Opportunities, challenges, and perspectives,” Journal of Automation and Intelligence , 2024

    2024

  14. [22]

    Code as policies: Language model programs for embodied control,

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in IEEE International Conference on Robotics and Automation . IEEE, 2023, pp. 9493–9500

    2023

  15. [23]

    Language to rewards for robotic skill synthesis,

    W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.- T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik et al. , “Language to rewards for robotic skill synthesis,” in Conference on Robot Learning . PMLR, 2023, pp. 374–404

    2023

  16. [24]

    Safety aware task planning via large language models in robotics,

    A. A. Khan, M. Andrev, M. A. Murtaza, S. Aguilera, R. Zhang, J. Ding, S. Hutchinson, and A. Anwar, “Safety aware task planning via large language models in robotics,” arXiv preprint arXiv:2503.15707 , 2025

    2025 arXiv

  17. [25]

    Large lan- guage models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large lan- guage models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  18. [26]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  19. [27]

    Connection of nonlinear model predictive controllers for smooth task switching in autonomous driving,

    K. Honda, H. Okuda, T. Suzuki, and A. Ito, “Connection of nonlinear model predictive controllers for smooth task switching in autonomous driving,” Asian Journal of Control , vol. 25, no. 3, pp. 1805–1822, 2023

    2023

  20. [28]

    An environment for autonomous driving decision-making,

    E. Leurent, “An environment for autonomous driving decision-making,” https://github.com/eleurent/highway-env, 2018

    2018

  21. [29]

    Information-theoretic model predictive control: Theory and applications to autonomous driving,

    G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou, “Information-theoretic model predictive control: Theory and applications to autonomous driving,” IEEE Transactions on Robotics , vol. 34, no. 6, pp. 1603–1622, 2018

    2018

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.