Pith. sign in

REVIEW 4 major objections 5 minor 39 references

The paper claims that command-following failures in vision-language driving models stem from conditional policy collapse under regression training, and that a single precomputed per-command latent shift—Latent-Centroid Steering—can replace

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 00:55 UTC pith:3R5DDQHW

load-bearing objection A practical inference-time trick for command-aligned driving, but the paper's central claim—that a fixed centroid shift can replace a per-sample CFG residual—is asserted rather than demonstrated, so the empirical gains are encouraging but not yet convincing. the 4 major comments →

arxiv 2608.00237 v1 pith:3R5DDQHW submitted 2026-07-31 cs.CV cs.RO

Latent-Centroid Steering: Single-Pass Classifier-Free Guidance for Command-Aligned Autonomous Driving

Meibo Hu , Jiamian Wang , Pichao Wang , Zhiqiang Tao This is my paper
classification cs.CV cs.RO
keywords autonomous drivingvision-language-action modelscommand followingclassifier-free guidancelatent-space steeringtrajectory predictionconditional policy collapsecentroid shift
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper identifies a command-following gap in vision-language action models for driving: when trained by regression against multimodal expert trajectories, the model averages over possible maneuvers and leans on dominant visual priors, so language commands like 'turn left' barely change the output. It then reformulates classifier-free guidance (CFG) as residual steering—subtracting the unconditional prediction from the conditional one to isolate the instruction's effect—and shows this amplifies command influence at inference. Because two-pass CFG is too slow for real-time control and its instance-level guidance is noisy, the paper replaces the per-sample residual with a precomputed class-level centroid shift in latent space. The proposed single-pass Latent-Centroid Steering adds a fixed per-command vector to the conditional latent before decoding, cutting inference latency by about 50% while improving success rate and reducing trajectory error on both closed-loop and open-loop evaluations. A sympathetic reader would care because it turns command adherence into a cheap, offline-computable correction rather than an architectural or training-time fix.

Core claim

Training a vision-language-action driving model with random command dropout produces a latent space in which samples conditioned on the same navigation command cluster together. The paper's central move is to exploit that geometry: compute the mean latent for each command and the global mean, form the shift vector as their difference, and at inference steer the conditional latent by adding a scaled version of that shift before decoding waypoints. This replaces the CFG residual, which requires two forward passes, with a single precomputed vector, motivated by a mean-shift interpretation of CFG. On the paper's benchmarks, LCS matches or exceeds two-pass CFG on driving score and success rate wh

What carries the argument

The central object is the class-level centroid shift, a precomputed vector per navigation command defined as the difference between the mean latent feature of samples carrying that command and the mean over all commands. The argument rides on two supports: command dropout during training, so the model also learns unconditional features and the centroids separate by command, and a mean-shift interpretation of classifier-free guidance that justifies replacing the instance-level residual with a cluster-level shift. The shift is applied as a gentle push on the conditional latent before the trajectory decoder, so scene-specific perception is preserved while command identity is amplified.

Load-bearing premise

The load-bearing premise is that every scene with the same navigation command has a similar latent direction of change, so one fixed shift vector improves most instances of, say, 'turn left' without harming scene-specific perception; if a command covers several geometrically distinct maneuvers, the averaged centroid may point between modes and the steering can fail on particular scenes.

What would settle it

Compute the per-sample guidance residuals on a held-out set of 'turn left' scenes from a command-dropout-trained model and measure their alignment with the precomputed shift for that command. If the cosine similarity is low or the residual directions have high variance—for example, separated clusters by intersection type—then the centroid shift is not a faithful stand-in for CFG, and one would predict LCS to underperform on those scenes; a direct test would evaluate LCS separately on scenes with different turn geometries and look for a success-rate drop relative to the aggregate.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Using LCS, the paper's closed-loop success rate rises from about 65.8% to 71.2% over the baseline, and driving score from 86.1 to 87.2, while keeping latency near the single-pass baseline's 285 ms instead of the 522–564 ms of two-pass CFG.
  • Open-loop trajectory error drops from 0.90 m to 0.85 m average L2 with no collision-rate penalty, indicating the correction generalizes beyond the training simulator.
  • The method needs no changes to training beyond command dropout and no extra online computation beyond a vector add; all centroid estimation is offline.
  • CFG and LCS are framed as two ends of a spectrum—instance-level versus class-level steering—so the paper offers a unified view of guidance in regression-based driving models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The single-global-shift assumption is likely to strain at complex intersections: a protected left turn, a yield left turn, and a multilane left turn may have different latent directions, and the averaged centroid could point between them. A testable extension is per-command multiple centroids, obtained by clustering within each command, or a similarity-weighted blend of shifts.
  • Because the shift is estimated from the training distribution, it is distribution-bound: if deployment scenes shift to new cities, weather, or sensor setups, the precomputed vector may need periodic re-estimation, and an online update of centroids from a small labeled stream would be a natural fix.
  • The mechanism suggests a general recipe for instruction following in other regression-based embodied agents: train with condition dropout, compute centroid shifts in a latent space, and steer at test time. The paper does not claim this, but the mechanism is agnostic to the modality of the conditioning signal.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses the command-following gap in vision-language-action (VLA) autonomous driving, attributing it to conditional policy collapse under regression-based training. It proposes classifier-free guidance (CFG) as a residual steering mechanism in both action space and latent space, then introduces Latent-Centroid Steering (LCS), which replaces the per-sample conditional–unconditional latent residual with a precomputed class-level centroid shift δ_c = μ_c − μ_g. This yields a single-pass inference method that avoids the two-forward-pass cost of CFG. The authors evaluate LCS on the closed-loop Bench2Drive benchmark and the open-loop nuScenes benchmark, reporting driving-score and success-rate improvements over SimLingo and a roughly 50% latency reduction relative to two-pass CFG.

Significance. If the central equivalence in Eq. (14) holds, the paper offers a practical and appealing alternative to CFG: a static, class-level latent shift that provides command steering at half the inference cost. The experimental design is generally sound—closed-loop evaluation with multiple seeds, comparison against a strong VLA baseline, and efficiency measurements are all valuable. The authors also commit to releasing code, which strengthens reproducibility. The main concern is that the core substitution from per-sample residuals to centroid shifts is asserted rather than derived or empirically validated. The paper does not show that a single shift per command approximates the per-sample CFG residual across diverse scenes, nor does it quantify the clustering structure beyond a t-SNE visualization. Hyperparameters are selected on the evaluation benchmark itself, which risks inflating the reported gains. These issues are load-bearing: the method is exactly the centroid-shift substitution, and without per-scene validation or a formal error analysis, the claim that LCS is a principled replacement for CFG remains under-supported. The efficiency advantage is clear, but the performance equivalen

major comments (4)
  1. [§III-D, Eq. (14)] The central substitution z_c − z_0 → μ_c − μ_g is asserted with an arrow, not derived or bounded. The paper does not provide conditions under which the class-level centroid shift approximates the per-sample residual, nor an error estimate. This is the load-bearing step: LCS is defined as this replacement. The authors should either derive the relationship (e.g., as an expected residual under a Gaussian or unimodal assumption) or empirically show that per-sample residuals are well approximated by δ_c, e.g., by reporting the distribution of cosine similarity / L2 distance between z_c − z_0 and μ_c − μ_g over the evaluation set.
  2. [§III-C, Fig. 2] The 'well-structured clustering property' is the motivation for using centroid shifts, but it is only visualized via t-SNE, which can create apparent clusters even in high-dimensional data with weak structure. No quantitative cluster metrics (silhouette score, within-class vs. between-class variance, or separation between command centroids) are given. More importantly, there is no per-scene breakdown showing whether the shift helps, hurts, or leaves individual scenes unchanged. The aggregate gains on Bench2Drive/nuScenes could be driven by a subset of scenes, with other scenes degraded. The authors should report per-command success-rate deltas and, ideally, a scatter or histogram of per-scene improvement vs. the alignment of the latent residual with δ_c.
  3. [§IV-D, Figs. 3–4] The hyperparameters p_drop, w (guidance scale), and γ are selected using the evaluation benchmark itself: Fig. 3 sweeps p_drop and w on Bench2Drive, and Fig. 4 sweeps γ on Bench2Drive, and the final reported numbers are the best configurations from these sweeps. This is a form of test-set tuning. The paper should state whether these sweeps were performed on a validation split or only on the final test routes, and ideally report results for a fixed default setting (e.g., γ=0.1 may generalize, but the claim of 8% SR improvement relies on the selected values). Without this, the reported superiority over baselines may be inflated by selection bias.
  4. [Table II and §IV-B] The paper claims LCS achieves 'stronger command adherence and improved driving performance' compared to CFG, but the differences in Table II (DS: 87.18±0.52 vs. 87.07±0.86; SR: 71.16±1.23 vs. 70.91±2.20) are within one standard deviation of the three-seed runs. The efficiency improvement (285.75 ms vs. 563.55 ms) is unambiguous, but the performance advantage over CFG is not statistically supported. The authors should either temper the claim or report paired significance tests (e.g., bootstrap or paired t-test across routes) to demonstrate that LCS is not just comparable but actually better.
minor comments (5)
  1. [§IV-A vs. Table III] The setup says the open-loop evaluation reports ADE, but Table III is titled 'Avg. L2 error' and lists Avg. L2 and Collision. Clarify whether ADE and Avg. L2 are the same metric; if not, use consistent terminology.
  2. [Table IV] The notation 'Δcmd' is used without definition. Presumably it should be δ_c (the centroid shift). Also clarify the exact γ values for the two steering-base rows; the table header only gives γ=0.1 and γ=0.30.
  3. [Algorithm 1 and §IV-A] The loss terms λ_p L_smooth-L1(p, p̂) + λ_s L_smooth-L1(s, ŝ) appear in Algorithm 1 but p, s, p̂, ŝ, λ_p, and λ_s are not defined in the main text. Define these symbols or reference a prior section.
  4. [Abstract/§V] The abstract and conclusion describe the method as 'principled' and 'theoretically grounded,' but no theorem or formal claim is proved in the paper. This wording overstates the current support; recommend softening until the Eq. (14) relationship is rigorously justified.
  5. [Fig. 5] The qualitative example for 'Follow the road' shows SimLingo bypassing a stopped vehicle. In real driving, bypassing a stopped vehicle may be the safer or more desirable behavior depending on context. Explain why the LCS behavior is considered correct here, or choose a less ambiguous example.

Circularity Check

0 steps flagged

No significant circularity: LCS's centroid shift is an empirically validated approximation, not a derivation that reduces to its own inputs.

full rationale

The paper's central step is Eq. (14), where the per-sample CFG residual z_c - z_0 is replaced by the precomputed class-level shift μ_c - μ_g. This is explicitly framed as a substitution ('LCS replaces this with a precomputed class-level shift'), i.e., an approximation, not an exact equivalence derived from the definition of CFG. The shift vectors are statistics of the model's own training latents, but they are not fitted to the evaluation metric; the reported gains are measured on external benchmarks (Bench2Drive closed-loop, nuScenes open-loop), so the empirical claim is independently testable. The paper does not claim that μ_c - μ_g is mathematically identical to each per-sample residual, and the ablation studies (Table IV, Fig. 4) treat γ as a tunable hyperparameter rather than as a quantity forced by construction. The only self-citations ([15], [31]) appear in related-work motivation about latent-space geometry and are not load-bearing for the main derivation. The unvalidated assumption that one centroid shift works across scenes is a correctness/rigor concern, not circularity. Therefore no circular step is established.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical entities are introduced. The free parameters are the dropout probability, CFG scale, and LCS strength, all chosen on the evaluation benchmark. The axioms are the conditional-policy-collapse diagnosis, the clustering property, and the centroid-shift approximation.

free parameters (3)
  • Dropout probability p_drop = 0.3
    Tuned on Bench2Drive (Fig. 3); controls the trade-off between conditional and unconditional training.
  • CFG guidance scale w = 3.0
    Tuned on Bench2Drive (Fig. 3) for the two-pass CFG baseline; not used by LCS but reported in comparisons.
  • LCS steering strength γ = 0.1
    Tuned on Bench2Drive (Fig. 4); directly controls the magnitude of the centroid shift and is central to LCS performance.
axioms (4)
  • domain assumption Regression training under multimodal trajectory distributions makes the optimal predictor approximate the conditional mean, dominated by visual priors, leading to command-insensitivity (Eqs. 3–4).
    Central motivation; no quantitative proof beyond the observed command-following gap.
  • domain assumption VLA latent features trained with command dropout exhibit command-specific clustering such that class centroids are meaningful steering targets (Fig. 2).
    Empirical observation presented without statistical validation.
  • ad hoc to paper The per-sample CFG residual z_c − z_0 can be approximated by the class-level centroid shift μ_c − μ_g (Eq. 14).
    This is the key substitution that defines LCS; it is asserted, not derived, and is the weakest load-bearing step.
  • domain assumption The mean-shift interpretation of CFG from [14] applies to regression-based VLA driving.
    Adopted from cited theory; not re-derived for this setting.

pith-pipeline@v1.3.0-alltime-deepseek · 11691 in / 11220 out tokens · 106461 ms · 2026-08-04T00:55:16.439021+00:00 · methodology

0 comments
read the original abstract

Vision-language models (VLMs) have recently emerged as a promising paradigm for end-to-end autonomous driving, enabling agents to map multimodal inputs and high-level navigation instructions directly to executable trajectories. However, in practice, these models exhibit a persistent command-following gap: predicted trajectories often show weak sensitivity to navigation commands, resulting in incorrect behavior at critical decision points. We identify this issue as a form of conditional policy collapse, where regression-based training under multimodal trajectory distributions encourages the model to rely on dominant visual priors while marginalizing the language-conditioned signal. To address this issue, we introduce a principled formulation of classifier-free guidance (CFG) for regression-based vision-language driving. We show that CFG can be interpreted as isolating the instruction-induced residual in the action space by contrasting conditional and unconditional predictions, thereby explicitly amplifying the effect of the navigation command at inference time. However, a standard two-pass CFG introduces prohibitive latency for real-time control and produces noisy instance-level guidance directions. Building on a mean-shift interpretation of CFG, we propose Latent-Centroid Steering (LCS), a single-pass guidance mechanism that replaces instance-level residuals with class-level latent shifts. By projecting conditional representations toward precomputed command-specific centroids, LCS performs class-level latent steering based on cluster geometry that is both more stable and computationally efficient. We demonstrate that LCS reduces inference latency by approximately 50% while achieving stronger command adherence and improved driving performance on both closed-loop (Bench2Drive) and open-loop (nuScenes) benchmarks. Code will be released.

Figures

Figures reproduced from arXiv: 2608.00237 by Jiamian Wang, Meibo Hu, Pichao Wang, Zhiqiang Tao.

Figure 1
Figure 1. Figure 1: Overview of Latent-Centroid Steering (LCS). [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of command-conditioned latent features [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Discussion on the LCS steering parameter [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison between LCS and Simlingo on Bench2Drive closed-loop evaluation by the CARLA simulator. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 13 linked inside Pith

  1. [1]

    Senna: Bridging large vision- language models and end-to-end autonomous driving,

    B. Jiang, S. Chen, B. Liao, X. Zhang, W. Yin, Q. Zhang, C. Huang, W. Liu, and X. Wang, “Senna: Bridging large vision- language models and end-to-end autonomous driving,”arXiv preprint arXiv:2410.22313, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Emma: End-to-end multimodal model for autonomous driving,

    J.-J. Hwang, R. Xu, H. Lin, W.-C. Hung, J. Ji, K. Choi, D. Huang, T. He, P. Covington, B. Sappet al., “Emma: End-to-end multimodal model for autonomous driving,”arXiv preprint arXiv:2410.23262, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Minddriver: Introducing progres- sive multimodal reasoning for autonomous driving,

    L. Zhang, Y . Yuan, C. Wu, X. Chang, X. Cai, S. Zeng, L. Shi, S. Wang, H. Zhang, and M. Xu, “Minddriver: Introducing progres- sive multimodal reasoning for autonomous driving,”arXiv preprint arXiv:2602.21952, 2026

    arXiv 2026

  4. [4]

    Lmdrive: Closed-loop end-to-end driving with large language models,

    H. Shao, Y . Hu, L. Wang, G. Song, S. L. Waslander, Y . Liu, and H. Li, “Lmdrive: Closed-loop end-to-end driving with large language models,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 15 120–15 130

    2024

  5. [5]

    Simlingo: Vision- only closed-loop autonomous driving with language-action alignment,

    K. Renz, L. Chen, E. Arani, and O. Sinavski, “Simlingo: Vision- only closed-loop autonomous driving with language-action alignment,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 11 993–12 003

    2025

  6. [6]

    Drivelm: Driving with graph visual question answering,

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li, “Drivelm: Driving with graph visual question answering,” inEuropean conference on computer vision. Springer, 2024, pp. 256–274

    2024

  7. [7]

    Drivevlm: The convergence of au- tonomous driving and large vision-language models,

    X. Tian, J. Gu, B. Li, Y . Liu, Y . Wang, Z. Zhao, K. Zhan, P. Jia, X. Lang, and H. Zhao, “Drivevlm: The convergence of au- tonomous driving and large vision-language models,”arXiv preprint arXiv:2402.12289, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Planning-oriented autonomous driving,

    Y . Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wanget al., “Planning-oriented autonomous driving,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 17 853–17 862

    2023

  9. [9]

    Vad: Vectorized scene representation for efficient autonomous driving,

    B. Jiang, S. Chen, Q. Xu, B. Liao, J. Chen, H. Zhou, Q. Zhang, W. Liu, C. Huang, and X. Wang, “Vad: Vectorized scene representation for efficient autonomous driving,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 8340–8350

    2023

  10. [10]

    Genad: Generative end-to-end autonomous driving,

    W. Zheng, R. Song, X. Guo, C. Zhang, and L. Chen, “Genad: Generative end-to-end autonomous driving,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 87–104

    2024

  11. [11]

    Nord: A data-efficient vision- language-action model that drives without reasoning,

    I. Rawal, S. Gupta, Y . Hu, and W. Zhan, “Nord: A data-efficient vision- language-action model that drives without reasoning,”arXiv preprint arXiv:2602.21172, 2026

    Pith/arXiv arXiv 2026

  12. [12]

    Hermes: A holistic end-to-end risk-aware multimodal em- bodied system with vision-language models for long-tail autonomous driving,

    W. Tang, J. You, J. Liu, Z. Wang, R. Gan, Z. Huang, F. Wei, and B. Ran, “Hermes: A holistic end-to-end risk-aware multimodal em- bodied system with vision-language models for long-tail autonomous driving,”arXiv preprint arXiv:2602.00993, 2026

    arXiv 2026

  13. [13]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,”arXiv preprint arXiv:2207.12598, 2022

    Pith/arXiv arXiv 2022

  14. [14]

    Towards understanding the mechanisms of classifier-free guidance,

    X. Li, R. Wang, and Q. Qu, “Towards understanding the mechanisms of classifier-free guidance,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=bRAm7A02Qm

    2025

  15. [15]

    Latent chain-of-thought for visual reasoning,

    G. Sun, H. Hua, J. Wang, J. Luo, S. Dianat, M. Rabbani, R. Rao, and Z. Tao, “Latent chain-of-thought for visual reasoning,”Advances in neural information processing systems, 2026

    2026

  16. [16]

    Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving,

    X. Jia, Z. Yang, Q. Li, Z. Zhang, and J. Yan, “Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving,”Advances in Neural Information Processing Systems, 2024

    2024

  17. [17]

    nuscenes: A multimodal dataset for autonomous driving,

    H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020

    2020

  18. [18]

    A language agent for autonomous driving,

    J. Mao, J. Ye, Y . Qian, M. Pavone, and Y . Wang, “A language agent for autonomous driving,”arXiv preprint arXiv:2311.10813, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    Vadv2: End-to-end vectorized autonomous driving via probabilistic planning,

    S. Chen, B. Jiang, H. Gao, B. Liao, Q. Xu, Q. Zhang, C. Huang, W. Liu, and X. Wang, “Vadv2: End-to-end vectorized autonomous driving via probabilistic planning,”arXiv preprint arXiv:2402.13243, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Generative scenario rollouts for end-to-end autonomous driving,

    R. Yasarla, D. Hegde, S. Han, H.-P. Cheng, Y . Shi, M. Sadeghi- googhari, S. Mahajan, A. Bhattacharyya, L. Liu, R. Garrepalliet al., “Generative scenario rollouts for end-to-end autonomous driving,” arXiv preprint arXiv:2601.11475, 2026

    arXiv 2026

  21. [21]

    Unidrive-wm: Unified understanding, planning and generation world model for autonomous driving,

    Z. Xiong, X. Ye, B. Yaman, S. Cheng, Y . Lu, J. Luo, N. Jacobs, and L. Ren, “Unidrive-wm: Unified understanding, planning and generation world model for autonomous driving,”arXiv preprint arXiv:2601.04453, 2026

    Pith/arXiv arXiv 2026

  22. [22]

    Drivetransformer: Unified trans- former for scalable end-to-end autonomous driving,

    X. Jia, J. You, Z. Zhang, and J. Yan, “Drivetransformer: Unified trans- former for scalable end-to-end autonomous driving,”arXiv preprint arXiv:2503.07656, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Open- drivevla: Towards end-to-end autonomous driving with large vision language action model,

    X. Zhou, X. Han, F. Yang, Y . Ma, V . Tresp, and A. Knoll, “Open- drivevla: Towards end-to-end autonomous driving with large vision language action model,”arXiv preprint arXiv:2503.23463, 2025

    arXiv 2025

  24. [24]

    Gpt-driver: Learning to drive with gpt,

    J. Mao, Y . Qian, J. Ye, H. Zhao, and Y . Wang, “Gpt-driver: Learning to drive with gpt,”arXiv preprint arXiv:2310.01415, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Vilta: A vlm-in-the-loop adversary for enhancing driving policy robustness,

    Q. Chen, F. Li, S. Xu, Z. Lai, Z. Xie, Y . Luo, S. Jiang, H. Li, L. Chen, B. Wanget al., “Vilta: A vlm-in-the-loop adversary for enhancing driving policy robustness,”arXiv preprint arXiv:2601.12672, 2026

    arXiv 2026

  26. [26]

    Photorealistic text-to-image diffusion models with deep lan- guage understanding,

    C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans et al., “Photorealistic text-to-image diffusion models with deep lan- guage understanding,”Advances in neural information processing systems, vol. 35, pp. 36 479–36 494, 2022

    2022

  27. [27]

    Video diffusion models,

    J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,”Advances in neural information processing systems, vol. 35, pp. 8633–8646, 2022

    2022

  28. [28]

    Natural language in- structions for scene-responsive human-in-the-loop motion planning in autonomous driving using vision-language-action models,

    A. Martinez-Sanchez, P. Roy, and R. Greer, “Natural language in- structions for scene-responsive human-in-the-loop motion planning in autonomous driving using vision-language-action models,”arXiv preprint arXiv:2602.04184, 2026

    arXiv 2026

  29. [29]

    Drive- r1: Bridging reasoning and planning in vlms for autonomous driving with reinforcement learning,

    Y . Li, M. Tian, D. Zhu, J. Zhu, Z. Lin, Z. Xiong, and X. Zhao, “Drive- r1: Bridging reasoning and planning in vlms for autonomous driving with reinforcement learning,”arXiv preprint arXiv:2506.18234, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Driveagent-r1: Advancing vlm-based autonomous driving with active perception and hybrid thinking,

    W. Zheng, X. Mao, N. Ye, P. Li, K. Zhan, X. Lang, and H. Zhao, “Driveagent-r1: Advancing vlm-based autonomous driving with active perception and hybrid thinking,”arXiv preprint arXiv:2507.20879, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Modeling mask uncertainty in hyperspectral image reconstruction,

    J. Wang, Y . Zhang, X. Yuan, Z. Meng, and Z. Tao, “Modeling mask uncertainty in hyperspectral image reconstruction,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 112–129

    2022

  32. [32]

    Think twice before driving: Towards scalable decoders for end-to-end autonomous driving,

    X. Jia, P. Wu, L. Chen, J. Xie, C. He, J. Yan, and H. Li, “Think twice before driving: Towards scalable decoders for end-to-end autonomous driving,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 21 983–21 994

    2023

  33. [33]

    Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving,

    X. Jia, Y . Gao, L. Chen, J. Yan, P. L. Liu, and H. Li, “Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 7953–7963

    2023

  34. [34]

    Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes,

    J.-T. Zhai, Z. Feng, J. Du, Y . Mao, J.-J. Liu, Z. Tan, Y . Zhang, X. Ye, and J. Wang, “Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes,”arXiv preprint arXiv:2305.10430, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    Don’t shake the wheel: Momentum-aware planning in end-to-end autonomous driving,

    Z. Song, C. Jia, L. Liu, H. Pan, Y . Zhang, J. Wang, X. Zhang, S. Xu, L. Yang, and Y . Luo, “Don’t shake the wheel: Momentum-aware planning in end-to-end autonomous driving,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 22 432–22 441

    2025

  36. [36]

    Orion: A holistic end-to-end au- tonomous driving framework by vision-language instructed action generation,

    H. Fu, D. Zhang, Z. Zhao, J. Cui, D. Liang, C. Zhang, D. Zhang, H. Xie, B. Wang, and X. Bai, “Orion: A holistic end-to-end au- tonomous driving framework by vision-language instructed action generation,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 24 823–24 834

    2025

  37. [37]

    Carla: An open urban driving simulator,

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “Carla: An open urban driving simulator,” inConference on robot learning. PMLR, 2017

    2017

  38. [38]

    Occworld: Learning a 3d occupancy world model for autonomous driving,

    W. Zheng, W. Chen, Y . Huang, B. Zhang, Y . Duan, and J. Lu, “Occworld: Learning a 3d occupancy world model for autonomous driving,” inEuropean conference on computer vision, 2024

    2024

  39. [39]

    St-p3: End- to-end vision-based autonomous driving via spatial-temporal feature learning,

    S. Hu, L. Chen, P. Wu, H. Li, J. Yan, and D. Tao, “St-p3: End- to-end vision-based autonomous driving via spatial-temporal feature learning,” inEuropean Conference on Computer Vision, 2022

    2022