Pith. sign in

REVIEW 3 major objections 4 minor 70 references

S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A self-supervised driving MLLM can beat supervised planners without human labels.

desk verdict A genuinely useful sparse-volume representation for MLLM driving, but the SOTA claim is undermined by a GT-future-derived behavior command that leaks the maneuver at inference. read the letter →

arxiv 2505.24139 v2 pith:YUJM7YQ4 submitted 2025-05-30 cs.CV cs.AI

Yichen Xie , Runsheng Xu , Tong He , Jyh-Jing Hwang , Katie Luo , Jingwei Ji , Hubert Lin , Letian Chen
show 4 more authors
Yiren Lu Zhaoqi Leng Dragomir Anguelov Mingxing Tan
This is my paper
classification cs.CVcs.AI
keywords autonomousdrivingend-to-endmotionplanningmultimodallargelanguagemodelself-supervisedlearningsparsevolumerepresentationspatio-temporalreasoningWaymoOpenDatasetnuScenes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a multimodal large language model can be turned into a state-of-the-art open-loop motion planner without any human annotations, using only camera images and the ego vehicle's own past and future trajectory as supervision. The proposed method, S4-Driver, lifts the frozen 2D visual features of an MLLM into a sparse 3D volume centered on the ego vehicle, aggregates several camera views and past frames, and then predicts future waypoints as text. If the claim is right, the main bottleneck of end-to-end driving—expensive perception and prediction labels—disappears, and planning performance can scale with the volume of unlabeled driving logs. The paper reports favorable results against supervised multi-task planners on both nuScenes and the much larger WOMD-Planning-ADE benchmark.

What carries the argument

The sparse volume representation carries the argument. From each camera view's frozen feature map, a small fully connected layer produces a low-dimensional gate feature; these are lifted into a 3D volume via bilinear projection (Simple-BEV-style sampling), and an MLP with sigmoid outputs a per-voxel gate score. The M voxels with the highest scores keep their semantic features (blended with a learnable vacant feature), while the rest are discarded, concentrating the model on space that matters for driving. A bin-wise relative position bias added to the self-attention of the multimodal encoder injects local 3D structure, and historical frames are fused after ego-motion compensation by concatenating per-frame volume features. Finally, nucleus sampling generates K candidate trajectories whose unweighted average forms the output, counteracting the model's bias toward simple behaviors.

What would settle it

Take the trained S4-Driver and re-run the nuScenes and WOMD-Planning-ADE evaluations with the high-level behavior command replaced by one derived from a map and past position only, so it contains no future information. If ADE and bADE degrade substantially or the gap to supervised baselines closes, the state-of-the-art claim rests on the future-information leak rather than on the spatio-temporal representation itself.

Watch

Extended reading notes

Core claim

The central claim is that MLLM-based planning underperforms because its visual representations live in the 2D image plane, while driving decisions are made in 3D space. S4-Driver addresses this by projecting multi-view, multi-frame image features into an ego-centered sparse volume: a lightweight gate selects the voxels most relevant to driving, sampled features plus positional embeddings form volume tokens, and a relative position bias in self-attention encodes local 3D geometry without retraining the vision encoder. With this spatio-temporal visual representation, the model predicts future ego waypoints directly as floating-point text, guided by a coarse meta-decision (stationary, keep speed, accelerate, decelerate) and a high-level behavior command. The paper claims state-of-the-art results on nuScenes and WOMD-Planning-ADE compared to prior end-to-end planners, all without perception or prediction labels.

Load-bearing premise

The results assume that giving the model a high-level behavior command computed from the ground-truth future trajectory is a fair way to evaluate open-loop planning; if that command leaks the intended maneuver, the reported advantage over planners without such commands may not hold under a future-blind evaluation.

Editorial extensions

If this is right

  • Human annotation is removed from the loop: the same pipeline can be pretrained on massive unlabeled driving logs, and the paper shows performance improves with pretraining data.
  • The frozen vision encoder means the representation can be dropped into stronger or larger MLLMs without retraining the image backbone.
  • The behavior-wise metrics (bADE) expose performance on rare but safety-critical maneuvers such as turns and U-turns, where the paper reports the largest gains.
  • A planner that reasons in ego-centric 3D volume space, instead of image space, is the direction that closes the gap between MLLM reasoning and the native coordinate frame of planning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the high-level behavior command is computed from the ground-truth future trajectory, as described in the supplementary material, the model receives a form of future information at inference; a comparison against planners that do not use such commands may overstate the advantage, and a fairer test would use a command derived without future knowledge.
  • The sparse-volume gating and relative-position-bias recipe is not specific to driving; the same machinery could be applied to other 3D reasoning tasks such as robotic manipulation or navigation, where an MLLM must ground language in egocentric 3D space.
  • Because the design keeps the vision encoder frozen, the gains are likely to transfer to newer MLLMs, but the paper's numbers should be re-checked if the behavior-command leakage is removed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes S4-Driver, a self-supervised end-to-end motion planner built on the PaLI multimodal large language model. It introduces a sparse 3D volume representation that lifts frozen 2D vision encoder features into an ego-centric 3D volume, aggregates multi-view and multi-frame information, adds relative-position attention bias, predicts a two-stage meta-decision before outputting waypoints, and averages multiple nucleus-sampled trajectory decodings. The model is trained with raw camera images and ego-vehicle trajectory supervision only, and is evaluated on nuScenes and a newly introduced WOMD-Planning-ADE benchmark. The paper reports state-of-the-art L2/ADE numbers on nuScenes and favorable results on WOMD-Planning-ADE, supported by ablations of each component.

Significance. If the evaluation protocol is corrected, this is a significant contribution: the sparse-volume lifting with a frozen vision encoder is a clean mechanism for providing MLLMs 3D spatial reasoning without finetuning the image encoder, and the ablations (Tabs. 4-11) are internally consistent and clearly presented. The WOMD-Planning-ADE benchmark with behavior-wise metrics addresses a real need for larger-scale evaluation of camera-based planners, and the scalability result with unlabeled driving logs is valuable. However, the headline SOTA claims currently rest on an evaluation protocol in which the model receives a high-level behavior command computed from the ground-truth future trajectory; this must be fixed before the empirical claims can be accepted.

major comments (3)
  1. [§3.1, Eq. (1); suppl. Sec. 7] Section 3.1, Eq. (1), and supplementary Sec. 7: The high-level behavior command b is an input to the planner at inference, and supplementary Sec. 7 defines it from the ground-truth future trajectory (stop if displacement <5 m; left turn if final heading >30 deg; left U-turn if final x-displacement < -5 m; if no command is satisfied, the future horizon is extended by 2 s until one is). This means the model is told at inference which maneuver class to produce, which is future information that a navigation system would not provide at time t. Feeding 'do left turn' or 'go straight right' can substantially reduce ADE/L2, especially in the behavior-wise bADE metric that averages per-maneuver errors. The paper itself excludes 'stop' from the command set because it 'may leak future information like traffic lights,' but turns and lane changes derived from future trajectories carry the same concern. This protocol issue is load-bearing for the SOTA claims in Sec. 6 and Tabs. 2-3; the internal ablations that compare S4-Driver with Vanilla PaLI under the same command protocol are not invalidated, but the absolute numbers and external comparisons are. Please rerun the evaluation with commands obtained from information available before the prediction horizon, add a no-command ablation, and state whether the baselines in Tab. 2 receive commands and from what source.
  2. [§4, Eq. (13); suppl. Sec. 7] Section 4, Eq. (13), and supplementary Sec. 7: The proposed behavior-wise metric bADE partitions the validation set with the same heuristic rules over the ground-truth future trajectory used to create the command input, so the evaluation is potentially circular: the model receives the behavior label as an input and is then averaged within that same label. Furthermore, 'stop' is included among the seven bADE behaviors but is deliberately excluded from the command set, making the per-behavior comparison inconsistent; Tab. 7 indeed shows S4-Driver is worse than Vanilla PaLI on stop (0.063 vs 0.048), consistent with the stop command being withheld. Please report bADE with and without the command input, keep the behavior partition fixed across methods, and clarify how stop-conditioned samples are handled for models that never receive a stop command.
  3. [Sec. 6 and Tab. 3] Section 6 and Tab. 3: The claim of state-of-the-art performance on WOMD-Planning-ADE is not supported by the evidence. Tab. 3 compares S4-Driver only with Vanilla PaLI and an internally reproduced MotionLM that additionally consumes auto-labeled objects, tracks, and roadgraph; no other camera-based end-to-end planner is included. Because WOMD-Planning-ADE is introduced by this paper, 'state-of-the-art' is a claim about a benchmark with no prior art, and the favorable comparison against MotionLM is acknowledged to be inequitable in input modality. Please broaden the comparison to existing camera-based planners adapted to this benchmark, or restrict the claim to 'favorable performance against the included baselines.'
minor comments (4)
  1. [Eq. (9)] Equation (9): the attention formula subtracts b(D) from the logits, but the text says a relative position bias is 'injected' into the attention; the sign convention is ambiguous. Please state explicitly whether larger bias values increase or decrease attention between nearby tokens, and confirm that suppl. Eq. (14) uses the same sign.
  2. [Tab. 2] Table 2: the check/cross symbols under 'perception', 'prediction', 'data', and 'labels' are not explained in the caption; please add a legend so readers can tell which columns indicate required annotations vs data use.
  3. [Tab. 5] Table 5: the row 'w/o MLLM pretraining fail to converge' would be more informative with a quantitative criterion (e.g., final ADE after a fixed number of steps or the training loss curve); as written, it is difficult to interpret.
  4. [suppl. Sec. 7] Supplementary Sec. 7: the fallback rule that labels a scenario 'go straight forward' when no command is satisfied by the end of the sequence can mislabel truncated turns or U-turns; please report how often this fallback occurs, since it directly affects the command distribution and the bADE partition.

Circularity Check

1 steps flagged · score 4.0 of 10

Evaluation protocol is self-referential: the high-level command input and the bADE behavior classifier are both computed from the same ground-truth future trajectory the model is asked to predict.

  1. self definitional [Sec. 3.1 Eq. (1); Sec. 4 'High-level Behavior Commands'; Supplementary Sec. 7 'Heuristics for high-level behaviors']
    "Instead of simply considering the last step position [6,25], we decide the behavior commands based on long-term future trajectories, which can handle the low-speed or stopping situations. ... For the high-level behavior command in the model inputs, 'stop' is excluded to avoid future information leakage. To determine the high-level behavior command input, we start from the ground-truth 8s future trajectories. If none of behaviors 2-7 is satisfied, we would prolong the future horizon by 2s until at least one of behaviors 2-7 is satisfied."

    Equation (1) defines the planning model as OTf = P(C, H_Th, b), treating b as a navigation-like input. But b is not a route signal available at time t; it is a deterministic function of the ground-truth future trajectory OTf^{GT}: 'do left turn' iff final heading >30 degrees, 'go straight right' iff final y-displacement < -5m, and so on. At inference the model is therefore told the maneuver class of the exact trajectory it is scored against, which sharply reduces the output hypothesis space and inflates ADE/L2. The paper acknowledges this leakage mode for 'stop' but not for the turn and lane-change commands, which are equally derived from the future.

full rationale

The core representation contribution—sparse volume lifting, temporal fusion with attention bias, and multi-decoding aggregation—is an empirical, non-circular engineering improvement. It is supported by ablations against a vanilla PaLI baseline and by standard sample-wise nuScenes L2 metrics that do not depend on the proposed bADE metric. There is no load-bearing self-citation chain or uniqueness theorem, and the final trajectory values are not mathematically forced by the command alone. However, the headline 'state-of-the-art without human annotations' claim rests on an evaluation protocol in which the high-level behavior command input is derived from the same ground-truth future trajectory that the planner must output, and the behavior-wise metric bADE is defined by the same future-derived heuristic rules. The paper even excludes 'stop' from the command because it 'may leak future information like traffic lights,' explicitly acknowledging the leakage mode while retaining turn and lane-change commands that carry the same problem. Because the comparison baselines' command provenance is not documented, the degree of inequity is unquantified. This is not full mathematical circularity, so the score is 4 rather than 6+: the self-reference is in the evaluation protocol, not in the derivation of the planning losses or the representation modules.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central empirical claim rests on the PaLI pretrained visual features, the known camera geometry projection, and the assumption that the gate mechanism selects relevant volumes without occupancy supervision. The evaluation also assumes that GT-derived behavior commands are a fair input. These are domain assumptions rather than free parameters in a derivation.

free parameters (7)
  • Sparse volume count M = 6000
    Chosen by hand; the paper fixes M per scene and ablates volume resolution but not M directly, so the value is an engineering choice that affects compute and performance.
  • Volume resolution = 1m x 1m x 2m (x,y,z)
    Chosen by hand; Tab. 6 shows bADE@5s ranges from 0.985 to 1.201 across resolutions, so this choice affects the reported headline numbers.
  • Temporal context T = 1 historical frame at -0.5s
    Chosen by hand; only the current frame plus one previous frame are used, so the reported temporal-fusion gain depends on this choice.
  • Decoded trajectory count K = 16
    Chosen by hand; Tab. 10 shows bADE@5s improves from 0.986 with greedy to 0.928 with K=16 nucleus sampling.
  • Gate feature dimension C' = 96
    Reduced from C=1536 via an FC layer; chosen by hand to allow a larger volume resolution.
  • Behavior command thresholds = stop <5m and <2m/s; turn angle >30 deg; straight displacement >5m; etc.
    These thresholds define the six high-level commands and the bADE metric (supplementary Sec. 7). They are chosen by hand and directly determine which behavior bucket each sample falls into.
  • Meta-decision thresholds = acceleration +/-0.5 m/s^2; stationary speed <2m/s and displacement <1.5m
    Heuristic rules convert ground-truth future trajectories into the four meta-decision labels used as intermediate supervision; thresholds are chosen by hand.
assumptions (6)
  • domain assumption The pretrained PaLI MLLM provides visual features that transfer to driving scenes without finetuning the vision encoder.
    Sec. 5.1 freezes the ViT and only tunes the multimodal encoder-decoder and inserted modules; the ablations in Tab. 4 show a large gain from MLLM pretraining.
  • standard math Camera intrinsics and extrinsics are known and the perspective projection in Eq. 3-4 is accurate.
    Sec. 3.3.1 projects 3D voxel coordinates to 2D image coordinates using calibrated cameras, as in SimpleBEV.
  • domain assumption The learned gate selects volumes relevant to planning without occupancy ground truth, i.e., low-gate volumes are empty or irrelevant.
    Sec. 3.3.2 states this assumption explicitly and assigns a learnable vacant feature to unselected positions.
  • domain assumption Relative-position bias added to self-attention injects 3D locality without disrupting the pretrained MLLM.
    Sec. 3.3.3 modifies attention as Eq. 9 with bin-wise bias; the ablation in Tab. 9 supports this empirically but it is still a modeling choice.
  • ad hoc to paper High-level behavior commands generated from ground-truth future trajectories are a valid navigation input that does not leak privileged information.
    Sec. 4 and supplementary Sec. 7 derive commands from the 8s GT future; this assumption is needed for the benchmark comparison and is the paper's weakest load-bearing premise.
  • ad hoc to paper The proposed behavior-wise metric bADE is a more meaningful evaluation than sample-wise metrics and collision rate.
    Sec. 4 argues sample-wise ADE and collision rate are insufficient; the paper defines bADE by the same behavior rules used for inputs, so the metric is coupled to the method.

how reviews work

0 comments
Cite this review

Pith. "Pith review of S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation." pith.science (2026). https://pith.science/paper/YUJM7YQ4

@misc{pith2026250524139,
  author       = {Pith},
  title        = {Pith review of: S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YUJM7YQ4}},
  note         = {Machine review of arXiv:2505.24139}
}
read the original abstract

The latest advancements in multi-modal large language models (MLLMs) have spurred a strong renewed interest in end-to-end motion planning approaches for autonomous driving. Many end-to-end approaches rely on human annotations to learn intermediate perception and prediction tasks, while purely self-supervised approaches--which directly learn from sensor inputs to generate planning trajectories without human annotations often underperform the state of the art. We observe a key gap in the input representation space: end-to-end approaches built on MLLMs are often pretrained with reasoning tasks in 2D image space rather than the native 3D space in which autonomous vehicles plan. To this end, we propose S4-Driver, a scalable self-supervised motion planning algorithm with spatio-temporal visual representation, based on the popular PaLI multimodal large language model. S4-Driver uses a novel sparse volume strategy to seamlessly transform the strong visual representation of MLLMs from perspective view to 3D space without the need to finetune the vision encoder. This representation aggregates multi-view and multi-frame visual inputs and enables better prediction of planning trajectories in 3D space. To validate our method, we run experiments on both nuScenes and Waymo Open Motion Dataset (with in-house camera data). Results show that S4-Driver performs favorably against existing supervised multi-task approaches while requiring no human annotations. It also demonstrates great scalability when pretrained on large volumes of unannotated driving logs.

Figures

Figures reproduced from arXiv: 2505.24139 by the authors.

Figure 1
Figure 1. S4-Driver scaling up with abundant unlabeled driv [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our proposed S4-Driver algorithm. We enhance the PaLI model for motion planning by incorporating meta-decision, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A roadmap for enhancing MLLM for planning. The [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: High-level behavior commands for motion planning. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Statistical distributions of WOMD-Planning-ADE. We [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results of motion planning. We show the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Meta-decision predic￾tion accuracy for each behavior [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Example input prompt and target output on WOMD [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Additional qualitative results of motion planning. We show the [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

70 extracted references · 34 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 1(2):3, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 2

    arXiv 2023

  4. [4]

    Paligemma: A versatile 3b vlm for trans- fer.arXiv preprint arXiv:2407.07726, 2024

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for trans- fer.arXiv preprint arXiv:2407.07726, 2024. 2

    arXiv 2024

  5. [5]

    nuscenes: A multi- modal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020. 2, 6

    2020

  6. [6]

    Mp3: A unified model to map, perceive, predict and plan

    Sergio Casas, Abbas Sadat, and Raquel Urtasun. Mp3: A unified model to map, perceive, predict and plan. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14403–14412, 2021. 2, 6

    work page 2021

  7. [7]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 14455–14465,

  8. [8]

    Driving with llms: Fusing object-level vec- tor modality for explainable autonomous driving

    Long Chen, Oleg Sinavski, Jan H ¨unermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vec- tor modality for explainable autonomous driving. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14093–14100. IEEE, 2024. 2

    work page 2024

Show all 70 references
  1. [9]

    Pali: A jointly- scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly- scaled multilingual language-image model. InThe Eleventh International Conference on Learning Representat...

  2. [10]

    Pali-3 vision language models: Smaller, faster, stronger

    Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul V oigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199, 2023. 1, 2, 3, 6, 8

    arXiv

  3. [11]

    Spatial- rgpt: Grounded spatial reasoning in vision language model

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2406.01584, 2024. 2

    2024 arXiv

  4. [12]

    Scaling instruction- finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024. 2

    2024

  5. [13]

    End-to-end driving via conditional imitation learning

    Felipe Codevilla, Matthias M ¨uller, Antonio L ´opez, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. In2018 IEEE international conference on robotics and automation (ICRA), pages 4693–

  6. [14]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  7. [15]

    Holistic autonomous driving un- derstanding by bird’s-eye-view injected multi-modal large models

    Xinpeng Ding, Jianhua Han, Hang Xu, Xiaodan Liang, Wei Zhang, and Xiaomeng Li. Holistic autonomous driving un- derstanding by bird’s-eye-view injected multi-modal large models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13668– 13...

    2024

  8. [16]

    Carla: An open urban driv- ing simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Anto- nio Lopez, and Vladlen Koltun. Carla: An open urban driv- ing simulator. InConference on robot learning, pages 1–16. PMLR, 2017. 2

    2017

  9. [17]

    An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint ar...

    2010 arXiv

  10. [18]

    Large scale interactive motion forecasting for autonomous driving: The waymo open mo- tion dataset

    Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open mo- tion dataset. InProceedings of the IEEE/CVF Interna...

    2021

  11. [19]

    Eva: Exploring the limits of masked visual representa- tion learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representa- tion learning at scale. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, ...

    2023

  12. [20]

    Simple-bev: What really mat- ters for multi-sensor bev perception? In2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 2759–2765

    Adam W Harley, Zhaoyuan Fang, Jie Li, Rares Ambrus, and Katerina Fragkiadaki. Simple-bev: What really mat- ters for multi-sensor bev perception? In2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 2759–2765. IEEE, 2023. 3, 5

    2023

  13. [21]

    The curious case of neural text degeneration.arXiv preprint arXiv:1904.09751, 2019

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration.arXiv preprint arXiv:1904.09751, 2019. 5, 1

    1904 arXiv

  14. [22]

    3d-llm: In- jecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494,

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: In- jecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494,

  15. [23]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. 2

    2021 arXiv

  16. [24]

    St-p3: End-to-end vision-based au- tonomous driving via spatial-temporal feature learning

    Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based au- tonomous driving via spatial-temporal feature learning. In European Conference on Computer Vision, pages 533–549. Springer, 2022. 2

    2022

  17. [25]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17853–17862, 2023. 1, 2, 6, 7

    2023

  18. [26]

    Emma: End-to-end multimodal model for autonomous driving, 2024

    Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo, Dragomir Anguelov, and Mingxing Tan. Emma: End-to-end multimodal model for autonomous driving, 2024. 3

    2024

  19. [27]

    Sym- phony: Learning realistic and diverse agents for autonomous driving simulation

    Maximilian Igl, Daewoo Kim, Alex Kuefler, Paul Mougin, Punit Shah, Kyriacos Shiarlis, Dragomir Anguelov, Mark Palatucci, Brandyn White, and Shimon Whiteson. Sym- phony: Learning realistic and diverse agents for autonomous driving simulation. In2022 International Conference on ...

  20. [28]

    Vad: Vectorized scene representa- tion for efficient autonomous driving

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representa- tion for efficient autonomous driving. InProceedings of the IEEE/CVF International Conference on Computer Vision, p...

    2023

  21. [29]

    Textual explanations for self-driving ve- hicles

    Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving ve- hicles. InProceedings of the European conference on com- puter vision (ECCV), pages 563–578, 2018. 6

    2018

  22. [30]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 2

    2023

  23. [31]

    Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers

    Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chong- hao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. InEuropean con- ference on computer vision, pages 1–18. Springer, 2022. 2, 3, 5

    2022

  24. [32]

    Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open- loop end-to-end autonomous driving? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14864–14873, 2024. 2, 3, 8

    2024

  25. [33]

    Maptr: Structured modeling and learning for online vectorized hd map construction

    Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. Maptr: Structured modeling and learning for online vectorized hd map construction. InThe Eleventh International Conference on Learning Representations. 2

  26. [34]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024. 2

    2024

  27. [35]

    Bevfusion: Multi- task multi-sensor fusion with unified bird’s-eye view repre- sentation

    Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi- task multi-sensor fusion with unified bird’s-eye view repre- sentation. In2023 IEEE international conference on robotics and automation (ICRA), pages 2774–2781. IE...

    2023

  28. [36]

    When llms step into the 3d world: A survey and meta-analysis of 3d tasks via multi-modal large language models.arXiv preprint arXiv:2405.10255, 2024

    Xianzheng Ma, Yash Bhalgat, Brandon Smart, Shuai Chen, Xinghui Li, Jian Ding, Jindong Gu, Dave Zhenyu Chen, Songyou Peng, Jia-Wang Bian, et al. When llms step into the 3d world: A survey and meta-analysis of 3d tasks via multi-modal large language models.arXiv preprint arXiv:2...

    2024

  29. [37]

    Dolphins: Multimodal language model for driving.arXiv preprint arXiv:2312.00438, 2023

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving.arXiv preprint arXiv:2312.00438, 2023. 2

    2023 arXiv

  30. [38]

    Gpt-driver: Learning to drive with gpt.arXiv preprint arXiv:2310.01415, 2023

    Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt.arXiv preprint arXiv:2310.01415, 2023. 2, 7

    2023 arXiv

  31. [39]

    A language agent for autonomous driving.arXiv preprint arXiv:2311.10813, 2023

    Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang. A language agent for autonomous driving.arXiv preprint arXiv:2311.10813, 2023. 2

    2023 arXiv

  32. [40]

    Mm1: Methods, analysis & insights from multimodal llm pre-training.arXiv preprint arXiv:2403.09611, 2024

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xi- anzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training.arXiv preprint arXiv:2403.09611, 2024. 2

    2024 arXiv

  33. [41]

    Wayformer: Motion forecasting via simple & efficient attention networks

    Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In2023 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 2980–2987. IEEE, 2023. 2

    2023

  34. [42]

    Gpt-4v(ision) system card, 2023

    OpenAI. Gpt-4v(ision) system card, 2023. 1, 2

    2023

  35. [43]

    Alvinn: An autonomous land vehicle in a neural network.Advances in neural information processing systems, 1, 1988

    Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network.Advances in neural information processing systems, 1, 1988. 1

    1988

  36. [44]

    Qi, Yin Zhou, Mahyar Najibi, Pei Sun, Khoa T

    C. Qi, Yin Zhou, Mahyar Najibi, Pei Sun, Khoa T. V o, Boyang Deng, and Dragomir Anguelov. Offboard 3d ob- ject detection from point cloud sequences.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6130–6140, 2021. 7

    2021

  37. [45]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  38. [46]

    Motionlm: Multi-agent motion forecast- ing as language modeling

    Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecast- ing as language modeling. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 8579– ...

    2023

  39. [47]

    Languagempc: Large language models as decision makers for autonomous driving.arXiv preprint arXiv:2310.03026, 2023

    Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. Languagempc: Large language models as decision makers for autonomous driving.arXiv preprint arXiv:2310.03026, 2023. 2

    arXiv

  40. [48]

    Lmdrive: Closed-loop end-to-end driving with large language models

    Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15120–15130, 2024. 2

    2024

  41. [49]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. InFirst Vision and Lan- guage for Autonomous Driving and Robotics Workshop. 2, 6, 8, 4

  42. [50]

    Ul2: Unifying language learning paradigms.arXiv preprint arXiv:2205.05131, 2022

    Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Ja- son Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shak- eri, Dara Bahri, Tal Schuster, et al. Ul2: Unifying language learning paradigms.arXiv preprint arXiv:2205.05131, 2022. 6

    2022 arXiv

  43. [51]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 1, 2, 3

    2023 arXiv

  44. [52]

    Tokenize the world into object-level knowledge to address long-tail events in autonomous driving

    Thomas Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Ed- ward Schmerling, Yue Wang, Boris Ivanovic, and Marco Pavone. Tokenize the world into object-level knowledge to address long-tail events in autonomous driving. InWorkshop on Language and Robot Learning: Language as an Inter- face. 2

  45. [53]

    Drivevlm: The convergence of autonomous driving and large vision-language models.arXiv preprint arXiv:2402.12289, 2024

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Chenxu Hu, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models.arXiv preprint arXiv:2402.12289, 2024. 1, 2, 3, 6, 7

    2024 arXiv

  46. [54]

    Multipath++: Efficient information fu- sion and trajectory aggregation for behavior prediction

    Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivas- tava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Multipath++: Efficient information fu- sion and trajectory aggregation for behavior prediction....

    2022

  47. [55]

    Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning.arXiv preprint arXiv:2405.01533, 2024

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning.arXiv preprint arXiv:2405.01533, 2024. 7, 8

    2024 arXiv

  48. [56]

    Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022. 2, 3

    2022

  49. [57]

    Para-drive: Parallelized architecture for real- time autonomous driving

    Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real- time autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15449–15458, 2024. 1, 7

    2024

  50. [58]

    Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong base- line.Advances in Neural Information Processing Systems, 35:6119–6132, 2022

    Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong base- line.Advances in Neural Information Processing Systems, 35:6119–6132, 2022. 2

    2022

  51. [59]

    Grok-1.5 vision preview, 2024

    xAI. Grok-1.5 vision preview, 2024. 2

    2024

  52. [60]

    Sparsefusion: Fusing multi-modal sparse rep- resentations for multi-sensor 3d object detection

    Yichen Xie, Chenfeng Xu, Marie-Julie Rakotosaona, Patrick Rim, Federico Tombari, Kurt Keutzer, Masayoshi Tomizuka, and Wei Zhan. Sparsefusion: Fusing multi-modal sparse rep- resentations for multi-sensor 3d object detection. InProceed- ings of the IEEE/CVF International Confer...

    2023

  53. [61]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model.IEEE Robotics and Automation Let- ters, 2024

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model.IEEE Robotics and Automation Let- ters, 2024. 1, 2, 6

    2024

  54. [62]

    Rethinking the open-loop evaluation of end- to-end autonomous driving in nuscenes.arXiv preprint arXiv:2305.10430, 2023

    Jiang-Tian Zhai, Ze Feng, Jihao Du, Yongqiang Mao, Jiang- Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jing- dong Wang. Rethinking the open-loop evaluation of end- to-end autonomous driving in nuscenes.arXiv preprint arXiv:2305.10430, 2023. 3, 8

    2023 arXiv

  55. [63]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 2

    2023

  56. [64]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonza- lez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 2

    2023

  57. [65]

    Occworld: Learning a 3d occupancy world model for autonomous driving.arXiv preprint arXiv:2311.16038, 2023

    Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Occworld: Learning a 3d occupancy world model for autonomous driving.arXiv preprint arXiv:2311.16038, 2023. 2

    2023 arXiv

  58. [66]

    Genad: Generative end-to-end autonomous driving.arXiv preprint arXiv:2402.11502, 2024

    Wenzhao Zheng, Ruiqi Song, Xianda Guo, and Long Chen. Genad: Generative end-to-end autonomous driving.arXiv preprint arXiv:2402.11502, 2024. 2 S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation Supplementary M...

    2024 arXiv

  59. [67]

    go straight forward

    Implementation Details WOMD-Planning-ADE benchmark.This benchmark contains 487k scenarios for model training and 44k for val- idation, which are divided from 103k sequences of20s length. Each scenario contains1shistory and8sfuture. We only consider the future5sfor the open-loo...

  60. [68]

    7, we report theADE@5smetric of S4-Driver for each ego-vehicle behavior on WOMD-Planning-ADE benchmark separately

    Behavior-wise Model Performance In Tab. 7, we report theADE@5smetric of S4-Driver for each ego-vehicle behavior on WOMD-Planning-ADE benchmark separately. Results show the superiority of S4- Driver especially in complicated scenarios like turnings, Methods ADE@5s for each beha...

  61. [69]

    10, we visualize more planning results on WOMD-Planning-ADE

    Additional Qualitative Results In Fig. 10, we visualize more planning results on WOMD-Planning-ADE. Examples cover different behav- iors, speeds, lighting conditions, and weathers. Results show the robust performance of S4-Driver in all these di- verse scenarios

  62. [70]

    Camera configuration.We apply different configurations of camera sensors in Tab

    Additional Ablation Studies In this part, we conduct several additional ablation stud- ies to further justify the design of our S4-Driver includ- ing the camera configuration, relative attention bias, multi- decoding aggregation, and motion tokens. Camera configuration.We appl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.