{"id":"bfc20fb6-6ff4-446e-842c-9defc25054b4","arxiv_id":"2505.24139","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"S4-Driver uses a multimodal LLM with a sparse 3D spatio-temporal volume representation to achieve self-supervised motion planning that rivals supervised methods on nuScenes and WOMD.","lead":"A self-supervised driving planner built on a multimodal language model lifts frozen 2D image features into a sparse 3D volume to predict future waypoints without human labels. On nuScenes and a new Waymo-based benchmark it reports lower average displacement error than many supervised baselines, though the evaluation protocol has caveats.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GT-future-derived high-level behavior command (suppl. Sec. 7) plausibly leaks the maneuver to be predicted; until the open-loop evaluation is rerun with non-future commands or a no-command ablation, the SOTA claim rests on an insecure protocol.","rationale":"The reader's identified weakest assumption is also the most load-bearing concern I can find. The central claim is that S4-Driver achieves state-of-the-art open-loop planning without human annotations. That claim is evaluated with a high-level behavior command computed from ground-truth future trajectories (supplementary Sec. 7). In an open-loop benchmark, this can be a direct leak of the maneuver being predicted, and the behavior-wise bADE metric amplifies the effect by averaging within maneuver classes. The paper even shows awareness of the leakage class by excluding the 'stop' command at inference. My reading of the architecture and ablations is otherwise fair: the sparse volume representation, temporal fusion, and multi-decoding aggregation are internally consistent and the ablations support their usefulness. But those contributions do not establish the SOTA claim unless the evaluation protocol is fair. Re-running with past/route-derived commands or no command is a concrete, minimal check that would settle the issue. Because the reader already assigned CONDITIONAL on exactly this basis, my assessment does not move the verdict; it strengthens the rationale for keeping it conditional until the protocol is fixed.","tokens_in":20307,"tokens_out":3683,"duration_ms":48676,"concrete_test":"Re-run the Tab. 2 and Tab. 3 evaluations with two command-ablated protocols: (i) no high-level command at all, and (ii) commands generated only from the past trajectory plus the route/HD map and intersection geometry available at time t, not from the 8s ground-truth future. Keep all model hyperparameters and training unchanged. If bADE@5s or the nuScenes average L2 degrades by more than ~10%, or if S4-Driver no longer beats MotionLM/OmniDrive under protocol (ii), the SOTA claim is attributable to future-command leakage. Also report per-behavior ADE and collision rate alongside these numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the way the high-level command is produced. Sec. 3.1 treats the command as a navigation input available at inference, but supplementary Sec. 7 defines it using the ground-truth future trajectory: stop if displacement <5m; left turn if final heading angle >30 degrees; left U-turn if final displacement is < -5m along x; go straight right/left if final lateral displacement exceeds 5m; and if no behavior is satisfied, the future horizon is extended by 2s until one is. This label is a function of the exact future behavior the model is asked to predict, not of a route or map available at time t. At inference, feeding 'do left turn' or 'go straight right' tells the model which maneuver class to generate, which can substantially reduce ADE, especially in the behavior-wise bADE metric that averages per-maneuver errors. The paper explicitly excludes 'stop' from commands because it 'may leak future information like traffic lights,' showing that the authors recognize this leakage mode; however, turn and lane-change commands derived from future trajectories carry the same concern. Baselines in Tab. 2 (UniAD, VAD, OmniDrive, etc.) may or may not receive equivalent GT-derived commands; the paper does not state their command provenance. The headline claim of state-of-the-art performance without annotations therefore depends on a protocol change: commands must come from information available before the prediction horizon, or the comparison must be run without commands.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes S4-Driver, a self-supervised end-to-end motion planner built on the PaLI multimodal large language model. It introduces a sparse 3D volume representation that lifts frozen 2D vision encoder features into an ego-centric 3D volume, aggregates multi-view and multi-frame information, adds relative-position attention bias, predicts a two-stage meta-decision before outputting waypoints, and averages multiple nucleus-sampled trajectory decodings. The model is trained with raw camera images and ego-vehicle trajectory supervision only, and is evaluated on nuScenes and a newly introduced WOMD-Planning-ADE benchmark. The paper reports state-of-the-art L2/ADE numbers on nuScenes and favorable results on WOMD-Planning-ADE, supported by ablations of each component.","tokens_in":20647,"tokens_out":10018,"duration_ms":100956,"significance":"If the evaluation protocol is corrected, this is a significant contribution: the sparse-volume lifting with a frozen vision encoder is a clean mechanism for providing MLLMs 3D spatial reasoning without finetuning the image encoder, and the ablations (Tabs. 4-11) are internally consistent and clearly presented. The WOMD-Planning-ADE benchmark with behavior-wise metrics addresses a real need for larger-scale evaluation of camera-based planners, and the scalability result with unlabeled driving logs is valuable. However, the headline SOTA claims currently rest on an evaluation protocol in which the model receives a high-level behavior command computed from the ground-truth future trajectory; this must be fixed before the empirical claims can be accepted.","major_comments":[{"comment":"Section 3.1, Eq. (1), and supplementary Sec. 7: The high-level behavior command b is an input to the planner at inference, and supplementary Sec. 7 defines it from the ground-truth future trajectory (stop if displacement <5 m; left turn if final heading >30 deg; left U-turn if final x-displacement < -5 m; if no command is satisfied, the future horizon is extended by 2 s until one is). This means the model is told at inference which maneuver class to produce, which is future information that a navigation system would not provide at time t. Feeding 'do left turn' or 'go straight right' can substantially reduce ADE/L2, especially in the behavior-wise bADE metric that averages per-maneuver errors. The paper itself excludes 'stop' from the command set because it 'may leak future information like traffic lights,' but turns and lane changes derived from future trajectories carry the same concern. This protocol issue is load-bearing for the SOTA claims in Sec. 6 and Tabs. 2-3; the internal ablations that compare S4-Driver with Vanilla PaLI under the same command protocol are not invalidated, but the absolute numbers and external comparisons are. Please rerun the evaluation with commands obtained from information available before the prediction horizon, add a no-command ablation, and state whether the baselines in Tab. 2 receive commands and from what source.","section":"§3.1, Eq. (1); suppl. Sec. 7"},{"comment":"Section 4, Eq. (13), and supplementary Sec. 7: The proposed behavior-wise metric bADE partitions the validation set with the same heuristic rules over the ground-truth future trajectory used to create the command input, so the evaluation is potentially circular: the model receives the behavior label as an input and is then averaged within that same label. Furthermore, 'stop' is included among the seven bADE behaviors but is deliberately excluded from the command set, making the per-behavior comparison inconsistent; Tab. 7 indeed shows S4-Driver is worse than Vanilla PaLI on stop (0.063 vs 0.048), consistent with the stop command being withheld. Please report bADE with and without the command input, keep the behavior partition fixed across methods, and clarify how stop-conditioned samples are handled for models that never receive a stop command.","section":"§4, Eq. (13); suppl. Sec. 7"},{"comment":"Section 6 and Tab. 3: The claim of state-of-the-art performance on WOMD-Planning-ADE is not supported by the evidence. Tab. 3 compares S4-Driver only with Vanilla PaLI and an internally reproduced MotionLM that additionally consumes auto-labeled objects, tracks, and roadgraph; no other camera-based end-to-end planner is included. Because WOMD-Planning-ADE is introduced by this paper, 'state-of-the-art' is a claim about a benchmark with no prior art, and the favorable comparison against MotionLM is acknowledged to be inequitable in input modality. Please broaden the comparison to existing camera-based planners adapted to this benchmark, or restrict the claim to 'favorable performance against the included baselines.'","section":"Sec. 6 and Tab. 3"}],"minor_comments":[{"comment":"Equation (9): the attention formula subtracts b(D) from the logits, but the text says a relative position bias is 'injected' into the attention; the sign convention is ambiguous. Please state explicitly whether larger bias values increase or decrease attention between nearby tokens, and confirm that suppl. Eq. (14) uses the same sign.","section":"Eq. (9)"},{"comment":"Table 2: the check/cross symbols under 'perception', 'prediction', 'data', and 'labels' are not explained in the caption; please add a legend so readers can tell which columns indicate required annotations vs data use.","section":"Tab. 2"},{"comment":"Table 5: the row 'w/o MLLM pretraining fail to converge' would be more informative with a quantitative criterion (e.g., final ADE after a fixed number of steps or the training loss curve); as written, it is difficult to interpret.","section":"Tab. 5"},{"comment":"Supplementary Sec. 7: the fallback rule that labels a scenario 'go straight forward' when no command is satisfied by the end of the sequence can mislabel truncated turns or U-turns; please report how often this fallback occurs, since it directly affects the command distribution and the bADE partition.","section":"suppl. Sec. 7"}],"recommendation":"major_revision","confidential_remarks":"The core representation story is credible and the internal ablations are consistent. The main risk is the open-loop evaluation protocol: the GT-future-derived command must be ablated or replaced before the SOTA claims are publishable. I would also note that WOMD-Planning-ADE uses in-house camera data; without a release or a clear data-sharing plan, the benchmark contribution will not be reproducible by the community. If the revision addresses the command-leak issue with a no-command or route-command ablation, I would be willing to re-review favorably."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a real methods contribution—a sparse volume representation that makes a frozen MLLM visual encoder do 3D spatio-temporal reasoning—and its internal ablations are clean. But the headline SOTA claim is not yet credible because the high-level behavior command fed to the model at inference is computed from the ground-truth future trajectory (supplementary Sec. 7). That leaks the maneuver the model is supposed to predict, and it biases both ADE and behavior-wise ADE. The claim should be treated as conditional until the evaluation is rerun without future-derived commands or with commands from information available before the prediction horizon.\n\nWhat's actually new: the gate-based sparse volume selection over lifted multi-view features, with a learned vacant feature and relative-position attention bias, applied to a frozen ViT and a PaLI decoder. That combination is not in the cited literature. The ablations (Tabs. 4-11) are consistent and show each component buys something. The scaling result with internal unlabeled data is also suggestive. I credit the paper for clearly writing out the command heuristics, which is exactly what allows a reviewer to see the problem.\n\nThe soft spot is the one the stress-test note names. The command is defined from the GT 8s future, and if no behavior is satisfied the horizon is extended by 2s. Feeding 'do left turn' or 'go straight right' to the prompt tells the model which maneuver class to generate. The paper excludes 'stop' from commands because it may leak future info like traffic lights, which shows the authors know this failure mode, but turns and lane changes are the same kind of leak. The nuScenes comparison has the same issue, and the paper does not state whether UniAD, VAD, etc. received equivalent commands. On WOMD, the bADE metric is defined by the same heuristics over the same GT trajectory, so the metric and the command are entangled. However, the internal comparisons against vanilla PaLI use the same protocol, so the representation gain is still credible. The proprietary in-house camera data also limits reproducibility, but that's a standard industry constraint.\n\nBottom line: for someone working on MLLM-based driving, the sparse volume idea is worth engaging and the paper deserves a serious referee. But the SOTA sentence should not be taken at face value; it needs a protocol fix.","headline":"A genuinely useful sparse-volume representation for MLLM driving, but the SOTA claim is undermined by a GT-future-derived behavior command that leaks the maneuver at inference.","tokens_in":21213,"tokens_out":3018,"would_cite":false,"duration_ms":34878,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A self-supervised driving MLLM can beat supervised planners without human labels.","keywords":["autonomous driving","end-to-end motion planning","multimodal large language model","self-supervised learning","sparse volume representation","spatio-temporal reasoning","Waymo Open Motion Dataset","nuScenes"],"falsifier":"Take the trained S4-Driver and re-run the nuScenes and WOMD-Planning-ADE evaluations with the high-level behavior command replaced by one derived from a map and past position only, so it contains no future information. If ADE and bADE degrade substantially or the gap to supervised baselines closes, the state-of-the-art claim rests on the future-information leak rather than on the spatio-temporal representation itself.","tokens_in":20083,"feed_emoji":"🚗","tokens_out":4984,"duration_ms":48605,"temperature":0.7,"pith_summary":"This paper argues that a multimodal large language model can be turned into a state-of-the-art open-loop motion planner without any human annotations, using only camera images and the ego vehicle's own past and future trajectory as supervision. The proposed method, S4-Driver, lifts the frozen 2D visual features of an MLLM into a sparse 3D volume centered on the ego vehicle, aggregates several camera views and past frames, and then predicts future waypoints as text. If the claim is right, the main bottleneck of end-to-end driving—expensive perception and prediction labels—disappears, and planning performance can scale with the volume of unlabeled driving logs. The paper reports favorable results against supervised multi-task planners on both nuScenes and the much larger WOMD-Planning-ADE benchmark.","feed_headline":"Self-supervised MLLM planner tops two driving benchmarks","feed_subtitle":"Lifting frozen camera features into sparse 3D volumes lets it match supervised planners using only raw video and ego trajectories.","key_machinery":"The sparse volume representation carries the argument. From each camera view's frozen feature map, a small fully connected layer produces a low-dimensional gate feature; these are lifted into a 3D volume via bilinear projection (Simple-BEV-style sampling), and an MLP with sigmoid outputs a per-voxel gate score. The M voxels with the highest scores keep their semantic features (blended with a learnable vacant feature), while the rest are discarded, concentrating the model on space that matters for driving. A bin-wise relative position bias added to the self-attention of the multimodal encoder injects local 3D structure, and historical frames are fused after ego-motion compensation by concatenating per-frame volume features. Finally, nucleus sampling generates K candidate trajectories whose unweighted average forms the output, counteracting the model's bias toward simple behaviors.","core_discovery":"The central claim is that MLLM-based planning underperforms because its visual representations live in the 2D image plane, while driving decisions are made in 3D space. S4-Driver addresses this by projecting multi-view, multi-frame image features into an ego-centered sparse volume: a lightweight gate selects the voxels most relevant to driving, sampled features plus positional embeddings form volume tokens, and a relative position bias in self-attention encodes local 3D geometry without retraining the vision encoder. With this spatio-temporal visual representation, the model predicts future ego waypoints directly as floating-point text, guided by a coarse meta-decision (stationary, keep speed, accelerate, decelerate) and a high-level behavior command. The paper claims state-of-the-art results on nuScenes and WOMD-Planning-ADE compared to prior end-to-end planners, all without perception or prediction labels.","pith_inferences":["If the high-level behavior command is computed from the ground-truth future trajectory, as described in the supplementary material, the model receives a form of future information at inference; a comparison against planners that do not use such commands may overstate the advantage, and a fairer test would use a command derived without future knowledge.","The sparse-volume gating and relative-position-bias recipe is not specific to driving; the same machinery could be applied to other 3D reasoning tasks such as robotic manipulation or navigation, where an MLLM must ground language in egocentric 3D space.","Because the design keeps the vision encoder frozen, the gains are likely to transfer to newer MLLMs, but the paper's numbers should be re-checked if the behavior-command leakage is removed."],"forward_implications":["Human annotation is removed from the loop: the same pipeline can be pretrained on massive unlabeled driving logs, and the paper shows performance improves with pretraining data.","The frozen vision encoder means the representation can be dropped into stronger or larger MLLMs without retraining the image backbone.","The behavior-wise metrics (bADE) expose performance on rare but safety-critical maneuvers such as turns and U-turns, where the paper reports the largest gains.","A planner that reasons in ego-centric 3D volume space, instead of image space, is the direction that closes the gap between MLLM reasoning and the native coordinate frame of planning."],"supporting_citations":[{"why":"Supplies the pretrained PaLI-3 5B MLLM whose frozen image encoder provides the 2D features lifted into 3D volumes.","marker":"[10]"},{"why":"Provides the WOMD dataset and the basis for the new WOMD-Planning-ADE benchmark with in-house camera data.","marker":"[18]"},{"why":"nuScenes is the standard benchmark used for the main planning comparison against prior end-to-end drivers.","marker":"[5]"},{"why":"UniAD, the supervised multi-task end-to-end baseline that S4-Driver claims to outperform on nuScenes.","marker":"[25]"},{"why":"VAD, another supervised vectorized end-to-end baseline in the nuScenes comparison table.","marker":"[28]"},{"why":"MotionLM, the object-input motion-forecasting baseline that S4-Driver is compared against on WOMD-Planning-ADE.","marker":"[46]"},{"why":"DriveVLM, the MLLM-based chain-of-thought driving baseline that the self-supervised design is contrasted with.","marker":"[53]"},{"why":"Simple-BEV, the lifting and sampling procedure that the sparse volume construction adapts.","marker":"[20]"},{"why":"Nucleus sampling, the decoding strategy used to generate multiple candidate trajectories for aggregation.","marker":"[21]"}],"fun_headline_variants":["Self-supervised planner uses sparse 3D volumes to beat benchmarks","S4-Driver: Self-supervised MLLM for 3D motion planning","Frozen vision meets sparse volumes for self-supervised driving","Self-supervised driving MLLM excels without labels via 3D volumes","Sparse 3D volumes boost self-supervised driving MLLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that giving the model a high-level behavior command computed from the ground-truth future trajectory is a fair way to evaluate open-loop planning; if that command leaks the intended maneuver, the reported advantage over planners without such commands may not hold under a future-blind evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised planner uses sparse 3D volumes to beat benchmarks","S4-Driver: Self-supervised MLLM for 3D motion planning","Frozen vision meets sparse volumes for self-supervised driving","Self-supervised driving MLLM excels without labels via 3D volumes","Sparse 3D volumes boost self-supervised driving MLLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1940,"prompt_tokens":996,"completion_tokens":944,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":848}},"tokens_in":612,"tokens_out":944,"duration_ms":6544,"temperature":1.0,"reasoning_tokens":848,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:33:28.042207+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained S4-Driver and re-run the nuScenes and WOMD-Planning-ADE evaluations with the high-level behavior command replaced by one derived from a map and past position only, so it contains no future information. If ADE and bADE degrade substantially or the gap to supervised baselines closes, the state-of-the-art claim rests on the future-information leak rather than on the spatio-temporal representation itself.","supporting_citations":[{"cited_title":"Large scale interactive motion forecasting for autonomous driving: The waymo open mo- tion dataset","cited_arxiv_id":null,"evidence_quote":"Provides the WOMD dataset and the basis for the new WOMD-Planning-ADE benchmark with in-house camera data."},{"cited_title":"Planning-oriented autonomous driving","cited_arxiv_id":null,"evidence_quote":"UniAD, the supervised multi-task end-to-end baseline that S4-Driver claims to outperform on nuScenes."},{"cited_title":"Vad: Vectorized scene representa- tion for efficient autonomous driving","cited_arxiv_id":null,"evidence_quote":"VAD, another supervised vectorized end-to-end baseline in the nuScenes comparison table."},{"cited_title":"Motionlm: Multi-agent motion forecast- ing as language modeling","cited_arxiv_id":null,"evidence_quote":"MotionLM, the object-input motion-forecasting baseline that S4-Driver is compared against on WOMD-Planning-ADE."},{"cited_title":"Simple-bev: What really mat- ters for multi-sensor bev perception? In2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 2759–2765","cited_arxiv_id":null,"evidence_quote":"Simple-BEV, the lifting and sampling procedure that the sparse volume construction adapts."}],"review_version":1}