{"id":"c4758b82-82fc-49a5-95b8-21c48c3d8f7b","arxiv_id":"2608.03084","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"One shared video generator predicts future RGB, semantics, depth, and instance tracks simultaneously, and a separate action expert reads those latents to output the ego trajectory, reaching 91.0 EPDMS on NAVSIM-v2 navtest.","lead":"SUV is an autonomous-driving model that treats prediction of future camera views, road layout, depth, and object tracks as a single video-generation task, using one pretrained video model for all four. It reports top scores on the NAVSIM-v2 planning benchmark with a single front camera and no candidate-trajectory selection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Future-scene metrics are evaluated against the same frozen SAM3/DA3 teachers that generated the training targets, so the structured-stream results may reflect teacher-specific artifacts rather than independent scene understanding; this self-reference is acknowledged but not resolved.","rationale":"The reader's weakest assumption is the same one I would put first: the structured future-stream metrics are self-referential. This is the load-bearing point for the paper's named contribution. The planning benchmark is external, so the strongest argument for the action expert survives; but the title and abstract lead with 'future scene understanding,' and the controlled ablations that connect future representations to planning are trained on exactly the teacher labels used for evaluation. If the teachers are biased, both the measured stream quality and the measured planning benefit of S/G/I supervision could be inflated. I do not think this changes the verdict from the reader's CONDITIONAL: the paper is a strong empirical application, but resolution of the circularity should be a stated condition. Other concerns—single training seed, small navtest margins, baseline EPDMS protocol consistency—are real but secondary; they affect confidence intervals rather than the validity of the central construction, and the paper already reports fixed-checkpoint reproducibility and scene-level variation. The independent-reference test above would settle the circularity directly.","tokens_in":22438,"tokens_out":12651,"duration_ms":131256,"concrete_test":"Select 150–200 held-out NAVSIM-v2 navtest clips and construct independent references: human-annotated semantic labels and instance associations for Car, Truck, Bus, Bicycle, and Pedestrian in the 50-m region, and depth references from a source other than DA3, such as projected nuScenes LiDAR or a different model family like Depth Anything V2. Recompute mIoU, AssA@50, δ1, and AbsRel for both SUV native generation and Generate-then-Perceive against these independent references using the same evaluation code. If the independent-reference scores are substantially lower than the teacher-reference scores (for example, more than 5 points in mIoU or AssA@50) or if the native-versus-post-hoc ordering in Table 5 reverses, then the structured-stream results are overfitted to the training teachers and the 'future scene understanding' claim should be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Load-bearing concern: the evidence for the 'future scene understanding' contribution is partially circular. The structured training targets for segmentation, relative depth, and instance tracks are produced by frozen SAM3 and DA3 (Supp. §4.1), and the evaluation references for mIoU, AssA@50, δ1, and AbsRel are produced by the same frozen teachers on the recorded future RGB frames (Experiments, Future-scene evaluation). Any systematic teacher bias—SAM3's prompt and confidence processing, DA3's clip-level percentile clipping and 256-color Turbo quantization, the track-identity modulo-7 color code and Hungarian decoder—is learned by the video expert and then counted as 'correct.' Table 5 also compares native generation against Generate-then-Perceive using these same teachers for both arms, so it cannot detect teacher-specific artifacts; it only shows which route best reproduces the teachers. The paper's explicit caveat concedes this. The planning results on NAVSIM-v2 and WOD-E2E are external and therefore not directly invalidated, but the integrated claim that structured future supervision improves planning inherits the risk if the planner exploits teacher-specific artifacts rather than genuine scene structure. As it stands, the central novelty—future scene understanding—is not yet validated against independent ground truth.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes SUV, an end-to-end driving framework that casts future scene understanding as video generation. A single video expert initialized from Wan2.2-5B is post-trained to generate four future streams (RGB, semantic segmentation, relative depth, instance tracks) as native videos, without stream-specific visual prediction heads. A separate action expert attends to the latent tokens of all streams during joint denoising to produce a trajectory. The model is evaluated on NAVSIM-v2 (91.0 EPDMS on navtest, 36.9 on navhard) and WOD-E2E (RFS 7.94) with a single front camera and no candidate selection, along with ablations isolating the contributions of generative initialization, structured supervision, and future-stream access. The paper also provides six-run reproducibility checks and detailed supplementary documentation of target construction and training.","tokens_in":22635,"tokens_out":8736,"duration_ms":75998,"significance":"If the results hold, SUV offers a conceptually clean demonstration that one pretrained video generator can serve as a shared predictor for heterogeneous future scene signals and that the resulting future-stream latents can improve trajectory planning. The external planning benchmark results are competitive or state-of-the-art, the experimental design includes controlled 2x2 ablations and per-stream access ablations, and the paper ships code and reproducibility details. The principal weakness is that the future-scene evaluation is self-referential: segmentation, depth, and tracking references are produced by the same frozen SAM3 and DA3 models that generated the training targets, so the reported mIoU, δ1, AbsRel, and AssA@50 values measure teacher-agreement rather than independent scene understanding. This gap is explicitly acknowledged in the paper but remains unresolved, and it weakens the headline 'future scene understanding' contribution.","major_comments":[{"comment":"The structured-stream metrics are computed against references generated by the same frozen SAM3 (segmentation, tracks) and DA3 (depth) models that produced the training targets, as the paper explicitly acknowledges: 'The segmentation, tracking, and depth metrics measure agreement with frozen teachers rather than accuracy against independent ground truth.' This circularity is load-bearing because the paper's headline contribution is future scene understanding, and the evidence for it is currently only teacher-agreement. Please either (a) add an independent evaluation subset (e.g., human annotations or a different off-the-shelf model for a sample of frames) and report the same metrics on that subset, or (b) explicitly reframe the future-scene claims as 'agreement with frozen teacher models' rather than 'accuracy' or 'understanding'. The Table 5 comparison between native generation and Generate-then-Perceive is useful for task-alignment but cannot detect teacher-specific artifacts because both arms use the same teacher-generated references.","section":"Future-scene evaluation (main text) and Supplementary §4.1"},{"comment":"The 2x2 ablation shows that adding segmentation/relative-depth/instance-track supervision (S/G/I) improves EPDMS on both navtest and navhard. Since the structured targets come from the same teachers used in the future-scene evaluation, this improvement could partly reflect the planner exploiting teacher-specific artifacts rather than general scene structure. The planning scores themselves are external and therefore not invalidated, but the interpretation of the improvement as evidence for 'future scene understanding' inherits the circularity concern. Please address this by either adding a control with targets from a different teacher on a subset (e.g., a different depth or segmentation model) or explicitly discussing the risk and tempering the corresponding conclusion.","section":"Planning Benefits of Future Representations, Table 6"}],"minor_comments":[{"comment":"The novelty claim ('the first end-to-end driving framework ...') should be cross-checked against the most recent concurrent work on generative world models for driving (e.g., EponaV2, GeoSem-WAM, WAM4D). A sentence explaining how SUV specifically differs in the format of the predicted streams (native video vs. task-specific heads) would strengthen the claim.","section":"Introduction / Contributions"},{"comment":"The abbreviation 'S/G/I' is used in the text without definition; it is only defined in the Table 6 caption. Please define it at first use.","section":"Planning Benefits of Future Representations"},{"comment":"'Rankr seeds Python, NumPy, PyTorch, and CUDA' appears to be a typo; it should read 'rank r seeds ...'.","section":"Supplementary §4.3"},{"comment":"The sentence 'Wan2.2-5B initialization yields better point estimates for all five metrics' is correct, but the large gaps (e.g., mIoU 64.2 vs 49.7) would benefit from a note on whether these differences are consistent across repeated training runs, or at least a statement that the six-run reproducibility checks were only reported for planning scores.","section":"Table 4"},{"comment":"The notation A(g) is slightly confusing because it is a set of visible key groups. Consider defining it explicitly as a function from a token group to the set of allowed key groups to improve readability.","section":"Method, Eq. (3)"}],"recommendation":"major_revision","confidential_remarks":"The authors have been transparent about the circularity of the future-scene evaluation, which is commendable. If they can add even a small independent validation (e.g., a few hundred annotated frames or a different off-the-shelf teacher for the reference set), the paper's central claim would be substantially stronger. The planning results appear solid and reproducible, and the paper is well documented. The novelty claim should be carefully checked against concurrent works on generative world models for driving."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this one. The central idea is clean and I haven't seen it in the cited literature: instead of task-specific heads or language serialization, SUV post-trains a single Wan2.2-5B video expert to generate RGB, segmentation, relative depth, and instance tracks as four native video streams, and a separate action expert attends to all stream latents during joint denoising. That is a real interface change, and the paper executes it carefully: controlled 2x2 ablations isolate structured supervision and future-stream access, the planning numbers on NAVSIM-v2 (91.0 EPDMS navtest, 36.9 navhard) and WOD-E2E (7.94 RFS) come from a single front camera with no candidate selection, and they report six-run fixed-checkpoint reproducibility. Credit where due: this is a serious empirical study with code released.\n\nThe main soft spot is exactly where the stress-test note lands. Segmentation, depth, and tracking metrics are evaluated against the same frozen SAM3/DA3 teachers that produced the training targets. The paper acknowledges this in the experiments section, but it means the future-scene understanding claims are about reproducing teacher behavior, not about independent ground truth. The Generate-then-Perceive comparison in Table 5 inherits the same problem—both arms are scored by the same teachers, so it only tells you which route best mimics them. This is a genuine limitation, not a fatal flaw. The planning results are external and the ablations are informative; the integrated claim that structured future supervision helps planning is plausible but does inherit some risk if the planner exploits teacher artifacts.\n\nOther soft spots are smaller. The navtest gains from future-stream access are tiny (90.7 to 91.0), and there is no multi-seed variance for those effect sizes—only scene-level dispersion. The \"corrected official EPDMS implementation\" is mentioned but not precisely specified, which matters for reproducing the leaderboard numbers. The multi-stream versus RGB-only RGB differences are within noise, which the paper honestly says. These are minor and fixable in revision.\n\nCitation pattern looks fine. No obvious missing related work, and the claims match the cited methods' scope.\n\nWho is this for? Anyone working on generative world models, end-to-end driving, or video foundation models as perception interfaces. It deserves a serious referee: the architecture is meaningful, the ablations are controlled, and the planning benchmark results are competitive. My recommendation: engage with it, send it to peer review, and ask the authors to add independent evaluation of the structured streams (even a human-rated subset or a different teacher family), report multi-seed variance for the key ablations, and spell out the corrected EPDMS protocol. The core planning claim should survive those additions.","headline":"A solid, genuinely new interface for future-scene prediction in driving—one shared video expert generating four stream types and an action expert reading their latents—with planning results that survive scrutiny, though the structured-stream evaluation is partly circular against its own frozen teachers.","tokens_in":23233,"tokens_out":1431,"would_cite":true,"duration_ms":15154,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One video model generates four road futures and steers by them","keywords":["video generation","end-to-end driving","future scene understanding","joint video-action attention","flow matching","trajectory planning","world model","multi-stream prediction"],"falsifier":"Replace the future-stream latents fed to the action expert with shape-matched Gaussian noise at inference; if navhard EPDMS stays near 36.9, the claimed causal contribution of direct future-stream access is refuted.","tokens_in":22185,"feed_emoji":"🚗","tokens_out":7096,"duration_ms":57073,"temperature":0.7,"pith_summary":"SUV asks whether a single pretrained video generator can serve as the shared predictor of everything a self-driving policy needs to know about the next four seconds: appearance (RGB), road semantics, relative geometry, and which object is which as it moves. The paper claims it can: one video expert, with no task-specific prediction heads, denoises all four streams natively, and a separate action expert that attends to their latent tokens during the same denoising pass produces the ego trajectory. The payoff would be a scaling path in which future-scene understanding grows by adding video targets rather than new decoders, heads, and losses. On NAVSIM-v2 with one front camera and no candidate-trajectory selection, the system scores 91.0 EPDMS on navtest and 36.9 on navhard, and controlled ablations link both structured future supervision and direct stream access to higher planning scores.","feed_headline":"One video model generates four road futures and steers by them","feed_subtitle":"The action expert reads those four stream latents to reach 91.0 EPDMS with one front camera.","key_machinery":"The load-bearing machinery is the token-interaction mask inside joint video-action attention, together with the shared latent video space. The mask implements directed information flow: observation tokens read only the clean prefix, each future stream reads the prefix and itself, and the action expert reads the prefix, all four future streams, and its own trajectory tokens, and this mask is applied at every transformer block and denoising step. What makes it work is that all four streams share one latent video layout, one observation prefix, and one flow-matching schedule with a shifted timestep $\\phi_\\kappa(\\rho)=\\kappa\\rho/(1+(\\kappa-1)\\rho)$, $\\kappa=5$, so the pretrained video expert generates them natively; the action expert needs no decoded video, only intermediate latents.","core_discovery":"The central claim is that future scene understanding for end-to-end driving can be recast as video generation: the same video expert that predicts future RGB also predicts segmentation, relative depth, and instance tracks as three-channel video streams in a shared latent space, initialized from generative video pretraining and post-trained with flow matching. A Mixture-of-Transformers action expert shares the denoising loop; a token-interaction mask lets action queries attend to every future-stream token group while blocking cross-stream and action-to-future information flow. The paper's evidence is that this setting reaches 91.0 EPDMS on NAVSIM-v2 navtest and 36.9 on navhard with a single front camera and no trajectory-candidate selection, outperforming a broad set of recent methods, and that the multi-stream model matches or slightly exceeds RGB-only future quality. It further argues, via a 2x2 ablation, that adding segmentation/depth/track supervision raises EPDMS on navtest from 89.7 to 90.7 and on navhard from 30.5 to 32.8, and that giving the action expert direct access to future-stream latents raises it further to 91.0 and 36.9.","pith_inferences":["(Editorial inference) If the shared-expert design transfers beyond roads, a single generative backbone could become the common future-predictor for other embodied tasks, such as manipulation or navigation, where semantic, geometric, and instance futures are rendered as color-coded streams.","(Editorial inference) Because the structured metrics judge agreement with the frozen teachers that built the targets, an independent-ground-truth evaluation could reorder the comparison between native generation and generate-then-perceive.","(Editorial inference) The per-stream access ablations suggest a cheaper deployment recipe: generate only RGB and read its latents while dropping the other streams at inference, which the paper's own numbers predict would cost little on navtest but several points on navhard."],"forward_implications":["Adding a new kind of future knowledge, such as drivable-area gradient or object velocities, reduces to rendering it as another video stream and adding a prompt, with no new visual head or decoder required.","Planning can happen entirely in latent space: the trajectory is denoised alongside the future streams, and the VAE decoder is needed only for evaluation or visualization.","The accuracy-latency trade-off is controllable through solver steps: one Euler step gives 89.8 EPDMS on navtest and 33.0 on navhard at 177 ms on an RTX 4090, while ten steps reach 36.9 on navhard at 1356 ms.","Both structured future supervision and direct future-stream access improve planning, with the larger gains on the long-tail navhard split.","Removing access to any single structured stream lowers navhard EPDMS, with track removal hurting Stage 2 most; no single stream alone explains the gain."],"supporting_citations":[{"why":"Supplies the pretrained Wan2.2-5B video generator that is post-trained as the shared video expert for all four future streams.","marker":"Team Wan et al. 2025"},{"why":"Supplies the Mixture-of-Transformers architecture that separates the video expert from the action expert while sharing one attention space.","marker":"Liang et al. 2025"},{"why":"Defines the NAVSIM-v2 benchmark and the navhard two-stage pseudo-simulation protocol used for planning evaluation.","marker":"Cao et al. 2025"},{"why":"Defines the PDMS/EPDMS planning metrics and the non-reactive simulation evaluation used for all NAVSIM scores.","marker":"Dauner et al. 2024"},{"why":"SAM 3 is the frozen teacher that generates the semantic segmentation and instance-track targets and references.","marker":"Carion et al. 2026"},{"why":"DA3 is the frozen teacher that provides the relative-depth targets and references for the depth stream.","marker":"Lin et al. 2026"},{"why":"Defines the WOD-E2E long-tail benchmark and the RFS metric used to report the 7.94 score.","marker":"Xu et al. 2026b"}],"fun_headline_variants":["Four future streams from one video expert guide the drive","Video generation as shared predictor for driving futures","SUV casts future scene understanding as video prediction","One camera, one video model, four predicted futures, no candidate paths","Unified driving framework: future scenes as video streams"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Future-scene quality and the planning benefit attributed to structured futures are measured against the same frozen teacher models that generated the training targets, so systematic teacher errors would inflate both the reported stream quality and, potentially, the planning gain.","fun_headline_variants_meta":{"raw":{"variants":["Four future streams from one video expert guide the drive","Video generation as shared predictor for driving futures","SUV casts future scene understanding as video prediction","One camera, one video model, four predicted futures, no candidate paths","Unified driving framework: future scenes as video streams"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000897,"raw_usage":{"total_tokens":3887,"prompt_tokens":993,"completion_tokens":2894,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":2818}},"tokens_in":609,"tokens_out":2894,"duration_ms":19474,"temperature":1.0,"reasoning_tokens":2818,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:52:34.234628+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the future-stream latents fed to the action expert with shape-matched Gaussian noise at inference; if navhard EPDMS stays near 36.9, the claimed causal contribution of direct future-stream access is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Mixture-of-Transformers architecture that separates the video expert from the action expert while sharing one attention space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the NAVSIM-v2 benchmark and the navhard two-stage pseudo-simulation protocol used for planning evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the PDMS/EPDMS planning metrics and the non-reactive simulation evaluation used for all NAVSIM scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SAM 3 is the frozen teacher that generates the semantic segmentation and instance-track targets and references."},{"cited_title":"H.; Chen, D","cited_arxiv_id":null,"evidence_quote":"DA3 is the frozen teacher that provides the relative-depth targets and references for the depth stream."}],"review_version":2}