DriveVA: Video Action Models are Zero-Shot Drivers

Diankun Zhang; Francesco Nex; Guang Chen; Hangjun Ye; Hao Cheng; Hongwei Xie; Jianfeng Cui; Jiuming Liu; Mengmeng Liu; Michael Ying Yang

REVIEW 3 major objections 3 minor 11 cited by

DriveVA jointly decodes future video forecasts and driving actions in one shared latent process, delivering 90.9 PDM zero-shot planning that sharply cuts error and collisions on unseen datasets.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 10:56 UTC pith:NEOE6VUY

load-bearing objection Joint video-action DiT from pretrained generators claims large zero-shot planning gains, but abstract-only leaves the numbers and transfer uncheckable. the 3 major comments →

arxiv 2604.04198 v2 pith:NEOE6VUY submitted 2026-04-05 cs.CV cs.RO

DriveVA: Video Action Models are Zero-Shot Drivers

Mengmeng Liu , Diankun Zhang , Jiuming Liu , Jianfeng Cui , Hongwei Xie , Guang Chen , Hangjun Ye , Michael Ying Yang

show 2 more authors

Francesco Nex Hao Cheng

This is my paper

classification cs.CV cs.RO

keywords autonomous drivingworld modelsvideo generationzero-shot planningtrajectory predictionDiT decodercross-domain generalizationNAVSIM

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DriveVA argues that autonomous-driving planners generalize better when future videos and action trajectories are generated together inside a single latent process rather than planned separately. It reuses motion and physical-plausibility priors already present in large pretrained video generators, then attaches a DiT decoder that emits both the next video frames and the corresponding control trajectory. A video-continuation step keeps long rollouts coherent. The resulting model reaches 90.9 PDM on NAVSIM and, without any task-specific world-model retraining, reduces average L2 error and collision rate by roughly 79 percent and 83 percent on nuScenes and by about 52 percent on Bench2Drive. If the claim holds, a single video-pretrained generative model can serve as a drop-in zero-shot driver across sensor domains and simulation-to-real gaps that currently force expensive retraining.

Core claim

Jointly decoding future visual forecasts and action sequences from one shared latent generative process, guided by priors from large-scale video models and stabilized by a video-continuation strategy, yields a zero-shot autonomous driver that scores 90.9 PDM on NAVSIM and substantially lowers trajectory error and collisions on held-out real and simulated benchmarks.

What carries the argument

A DiT-based decoder that jointly predicts action trajectories and future video frames inside a shared latent generative process, reinforced by a video-continuation strategy that preserves long-horizon spatiotemporal consistency.

Load-bearing premise

The method rests on the premise that motion and physical-plausibility knowledge already stored in large video generators transfers well enough to produce safe, consistent driving trajectories on completely unseen datasets and sensor setups without any task-specific world-model training.

What would settle it

Evaluate the same pretrained DriveVA zero-shot on a new real-world driving corpus that uses different camera rigs and traffic conventions; if average L2 error or collision rate does not drop by a large margin relative to the prior world-model planner, the transfer claim is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Joint video-action DiT from pretrained generators claims large zero-shot planning gains, but abstract-only leaves the numbers and transfer uncheckable.

read the letter

The one thing to know is that DriveVA claims a shared DiT decoder that jointly generates future video and trajectories from large pretrained video models, plus a video-continuation rollout, and then reports very large zero-shot cuts: 90.9 PDM on NAVSIM and roughly 50–80 % drops in L2 and collisions on nuScenes and Bench2Drive versus prior world-model planners. That is the load-bearing claim.

What looks new is the tight coupling: instead of the usual loosely coupled “imagine then plan” loop, they put actions and pixels in one generative process so the trajectory has to stay consistent with the visual forecast. Inheriting motion and physics priors from video generators is a sensible way to attack cross-domain and sensor-shift generalization, and the video-continuation idea is a concrete engineering fix for long-horizon drift. If the numbers hold under matched protocols, this is a real step for world-model planners.

The soft spots are exactly what you would expect from an abstract-only read. We cannot see the training split, whether any evaluation domains leaked into the video backbone, the precise baselines, ablations, or error bars. The relative gains are large enough that they need full tables and failure cases before anyone should trust the zero-shot story. The transfer premise—that video priors alone give safe, consistent driving without task-specific world-model retraining—is plausible but still an assumption until the methods section is inspected. Nothing in the abstract looks circular or invented; it is just unverifiable right now.

This is for people already working on generative world models and closed-loop planning. They will get value from the architecture sketch and the reported numbers even if the final paper needs heavy revision. It is not for someone looking for a finished, reproducible system.

I would send it to peer review. The idea is clear enough and the claimed effect size is large enough that a serious referee should look at the full paper rather than desk-reject on the abstract alone.

Referee Report

3 major / 3 minor

Summary. DriveVA proposes an autonomous-driving world model that jointly decodes future visual forecasts and action (trajectory) sequences in a shared latent generative process via a DiT-based decoder, inheriting motion and physical-plausibility priors from large-scale pretrained video generation models, and adding a video-continuation strategy for long-horizon rollout consistency. The abstract claims this yields tighter video–trajectory alignment than loosely coupled imagine-then-plan pipelines, reports 90.9 PDM on NAVSIM, and asserts large zero-shot/cross-domain gains versus prior world-model planners: ~78.9% / 83.3% reductions in average L2 and collision rate on nuScenes, and ~52.5% / 52.4% on Bench2Drive (CARLA v2).

Significance. If the joint latent decoding, video-continuation design, and the reported zero-shot/cross-domain numbers hold under matched protocols, the work would be a meaningful step for generalization in world-model-based planning: it targets a known failure mode (video–trajectory inconsistency) and claims substantial safety-metric gains without task-specific world-model retraining. The use of pretrained video-generation priors for driving is a timely direction. Significance, however, is entirely conditional on verification of the quantitative claims, baselines, and domain-contamination controls, none of which can be assessed from the abstract alone.

major comments (3)

Only the abstract is available for review. The load-bearing quantitative claims (90.9 PDM on NAVSIM; 78.9%/83.3% L2/collision reductions on nuScenes; 52.5%/52.4% on Bench2Drive) cannot be checked against tables, error bars, matched evaluation protocols, or baseline reimplementations. These relative reductions are large for planning metrics and require full-table verification before the central claim can be accepted or rejected.
Abstract: the zero-shot and cross-domain generalization claims rest on transfer of video-generation priors without task-specific world-model retraining. The manuscript must document pretraining/fine-tuning data sources, sensor configurations, and explicit checks that evaluation domains (nuScenes, Bench2Drive/CARLA, NAVSIM) did not leak into the video backbone or joint decoder. Without that, the zero-shot attribution remains a domain-contamination risk rather than a demonstrated result.
Abstract: the causal claim that joint DiT decoding in a shared latent process (plus video continuation) is what produces the gains over loosely coupled imagine-then-plan pipelines is not yet supported by inspectable ablations. The full paper needs controlled ablations isolating (i) joint vs. separate decoding, (ii) video continuation on/off, and (iii) frozen vs. adapted video priors, with the same metrics reported in the abstract.

minor comments (3)

Abstract wording: 'impressive PDM-based planning performance' is promotional; prefer a neutral statement of the score and the comparison set.
Abstract: 'Video Action Models' in the title is not defined in the abstract; a one-sentence definition of what constitutes a VAM in this work would help readers place the contribution.
Abstract: relative reductions are given without absolute baseline and DriveVA numbers; reporting both (e.g., L2 and collision rate absolute values) would make the claims self-contained even at abstract length.

Circularity Check

0 steps flagged

Abstract-only text shows no equation-level or definitional circularity; reported metrics are presented as empirical outcomes, not constructions from fitted inputs.

full rationale

Only the abstract is available. It claims a joint DiT-based decoder for future video and action sequences, inheritance of motion/physical priors from pretrained video models, a video-continuation strategy, and empirical numbers (90.9 PDM on NAVSIM; large relative cuts in L2 and collision rate on nuScenes and Bench2Drive versus a prior world-model planner). None of these statements reduce by construction to their own inputs: there are no equations, no fitted parameters renamed as predictions, no uniqueness theorems, no self-citation chains, and no ansatz smuggled in via prior author work that can be quoted and shown to force the result. Standard train/eval domain-contamination risk for zero-shot claims is a correctness/generalization concern, not circularity under the stated criteria. With no quotable reduction of a load-bearing claim to its inputs, the honest finding is no significant circularity (score 0, empty steps).

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

Abstract-only audit: no free parameters or invented physical entities are numerically specified. The claim rests on domain assumptions that large video generators encode transferable driving dynamics and that joint latent decoding plus continuation yields planning-relevant consistency. Architecture name DriveVA is a method, not a new ontological entity.

axioms (4)

domain assumption Large-scale pretrained video generation models encode motion dynamics and physical plausibility priors useful for driving trajectory generation.
Stated as the reason DriveVA can inherit continuous spatiotemporal evolution and causal interaction patterns for zero-shot driving.
ad hoc to paper Joint decoding of future video and action sequences in one shared latent generative process improves video-trajectory consistency over loosely coupled imagine-then-plan pipelines.
Core design hypothesis of DriveVA; not independently established in the abstract.
ad hoc to paper A video continuation strategy strengthens long-duration rollout consistency for planning.
Introduced as a method component without external proof in the abstract.
domain assumption Standard autonomous-driving benchmarks (NAVSIM PDM, nuScenes L2/collision, Bench2Drive on CARLA v2) are valid proxies for cross-domain generalization.
All headline scores are defined on these suites; validity of the claim tracks validity of the metrics.

pith-pipeline@v1.1.0-grok45 · 6200 in / 2662 out tokens · 27009 ms · 2026-07-13T10:56:01.690791+00:00 · methodology

0 comments

read the original abstract

Generalization is a central challenge in autonomous driving, as real-world deployment requires robust performance under unseen scenarios, sensor domains, and environmental conditions. Recent world-model-based planning methods have shown strong capabilities in scene understanding and multi-modal future prediction, yet their generalization across datasets and sensor configurations remains limited. In addition, their loosely coupled planning paradigm often leads to poor video-trajectory consistency during visual imagination. To overcome these limitations, we propose DriveVA, a novel autonomous driving world model that jointly decodes future visual forecasts and action sequences in a shared latent generative process. DriveVA inherits rich priors on motion dynamics and physical plausibility from well-pretrained large-scale video generation models to capture continuous spatiotemporal evolution and causal interaction patterns. To this end, DriveVA employs a DiT-based decoder to jointly predict future action sequences (trajectories) and videos, enabling tighter alignment between planning and scene evolution. We also introduce a video continuation strategy to strengthen long-duration rollout consistency. DriveVA achieves an impressive PDM-based planning performance of 90.9 PDM score on the NAVSIM benchmark. Extensive experiments also demonstrate the zero-shot capability and cross-domain generalization of DriveVA, which reduces average L2 error and collision rate by 78.9% and 83.3% on nuScenes and 52.5% and 52.4% on the Bench2Drive built on CARLA v2 compared with the state-of-the-art world-model-based planner.

discussion (0)

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
cs.CV 2026-06 unverdicted novelty 7.0

MemoBench is a new diagnostic benchmark with 360 synthetic and real clips plus VQA evaluation that tests memory consistency in video models under the disappear-and-reappear paradigm in dynamically changing environments.
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
cs.CV 2026-06 conditional novelty 7.0

Current video world models do not reliably recover an object's updated state after it disappears and reappears under simultaneous camera and scene dynamics.
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
cs.CV 2026-06 unverdicted novelty 7.0

MemoBench is a new diagnostic benchmark with automated and VQA metrics that evaluates memory consistency in video models under disappear-and-reappear in dynamic environments.
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
cs.CV 2026-06 unverdicted novelty 7.0

MemoBench curates 360 ground-truth clips and an evaluation suite to diagnose memory consistency failures in video models when objects change state while out of view.
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
cs.CV 2026-06 unverdicted novelty 6.0

MemoBench curates 360 clips and an evaluation suite to test video models on recovering updated object states after disappear-and-reappear in changing environments.
UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation
cs.CV 2026-07 conditional novelty 5.0

A single mask-modulated DiT that co-trains future video and trajectories yields stronger autonomous-driving action generalization and 4.3× faster trajectory-only inference than dual-DiT designs.
ReWorld: Learning Better Representations for World Action Models
cs.CV 2026-06 unverdicted novelty 5.0

ReWorld applies future-predictive, cross-modal, and hard-negative supervision directly to intermediate representations in Video and Action DiTs for WAMs, reporting 23.9% FVD improvement and PDMS rise from 89.1 to 90.4...
Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Models
cs.CV 2026-06 unverdicted novelty 5.0

Applies differentiable search over prompt fusion schemes (concatenation, addition, affine, cross-attention) per ViT layer to improve visual prompt tuning, reporting gains across 34 datasets.
LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model
cs.CV 2026-05 unverdicted novelty 5.0

LVDrive improves closed-loop driving on Bench2Drive by adding latent future scene prediction to VLA models via unified embedding space processing and two-stage trajectory decoding.
Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning
cs.CV 2026-07 unverdicted novelty 4.0

TCA-Captioner introduces an Observer-Checker-Corrector refinement loop and TCA-Bench to address modality detachment and temporal incoherence in audiovisual video captioning.
World Action Models: A Survey
cs.RO 2026-06 unverdicted novelty 3.0

A survey that clarifies boundaries and organizes World Action Models by generation requirements and predictive substrates, identifying a trend toward generating less of the future.