Zero-Label Driving Scenario Complexity Detection via Joint Embedding Predictive Architecture

Santosh Jaiswal

arxiv: 2606.28383 · v1 · pith:JUVGDZGAnew · submitted 2026-06-21 · 💻 cs.CV · cs.LG

Zero-Label Driving Scenario Complexity Detection via Joint Embedding Predictive Architecture

Santosh Jaiswal This is my paper

Pith reviewed 2026-06-30 10:47 UTC · model grok-4.3

classification 💻 cs.CV cs.LG

keywords self-supervised learningdriving scenario complexitytemporal prediction errorzero-label detectionjoint embedding predictive architectureanomaly detectionautonomous drivingnuplan dataset

0 comments

The pith

A self-supervised world model can score driving scenario complexity using only its temporal prediction errors.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether a model can identify complex driving scenarios in large unlabeled datasets without any human annotations or rule sets. It trains a minimal Joint Embedding Predictive Architecture on structured agent state data and treats the model's temporal prediction error as a zero-shot complexity score. Scenarios involving unprotected turns, crosswalks, and pedestrians receive significantly higher scores than lane-following or stationary traffic. The signal is isolated through four ablation experiments and shown to support anomaly detection at 0.512 average precision against a 0.436 baseline. The work establishes that prediction error in a latent world model serves as a practical proxy for scenario complexity.

Core claim

The central claim is that temporal prediction error in a self-supervised Joint Embedding Predictive Architecture trained on nuPlan mini agent data functions as a zero-label proxy for driving scenario complexity, assigning higher values to unprotected turns and pedestrian interactions while assigning lower values to simple lane-following cases, with the pattern confirmed by ablations and downstream anomaly detection.

What carries the argument

The temporal prediction error of the Joint Embedding Predictive Architecture, which measures mismatch between predicted and observed latent states over time and is used directly as the complexity score.

If this is right

Unprotected turns and pedestrian-proximity cases produce measurably higher prediction errors than lane-following or stationary traffic.
The same error signal yields 0.512 average precision in anomaly detection, above the 0.436 random baseline.
Four ablation experiments confirm that the predictive component, not other architectural elements, supplies the complexity signal.
No ground-truth labels are required at any training or scoring stage.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The proxy could be applied to rank or filter large unlabeled driving datasets for targeted simulation or data collection.
Similar prediction-error scoring might transfer to other sequential domains such as robotics trajectories or video event detection.
If the error signal proves stable across different model sizes, it could reduce reliance on engineered rules for scenario curation.
The approach implicitly assumes the learned latent space encodes aspects of interaction difficulty that align with safety relevance.

Load-bearing premise

That the prediction error specifically tracks human-interpretable scenario complexity rather than dataset biases or model-specific artifacts.

What would settle it

A direct comparison on a held-out set of driving scenarios where human raters label complexity and the model's error scores show no correlation.

Figures

Figures reproduced from arXiv: 2606.28383 by Santosh Jaiswal.

**Figure 2.** Figure 2: Mean surprise score per nuPlan scenario tag. Left panel: top 20 most surprising tags (complex [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Figure 3: Ablation: per-tag surprise scores for the trained JEPA model (left) vs. no-EMA model (right); top [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: Precision@K curves for all four methods. The trained JEPA model (blue solid line) is the only one [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

**Figure 5.** Figure 5: Top 9 most surprising scenarios ranked by JEPA surprise score. Each subplot shows agent [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

**Figure 6.** Figure 6: Bottom 9 most routine scenarios. Sparse agents moving in parallel lanes with little interaction; the [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

read the original abstract

Identifying complex and safety-critical driving scenarios in large unlabelled datasets is an important but expensive problem. Existing approaches rely on human annotators, supervised classifiers, or carefully engineered rule sets, all of which require substantial prior knowledge about what constitutes a difficult scenario. We ask whether a model can discover scenario complexity on its own, with no labels at any stage. We train a minimal Joint Embedding Predictive Architecture (JEPA) on structured agent state data from the nuPlan mini dataset and use the temporal prediction error as a zero-shot complexity score. Without access to any ground-truth labels during training or evaluation setup, the model assigns significantly higher scores to scenarios involving unprotected turns, crosswalk interactions, and pedestrian proximity, and significantly lower scores to lane-following and stationary-traffic scenarios. We validate this finding through four ablation experiments that isolate the source of the signal, and through a downstream anomaly detection evaluation that achieves Average Precision of 0.512 against a 0.436 chance baseline. The results show that temporal prediction error in a self-supervised latent world model is a practical proxy for driving scenario complexity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

JEPA error gives a label-free signal on nuPlan that lines up with complex scenarios, but it may just track agent count and motion stats.

read the letter

The paper's core result is that prediction error from a minimal JEPA trained on structured agent states from nuPlan mini assigns higher scores to unprotected turns, crosswalks, and pedestrian cases, and lower scores to simple lane following, all with zero labels at any point. It also reports a downstream anomaly detection AP of 0.512 against a 0.436 chance baseline plus four ablations.

What is new is the direct use of JEPA temporal error as the complexity proxy itself rather than as an auxiliary loss. The setup is clean on that point: no ground-truth labels enter training or the scoring step, and the circularity burden stays low.

The numbers are modest and the support is thin. The AP lift is small, no statistical tests are described in the abstract, and the ablations are only summarized at high level. More importantly, the stress-test concern holds up on the given information: scenarios humans call complex differ systematically in agent density, interaction count, and velocity variance, so any latent predictor will naturally incur higher error on them for statistical reasons alone. Without controls that regress those factors out, it is unclear whether the signal reflects interpretable complexity or just those surface statistics.

This is for researchers in self-supervised world models or AV data curation who want cheap ways to surface hard cases from unlabeled logs. A reader already working on JEPA-style models could extract the basic recipe and test the confounder issue themselves.

It deserves a serious referee because the framing is testable and the data source is public, even though the current evidence is preliminary and the central claim needs tighter validation against simple motion features.

Referee Report

3 major / 1 minor

Summary. The paper claims that temporal prediction error from a self-supervised Joint Embedding Predictive Architecture (JEPA) trained on structured agent state data from the nuPlan mini dataset can serve as a zero-label proxy for driving scenario complexity. It reports that this score assigns significantly higher values to scenarios involving unprotected turns, crosswalk interactions, and pedestrian proximity (and lower to lane-following and stationary traffic), supported by four ablation experiments and a downstream anomaly detection task achieving AP 0.512 versus a 0.436 chance baseline.

Significance. If the central claim holds after addressing potential statistical confounders, the work demonstrates a practical label-free method for surfacing safety-critical scenarios in large driving datasets. This could reduce annotation costs and leverage self-supervised world models for data curation in autonomous driving, with the zero-shot nature and use of prediction error as a complexity signal being notable strengths.

major comments (3)

[Abstract and ablation experiments section] Abstract and ablation experiments section: the four ablations are described only at high level and do not include controls (e.g., regression of prediction error on agent count, interaction density, or velocity variance) to test whether the signal specifically indexes human-interpretable complexity rather than dataset motion statistics; this is load-bearing for the claim that the error is a 'practical proxy' for complexity.
[Downstream evaluation paragraph] Downstream evaluation paragraph: the reported AP of 0.512 versus 0.436 baseline is only marginally above chance; the manuscript must specify the exact positive-class definition (without ground-truth labels), the baseline construction, and any statistical significance test to establish that the improvement supports the complexity-proxy interpretation.
[Methods section] Methods section: insufficient detail is given on the JEPA architecture (e.g., predictor and target encoder sizes, loss formulation, exact temporal prediction error computation), training hyperparameters, and nuPlan mini subset used, which prevents verification that the reported scores arise from the self-supervised objective rather than implementation artifacts.

minor comments (1)

[Abstract] The abstract should include the size of the nuPlan mini subset and a one-sentence description of the JEPA variant to improve reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback, which highlights opportunities to strengthen the clarity and rigor of our claims. We address each major comment point-by-point below. Where the manuscript requires expansion or additional analysis, we will revise accordingly.

read point-by-point responses

Referee: [Abstract and ablation experiments section] Abstract and ablation experiments section: the four ablations are described only at high level and do not include controls (e.g., regression of prediction error on agent count, interaction density, or velocity variance) to test whether the signal specifically indexes human-interpretable complexity rather than dataset motion statistics; this is load-bearing for the claim that the error is a 'practical proxy' for complexity.

Authors: We agree that the current high-level description of the ablations leaves room for stronger isolation of the complexity signal. In the revised manuscript we will expand the ablation section with detailed descriptions of each experiment and add explicit controls, including regressions of prediction error on agent count, interaction density, and velocity variance, to demonstrate that the signal is not reducible to these basic motion statistics. revision: yes
Referee: [Downstream evaluation paragraph] Downstream evaluation paragraph: the reported AP of 0.512 versus 0.436 baseline is only marginally above chance; the manuscript must specify the exact positive-class definition (without ground-truth labels), the baseline construction, and any statistical significance test to establish that the improvement supports the complexity-proxy interpretation.

Authors: We will revise the downstream evaluation paragraph to provide the exact positive-class definition used (constructed without ground-truth complexity labels, consistent with the zero-label protocol), a precise description of the baseline, and the results of a statistical significance test (e.g., bootstrap) on the AP difference. These additions will allow readers to assess whether the observed improvement supports the complexity-proxy claim. revision: yes
Referee: [Methods section] Methods section: insufficient detail is given on the JEPA architecture (e.g., predictor and target encoder sizes, loss formulation, exact temporal prediction error computation), training hyperparameters, and nuPlan mini subset used, which prevents verification that the reported scores arise from the self-supervised objective rather than implementation artifacts.

Authors: We acknowledge that the Methods section lacks sufficient implementation detail. The revised manuscript will include complete specifications of the JEPA architecture (encoder sizes, loss formulation, temporal prediction error computation), all training hyperparameters, and the exact nuPlan mini subset, enabling independent verification that the results derive from the self-supervised objective. revision: yes

Circularity Check

0 steps flagged

No circularity: complexity score defined directly as self-supervised prediction error

full rationale

The paper trains a JEPA model self-supervised on unlabeled nuPlan agent states and defines the zero-shot complexity score explicitly as the resulting temporal prediction error. This quantity is not fitted to any complexity labels, not derived from human annotations, and not reduced by construction to the downstream scenarios it is later observed to rank. The anomaly detection AP (0.512 vs 0.436 chance) and ablations serve as external validation of the proxy but do not enter the definition of the score itself. No self-citations, uniqueness theorems, or ansatzes are invoked to force the result; the derivation chain remains independent of its evaluation targets.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

Only the abstract is available, so the ledger is necessarily incomplete. The central assumption is the link between prediction error and complexity.

free parameters (1)

JEPA architecture and training hyperparameters
Specific model size, prediction horizon, and optimization choices are not stated in the abstract.

axioms (1)

domain assumption Temporal prediction error in the learned latent space measures scenario complexity
This assumption enables the entire zero-label claim and is not derived from external benchmarks.

pith-pipeline@v0.9.1-grok · 5715 in / 1137 out tokens · 49020 ms · 2026-06-30T10:47:20.229029+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

8 extracted references · 4 canonical work pages · 2 internal anchors

[1]

OpenReview , year =

A Path Towards Autonomous Machine Intelligence , author =. OpenReview , year =
[2]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =
[3]

Revisiting Feature Prediction for Learning Visual Representations from Video

Revisiting Feature Prediction for Learning Visual Representations from Video , author =. arXiv preprint arXiv:2404.08471 , year =

work page internal anchor Pith review Pith/arXiv arXiv
[4]

NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles

nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles , author =. arXiv preprint arXiv:2106.11810 , year =

work page internal anchor Pith review Pith/arXiv arXiv
[5]

arXiv preprint arXiv:2502.05677 , year =

Surprise Potential as a Measure of Interactivity in Driving Scenarios , author =. arXiv preprint arXiv:2502.05677 , year =

work page arXiv
[6]

arXiv preprint arXiv:2601.22032 , year =

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving , author =. arXiv preprint arXiv:2601.22032 , year =

work page arXiv
[7]

Advances in Neural Information Processing Systems , volume =

Attention Is All You Need , author =. Advances in Neural Information Processing Systems , volume =
[8]

Advances in Neural Information Processing Systems , volume =

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning , author =. Advances in Neural Information Processing Systems , volume =

[1] [1]

OpenReview , year =

A Path Towards Autonomous Machine Intelligence , author =. OpenReview , year =

[2] [2]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

[3] [3]

Revisiting Feature Prediction for Learning Visual Representations from Video

Revisiting Feature Prediction for Learning Visual Representations from Video , author =. arXiv preprint arXiv:2404.08471 , year =

work page internal anchor Pith review Pith/arXiv arXiv

[4] [4]

NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles

nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles , author =. arXiv preprint arXiv:2106.11810 , year =

work page internal anchor Pith review Pith/arXiv arXiv

[5] [5]

arXiv preprint arXiv:2502.05677 , year =

Surprise Potential as a Measure of Interactivity in Driving Scenarios , author =. arXiv preprint arXiv:2502.05677 , year =

work page arXiv

[6] [6]

arXiv preprint arXiv:2601.22032 , year =

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving , author =. arXiv preprint arXiv:2601.22032 , year =

work page arXiv

[7] [7]

Advances in Neural Information Processing Systems , volume =

Attention Is All You Need , author =. Advances in Neural Information Processing Systems , volume =

[8] [8]

Advances in Neural Information Processing Systems , volume =

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning , author =. Advances in Neural Information Processing Systems , volume =