{"id":"d00e9bd9-f45c-451f-af08-a2228abea3dd","arxiv_id":"2607.29031","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A driving planner that predicts a future-ego-trajectory latent and retrieves executable trajectories from a fixed memory reaches 91.3 PDMS on NAVSIM v1 without reconstructing the scene.","lead":"Auto-JEPA predicts a short high-level \"intent\" of where the ego car is going, then looks up similar real recorded trajectories instead of generating them. It reports top results on NAVSIM driving benchmarks with only a front camera and no dense future scene reconstruction.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Frozen trajectory encoder's flat-cosine space is never validated as a maneuver-compatibility metric; high PDMS could come from scorer plus memory density rather than intent retrieval.","rationale":"The reader's weakest assumption is the same one I find most load-bearing: the paper assumes cosine similarity in the frozen trajectory encoder's latent space is a reliable retrieval measure without directly validating it. The paper is transparent, has strong ablations, and the component ablation shows that retrieval conditioning matters, but it does not establish that the latent-space metric ranks trajectories by future-maneuver compatibility. The proposed retrieval-recall test would settle this directly. Because this concern is already reflected in the reader's CONDITIONAL verdict, I do not change the verdict.","tokens_in":15960,"tokens_out":13110,"duration_ms":141475,"concrete_test":"On the NAVSIM validation split, encode both the GT future ego trajectories and the model's predicted intents with the frozen E_traj; run Top-K retrieval (Eqs. 15–17) against the 110k memory. Report (i) top-1/top-5 ADE/FDE of retrieved memory trajectories relative to the GT trajectory and (ii) the fraction of scenes where at least one of the top-300 candidates has ADE <= 1.0 m or passes the NAVSIM utility threshold. Compare against retrieval using Euclidean waypoint distance. If predicted-intent retrieval recall is close to random or much worse than waypoint retrieval, the scoring module is carrying the planner and the central intent-retrieval claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing mechanism is that flat-cosine similarity in the frozen trajectory encoder's latent space is a faithful proxy for whether a memory trajectory matches the predicted future maneuver. This enters at Eqs. 15–17: query and memory latents are flattened, L2-normalized, and ranked by r_n = q^T m_n. But E_traj is trained only to reconstruct waypoints (Eq. 3); nothing in that objective encourages cosine closeness to correspond to geometric or maneuver compatibility. The paper never reports retrieval recall or discrimination on validation scenes—e.g., what fraction of top-K retrieved trajectories are within a small ADE of the GT future or are rated as high-utility by the evaluator. Consequently, the high final PDMS could in principle be produced mostly by the CLOVER-initialized scene scorer selecting a good trajectory from 300 candidates that are only weakly conditioned by the predicted intent, together with the density of a 110k ground-truth memory. The fixed-medoid ablation (52.6 PDMS) shows that some retrieval conditioning is necessary, but it does not validate the cosine metric itself; a poorly calibrated metric can still condition strongly and yet rank compatible trajectories poorly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Auto-JEPA, a camera-only end-to-end planner that predicts a continuous latent 'intent' aligned with a frozen trajectory encoder's representation of the ground-truth future ego trajectory. At inference, the predicted intent retrieves 300 trajectories from a fixed memory of 110k logged trajectories; a CLOVER-initialized scene-conditioned scorer ranks the candidates and a learned drivable-area gate filters infeasible ones. The authors report 91.3 PDMS on NAVSIM v1 and 89.1 EPDMS on NAVSIM v2, ablate the intent/scorer/gate components, and present semantic occlusion experiments showing that masking dynamic-agent regions changes the predicted intent 2.97x more than equal-area random masks. The central conceptual claim is that planning-oriented latent prediction need not reconstruct the complete future scene, only features relevant to future ego action.","tokens_in":16266,"tokens_out":6713,"duration_ms":77911,"significance":"If the results hold, the paper is a useful demonstration that a compact, action-oriented latent world model plus non-parametric retrieval can be competitive with dense world-model reconstructions. Strengths include a clearly specified staged training procedure, explicit objectives (Eqs. 8-12), reproducible code/model release, controlled occlusion diagnostics, and unusually candid implementation details including the single-run nature of evaluation and the exact scorer label-generation protocol. The main caveats are that the retrieval metric itself is not independently validated, that reported margins over prior work may be within evaluation noise, and that the selectivity claim is supported only partly by aggregate controls. These are addressable with additional experiments rather than fundamental architectural flaws.","major_comments":[{"comment":"The main benchmark numbers are stated to come from one deterministic full-navtest evaluation, not averages over independently retrained models, and the semantic-occlusion ratio is from one checkpoint. PDMS differences against the closest published systems in Table 1 are on the order of 1 point (91.3 vs 90.3/90.2); without noise estimates it is unclear whether the reported ordering is significant. Please provide repeated evaluations (at least three seeds) or, if official evaluation cost makes this infeasible, report bootstrap/confidence intervals over navtest scenarios and state the selection procedure used to choose the reported checkpoint.","section":"Supplementary, Randomness, Runs, and Computing Infrastructure"},{"comment":"The dataset-level result (2.97x) only compares agent-region masking with random masking; it shows sensitivity to dynamic agents, not selectivity among dynamic agents according to their influence on the ego plan. The claim that the model 'focuses more strongly on vehicles that may affect future driving decisions' is supported only by selected examples (Figure 4). Please add an aggregate control in which occlusions of influential vs non-influential agents are matched for area/class and measured across the full validation split (e.g., defined by distance, TTC, or the evaluator's interaction signal). Without this, the central qualitative claim about selective planning-relevant focus is softer than the abstract suggests.","section":"§Analysis; Supplementary Semantic Occlusion Protocol"}],"minor_comments":[{"comment":"The PDMS formula is typeset without parentheses: it should be (NC · DAC · 5 · (EP + TTC) + 2C)/12. Please fix for clarity.","section":"Eq. (22)"},{"comment":"The headline v2 number uses the updated official evaluator; Table 2 also reports 85.6 EPDMS under the original implementation. Make this explicit in the abstract and introduction to avoid the impression that the same protocol yields both numbers.","section":"Abstract and Table 2"},{"comment":"The 'Intent ✗' row uses a fixed codebook medoid as the retrieval query. This is a good control, but the caption should state it explicitly, since the same symbol in the main text could be read as removing intent prediction entirely.","section":"Table 3 / component ablation"},{"comment":"Several references are dated 2026 and are unpublished preprints or accepted papers; please check that all citations have verifiable identifiers and that the related-work claims about each method are accurate.","section":"References / Related Work"}],"recommendation":"major_revision","confidential_remarks":"Editorial note: the stress-test concern about the unvalidated latent retrieval metric is genuine and should be addressed before acceptance. The single-run evaluation also needs to be addressed. The authors are transparent about the CLOVER initialization and the evaluation protocol, which is a point in their favor. If the requested retrieval diagnostics and uncertainty estimates are added without changing the architecture, the paper could be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Auto-JEPA is a genuine architectural contribution — it predicts a future ego-trajectory latent (intent) with JEPA-style objectives and uses that latent directly as a retrieval key over a fixed ground-truth trajectory memory, with a scene-conditioned scorer and gating. That exact combination is new, and the paper does the empirical legwork to back it up: clean component ablations, controlled semantic-occlusion experiments, and a transparent write-up of training details and limitations.\n\nWhat impressed me: the frozen V-JEPA visual encoder, no perception annotations, no learned trajectory generator, and still 91.3 PDMS on NAVSIM v1 and 89.1 EPDMS on v2. The ablations show intent retrieval is load-bearing: replacing the predicted intent with a fixed codebook medoid craters PDMS from 91.3 to 52.6. The occlusion analysis is a nice controlled demonstration that the model responds more strongly to influential agents than to random or non-influential ones.\n\nThe soft spots are real but addressable. First, the central retrieval mechanism relies on flat cosine similarity in the trajectory encoder's latent space, but that space is trained only to reconstruct waypoints. Nothing in Eq. 3 encourages cosine closeness to correspond to maneuver compatibility. The paper never reports retrieval recall — for instance, what fraction of top-K candidates land within a small ADE of the ground-truth future or are rated as feasible by the evaluator. The medoid ablation confirms that the query matters, but it doesn't show the ranking is picking the right kind of trajectory. This is the strongest thing to push on in revision. Second, all benchmark numbers come from one deterministic full-navtest evaluation; no error bars or multiple runs. The paper is honest about this, but it does limit confidence in the single-point claims. Third, the scorer is initialized from CLOVER and re-optimized; the paper discloses this, and the boost from the scorer is 3.7 PDMS, so it's not hiding the main effect, but it's worth noting that the final number carries some CLOVER DNA.\n\nNone of these are fatal. The architecture, ablations, and occlusion results are strong enough that the paper deserves serious referee time. The revision should add retrieval-space validation and multi-run or bootstrap error estimates. I'd bring it to reading group and would cite it.","headline":"Auto-JEPA is a legitimate new combination with strong empirical results, but the paper should directly validate its latent retrieval space before we fully trust the mechanism.","tokens_in":16699,"tokens_out":2748,"would_cite":true,"duration_ms":29170,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that autonomous driving can be planned by predicting a latent embedding of the future ego trajectory — a 'driving intent' — and retrieving executable trajectories from a fixed memory, without dense future-world prediction.","keywords":["autonomous driving","world model","latent world model","joint-embedding prediction","trajectory retrieval","driving intent","NAVSIM","end-to-end planning"],"falsifier":"Retrain the trajectory encoder with a different random seed and rebuild the memory from scratch; if the final NAVSIM PDMS changes by more than about 1 point, retrieval depends on accidental latent geometry rather than on stable maneuver compatibility.","tokens_in":15890,"feed_emoji":"🚗","tokens_out":6007,"duration_ms":61055,"temperature":0.7,"pith_summary":"Auto-JEPA is a planner that learns to predict a latent embedding of the future ego trajectory, called the driving intent, directly from front-camera frames, egomotion history, and a navigation command. Instead of reconstructing future video, occupancy, agent motion, or other dense scene states, the predicted intent is used as a retrieval key into a fixed memory of recorded trajectories; a scene-conditioned scorer and a drivable-area feasibility gate then select the final plan. The paper reports 91.3 PDMS on NAVSIM v1 and 89.1 EPDMS on NAVSIM v2 using only one front camera, no perception annotations, and no learned trajectory generator. Semantic occlusion experiments show that masking dynamic-agent regions changes the predicted intent on average 2.97 times more than equal-area random masking, and that occluding only vehicles that affect the future plan substantially shifts the selected trajectory. The central message is that action-oriented latent prediction provides a planning-focused world model that avoids the cost and error-propagation of full future-scene modeling.","feed_headline":"Plan driving by predicting intent, not the whole future scene","feed_subtitle":"Reports 91.3 PDMS on NAVSIM v1 with only a front camera, no perception labels, no learned trajectory generator.","key_machinery":"The central mechanism is the continuous driving-intent latent: an 8-token embedding of the future ego trajectory (waypoints over a four-second horizon) produced by a frozen trajectory encoder. The same encoder maps every candidate trajectory in a fixed ground-truth memory into the same latent space, so the predicted intent can serve directly as a retrieval key via flat cosine similarity. The retrieval is followed by a scene-conditioned trajectory scorer and a drivable-area feasibility gate, separating the questions of what kind of future motion is appropriate, which candidate is best for this scene, and which candidate is drivable.","core_discovery":"Auto-JEPA establishes that a planning-oriented world model can be built by predicting the latent representation of the future ego trajectory through joint-embedding predictive learning, rather than predicting the future state of the scene. The predicted embedding is aligned with the latent of the ground-truth future trajectory from a frozen trajectory encoder, and at inference it retrieves near neighbors from a fixed memory of 110,335 recorded trajectories. A scene-conditioned scorer and a drivable-area feasibility gate then select the final trajectory. The paper reports 91.3 PDMS on NAVSIM v1 and 89.1 EPDMS on NAVSIM v2, and that systematically occluding dynamic agents produces a 2.97x larg","pith_inferences":["A natural extension the authors leave implicit is to replace the fixed memory with an intent-conditioned trajectory generator at inference, which could remove the coverage limit while preserving the selective-attention property.","The occlusion protocol could become a planning-relevance attribution benchmark: measuring how much a planner's output changes when each traffic participant is masked yields a per-scene map of which objects drive decisions.","Because the intent is predicted from appearance and egomotion alone, the method may transfer to domains with scarce trajectory labels, as long as a trajectory encoder can be pretrained on any trajectory dataset; a testable prediction is that fine-tuning the predictor on a small new-route dataset suffices to shift retrieval behavior.","The selective-attention claim could be probed directly by inspecting the predictor's attention weights: if visual tokens corresponding to interacting vehicles carry the most weight in the intent output, that would corroborate the interpretation."],"forward_implications":["If the approach holds, latent-intent prediction can replace dense future-world reconstruction in end-to-end driving, reducing compute and avoiding error propagation from perception and forecasting into planning.","A frozen visual encoder and frozen trajectory encoder mean only the predictor, scorer, and gate need training, so adaptation to new sensors or routes could be done with modest fine-tuning.","The fixed trajectory memory bounds the reachable motion space; the paper observes saturation at K=300, implying that memory coverage or an intent-conditioned generator would be needed to extend maneuverability.","The 2.97x occlusion result suggests the model learns planning-relevant attention without any object-level supervision, which could support interpretability and safety analysis.","Separating retrieval from scoring allows the scorer or gate to be replaced or refined independently without retraining the intent predictor."],"fun_headline_variants":["Predict driving intent, not the whole future scene","Autonomous driving world model predicts intent, not pixels","No perception labels, no generated trajectories: just predict intent","Latent intent prediction beats dense future modeling in driving","Auto-JEPA: planning by predicting future ego intent"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central assumption is that the representation learned for a trajectory by the frozen encoder makes nearby (by cosine similarity) trajectories genuinely similar in driving relevance — if that is not true, retrieval will pick unsuitable plans.","fun_headline_variants_meta":{"raw":{"variants":["Predict driving intent, not the whole future scene","Autonomous driving world model predicts intent, not pixels","No perception labels, no generated trajectories: just predict intent","Latent intent prediction beats dense future modeling in driving","Auto-JEPA: planning by predicting future ego intent"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1234,"prompt_tokens":802,"completion_tokens":432,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":356}},"tokens_in":546,"tokens_out":432,"duration_ms":4503,"temperature":1.0,"reasoning_tokens":356,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T14:57:21.270163+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the trajectory encoder with a different random seed and rebuild the memory from scratch; if the final NAVSIM PDMS changes by more than about 1 point, retrieval depends on accidental latent geometry rather than on stable maneuver compatibility.","supporting_citations":[],"review_version":1}