Pith. sign in

REVIEW 2 major objections 4 minor 59 references

Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A rollout-level distance between clean and perturbed histories provably bounds how much visual noise can change prediction error and planning cost — and the screen built on it flags JEPA checkpoints that stay reliable.

desk verdict A correctly proven and carefully evaluated diagnostic for perturbation robustness in JEPA world models; the central bound is real but its advertised meaning needs an explicit task-preservation assumption. read the letter →

arxiv 2608.12939 v1 pith:LEPCACI6 submitted 2026-08-13 cs.LG

Guo An , Zijing Wu , Honghua Dong , Yuhao Yan , Zixuan Gui , Haochong Chen , Shanzhao Ruan , Xiang Wang
show 2 more authors
Yurong Ling Qi Tian
This is my paper · ORCID
classification cs.LG
keywords Action-ConditionedPredictiveConsistencyJEPAworldmodelslatentpredictionvisualperturbationsplanningcostboundsInvarianceRadiusSeparationRatestate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Joint-embedding predictive world models (JEPAs) forecast in latent space rather than pixels, which frees them from reproducing nuisance appearance but gives no guarantee against visual perturbations that still shift the encoded representation. This paper tries to establish that a single, model-only measurement — rolling a clean history and its visually perturbed copy forward under the same recorded action sequence and measuring how far the two predicted trajectories drift, called Action-Conditioned Predictive Consistency (ACPC) — bounds how much that perturbation can change prediction error and planning cost, samplewise and with no distributional or smoothness assumptions. From pairwise ACPC the paper builds a checkpoint screen: the Invariance Radius (IR) summarizes how much perturbation spreads same-state rollouts, and the Separation Rate (SR) checks that different states remain distinguishable after rollout, so a representation collapse that would fool a low-IR test still fails on SR. Four visual control tasks, three perturbation types, and two model families support the claim that multi-step ACPC carries information about prediction-error drift and CEM selection regret that encoder distance, one-step ACPC, and same-horizon action-destroyed controls lack. A reader should care because this supplies an assumption-light diagnostic for the visual robustness of latent world models that needs no retraining and relies only on loggable data.

What carries the argument

The central object is the action-conditioned predictive consistency (ACPC) quantity: a weighted rollout vector $\bar G_a(z)$ stacks the first $H$ predicted representations of a history under a fixed action sequence, and ACPC is the Euclidean distance between the clean and perturbed versions of this vector. It carries the argument in three ways. The prediction-error bound is the reverse triangle inequality applied to the two errors measured against the same observed future. The planning-cost bound factors the difference of two squared distances to the goal and applies Cauchy–Schwarz, giving the per-candidate cost-change bound $b_j$; ACPC enters by bounding the endpoint displacement $r_j$. At checkpoint level, normalized ACPC values are aggregated into the Invariance Radius, the q90 of clean–perturbed rollout distances scaled by each history's typical one-step motion, and the Separation Rate, the fraction of different-label state pairs whose rollout distance exceeds that radius plus a fixed margin, with a relative-IR comparison against an unaugmented reference checkpoint.

What would settle it

Keep the underlying state fixed and translate the goal image — which the paper deliberately leaves clean — in a TwoRoom or PushT episode, recompute ACPC between the clean and perturbed histories, and record planning success: if success collapses while ACPC stays small, the task-preserving assumption, not the bound, is carrying the diagnostic. A direct numerical check also exists: any logged pair violating $|e_{\tilde h} - e_h| \le \mathrm{ACPC}_H(h, \tilde h, a)$ would refute the proven claim, and the paper's own sweep found zero violations.

Watch

Extended reading notes

Core claim

The central claim is that rolling paired observations forward under identical actions exposes downstream perturbation effects that encoder-only and one-step comparisons miss, and that the measured divergence is quantitatively tied to downstream failure. Concretely, for every paired sample, the difference between the two multi-step prediction errors evaluated against the same observed future is bounded by the ACPC value, $|e_{\tilde h} - e_h| \le \mathrm{ACPC}_H(h, \tilde h, a)$; when a planner scores candidate action sequences by squared distance to a goal embedding, each candidate's cost can shift by at most $b_j = r_j(\|x_j - g\|_2 + \|\tilde x_j - g\|_2)$, so if every clean winner-to-competitor cost gap exceeds the combined bounds, the perturbation cannot reverse the selected plan or elite set. Because these bounds hold per pair and require no distributional or smoothness assumptions, a small ACPC together with large clean cost gaps is a certificate that planning decisions are stable under that perturbation, while a large ACPC diagnoses fragility before the environment is consulted. Summarized across histories, the Invariance Radius and Separation Rate jointly characterize a checkpoint, and the paper reports that checkpoints passing the low-IR/high-SR screen recover planning success under Gaussian noise on four tasks, that thresholds chosen on some tasks transfer to held-out tasks, and that the same qualitative pattern appears on a second architecture and under blur and resize.

Load-bearing premise

The load-bearing premise is that the evaluated perturbation is task-preserving — it changes how the history looks but not the true underlying state, so the same recorded action sequence and the same observed future are the correct comparison target for both the clean and the perturbed rollout; if a perturbation rewrote task-relevant state, a small ACPC would say nothing about whether the planner still works.

Editorial extensions

If this is right

  • A trained JEPA world-model checkpoint can be screened for robustness to a chosen visual perturbation using only logged histories, the frozen model, and an unaugmented reference — no retraining, no environment interaction, and no modification of the planner.
  • On a fixed candidate pool, ACPC plus clean cost gaps yields a per-pair certificate: whenever every clean winner-to-competitor gap exceeds the summed cost-change bounds, the perturbation provably cannot change the planner's chosen action or its elite set, and if the condition holds across CEM iterations the two runs return identical actions.
  • Multi-step ACPC under the recorded actions predicts perturbation-induced prediction-error drift better than encoder distance, one-step ACPC, or same-horizon controls with destroyed action information across all 12 task-run cells, and planner-horizon ACPC lowers cross-task prediction error for CEM selection regret in all 12 test cases.
  • Across the augmentation sweep, lower relative IR and higher SR align with recovery of planning success under Gaussian noise; thresholds selected on held-out tasks identify recovery with balanced accuracy 0.900, and the same low-IR/high-SR pattern appears on a second architecture and under blur and resize, with the sign of the IR-SR score change agreeing with the success criterion in 22 of 24 check
  • SR closes the collapse loophole: a representation whose latent distances shrink to near zero has the lowest possible IR but its SR falls to 0.066 and clean success to 33%, so the two measures are jointly necessary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper measures ACPC only on frozen models; a natural extension is to minimize clean–perturbed rollout distance during JEPA training as an explicit visual-robustness regularizer, since the quantity needs nothing beyond logged histories and the perturbation itself.
  • Because the paper shows both bounds can hold with equality, ACPC is not a loose worst-case estimate; it could serve as a vulnerability score ranking different perturbations by how much they can move a planner's costs on a fixed candidate pool, which the paper does not attempt.
  • The paper always keeps the goal image clean; perturbing the goal embedding instead would move the target that both costs are measured against, so the present bounds would no longer directly apply — an obvious testable extension is a goal-perturbation analogue of ACPC and a check of whether an analogous bound survives.
  • The bisimulation framing makes ACPC an empirical, rollout-level equivalence induced by the model itself; one could audit whether a world model's internal equivalence classes coincide with task state across a wider perturbation family, including closed-loop replanning rather than only the recorded open-loop action sequence.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces Action-Conditioned Predictive Consistency (ACPC), a diagnostic that measures the rollout-level divergence between a clean history and a visually perturbed view when both are rolled forward under the same action sequence. It proves two samplewise bounds: Proposition 1 bounds the change in multi-step prediction error by ACPC, and Proposition 2 bounds the change in a model-based planner's predicted cost and gives conditions under which the winner or elite set is preserved. The paper then defines checkpoint-level summaries—Invariance Radius (IR) and Separation Rate (SR)—and empirically evaluates them on four visual control tasks with LeWM and one with PLDM, under Gaussian noise, blur, and resize. The experiments include cross-validated regressions for prediction-error drift and CEM selection regret, destroyed-action controls, and a threshold screen selected on some tasks and applied to held-out tasks.

Significance. If the results hold, the paper provides a useful and inexpensive diagnostic for JEPA world models: pairwise ACPC is a computable quantity that gives samplewise, assumption-light bounds on how a visual perturbation changes prediction error and model-based planning cost, and the IR-SR screen offers a practical way to compare checkpoints across training conditions. The empirical work is careful in several respects: the prediction-error regression uses group-excluded cross-validation and conservative destroyed-action controls; the threshold transfer experiment uses held-out tasks; and the paper explicitly verifies Proposition 1 numerically on all logged pairs. The authors are also unusually candid about the scope of their claims, including the observation that the screen requires an unaugmented reference and that the planner experiment measures predicted cost rather than simulator return. The main gap is that the formal statements of Propositions 1 and 2 do not state the task-preservation assumption on which their interpretation depends.

major comments (2)
  1. [Section 3.2, Proposition 1 and Eq. (4)] The inequality in Proposition 1 is mathematically correct, but its advertised meaning as a bound on 'perturbation-induced change in prediction error' requires the perturbation to be task-preserving: the perturbed history must be an observation-level view of the same underlying state, so that the same recorded future Y_H^a is the correct comparison target for both rollouts. The proof uses Y_H^a for both errors, and the text says 'the perturbed history is created by applying a visual perturbation to the clean history,' but Proposition 1, the abstract, and the opening of Section 3 state the bound without this qualifier. If the perturbation changes the task-relevant state, the correct target for the perturbed rollout is the future of the perturbed state, and |e_tilde_true - e_h| can be large even when ACPC is near zero. The experiments apply perturbations to history observations only and appear to satisfy the assumption, so the empirical conclusions are not affected; however, the formal statement and the abstract should be restricted to observation-level, task-preserving perturbations, with the assumption stated before Proposition 1.
  2. [Section 3.3, Proposition 2 and Eq. (5)-(7)] The same task-preservation issue affects the planning-cost interpretation. Proposition 2 bounds the change in the model's predicted cost to a fixed goal embedding g; this is a statement about the model's internal cost, not about actual planning performance. If a perturbation changes the true task-relevant state, the fixed goal is no longer the correct target for the perturbed input, and a small ACPC (or small b_j) does not imply that the planner's real-world behavior is preserved. The paper does mention 'task-preserving visual perturbation' once in the Related Work discussion, but this assumption should be made a standing, explicit condition of Propositions 2 and 3 and of the checkpoint screen's interpretation. The adaptive-CEM certificate in Corollary 1 also inherits this scope condition, since equal elite sets in the model guarantee only equal model-based choices, not equal outcomes in the environment.
minor comments (4)
  1. [Reproducibility] The text says 'Code is available here' but no URL is visible; please include the repository link in the final version.
  2. [Eq. (12)] The screening score normalizes the IR margin by |tIR| and the SR margin by |tSR|, making the relative weighting of the two conditions depend on the chosen threshold magnitudes. The authors should justify this scaling or state explicitly that it is a heuristic.
  3. [Section 3.4 and Table 1] Table 1 reports sensitivity of the IR reduction to the horizon and quantile, but the text also claims stability of the diagnostic; please either report the corresponding sensitivity for SR or revise the sentence to refer only to IR.
  4. [Appendix H] The cost is divided by the embedding dimension d in the experiments ('C_j = ||x_j - g||_2^2 / d'), while Proposition 2 is stated for the summed squared cost. The rescaling is correct, but the appendix should say this explicitly before the claim that the regret bound is unchanged, since a reader may otherwise think the constant factor affects the bound.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular dependency found: the bounds are direct consequences of triangle/Cauchy-Schwarz inequalities, and the empirical IR/SR screens are calibrated on source tasks and tested on held-out tasks.

full rationale

The paper's central derivation chain is self-contained rather than circular. Proposition 1 defines ACPC as the distance between two predicted rollouts and defines e_h and e_tilde_h as the distances of those rollouts to the same observed future, then applies the reverse triangle inequality to obtain |e_tilde_h - e_h| <= ACPC. The right-hand side is a computable function of the frozen encoder, frozen predictor, and action sequence, with no fitted parameter and no dependence on the target quantity. Proposition 2 uses Cauchy-Schwarz and the triangle inequality to bound the difference of squared distances to a fixed goal, with b_j computed from evaluated endpoints; again no fitted parameter is renamed as a prediction. The empirical regressions in Sections 4.4 and 4.5 use ACPC as a feature to predict held-out error drift and CEM selection regret; an upper bound does not force predictive correlation, so the held-out improvement is genuine empirical evidence rather than a construction artifact. The IR/SR screen is explicitly calibrated on source tasks and then applied unchanged to held-out tasks: the paper states 'We choose the two thresholds using planning success on a set of source tasks and apply them unchanged to the remaining tasks,' and it separately notes that 'success rates are used to select and evaluate thresholds, but they are not inputs to ACPC, IR, or SR.' The paper also explicitly disclaims that the checkpoint-level findings follow from the pairwise bounds, saying 'These checkpoint-level findings are empirical; they do not follow from the pairwise bounds.' The remaining caveat identified by the skeptical reader, namely that the diagnostic is meaningful for task-preserving perturbations, is an explicitly stated scope condition ('a task-preserving visual perturbation' in Related Work; 'passing is specific to the evaluated visual shift and state labels and does not certify robustness' in the Figure 1 caption) and is a correctness/scope issue, not a circularity issue. No load-bearing self-citation, imported uniqueness theorem, or ansatz-smuggling citation appears in the derivation. The paper's own limitation statements are consistent with the actual scope of the claims and do not reveal a hidden dependence of the results on their own conclusions.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central theoretical results rest only on standard inequalities and the explicit domain assumption that perturbations do not alter the true state. The checkpoint screen introduces hyperparameters chosen by hand or by calibration on source tasks, all disclosed. No new physical or causal entities are postulated.

free parameters (6)
  • IR threshold tIR = 0.3 (0.1 for Reacher-only selection)
    Chosen by grid search on source tasks using the planning-success criterion; applied unchanged to held-out tasks. The upper edge of the useful range is unresolved.
  • SR threshold tSR = 0.95
    Chosen jointly with tIR on source tasks; fixed before evaluating held-out tasks.
  • SR margin delta = 0.10
    Fixed margin in Equation (10); chosen by hand, not tuned.
  • IR summarization quantile q = 0.90
    Chosen to reflect high-sensitivity anchors; stability checked for q in 0.80 to 0.95.
  • rollout horizon H = 8 for diagnostics, 5 for planner analysis
    8 is the longest horizon with complete logged futures; 5 matches the CEM planning horizon.
  • success-rate criterion constants = 0.8 improvement fraction and 5 percentage points clean loss
    Fixed a priori in Section 4.1; not optimized.
assumptions (5)
  • standard math Reverse triangle inequality, triangle inequality, and Cauchy-Schwarz
    Used in the proofs of Propositions 1 and 2 and the fixed-pool selection certificates in Appendix A.
  • domain assumption The visual perturbation is task-preserving
    Both predicted rollouts are compared with the same observed future Y_H^a; if the perturbation changed the true state, this target would not be valid for both.
  • domain assumption Frozen encoder and predictor
    ACPC evaluates a checkpoint without retraining; all results assume F_theta and E_theta are fixed.
  • domain assumption Paired CEM runs share initial proposal and random samples
    Corollary 1 requires both runs to evaluate corresponding candidate pools; only then do the selection certificates propagate across iterations.
  • domain assumption Endpoint-state labels from median splits define relevant distinctions
    SR separation is measured only with respect to these labels; if labels miss control-relevant state, SR could overstate distinguishability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency." pith.science (2026). https://pith.science/paper/LEPCACI6

@misc{pith2026260812939,
  author       = {Pith},
  title        = {Pith review of: Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LEPCACI6}},
  note         = {Machine review of arXiv:2608.12939}
}
read the original abstract

Joint-embedding predictive architectures (JEPAs) learn world models that predict in a compact latent space rather than in pixels, reducing the pressure to model nuisance appearance. Yet this provides no guarantee against visual perturbations: they can still alter the encoded representation and affect subsequent action-conditioned predictions. Bisimulation captures this requirement precisely: two observations should be treated as the same state only when their action-conditioned consequences agree. Guided by this criterion, we introduce Action-Conditioned Predictive Consistency (ACPC), a diagnostic that measures how far a clean history and a visually perturbed view of it diverge after being rolled forward under the same action sequence. We prove that this divergence bounds the perturbation-induced change in multi-step prediction error and planner cost. Building on pairwise ACPC, we define two complementary measures: the Invariance Radius (IR) summarizes clean-perturbed rollout spread, while the Separation Rate (SR) checks whether different states remain distinguishable after rollout. Experiments on four visual control tasks show that pairwise ACPC predicts perturbation-induced prediction and cost changes. On LeWM, the IR-SR screen transfers across tasks, and the joint diagnostic remains informative under blur and resize. PLDM exhibits similar diagnostic trends under a different architecture.

Figures

Figures reproduced from arXiv: 2608.12939 by the authors.

Figure 1
Figure 1. Conceptual overview of ACPC, IR, and SR. (a) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Descriptive local-geometry case study motivating the controlled ACPC comparison. Panels (a)–(d) compare a matched pair of PushT checkpoints—the left two unaugmented, the right two trained with Gaussian-noise augmentation at σmax = 0.08—at the encoder output and after eight autoregressive rollout steps. Each color denotes one of 16 highlighted histories: the large black-rimmed marker is its clean anchor, smaller same… view at source ↗
Figure 3
Figure 3. Checkpoint-level planning performance and diagnostics across the complete Gaussian-noise training sweep. We report planning success rate at evaluation noise σ = 0.08, relative IR, and SR. Points are across-run means; success-rate error bars span the three training runs, and success-rate axes are scaled per task. The dashed line is the clean success rate of the unaugmented checkpoint; the leftmost black point is that… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Cross-validated MAE for predicting the error drift d on the unaugmented checkpoints; lower is better. Each model is evaluated on trajectory groups excluded from fitting. Small points are training runs, thin lines join the same run, and diamonds are task means. The in-f…
Figure 5
Figure 5. Figure 5: Adding planner-horizon ACPC improves prediction of the extra model cost associated with changes in the plan selected by adaptive CEM. Each colored line connects the baseline and expanded-model errors for one test task excluded from regression fitting in one training ru…
Figure 6
Figure 6. Figure 6: PLDM Gaussian-noise training sweep over three independent training runs. Curves show run means, and success-rate error bars span the run range. The dashed gray line marks clean success of the unaugmented checkpoint, and the leftmost black point is its success at evalua…
Figure 7
Figure 7. Figure 7: Relative checkpoint comparison under blur and resize. Bars average the three training runs for each task and shift; per-pair values are in [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Ratio of local sensitivity for the Gaussian-noise-augmented checkpoint to that of the unaugmented checkpoint. We report a finite-difference estimate and a JVP-based Hutchinson estimate of the composed Jacobian’s squared Frobenius norm. Both estimates are below one for …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 45 canonical work pages

  1. [1]

    Mastering diverse control tasks through world models.Nature, 640(8059):647–653, 2025

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models.Nature, 640(8059):647–653, 2025

    2025

  2. [2]

    TD-MPC2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. InProceedings of the International Conference on Learning Representations (ICLR), 2024

    work page 2024

  3. [3]

    A path towards autonomous machine intelligence

    Yann LeCun. A path towards autonomous machine intelligence. OpenReview preprint, 2022. Version 0.9.2

    work page 2022

  4. [4]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15619–15629, 2023

    work page 2023

  5. [5]

    Revisiting feature prediction for learning visual representations from video.Transactions on Machine Learning Research, 2024

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video.Transactions on Machine Learning Research, 2024

    2024

  6. [6]

    Joint- embedding vs reconstruction: Provable benefits of latent space prediction for self-supervised learning

    Hugues Van Assel, Mark Ibrahim, Tommaso Biancalani, Aviv Regev, and Randall Balestriero. Joint- embedding vs reconstruction: Provable benefits of latent space prediction for self-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 38, pages 21897–21937, 2025

    work page 2025

  7. [7]

    How JEPA avoids noisy features: The implicit bias of deep linear self distillation networks

    Etai Littwin, Omid Saremi, Madhu Advani, Vimal Thilak, Preetum Nakkiran, Chen Huang, and Joshua Susskind. How JEPA avoids noisy features: The implicit bias of deep linear self distillation networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 91300–91336, 2024

    work page 2024

  8. [8]

    Bellemare

    Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G. Bellemare. DeepMDP: Learning continuous latent space models for representation learning. InProceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 2170–2179. PMLR, 2019

    work page 2019

Show all 59 references
  1. [9]

    McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine

    Amy Zhang, Rowan T. McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. InProceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  2. [10]

    LeWorldModel: Stable end-to-end joint-embedding predictive architecture from pixels.arXiv preprint arXiv:2603.19312, 2026

    Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. LeWorldModel: Stable end-to-end joint-embedding predictive architecture from pixels.arXiv preprint arXiv:2603.19312, 2026

    2026 arXiv

  3. [11]

    Kroese, Sergey Porotsky, and Reuven Y

    Dirk P. Kroese, Sergey Porotsky, and Reuven Y. Rubinstein. The cross-entropy method for continuous multi-extremal optimization.Methodology and Computing in Applied Probability, 8(3):383–407, 2006

    2006

  4. [12]

    Uladzislau Sobal, Wancong Zhang, Kyunghyun Cho, Randall Balestriero, Tim G. J. Rudner, and Yann LeCun. Learning from reward-free offline data: A case for planning with latent dynamics models. In Advances in Neural Information Processing Systems (NeurIPS), volume 38, pages 4390...

    2025

  5. [13]

    V-JEPA 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patr...

    2025 arXiv

  6. [14]

    Joint embedding predictive architectures focus on slow features

    Vlad Sobal, Jyothir S V, Siddhartha Jalagam, Nicolas Carion, Kyunghyun Cho, and Yann LeCun. Joint embedding predictive architectures focus on slow features. arXiv preprint arXiv:2211.10831, 2022. Self- Supervised Learning: Theory and Practice Workshop at NeurIPS 2022

    2022 arXiv

  7. [15]

    Bootstrap latent-predictive representations for multitask reinforcement learning

    Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean-Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent-predictive representations for multitask reinforcement learning. InProceedings of the 37th International Conference on Machine L...

    2020

  8. [16]

    Devon Hjelm, Aaron Courville, and Philip Bachman

    Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. InProceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  9. [17]

    Understanding self-predictive learning for reinforcement learning

    Yunhao Tang, Zhaohan Daniel Guo, Pierre Harvey Richemond, Bernardo Avila Pires, Yash Chandak, Rémi Munos, Mark Rowland, Mohammad Gheshlaghi Azar, Charline Le Lan, Clare Lyle, András György, Shantanu Thakoor, Will Dabney, Bilal Piot, Daniele Calandriello, and Michal Valko. Unde...

    2023

  10. [18]

    Borsa, Arthur Guez, and Will Dabney

    Khimya Khetarpal, Zhaohan Daniel Guo, Bernardo Avila Pires, Yunhao Tang, Clare Lyle, Mark Rowland, Nicolas Heess, Diana L. Borsa, Arthur Guez, and Will Dabney. A unifying framework for action-conditional self-predictive reinforcement learning. InProceedings of the 28th Interna...

    2025

  11. [19]

    When does LeJEPA learn a world model? arXiv preprint arXiv:2605.26379, 2026

    David Klindt, Yann LeCun, and Randall Balestriero. When does LeJEPA learn a world model? arXiv preprint arXiv:2605.26379, 2026

    2026 arXiv

  12. [20]

    Muller, and Shahab Bakhtiari

    Hafez Ghaemi, Eilif B. Muller, and Shahab Bakhtiari. seq-JEPA: Autoregressive predictive learning of invariant-equivariant world models. InAdvances in Neural Information Processing Systems (NeurIPS), volume 38, pages 32943–32973, 2025

    2025

  13. [21]

    Image augmentation is all you need: Regularizing deep rein- forcement learning from pixels

    Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep rein- forcement learning from pixels. InProceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  14. [22]

    Mastering visual continuous control: Improved data-augmented reinforcement learning

    Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. InProceedings of the International Conference on Learning Representations (ICLR), 2022

    2022

  15. [23]

    Generalization in reinforcement learning by soft data augmentation

    Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmentation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 13611–13617, 2021

    2021

  16. [24]

    Zero-shot visual generalization in model-based reinforcement learning via latent consistency

    Mingyu Park, Samyeul Noh, Hyun Myung, and Donghwan Lee. Zero-shot visual generalization in model-based reinforcement learning via latent consistency. OpenReview (ICLR 2026 submission), 2025

    2026

  17. [25]

    VIBR: Learning view-invariant value functions for robust visual control

    Tom Dupuis, Jaonary Rabarisoa, Quoc-Cuong Pham, and David Filliat. VIBR: Learning view-invariant value functions for robust visual control. InProceedings of The 2nd Conference on Lifelong Learning Agents, volume 232 ofProceedings of Machine Learning Research, pages 658–682. PM...

    2023

  18. [26]

    Reimagination with test-time observation interventions: Distractor-robust world model predictions for visual model predictive control

    Yuxin Chen, Jianglan Wei, Chenfeng Xu, Boyi Li, Masayoshi Tomizuka, Andrea Bajcsy, and Ran Tian. Reimagination with test-time observation interventions: Distractor-robust world model predictions for visual model predictive control. InProceedings of the IEEE International Confe...

    2026

  19. [27]

    Nguyen, Rui Shu, Tuan Pham, Hung Bui, and Stefano Ermon

    Tung D. Nguyen, Rui Shu, Tuan Pham, Hung Bui, and Stefano Ermon. Temporal predictive coding for model-based planning in latent space. InProceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 8130–8139....

    2021

  20. [28]

    DreamerPro: Reconstruction-free model-based reinforcement learning with prototypical representations

    Fei Deng, Ingook Jang, and Sungjin Ahn. DreamerPro: Reconstruction-free model-based reinforcement learning with prototypical representations. InProceedings of the 39th International Conference on Machine Learning, volume 162 ofProceedings of Machine Learning Research, pages 49...

    2022

  21. [29]

    Learning task informed abstractions

    Xiang Fu, Ge Yang, Pulkit Agrawal, and Tommi Jaakkola. Learning task informed abstractions. In Proceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 3480–3491. PMLR, 2021

    2021

  22. [30]

    Du, Antonio Torralba, Phillip Isola, Amy Zhang, and Yuandong Tian

    Tongzhou Wang, Simon S. Du, Antonio Torralba, Phillip Isola, Amy Zhang, and Yuandong Tian. Denoised MDPs: Learning world models better than the world itself. InProceedings of the 39th International Conference on Machine Learning, volume 162 ofProceedings of Machine Learning Re...

    2022

  23. [31]

    Iso-Dream: Isolating and leveraging noncontrollable visual dynamics in world models

    Minting Pan, Xiangming Zhu, Yunbo Wang, and Xiaokang Yang. Iso-Dream: Isolating and leveraging noncontrollable visual dynamics in world models. InAdvances in Neural Information Processing Systems (NeurIPS), volume 35, pages 23178–23191, 2022

    2022

  24. [32]

    Learning latent dynamic robust representations for world models

    Ruixiang Sun, Hongyu Zang, Xin Li, and Riashat Islam. Learning latent dynamic robust representations for world models. InProceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 47234–47260. PMLR, 2024

    2024

  25. [33]

    AD3: Implicit action is the key for world models to distinguish the diverse visual distractors

    Yucen Wang, Shenghua Wan, Le Gan, Shuai Feng, and De-Chuan Zhan. AD3: Implicit action is the key for world models to distinguish the diverse visual distractors. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning...

    2024

  26. [34]

    Policy-shaped prediction: Avoiding distractions in model-based reinforcement learning

    Miles Hutson, Isaac Kauvar, and Nick Haber. Policy-shaped prediction: Avoiding distractions in model-based reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), volume 37, pages 13124–13148, 2024

    2024

  27. [35]

    Bisimulation metric for model predictive control

    Yutaka Shimizu and Masayoshi Tomizuka. Bisimulation metric for model predictive control. InProceedings of the International Conference on Learning Representations (ICLR), 2025

    2025

  28. [36]

    Toso, Davit Shadunts, Yunyang Lu, Nihal Sharma, Donglin Zhan, Nam H

    Leonardo F. Toso, Davit Shadunts, Yunyang Lu, Nihal Sharma, Donglin Zhan, Nam H. Nguyen, and James Anderson. Learning invariant visual representations for planning with joint-embedding predictive world models.arXiv preprint arXiv:2602.18639, 2026

    2026

  29. [37]

    Singh, and David Silver

    Christopher Grimm, André Barreto, Satinder P. Singh, and David Silver. The value equivalence principle for model-based reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), volume 33, pages 5541–5552, 2020

    2020

  30. [38]

    Voelcker, Anastasiia Pedan, Arash Ahmadian, Romina Abachi, Igor Gilitschenski, and Amir- Massoud Farahmand

    Claas A. Voelcker, Anastasiia Pedan, Arash Ahmadian, Romina Abachi, Igor Gilitschenski, and Amir- Massoud Farahmand. Calibrated value-aware model learning with probabilistic environment models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267...

    2025

  31. [39]

    MWM: Mobile world models for action-conditioned consistent prediction

    Han Yan, Zishang Xiang, Zeyu Zhang, and Hao Tang. MWM: Mobile world models for action-conditioned consistent prediction. arXiv preprint arXiv:2603.07799, 2026

    2026

  32. [40]

    Objective mismatch in model-based reinforcement learning

    Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model-based reinforcement learning. InProceedings of the 2nd Conference on Learning for Dynamics and Control, volume 120 ofProceedings of Machine Learning Research, pages 761–770. PMLR, 2020. 16

    2020

  33. [41]

    McDonald, Alfredo Garcia, and Roberto Calandra

    Ran Wei, Nathan Lambert, Anthony D. McDonald, Alfredo Garcia, and Roberto Calandra. A unified view on solving objective mismatch in model-based reinforcement learning.Transactions on Machine Learning Research, 2024

    2024

  34. [42]

    How should world models be evaluated for embodied decision-making? a decision-making-centric position.arXiv preprint arXiv:2606.15032, 2026

    Yang Yu, Shiyuan Zhang, Yifei Sheng, Haoxiang Ren, and Haoxin Lin. How should world models be evaluated for embodied decision-making? a decision-making-centric position.arXiv preprint arXiv:2606.15032, 2026

    2026 arXiv

  35. [43]

    ATM: Action-consistency transfer matrix for diagnosing and improving latent world models

    Jiaheng Chen. ATM: Action-consistency transfer matrix for diagnosing and improving latent world models. arXiv preprint arXiv:2606.09028, 2026

    2026 arXiv

  36. [44]

    Delta-JEPA: Learning action-sensitive world models via latent difference decoding

    Zhenghao Zhang, Yuanxiang Wang, Zhenyu Guan, Yujia Yang, Bingkang Shi, Tianyu Zong, Hongzhu Yi, Guoqing Chao, Xingchen Chen, Tiankun Yang, Chenxi Bao, Tao Yu, Jingjing Zhou, and Jungang Xu. Delta-JEPA: Learning action-sensitive world models via latent difference decoding. arXi...

    2026 arXiv

  37. [45]

    ACID: Action consistency via inverse dynamics for planning with world models

    Gawon Seo, Dongwon Kim, and Suha Kwak. ACID: Action consistency via inverse dynamics for planning with world models. arXiv preprint arXiv:2607.02403, 2026

    2026 arXiv

  38. [46]

    Is the future compatible? Diagnosing dynamic consistency in world action models

    Bo-Kai Ruan, Teng-Fang Hsiao, Ling Lo, and Hong-Han Shuai. Is the future compatible? Diagnosing dynamic consistency in world action models. arXiv preprint arXiv:2605.07514, 2026

    2026 arXiv

  39. [47]

    World models as group actions

    Zijie Wang, Wei Zhang, Weiming Zhang, Fanqi Zhang, Xiao Tan, Yipeng Qin, and Guanbin Li. World models as group actions. arXiv preprint arXiv:2605.24578, 2026

    2026 arXiv

  40. [48]

    Imagined rollouts are kinematic, not dynamic: A diagnosis of long-horizon world-model failure

    Finn Rasmus Schäfer, Korbinian Moller, Yuan Gao, Christian Oefinger, Sebastian Schmidt, and Johannes Betz. Imagined rollouts are kinematic, not dynamic: A diagnosis of long-horizon world-model failure. arXiv preprint arXiv:2607.05966, 2026

    2026 arXiv

  41. [49]

    What makes video world model latents action-relevant: Prediction over reconstruction.arXiv preprint arXiv:2606.07687, 2026

    Jewon Yeom, Hanseul Kim, Jeongjae Park, Sungmok Jung, Jaejin Lee, and Taesup Kim. What makes video world model latents action-relevant: Prediction over reconstruction.arXiv preprint arXiv:2606.07687, 2026

    2026 arXiv

  42. [50]

    WMAttack: Automated attack search for adversarial evaluation of world-model agents.arXiv preprint arXiv:2605.23220, 2026

    Zhixiang Guo, Siyuan Liang, Shi Fu, Cheng Guo, András Balogh, Márk Jelasity, and Dacheng Tao. WMAttack: Automated attack search for adversarial evaluation of world-model agents.arXiv preprint arXiv:2605.23220, 2026

    2026 arXiv

  43. [51]

    ARB4WM: An adversarial robustness benchmark for world models in continuous control.arXiv preprint arXiv:2606.16605, 2026

    Junjian Zhang, Hao Tan, Ruonan Li, Dong Zhu, Aiping Li, and Zhaoquan Gu. ARB4WM: An adversarial robustness benchmark for world models in continuous control.arXiv preprint arXiv:2606.16605, 2026

    2026

  44. [52]

    Operator-on-F complements value-equivalence: A planning-time diagnostic for latent world models.arXiv preprint arXiv:2607.04464, 2026

    Donna Vakalis. Operator-on-F complements value-equivalence: A planning-time diagnostic for latent world models.arXiv preprint arXiv:2607.04464, 2026

    2026 arXiv

  45. [53]

    A control theory of predictability in latent world models.arXiv preprint arXiv:2607.10362, 2026

    HanzheYou, YonggangZhang, MaohaoRan, ZhiqinYang, ZhenyuanZhang, WeiXue, JunSong, XinmeiTian, and Yike Guo. A control theory of predictability in latent world models.arXiv preprint arXiv:2607.10362, 2026

    2026 arXiv

  46. [54]

    Björn Lütjens, Michael Everett, and Jonathan P. How. Certified adversarial robustness for deep reinforcement learning. InProceedings of the Conference on Robot Learning, volume 100 ofProceedings of Machine Learning Research, pages 1328–1337. PMLR, 2020

    2020

  47. [55]

    CROP: Certifying robust policies for reinforcement learning through functional smoothing

    Fan Wu, Linyi Li, Zijian Huang, Yevgeniy Vorobeychik, Ding Zhao, and Bo Li. CROP: Certifying robust policies for reinforcement learning through functional smoothing. InProceedings of the International Conference on Learning Representations (ICLR), 2022

    2022

  48. [56]

    Visualizing data using t-SNE.Journal of Machine Learning Research, 9(86):2579–2605, 2008

    Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE.Journal of Machine Learning Research, 9(86):2579–2605, 2008

    2008

  49. [57]

    Contractive auto-encoders: Explicit invariance during feature extraction

    Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto-encoders: Explicit invariance during feature extraction. In Lise Getoor and Tobias Scheffer, editors,Proceedings of the 28th International Conference on Machine Learning, pages 833–8...

    2011

  50. [58]

    Hutchinson

    Michael F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines.Communications in Statistics - Simulation and Computation, 18(3):1059–1076, 1989. 17 A Proofs and Fixed-Pool Analysis Proof of Proposition 1.Let u = ¯Ga(Eθ(h)), v...

    1989

  51. [59]

    Selected control

    If their proposals agree at iterationt, the shared samples generate the same candidate pool. When Equation (16) holds, the two runs also select the same elite candidates and therefore fit the same next proposal: ϕc t =ϕ p t =⇒ Ac t =A p t =⇒ Ec t =E p t =⇒ϕ c t+1 =ϕ p t+1. Thu...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.