Pith.

Robotics · arXiv:2610.12407 v2 · submitted 8 Oct 2026

Diffusion-Steering-Based MPC with a JEPA World Action Model

Shashank Hegde, Alexander Popov, Elie Aljalbout, Nikolai Smolyanskiy

The paper says

“World action models (WAMs) predict actions and future observations, typically from a reconstruction-based representation that carries noisy, redundant information which can complicate downstream predictions.”

more of the abstract

We introduce LeWAM, a bidirectional transformer for forward, backward, inverse dynamics and policy prediction, on a decoder-free JEPA latent trained end-to-end through all four modes. We see the following benefits: 1) Alignment: linear probes read robot and object state from LeWAM's latent better than from a regular Le World Model (a forward-only JEPA world model), while the latent ignores visual distractors as well as LeWM does and far better than a reconstruction-based WAM. 2) Acting: Closed-loop evaluations of LeWAM match a regular flow-matching policy trained on the same encoder at matched size, while also providing a world model. 3) Planning: Sampling raw actions when planning with WAMs lets MPC exploit dynamics-model inaccuracies; planning in the noise space of the policy head instead improves the closed-loop performance of these WAMs.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 9 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A genuinely new decoder-free JEPA action model with careful multi-seed evaluation; diffusion-steering planning works empirically, but the goal-latent proxy is not yet shown to be doing the work. This line is wrong

3 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Section 4.1, Eq. (1); Section 5.4; Appendix D

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The planning claim depends on the assumption that the Euclidean latent distance in Eq. (1) ranks action chunks by task progress, but this equivalence is never tested. The goal latent is obtained by encoding the observation 25 steps after the demonstration frame nearest the current latent, in the model's own latent space, so the ranking can be internally self-consistent without tracking the true block or end-effector state.

more of objection 1

This is load-bearing because the projection ablation (Table 4) shows a large effect of staying on the sampler's typical set (Can: 78.8% with the sqrt(A) shell versus 23.0% without), so the DS gain could partly or wholly come from policy-prior selection rather than from goal-directed search through Eq. (1). Please add (a) a correlation study between Eq. (1) scores and task-state distance or success on held-out starts, (b) a DS-CEM control that scores candidates by ground-truth task-state distance to the goal, and (c) a sensitivity check on the 25-step retrieval horizon. The paper acknowledges that a goal is needed at inference time (Section 6), but it does not acknowledge or address the validity of the goal proxy itself.

The paper says, at Section 5.4; Figures 5 and 6

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The headline closed-loop planning result is presented only graphically, with no numeric table giving per-task, per-planner means and error bars. The text's quantitative claims—that action-space MPC collapses every model to 3–8% success and that sampling in noise space gives both LeWAM and RecWAM a performance gain—cannot be verified from the manuscript as written.

more of objection 2

Appendix A points to per-task results in Figure 6, but again without a numeric table or seed-level statistics. Please add a table reporting means and standard deviations over the 5 training seeds and 50 evaluation seeds for each policy, each action-space MPC variant (CEM, MPPI, CMA-ES), and each diffusion-steering variant on every task.

The paper says, at Section 4.3; Algorithm 1

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The paper does not separate the benefit of iterative planning from the benefit of best-of-N proposal ranking over the policy prior. Section 4.3 states that the first iteration of every DS planner is best-of-N ranking with N=S over projected noise, yet no experiment compares this one-iteration procedure against the full I=3 DS-CEM.

more of objection 3

Without this control, the improvement attributed to MPC-style iterative refinement is not distinguished from the simpler explanation that sampling from the policy's typical set and ranking by a self-consistent latent distance is already sufficient for most of the gain. Please report I=1 versus I=3 DS-CEM, ideally matched in total proposal count, for LeWAM and RecWAM.

6 minor objections

The paper says, at Section 4.4

Pith says: minor objection 1 of 6

The sentence "The search distribution stays Euclidean; only proposals are projected" appears twice in immediate succession; one occurrence should be removed.

The paper says, at Section 4.3, Eq. (2)

Pith says: minor objection 2 of 6

The notation in Eq. (2) is confusing: the update is written as z_hat_h = shift(z_hat_{h-1}, z_hat_h), where the same symbol z_hat_h appears on both sides. Use a temporary variable or a different subscript to make the recurrence clear.

show 4 more

The paper says, at Table 2

Pith says: minor objection 3 of 6

The MSE column for distractors is labelled "Distractors↑", but higher MSE means the distractor state is not encoded; the arrow notation is counterintuitive and should be explained in the caption.

The paper says, at Section 5.3 and Conclusion

Pith says: minor objection 4 of 6

The statement that the latent "ignores" visual distractors is stronger than what the evidence supports; the linear probe results show that distractor state is not linearly decodable, and the qualitative decoder rollouts are suggestive but not a quantitative invariance measure. Please qualify this as "not linearly decodable" or add a nonlinear-probe control.

The paper says, at Table 3

Pith says: minor objection 5 of 6

The "No policy" row is confounded by construction: that variant has no policy head, so its DS-CEM column steers the inverse-trained flow head, and its near-zero DS-CEM success shows that diffusion steering requires a policy head rather than that the policy mode improves the latent representation. The conclusion that "all modes add value" should be reworded to acknowledge this.

The paper says, at Appendix D

Pith says: minor objection 6 of 6

For Dreamer, action-space MPC is scored by pixel error of the decoded rollout rather than by the latent distance of Eq. (1); the main text should state this difference when comparing "action-space MPC" across models, since the comparison is not a pure ablation of the decision variable.

What would settle it

On PushT and Lift/Can, fix the policy head, budget (64x3), and projection, and compare DS-CEM goals: (a) the Appendix D retrieved goal, (b) a fixed goal latent from the final successful demonstration frame, (c) a random goal latent from another demonstration. Also, for a fixed set of start latents, score 64 policy-prior chunks by Eq. 1 and by actual simulator success and report the Spearman rank correlation. If closed-loop success is insensitive to goal choice, or the rank correlation is not significantly positive, Eq. 1 is not steering behavior and the planning claim needs revision.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 9 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 2 min 13 s · 14,902 tokens read, 9,182 written
  3. Editor wrote the verdict above 45 s · 15,640 tokens read, 5,980 written
  4. Referee major revision: 3 major objections, 6 minor 1 min 7 s · 15,119 tokens read, 9,063 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Here's my quick read. LeWAM is worth taking seriously. It isn't a rebranding: a bidirectional transformer on a LeWM-style JEPA latent, trained end-to-end with forward, backward, inverse, and policy modes on every batch, giving one model both a dynamics head and a flow-matching action head. The evaluation is properly structured—five training seeds, fifty eval seeds, parameter-matched baselines, public demonstration data, and a mode ablation. The acting claim in b2 holds up: LeWAM roughly matches the standalone flow-matching policy on the same encoder. The probe results are also credible: task-state R2 goes up and distractor R2 stays near zero. The no-policy ablation kills DS-CEM, which is what you'd expect if the planner is genuinely decoding through the policy head. The LeWM baseline is disclosed as frozen-LeWM plus their action head, and the DSRL/Golden Ticket lineage for noise-space search is credited. Fine so far.

The planning claim is the weak point, and the stress-test note is on target. Eq. 1 scores chunks by Euclidean distance to a goal latent retrieved from the model's own encoder; the same forward model both rolls out and scores, and the paper never checks that this proxy ranks chunks by task progress. Because the same encoder produces the goal and the imagined final latent, the ranking can be self-consistent without tracking true task state. The projection ablation makes the worry concrete: removing the sphere projection drops Can from 78.8% to 23.0%, a much larger effect than you'd expect from a temperature fix. There is also no table for Figure 5, and the design doesn't isolate one-iteration best-of-N proposal ranking from iterative MPC—the first DS iteration is already best-of-N. A random-goal or fixed-goal control, plus a numeric per-task table, would let a reader attribute the gain. The authors do acknowledge planners are slow and need a goal at inference (b123), but they don't acknowledge the proxy itself as the unresolved step.

Minor: no code or weights.

Bottom line: this is a serious, carefully run paper with one central empirical claim that is under-supported. I'd send it to review; the architecture and acting results justify referee time, and the planning section can be fixed with targeted experiments. I'd want goal-proxy validation and a numeric table before signing off. Would cite it.

The strongest objection

Claim (3) requires that minimizing ||f(z_t,a)-z_g||_2 in Eq. 1 ranks action chunks by task progress. That equivalence is never checked. Section 4.1 imports the proxy from DINO-WM/LeWM; Section 5.4 defines z_g as the latent 25 steps after the demonstration frame nearest the current latent (Appendix D); Figure 5 reports only aggregate closed-loop success, not the goal-ranking quality. Since the same encoder produces both the retrieved goal and the scored final latent, ranking can be self-consistent without tracking true task state, and because DS samples only through the flow head g(epsilon; z), gains may come from staying on the policy's typical set rather than from Eq. 1. Table 4 makes this salient: removing the spherical projection changes Can from 78.8% to 23.0%. The authors acknowledge needing a goal at inference time, but not the validity of the proxy; that is the unresolved load-bearing step.

The reader identifies the latent-distance proxy as the weakest assumption; I agree. The architecture and acting results are supported by multi-seed closed-loop evaluations and mode ablations, and the projection ablation in Table 4 is a genuine falsifiable check. But the planning benefit is the least supported of the three abstract claims: because the same latent space is used for goal retrieval and for scoring, the planner could select actions that merely keep rollouts consistent with the model's own representation, while the proposal distribution is simultaneously changed from raw actions to the flow head's typical set. A control that holds proposals fixed while changing the goal, plus a direct rank-correlation check between Eq. 1 and environment success, would separate goal-directed planning from policy-prior selection. The lack of tabulated per-task numbers and the absence of a best-of-N-versus-iterative-update baseline weaken verification further, but the proxy validity is the one condition on which the entire planning claim rests. This does not overturn the paper; it confirms the CONDITIONAL verdict already given.

The referee's report

If the claims hold, the paper is a useful step toward reconstruction-free world action models: it shows that a single backbone can provide forward, backward, inverse, and policy prediction, that the resulting latent supports linear readout of task state while being robust to distractors, and that an inference-only change to the MPC decision variable can improve closed-loop success. The strengths are genuine and should be credited: multi-seed closed-loop evaluation (5 training seeds × 50 evaluation seeds), parameter-matched baselines, public tasks and demonstrations, an ablation table for the four training modes, and an ablation of the spherical projection step. The headline planning claim, however, currently rests on figures rather than auditable numeric tables, and the latent-distance goal proxy that Eq. (1) relies on is not validated against any task-reward signal; both are fixable within the scope of the manuscript.

Pith's reading of the paper (a machine's summary, not the authors')

LeWAM tries to give robots a single world action model that needs no pixel decoder: one bidirectional transformer, trained on a joint-embedding-predictive (JEPA) latent, that at once predicts future latents, back-predicts earlier latents, infers the actions bridging two observed states, and emits the next action chunk. The paper reports three payoffs. First, linear probes read robot and object state from LeWAM's latent more accurately than from the forward-only JEPA baseline, while visual distractors leave almost no trace in the latent. Second, acting with the learned flow-matching policy matches a standalone behavior-cloning policy at matched model size, while the same model still provides a dynamics model. Third, closed-loop model-predictive control improves when the planner searches the policy head's noise space, the Gaussian draw the flow sampler integrates, rather than raw action chunks; the paper's diagnosis is that sampling raw actions lets MPC exploit dynamics-model inaccuracies and optimize for hallucinated futures that fail in the simulator.

What carries the argument. Three mechanisms carry the argument. The latent: each frame becomes a single 384-dimensional vector from a ViT-tiny encoder trained from scratch, pushed toward an isotropic Gaussian by SIGReg, with no decoder, no tokenizer and no EMA target network; this one vector is what the probes read, the dynamics regress, and the planner compares. The backbone: a depth-8 bidirectional transformer reads the context latents and action tokens, and the four training modes are different maskings of the same input window - the final frame masked for forward dynamics, the first frame for backward dynamics, the target action chunk replaced by a noisy flow sample for inverse dynamics, and both the final frame and recent actions masked for policy prediction. The planner: the rectified-flow head is a deterministic map $a = g(\epsilon; z_t)$ from a Gaussian draw to an action chunk, and diffusion steering moves the MPC decision variable to $\epsilon$, projecting every proposal onto the sphere of radius $\sqrt{A}$ before decoding so the search stays on the directions the head was trained to produce.

Core claim. Stated on the paper's own terms, the central claim is that a decoder-free JEPA latent trained end-to-end through four masked modes of one bidirectional transformer is simultaneously a cleaner state representation, a deployable policy, and a forward model good enough to plan with. The paper's argument is that reconstruction-based world action models such as Cosmos Policy force the latent to encode everything a pixel decoder could render, wasting capacity on distractors, while forward-only JEPA models such as LeWM offer no policy head; LeWAM couples LeWM's regularized latent with the UWM-style multi-mode skeleton and adds backward dynamics. On six manipulation tasks with visual distractors, the reported results are: LeWAM's latent beats LeWM on task-state probes (average R-squared of 0.80 vs 0.62) and slightly exceeds the reconstruction-based RecWAM (0.78), while reading essentially no distractor state (0.00 vs 0.73); its closed-loop acting holds the level of the flow-matching policy reference; and every action-space planner collapses to 3-8% closed-loop success, whereas the same planners steered in the policy's noise space improve on the policy itself.

The premise it rests on. The load-bearing premise is that the Euclidean distance between an imagined final latent and a goal latent, retrieved as the latent 25 steps after the demonstration frame nearest the current state, ranks action chunks the way real task success would; the paper does not validate this proxy against a task-reward signal.

If the paper is right (Pith's inferences, not the authors')

One architecture suffices for acting and imagining: LeWAM's policy, forward, inverse and backward modes share a single latent and backbone, so no separately trained dynamics model is needed to plan.

Diffusion steering is inference-only and planner-agnostic: the same search recipe hosts CEM, MPPI and CMA-ES updates, and any flow or diffusion policy with a deterministic sampler could be steered the same way.

Projecting proposals onto the $\sqrt{A}$ shell is load-bearing: the ablation reports closed-loop success dropping by about 5 points on Lift, 12 on Square and 56 on Can when the projection is removed.

Raw-action MPC on these human-demonstration datasets collapses every model to 3-8% closed-loop success, so the choice of decision variable, not the particular world model, is the practical failure point for WAM-based planning.

Every training mode contributes: the ablation removes forward, backward, inverse or policy training one at a time and each removal lowers latent informativeness or closed-loop success, and removing the policy mode leaves no steerable head with DS-CEM success falling to 0.8% or 0.0%.

A test the authors do not run: vary the number of distractors or their motion and track distractor-probe readout and closed-loop success; the near-zero distractor readout in Table 2 predicts graceful degradation under clutter, which is the property that would matter most on real-world data.

The paper's own explanation, that imperfect dynamics under small data make action-space search exploit model errors, predicts that noise-space planning's advantage shrinks as the dataset grows; a data-scaling curve of DS-CEM versus action-space CEM would separate the steering mechanism from mere model quality.

Because ProjectedDS projects to the $\sqrt{A}$ shell unconditionally, the 56-point swing on Can suggests the projection acts as a strong prior that can suppress useful low-norm proposals; interpolating the projection radius between 0 and $\sqrt{A}$ would reveal whether the shell is a correction or a constraint.

The planner needs a demonstration-derived goal latent at every decision, a limitation the authors acknowledge; a learned goal encoder or task embedding would let the same steering mechanism plan toward goals that no demonstration instantiates.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

World action models (WAMs) predict actions and future observations, typically from a reconstruction-based representation that carries noisy, redundant information which can complicate downstream predictions. We introduce LeWAM, a bidirectional transformer for forward, backward, inverse dynamics and policy prediction, on a decoder-free JEPA latent trained end-to-end through all four modes. We see the following benefits: 1) Alignment: linear probes read robot and object state from LeWAM's latent better than from a regular Le World Model (a forward-only JEPA world model), while the latent ignores visual distractors as well as LeWM does and far better than a reconstruction-based WAM. 2) Acting: Closed-loop evaluations of LeWAM match a regular flow-matching policy trained on the same encoder at matched size, while also providing a world model. 3) Planning: Sampling raw actions when planning with WAMs lets MPC exploit dynamics-model inaccuracies; planning in the noise space of the policy head instead improves the closed-loop performance of these WAMs.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-09

The authors' main claim, as the classifier picked it out 0.93

“We see the following benefits: 1) Alignment: linear probes read robot and object state from LeWAM's latent better than from a regular Le World Model (a forward-only JEPA world model), while the latent ignores visual distractors as well as LeWM does and far better than a reconstruction-based WAM. 2) Acting: Closed-loop evaluations of LeWAM match a regular flow-matching policy trained on the same encoder at matched size, while also providing a world model. 3) Planning: Sampling raw actions when planning with WAMs lets MPC exploit dynamics-model inaccuracies; planning in the noise space of the policy head instead improves the closed-loop performance of these WAMs.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.29 · human participants 0.03 · controlled comparison 0.93 · randomized assignment 0.02 · states quantitative result 0.05 · compares with prior results 0.97 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
World action models (WAMs) predict actions and future observations, typically from a reconstruction-based representation that carries noisy, redundant information which can complicate downstream predictions. background 0.99 0.06 0.04 asserted 0.84
We introduce LeWAM, a bidirectional transformer for forward, backward, inverse dynamics and policy prediction, on a decoder-free JEPA latent trained end-to-end through all four modes. contribution 0.78 0.93 0.16 asserted 1.00
We see the following benefits: 1) Alignment: linear probes read robot and object state from LeWAM's latent better than from a regular Le World Model (a forward-only JEPA world model), while the latent ignores visual distractors as well as LeWM does and far better than a reconstruction-based WAM. 2) Acting: Closed-loop evaluations of LeWAM match a regular flow-matching policy trained on the same encoder at matched size, while also providing a world model. 3) Planning: Sampling raw actions when planning with WAMs lets MPC exploit dynamics-model inaccuracies; planning in the noise space of the policy head instead improves the closed-loop performance of these WAMs. result 0.97 0.94 0.57 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.71

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-09: 1 request, 3,161 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (6)
Figure 1 of the paper
Figure 1. The four training modes on one window: the mode is set by what is masked at the trans
Figure 2 of the paper
Figure 2. LeWAM: a shared ViT-tiny encoder with SIGReg on its single latent per frame, and one
Figure 3 of the paper
Figure 3. The six tasks as the models see them, with their distractors: painted dot and bar on PushT,
Figure 4 of the paper
Figure 4. Imagined rollouts with LeWAM using ground truth actions. The top image is for PushT
Figure 5 of the paper
Figure 5. Closed-loop success of all models and planners,
Figure 6 of the paper
Figure 6. Average closed-loop performance of all models/MPC, for each task, over 5 training seeds,

Figures are the authors', reproduced from the paper.

Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment