Enfold: Folding World Model Imagination into Predictive Representations for Ultra-Efficient Embodied Control

Antao Xiang; Chengqun Yang; Feng Tian; Fulong Liu; Jingnan Gao; Jisong Cai; Weili Zeng; Xiaokang Yang; Xiaomin Wu; Xin Wang

A world generator's future-building computation can be folded into a representation computed from the present alone, removing the generator from the deployment-time control loop.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:21 UTC pith:V4PR57ZV

load-bearing objection A well-executed empirical recipe for amortizing a video generator into a current-only policy; the central interpretation as internalizing future computation is plausible but not fully pinned down, and the self-conditioning loop needs a direct test. the 3 major comments →

arxiv 2607.26657 v2 pith:V4PR57ZV submitted 2026-07-29 cs.RO

Enfold: Folding World Model Imagination into Predictive Representations for Ultra-Efficient Embodied Control

Weili Zeng , Yitong Xing , Fulong Liu , Chengqun Yang , Antao Xiang , Feng Tian , Jingnan Gao , Jisong Cai

show 5 more authors

Xin Wang Xiaomin Wu Yao Mu Xiaokang Yang Yichao Yan

This is my paper

classification cs.RO

keywords world generative modelspredictive representation learningvisuomotor controlrepresentation distillationlatent video generationflow matchingrobotic manipulationaction latency

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a video world generator's internal computation—the way its intermediate states organize appearance, layout, and interaction while constructing a future—can be distilled into a representation inferred from the current observation and instruction alone. If true, it means generative world models can serve as a training-time source of predictive control structure rather than a mandatory component of the control loop: the future need not be materialized at every step. Enfold achieves this by supervising a current-only encoder with multi-level states extracted during a teacher-forced generator pass, feeding the learned representation back to condition generation, and reading actions from it through detached task heads. The paper reports 97.8% success on LIBERO and 91.77% on RoboTwin2.0 without executing the generator at deployment, alongside a 3.7x action-latency reduction (134 ms) and 10.1x with TensorRT acceleration (49 ms). A sympathetic reader would care because this reframes the role of world generators from action-time simulators to reusable sources of anticipatory representations.

Core claim

The central claim is that future-conditioned internal states of a world generator can be predicted from the present alone, and that the resulting representation retains enough transition-relevant structure to support both control and future generation. The key asymmetry is that the encoder never sees the future, so it must preserve structure anticipatable from the current state while discarding nuisance variation; detached task heads then read the representation without task gradients reshaping it into a policy backbone. Because no single generator layer is universally informative across corruption levels, Enfold predicts a multi-level target through a timestep-conditioned head. The paper sh

What carries the argument

The governing mechanism is a bidirectional coupling between a current-only predictive encoder and a video generator. In generation-to-representation (G2R), a teacher-forced pass over the observed future exposes hidden states at selected generator layers; these are concatenated, layer-normalized, and used—with a timestep-conditioned head—as supervision targets for the encoder. In representation-to-generation (R2G), the same representation, detached, conditions the generator's flow-matching objective, making it a functional test of whether the representation carries predictive structure beyond the visible context. Detached task readouts complete the loop, ensuring the encoder is shaped only by

Load-bearing premise

The closed loop between the generator and the encoder must avoid a trivial fixed point: the generator's states must carry future structure beyond re-encoding the learner's own conditioning output, otherwise the distilled computation is just the generator reading back its own conditioning and the control gains come from the encoder's image features.

What would settle it

Train Enfold with G2R targets computed from a future that belongs to a different episode while keeping the current context and instruction fixed; if the resulting representation still achieves near-identical control success, then the supervision transferred no future-specific structure, and the reported gains would be attributable to the encoder's image features rather than to the generator's computation.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

World generators no longer need to run in the control loop: Enfold cuts action latency to 134 ms (3.7x) and 49 ms (10.1x) with Flash mode, while keeping LIBERO success at 97.8% and RoboTwin2.0 at 91.77%.
Generator-state supervision is stronger than future-pixel or action-only targets, and multi-level targets add 1.5 points over single-level supervision, concentrated on goal-directed and multi-stage tasks.
The learned representation improves future-video prediction when fed back to the generator (PSNR 25.92 to 27.27, SSIM 0.885 to 0.902, LPIPS down 12.2%), indicating a shared predictive interface rather than a policy-specific summary.
Under human intervention to the scene, both the imagined continuation and the executed actions adapt, which is inconsistent with fixed trajectory replay.
Representation analyses show that Enfold suppresses nuisance variation (7.9-10.4x lower lighting sensitivity) while retaining higher effective rank than generator features, and its predictive advantage grows with future horizon and concentrates on regions that change.
The generator still needs to exist during training; only deployment becomes generator-free. This suggests the same principle could be applied to other expensive generative components, with the cost shifted entirely to training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the same distillation principle could apply to other generative backbones, not just video: if a diffusion- or flow-based planner's internal states can be amortized into a present-only encoder, latent imagination could be skipped more aggressively in model-based control.
Editorial extension: the depth- and timestep-dependent utility of generator states raises the possibility of adaptively selecting supervision layers per task or horizon, turning the multi-level target into a learned or curriculum-based choice.
Editorial extension: since R2G shows that the representation reorganizes predictive structure into coordinates a finite-capacity generator can use, one could test whether the same representation serves as a shared latent interface across multiple generators or as a compressed memory for long-horizon tasks.
Editorial extension: a direct test of counterfactual consistency would measure, under intervention, whether the representation's change is aligned with the altered scene's effect on the future, rather than merely observing coordinated action and generation shifts qualitatively.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A well-executed empirical recipe for amortizing a video generator into a current-only policy; the central interpretation as internalizing future computation is plausible but not fully pinned down, and the self-conditioning loop needs a direct test.

read the letter

The short version: Enfold is a useful, clearly-written empirical paper. It proposes a new interface for using video generators in control — distill multi-level hidden states from a teacher-forced generator into a current-only encoder, then run only the encoder at deployment. I'd rather see this in the literature than not. But the central interpretation — that this transfers the generator's future-constructing computation — is underdetermined by the evidence, and the self-conditioning loop is a real worry that the paper does not fully close.

What's genuinely new: prior work gets predictive information from encoders, video-model features, caches, or coupled training. Enfold instead supervises the current-only encoder with the generator's hidden states across depth and corruption timestep, using a two-way G2R/R2G coupling with detached readouts. The method is clearly specified, the ablations are well-designed (multi-level vs single-level vs pixels vs actions), and the latency reduction is substantial — 3.7x, 10.1x in Flash mode — with competitive performance on LIBERO and RoboTwin2.0. The authors are also unusually candid: they explicitly say the RoboTwin2.0 margins are too small to rank, the intervention evidence is qualitative, and the probe dataset is small. That honesty earns real credit.

The soft spot is the one flagged in the stress-test note. Eq. (3) computes the teacher states conditioned on \bar{z}=sg(z), so the target is partly the student's own output. Without a fixed-point or collapse analysis, it remains possible that the G2R loss mostly teaches the student to predict the generator re-encoding its own conditioning, not anything about the future. The indirect evidence — R2G improving future prediction over a video-only baseline, lower nuisance sensitivity, higher effective rank — is suggestive but does not directly establish that future-dependent computation is being transferred. The paper's own Appendix A stops the target-producing branch and disclaims a convergence claim for the joint system, which is exactly where collapse would live. This is a genuine gap, though not a reason to throw the paper out.

Who should read this: anyone working on world models for control or efficient action prediction. It gives a concrete recipe and a competitive accuracy/latency trade-off. The conceptual claim should be treated as an open hypothesis rather than established fact.

Recommendation: yes, send it to peer review. It deserves a serious referee. The main revision requests should be: release code and data, report confidence intervals, and directly test the collapse concern — at minimum ablate the teacher's conditioning on the student (e.g., distil from a generator that never receives \bar{z}) to show the transferred signal is not just the self-conditioning loop. A scaling or failure analysis would be nice but is secondary.

Referee Report

3 major / 5 minor

Summary. The paper introduces Enfold, a framework that trains a current-only encoder U_phi(c,e) to predict multi-level hidden states of a video generator during a teacher-forced pass over the observed future, while the generator is additionally conditioned on the detached encoder output. At deployment the generator is removed from the action path; actions are read directly from the predictive representation. On LIBERO, RoboTwin2.0, and four real-robot tasks, Enfold achieves competitive success rates while reducing action latency by 3.7x (134 ms) and, in TensorRT-accelerated Flash mode, 10.1x (49 ms) relative to Fast-WAM. Auxiliary analyses probe representation stability, future-feature predictability, and adaptation to human intervention.

Significance. If the transfer mechanism is taken at face value, the paper offers a useful efficiency-oriented interface: predictive control representations distilled from generative world-model states. The experimental design is strong in several respects: the supervision-target ablation (Table 4), the bidirectional R2G check (Table 5), the representation diagnostics, and the explicit acknowledgment of qualitative intervention evidence and of the unrankable RoboTwin margins. However, the central claim that future-generative computation is internalized rests on a closed loop in Eqs. (3)-(5), and Appendix A explicitly disclaims joint-convergence analysis. The paper's own candor about this and about missing uncertainty estimates makes the remaining gaps clear; they are fixable with additional analysis and experiments.

major comments (3)

[§4.2-4.3, Eqs. (3)-(5), Appendix A] The G2R target is defined using a generator conditioned on \bar{z}=sg(z), and R2G explicitly trains that generator to rely on \bar{z}. This closes a loop: the student is regressed on generator states that are partly a function of the student's own output. Appendix A's predictive-projection analysis holds the target-producing branch fixed and explicitly disclaims a convergence claim for the jointly evolving system. This is exactly the regime in which L_G2R could be minimized by predicting the generator's re-encoding of its own conditioning rather than by internalizing future-dependent computation. The indirect evidence in Table 5 and Figure 6 is suggestive but does not isolate the mechanism; the R2G gain could reflect better context encoding, and the G2R loss could be dominated by \bar{z}-explainable variance. Please provide a collapse analysis or a loop-breaking ablation, for example com
[§5.2, Tables 1-3] No confidence intervals or repeated-seed statistics are reported. The authors themselves state that the RoboTwin2.0 margins among Enfold, Fast-WAM, and LingBot-VA are too small to support a ranking claim without uncertainty estimates (Table 2: 91.77 vs 91.83 vs 92.20), yet no such estimates are provided. The LIBERO average gap over Fast-WAM is 0.2 points (97.8 vs 97.6) and per-suite differences are similarly within plausible noise; the real-robot comparison in Table 3 uses N=30 rollouts but reports no CI. Report bootstrap CIs or multiple-seed means for the main comparisons, or soften the comparative wording. This is necessary before the control-quality comparisons can be evaluated.
[§5.3 and §B.5] The intervention evidence is based on two qualitative episodes and is used to argue against fixed trajectory replay. The paper itself notes that no recovery rate is estimated and no matched no-intervention baseline is reported. Since this section is framed as 'Reimagining and Recovery after Intervention,' it should either be explicitly labeled as anecdotal illustration or be supported by a quantitative intervention protocol, e.g., success/recovery rate over multiple interventions with a matched no-intervention condition. As written, it cannot bear the weight of the counterfactual-consistency claim.

minor comments (5)

[§B.4] The text refers to 'Equation 25' for effective rank, but Eq. (25) is defined in Appendix C.4. Please add a cross-reference to the correct location.
[Table 1] Several entries have missing spacing, e.g., '98.298.896.9' in the pi0.5 row. Also, the notation 'Fast-WAM' and 'Fast–W AM' is used inconsistently.
[Figure 6 caption] Typo: 'F rom stochastic teacher features' should be 'From stochastic teacher features.'
[§B.1] The language features are said to come from Cosmos-Reason1-7B, while the generator is Cosmos-Predict 2.5 2B. Clarify the relationship between these components.
[§5.3] The phrase 'reimagining and recovery' overstates the qualitative protocol. Consider retitling as 'intervention-conditioned replanning (illustrative)' to match the evidence.

Circularity Check

1 steps flagged

G2R teacher states are computed from the student's own detached representation, so the central 'future-generative computation transfer' claim is partially self-referential; the paper's own Appendix A disclaims convergence analysis for the joint system.

specific steps

self definitional [Section 4.2, Eqs. (3)-(4); Appendix A]
"While processing y_t under the current context, instruction, and detached representation, the generator produces Hθ(yt,t,c,e,¯z) = {hG_{ℓ,t}}, r^G_t = LN(A(Hθ(yt,t,c,e,¯z))), (3) ... It is not a convergence claim for the jointly evolving training system."

G2R's target is not defined purely by the observed future: Eq. (3) feeds the student's detached representation z̄=sg(z) into the generator, and Eq. (4) trains z to predict r^G_t. Thus part of the 'future-generative computation' distilled is the generator re-encoding the student's conditioning, making the prediction partly a fixed-point of z. R2G (Eq. 5) separately trains the generator to rely on z̄, reinforcing the loop. Appendix A analyzes G2R only with the target-producing branch fixed and says it 'is not a convergence claim for the jointly evolving training system,' so no collapse/fixed-point analysis rules out the trivial fixed point. Control scores and ablations show z reorganizes context, but do not isolate how much G2R gain is future-dependent versus self-reconstruction.

full rationale

The paper's central claim is that multi-level generator states expose future-generative computation that can be transferred to a current-only encoder. The transfer path is load-bearing and partially circular: Eq. (3) defines the teacher target r^G_t as a function of the generator conditioned on z̄=sg(z), the student's own representation, and Eq. (4) trains z to predict that target. This is a self-conditioning loop, and the paper provides no fixed-point or collapse analysis; Appendix A explicitly says its predictive-projection argument is 'not a convergence claim for the jointly evolving training system.' Because the target-producing branch is not held fixed across training and the generator is itself trained by R2G to use z̄, the G2R signal can be dominated by the student re-encoding itself. The external benchmarks, R2G-vs-video-only improvement, and representation probes provide independent evidence that z has useful structure, but they do not establish that the structure is future-generative rather than self-conditioning. No load-bearing self-citation chain or imported uniqueness theorem was found; the self-citations in related work are not load-bearing. Weighing the explicit limitation in Appendix A, the central prediction partially reduces by construction, so the score is 6 rather than 0-2.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The method's dependency stack is: pretrained Cosmos generator + DINOv3 encoder (external, fixed), flow matching (standard), the non-collapse of the G2R/R2G loop (ad hoc, empirically probed), and the validity of the fixed-target projection interpretation (ad hoc, explicitly disclaimed as a formal proof). Free parameters are architecture and loss choices, several tuned on a 20-video probe.

free parameters (5)

G2R supervision layers = {7,15,23,27}
Selected via the 20-video generator-state probe (Figures 2–3, Appendix C); the paper notes no single layer is universally best, so the multi-level set is a design choice that materially defines the target in Eq. (3).
DINO feature concat layers = {7,15,23,31}
Chosen for the predictive head input (Table 6); no selection analysis is provided, so it is a hand-set architectural choice.
Loss weights λ_R2G, λ_task = 1, 1
Set to unit coefficients in Eq. (7); no ablation or sweep is reported.
Noise shift for flow-matching action sampling = 5.0
Hyperparameter used in training and inference (Table 6); typical for flow-matching policies but hand-set.
Prediction-head capacity = 10 layers, 5120→1024→8192
Depth and width of the timestep-conditioned head F_ω (Eq. 4, Table 6); chosen without reported tuning analysis.

axioms (6)

standard math Flow matching (Lipman et al. 2022) is a valid objective for both action generation and video latent generation (Eq. 5, Table 6).
Assumed from prior literature; used without modification.
domain assumption Cosmos-Predict 2.5 2B and DINOv3 ViT-H+/16 are fixed pretrained models whose internal features are suitable for the G2R/R2G distillation (Section B.1).
The method runs inside these two backbones; no experiments with alternative generators or encoders are reported.
ad hoc to paper The teacher-forced generator states form a non-degenerate target: they carry future- and transition-relevant structure that is predictable from (c,e) and do not collapse to a function of \bar{z} alone (Eq. 3).
Core premise of G2R; the paper provides indirect empirical evidence (probes, ablation, R2G improvement) but no formal guarantee against bootstrap collapse.
ad hoc to paper The fixed-target population analysis in Appendix A (Eqs. 9–12) approximately describes the trained system, despite the target-producing branch changing during training.
The paper explicitly labels it an idealized interpretation, so relying on it to interpret G2R is an extra assumption.
domain assumption Cosmos-Reason1-7B language features, linearly projected into the DINO stream from block 8, provide an effective instruction conditioning (Section B.1).
External language encoder; no alternative language conditioning is studied.
domain assumption The comparison baselines (Fast-WAM, Motus, π0.5, LingBot-VA, etc.) are evaluated under comparable conditions because 'all methods use the same initial-state sampler, trial allocation, camera observations, and termination limits within each setting' (Appendix B.2).
The paper asserts protocol matching, but without released code or checkpoints we cannot verify the comparability of transcribed baselines.

pith-pipeline@v1.3.0-daily-deepseek · 22526 in / 17928 out tokens · 178202 ms · 2026-08-01T11:21:41.421620+00:00 · methodology

0 comments

read the original abstract

World generative models are typically used through what they produce: a rendered future, a video-conditioned action, or latent context computed by a costly generative branch. We argue that their more reusable asset is the computation that constructs a future. As a generator transforms a corrupted future into a coherent trajectory, its intermediate states organize appearance, spatial layout, and interaction across levels of abstraction. Can this future-generative computation be internalized in a representation inferred from the present alone? We present Enfold, which transfers this computation into a representation predicted from the current visual context and language instruction. During training, multi-level states exposed as the generator processes the observed future supervise a current-only encoder. The learned representation is fed back to condition future generation and is read by task heads without allowing task gradients to reshape the encoder. At deployment, action prediction no longer executes the generator. Across LIBERO, RoboTwin2.0, and real-robot tasks, Enfold supports strong control while reducing action latency by $3.7\times$ relative to Fast--WAM, Enfold-Flash reaches $10.1\times$. Representation analyses show that it suppresses nuisance variation and preferentially captures changes that emerge over longer horizons. When the current scene is altered by human intervention, both the generated continuation and the executed actions adapt, which is inconsistent with fixed trajectory replay. These results recast a world generator as a source of predictive control representations: its future need not be materialized at every step if its internal structure can be enfolded into the present.

Figures

Figures reproduced from arXiv: 2607.26657 by Antao Xiang, Chengqun Yang, Feng Tian, Fulong Liu, Jingnan Gao, Jisong Cai, Weili Zeng, Xiaokang Yang, Xiaomin Wu, Xin Wang, Yao Mu, Yichao Yan, Yitong Xing.