Pith. sign in

REVIEW 3 major objections 74 references

A single unified multimodal model that jointly predicts actions and future frames outperforms task-specific robot policies and learns more structured action representations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 02:18 UTC pith:UO2R7SRW

load-bearing objection Solid systems paper with a reusable Fourier-action recipe and clean multi-benchmark numbers; the “unification itself” claim is only partially isolated. the 3 major comments →

arxiv 2607.03461 v1 pith:UO2R7SRW submitted 2026-07-03 cs.CV cs.LG

WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modeling

Zelin Zhao , Min Shi , Bo Yuan , Haotian Xue , Jialuo Li , Lama Moukheiber , Humphrey Shi , Yongxin Chen This is my paper
classification cs.CV cs.LG
keywords world modelsvision-language-actionVLAWunified multimodal modelsrobotic manipulationFourier feature action tokenizationaction-conditioned generationsequence plans
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that modern unified vision–language models can serve as full Vision–Language–Action–World (VLAW) models, not just perception or generation engines. The authors build WorldBagel on a two-tower multimodal backbone so that one network performs language-grounded understanding, continuous action prediction, and action-conditioned future-image generation in a shared token space. Across multi-task robotic manipulation benchmarks, the unified model beats strong task-specific and separately weighted baselines, produces action embeddings that better encode task identity, and stays more stable when action dynamics are perturbed. The practical claim is that architectural unification itself supplies complementary supervision that improves both control success and predictive world modeling, rather than being a mere engineering convenience.

Core claim

Unification is a key factor in learning effective VLAW models: a single BAGEL-based system that jointly handles multimodal understanding, Fourier-tokenized action prediction, and action-conditioned future-frame generation consistently outperforms task-specific alternatives on multi-task robotic manipulation and yields action representations that are more structured, semantically aligned with vision and language, and more stable under distribution shifts in action dynamics.

What carries the argument

Fourier Feature Action Tokenizer and Decoder (FFAT/FFAD): continuous robot actions are mapped into multi-frequency sine–cosine features, embedded as tokens in the shared multimodal space for generation, and inverted by phase-consistent averaging for prediction; together with interleaved sequence plans that mix observations, language, actions, and future frames inside one autoregressive sequence.

Load-bearing premise

The performance gains come mainly from sharing one unified architecture, not from the many other design choices that also differ from the closest baseline (Fourier action coding, priority sampling of joint plans, loss weights, and full fine-tuning of both experts).

What would settle it

Train an otherwise identical system that keeps the Fourier action modules, sequence-plan sampling, and loss weights but uses completely separate weights for the action-prediction branch and the world-modeling branch; if the unified model no longer beats that controlled separate-weight baseline on LIBERO average success and on action-conditioned FVD/PSNR, the claim that unification itself is the key factor fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Jointly training action prediction and future-frame generation inside one model supplies complementary gradients that raise multi-task success rates on language-conditioned manipulation.
  • Fourier feature action tokens produce embeddings that linear probes can read task identity from far more accurately than regression, bin, or FAST baselines.
  • Action-conditioned world models remain more accurate under action noise, scaling, and temporal shifts than models that mainly replay visual history.
  • Sequence-plan sampling that prioritizes joint VLAW objectives over pure policy objectives stabilizes heterogeneous multimodal training.
  • The same backbone can serve both open-loop multi-step control and predictive environment imagination without a separate world-model network.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If unification truly regularizes action representations, the same Fourier-token plus shared-backbone pattern may transfer to other continuous control domains (mobile robots, humanoids) without re-deriving discrete action codebooks.
  • Priority sampling of joint plans resembles prioritized experience replay; the same reweighting idea could be tested for balancing safety-critical versus routine trajectories in real-robot data mixtures.
  • Eigenvalue-spectrum richness under distribution shift suggests a diagnostic: future work could track effective rank of action embeddings online as a cheap proxy for world-model robustness before deploying a policy.
  • Because reconstruction from Fourier features is locally Lipschitz, small prediction errors stay small in action space; this may reduce compounding open-loop error relative to hard discretization, a testable claim on longer-horizon suites.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. WorldBagel extends BAGEL’s two-tower GEN/UND architecture into a Vision–Language–Action–World (VLAW) model that jointly performs multimodal understanding, continuous action prediction, and action-conditioned future-frame generation. Continuous actions are mapped via multi-frequency Fourier features (FFAT for conditioning, FFAD for prediction/reconstruction), multimodal sequences are organized by sampled sequence plans, and heterogeneous data are balanced with mixture and priority sampling. On LIBERO, Language Table, and Franka the model reports multi-task success rates up to 98.0% average on LIBERO and improved action-conditioned world-model metrics (FVD/PSNR/SSIM/LPIPS) relative to RynnVLA-002 and strong VLA baselines, plus ablations and eigenvalue/linear-probe analyses arguing for more structured, shift-stable action representations.

Significance. If the results hold under cleaner controls, the paper is a useful empirical demonstration that a modern unified multimodal backbone can be adapted to joint VLAW modeling without a separate action expert, and that Fourier action tokens improve continuous control and representation structure relative to regression, binning, and FAST. The appendix Lipschitz/injectivity/reconstruction arguments for the Fourier map, the planned code/checkpoint release, and the multi-benchmark evaluation (including distribution-shift tests) are concrete strengths. The work sits at a timely intersection of unified VLMs and robotic world models; even a carefully qualified version would be of interest to the VLA/world-model community.

major comments (3)
  1. Section 4.2 and Table 2 attribute the +0.6% LIBERO average edge over RynnVLA-002 (98.0 vs 97.4) and the world-model gains in Table 3 primarily to architectural unification (shared backbone / joint VLAW training vs separate VLA and world-model weights). The systems also differ in action representation (FFAD/FFAT vs bin discretization), priority sequence-plan sampling (w_joint=2), L_vision weight 0.1, and full expert fine-tuning. Table 4 only varies components inside WorldBagel; it never re-implements the separate-weight baseline with FFAD/FFAT (or WorldBagel with bin/FAST tokens and decoupled heads). Without that isolation, the central claim that “unification itself is a key factor” is not supported by the reported comparisons and should be either experimentally controlled or substantially softened in the abstract, introduction, and Section 4.2.
  2. The manuscript states results are averaged over five seeds (Section 4.1) yet Tables 2–4 report point estimates only, with no standard deviations or significance tests. The headline multi-task margin is 0.6 percentage points on already saturated suites (OpenVLA-OFT 97.1, RynnVLA-002 97.4). Without error bars it is impossible to judge whether the multi-task and world-model improvements are reliable; please add seed-wise variance (or bootstrap CIs) to Tables 2–3 and the stability panels 4e–g, and qualify claims that depend on sub-1% gaps.
  3. Section 3.5 and the axiom that priority sampling of joint VLAW plans (w_joint > w_policy) improves both control and world-model metrics is load-bearing for the training recipe, but the main experiments fix w_joint=2, w_policy=1 with no ablation against uniform plan sampling or policy-only plans. An ablation of plan priorities (and of the 0.1 vision-loss weight) on LIBERO success and Table 3 metrics is needed to justify the claimed training design, or the claim should be demoted to a hyperparameter choice.

Circularity Check

0 steps flagged

No circularity: empirical multi-task success rates and world-model metrics on held-out suites do not reduce by construction to any fitted inputs or self-definitional premises.

full rationale

The paper's load-bearing claims are experimental comparisons (LIBERO average success 98.0 % vs. RynnVLA-002 97.4 %, action-conditioned FVD/PSNR/SSIM/LPIPS improvements on three benchmarks, linear-probe accuracy of action embeddings, eigenvalue-spectrum stability under action noise/scaling/temporal shift). These quantities are measured after supervised fine-tuning on trajectory data and evaluated on held-out task suites; they are not algebraic rearrangements of parameters fitted to the same quantities. The Fourier Feature Action Tokenizer/Decoder is introduced by explicit construction (Eqs. 2–6) and then analyzed in Appendix A via standard Lipschitz, injectivity, phase-reconstruction, and Stone–Weierstrass arguments; the theorems establish properties of that construction rather than deriving a target result from a fit that already encodes it. Sequence-plan sampling and loss weighting (L = L_action + 0.1 L_vision) are design choices whose effects are ablated inside the same architecture (Table 4); they do not force the reported numbers by definition. Citations to BAGEL and concurrent VLAW systems supply architectural starting points and baselines, not uniqueness theorems or ansatzes that smuggle the final performance claims. Consequently the derivation chain contains no self-definitional loop, no fitted-input-called-prediction, and no load-bearing self-citation reduction.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 3 invented entities

The load-bearing empirical claim rests on a handful of hand-chosen hyper-parameters (Fourier band count, loss weight, plan priorities) and on the domain assumption that BAGEL’s pretrained two-tower weights remain a suitable inductive bias after full fine-tuning on robot trajectories. No new physical entities are postulated; the invented components are engineering modules whose utility is measured by the reported tables.

free parameters (4)
  • Fourier band count K = 32
    Chosen by ablation (Table 4b/c); K=32 is selected as the operating point that maximizes success/PSNR.
  • vision loss weight = 0.1
    Fixed coefficient 0.1 in the joint loss L = L_action + 0.1 L_vision (Eq. 1); not derived.
  • sequence-plan priority weights = w_joint=2, w_policy=1
    w_joint=2, w_policy=1 set by hand to emphasize joint VLAW plans (Section 3.5).
  • learning rate / batch / steps = 2e-5 / 32 / 80k
    AdamW 2e-5, batch 32, 80k steps; standard but free choices that affect final numbers.
axioms (3)
  • standard math Fourier feature embeddings of continuous actions are Lipschitz and approximately injective on the bounded action domain, and therefore suitable for both prediction and conditioning.
    Appendix Theorems 1–3 restate classical properties of multi-frequency sine/cosine maps; used to justify FFAD/FFAT.
  • domain assumption BAGEL’s pretrained UND/GEN two-tower weights provide a useful inductive bias for robotic VLAW after supervised fine-tuning.
    All experiments initialize from a BAGEL checkpoint and freeze only the vision/language tokenizers (Section 3.1).
  • ad hoc to paper Priority sampling of joint VLAW sequence plans (w_joint > w_policy) improves both control and world-model metrics relative to uniform sampling.
    Stated as design choice in Section 3.5; no exhaustive ablation of alternative weightings is provided.
invented entities (3)
  • Fourier Feature Action Decoder (FFAD) no independent evidence
    purpose: Map continuous actions into multi-frequency features, predict in Fourier space, and invert via phase averaging.
    New adapter module (33 M parameters) whose superiority is claimed via Table 4a/d; independent evidence is limited to the paper’s own ablations.
  • Fourier Feature Action Tokenizer (FFAT) no independent evidence
    purpose: Encode continuous actions as conditioning tokens for the GEN expert’s future-frame prediction.
    Companion module to FFAD; consistency between prediction and conditioning is asserted but not independently verified outside the paper.
  • VLAW sequence plans no independent evidence
    purpose: Define flexible interleavings of multi-view observations, language, multi-step actions, and future frames inside one autoregressive sequence.
    Extension of BAGEL’s sequence-plan concept to robotics; utility measured only by the reported training runs.

pith-pipeline@v1.1.0-grok45 · 22462 in / 3146 out tokens · 30505 ms · 2026-07-12T02:18:48.534795+00:00 · methodology

0 comments
read the original abstract

World models aim to capture environment dynamics in ways that support perception, reasoning, and action, and have recently become a central direction in Vision-Language-Action-World (VLAW) modeling. Meanwhile, unified vision-language models have demonstrated strong multimodal generation capabilities, yet their potential as world models remains underexplored. In this work, we introduce \texttt{WorldBagel}, a unified VLAW framework built on BAGEL, a modern multimodal unified model, and use it to systematically investigate the role of unification in world modeling. Across multi-task robotic manipulation and cross-domain experiments, \texttt{WorldBagel} consistently outperforms task-specific alternatives and learns action representations that are more structured and semantically aligned with visual and linguistic context. Experiments on LIBERO, Language Table, and Franka show that unification is not only an architectural convenience, but also a key factor in learning effective VLAW models, leading to consistent empirical gains and deeper insights into multimodal world modeling. Code and model checkpoints will be released upon acceptance.

Figures

Figures reproduced from arXiv: 2607.03461 by Bo Yuan, Haotian Xue, Humphrey Shi, Jialuo Li, Lama Moukheiber, Min Shi, Yongxin Chen, Zelin Zhao.

Figure 1
Figure 1. Figure 1: Overview of WorldBagel. The model is built on a shared multimodal backbone with two experts: an Understanding (UND) expert and a Generation (GEN) expert, operating over a unified token embedding space. Visual observations and language instructions are encoded by a ViT encoder and a language tokenizer, while continu￾ous control actions are represented using a Fourier Feature Action Tokenizer (FFAT). WorldBa… view at source ↗
Figure 2
Figure 2. Figure 2: World modeling results on three environments: LIBERO [31], Language Ta￾ble [32] and Franka [34]. Please refer to the supplementary to see more video demos. the action embeddings from the trained policy by taking the decoder input rep￾resentation corresponding to each action step. Using these frozen embeddings, we train a linear classifier (linear probe) to predict the task identity among the LIBERO task su… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

74 extracted references · 31 linked inside Pith

  1. [1]

    Advances in neural information processing systems35, 23716– 23736 (2022)

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Men- sch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems35, 23716– 23736 (2022)

    2022

  2. [2]

    arXiv preprint arXiv:2506.09985 (2025)

    Assran, M., Bardes, A., Fan, D., Garrido, Q., Howes, R., Muckley, M., Rizvi, A., Roberts, C., Sinha, K., Zholus, A., et al.: V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985 (2025)

    Pith/arXiv arXiv 2025

  3. [3]

    arXiv preprint arXiv:2510.19818 (2025)

    Berg, J., Zhu, C., Bao, Y., Durugkar, I., Gupta, A.: Semantic world models. arXiv preprint arXiv:2510.19818 (2025)

    arXiv 2025

  4. [4]

    arXiv preprint arXiv:2212.06817 (2022)

    Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakr- ishnan, K., Hausman, K., Herzog, A., Hsu, J., et al.: Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 (2022)

    Pith/arXiv arXiv 2022

  5. [5]

    arXiv preprint arXiv:2505.06111 (2025)

    Bu, Q., Yang, Y., Cai, J., Gao, S., Ren, G., Yao, M., Luo, P., Li, H.: Uni- vla: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111 (2025)

    Pith/arXiv arXiv 2025

  6. [6]

    Cen, J., Huang, S., Yuan, Y., Li, K., Yuan, H., Yu, C., Jiang, Y., Guo, J., Li, X., Luo, H., Wang, F., Zhao, D., Chen, H.: Rynnvla-002: A unified vision-language- action and world model (2025),https://arxiv.org/abs/2511.17502

    Pith/arXiv arXiv 2025

  7. [7]

    arXiv preprint arXiv:2510.13778 (2025)

    Chen, X., Chen, Y., Fu, Y., Gao, N., Jia, J., Jin, W., Li, H., Mu, Y., Pang, J., Qiao, Y., et al.: Internvla-m1: A spatially guided vision-language-action framework for generalist robot policy. arXiv preprint arXiv:2510.13778 (2025)

    Pith/arXiv arXiv 2025

  8. [8]

    The International Journal of Robotics Research44(10-11), 1684–1704 (2025)

    Chi,C.,Xu,Z.,Feng,S.,Cousineau,E.,Du,Y.,Burchfiel,B.,Tedrake,R.,Song,S.: Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research44(10-11), 1684–1704 (2025)

    2025

  9. [9]

    CoRRabs/2510.26583(2025).https: //doi.org/10.48550/ARXIV.2510.26583

    Cui, Y., Chen, H., Deng, H., Huang, X., Li, X., Liu, J., Liu, Y., Luo, Z., Wang, J., Wang, W., Wang, Y., Wang, C., Zhang, F., Zhao, Y., Pan, T., Li, X., Hao, Z., Ma, W., Chen, Z., Ao, Y., Huang, T., Wang, Z., Wang, X.: Emu3.5: Native multimodal models are world learners. CoRRabs/2510.26583(2025).https: //doi.org/10.48550/ARXIV.2510.26583

    Pith/arXiv arXiv doi:10.48550/arxiv.2510.26583 2025

  10. [10]

    arXiv preprint arXiv:2505.14683 (2025)

    Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., et al.: Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683 (2025)

    Pith/arXiv arXiv 2025

  11. [11]

    The C Users Journal12(2), 23–38 (1994)

    Gage, P.: A new algorithm for data compression. The C Users Journal12(2), 23–38 (1994)

    1994

  12. [12]

    arXiv preprint arXiv:2602.06949 (2026)

    Gao,S.,Liang,W.,Zheng,K.,Malik,A.,Ye,S.,Yu,S.,Tseng,W.C.,Dong,Y.,Mo, K., Lin, C.H., et al.: Dreamdojo: A generalist robot world model from large-scale human videos. arXiv preprint arXiv:2602.06949 (2026)

    Pith/arXiv arXiv 2026

  13. [13]

    arXiv preprint arXiv:2510.10125 (2025)

    Guo, Y., Shi, L.X., Chen, J., Finn, C.: Ctrl-world: A controllable generative world model for robot manipulation. arXiv preprint arXiv:2510.10125 (2025)

    Pith/arXiv arXiv 2025

  14. [14]

    CoRRabs/1803.10122(2018),http: //arxiv.org/abs/1803.10122

    Ha, D., Schmidhuber, J.: World models. CoRRabs/1803.10122(2018),http: //arxiv.org/abs/1803.10122

    Pith/arXiv arXiv 2018

  15. [15]

    In: International Conference on Learning Representations (2020)

    Hafner, D., Lillicrap, T., Ba, J., Norouzi, M.: Dream to control: Learning behaviors by latent imagination. In: International Conference on Learning Representations (2020)

    2020

  16. [16]

    In: International Conference on Machine Learning

    Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., Davidson, J.: Learning latent dynamics for planning from pixels. In: International Conference on Machine Learning. pp. 2555–2565. PMLR (2019) WorldBagel17

    2019

  17. [17]

    Hafner, D., Yan, W., Lillicrap, T.: Training agents inside of scalable world models (2025),https://arxiv.org/abs/2509.24527

    Pith/arXiv arXiv 2025

  18. [18]

    arXiv preprint arXiv:2410.15959 (2024)

    Hou, Z., Zhang, T., Xiong, Y., Pu, H., Zhao, C., Tong, R., Qiao, Y., Dai, J., Chen, Y.: Diffusion transformer policy. arXiv preprint arXiv:2410.15959 (2024)

    Pith/arXiv arXiv 2024

  19. [19]

    arXiv preprint arXiv:2602.09849 (2026)

    Hu, Y., Zhang, J., Luo, Y., Guo, Y., Chen, X., Sun, X., Feng, K., Lu, Q., Chen, S., Zhang, Y., et al.: Bagelvla: Enhancing long-horizon manipulation via interleaved vision-language-action generation. arXiv preprint arXiv:2602.09849 (2026)

    arXiv 2026

  20. [20]

    In: The Thirty- ninth Annual Conference on Neural Information Processing Systems

    Huang, C.P., Wu, Y.H., Chen, M.H., Wang, Y.C.F., Yang, F.E.: Thinkact: Vision- language-action reasoning via reinforced visual latent planning. In: The Thirty- ninth Annual Conference on Neural Information Processing Systems

  21. [21]

    Intelligence, P., Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Galliker, M.Y., Ghosh, D., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., LeBlanc, D., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Ren, A.Z., Shi, L.X., Smith, L., Springenberg, J.T., Stachow...

    Pith/arXiv arXiv 2025

  22. [22]

    In: International Conference on Machine Learning

    Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., Carreira, J.: Per- ceiver: General perception with iterative attention. In: International Conference on Machine Learning. pp. 4651–4664. PMLR (2021)

    2021

  23. [23]

    In: Conference on Robot Learning

    Jia, X., Wang, Q., Donat, A., Xing, B., Li, G., Zhou, H., Celik, O., Blessing, D., Lioutikov,R.,Neumann,G.:Mail:Improvingimitationlearningwithselectivestate space models. In: Conference on Robot Learning. pp. 3888–3907. PMLR (2025)

    2025

  24. [24]

    Jiang, M., Wu, Y., Zhao, T., Zhao, Z., Lu, C.: Pointsift: A sift-like network module for 3d point cloud semantic segmentation (2018),https://arxiv.org/abs/1807. 00652

    2018

  25. [25]

    arXiv preprint arXiv:2502.19645 (2025)

    Kim, M.J., Finn, C., Liang, P.: Fine-tuning vision-language-action models: Opti- mizing speed and success. arXiv preprint arXiv:2502.19645 (2025)

    Pith/arXiv arXiv 2025

  26. [26]

    arXiv preprint arXiv:2601.16163 (2026)

    Kim, M.J., Gao, Y., Lin, T.Y., Lin, Y.C., Ge, Y., Lam, G., Liang, P., Song, S., Liu, M.Y., Finn, C., et al.: Cosmos policy: Fine-tuning video models for visuomotor control and planning. arXiv preprint arXiv:2601.16163 (2026)

    Pith/arXiv arXiv 2026

  27. [27]

    arXiv preprint arXiv:2406.09246 (2024)

    Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al.: Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246 (2024)

    Pith/arXiv arXiv 2024

  28. [28]

    CoRRabs/2507.16746(2025)

    Li, A., Wang, C., Yue, K., Cai, Z., Liu, O., Fu, D., Guo, P., Zhu, W.B., Sharan, V., Jia, R., Neiswanger, W., Huang, F., Goldstein, T., Goldblum, M.: Zebra-cot: A dataset for interleaved vision language reasoning. CoRRabs/2507.16746(2025). https://doi.org/10.48550/ARXIV.2507.16746,https://doi.org/10.48550/ arXiv.2507.16746

    doi:10.48550/arxiv.2507.16746 2025

  29. [29]

    In: International Conference on Machine Learning

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International Conference on Machine Learning. pp. 19730–19742. PMLR (2023)

    2023

  30. [30]

    Liu,B.,Gong,X.,Zhao,Z.,Song,Z.,Lu,Y.,Wu,S.,Zhang,J.,Banerjee,S.,Zhang, H.: Byteloom: Weaving geometry-consistent human-object interactions through progressive curriculum learning (2026),https://arxiv.org/abs/2512.22854

    arXiv 2026

  31. [31]

    Advances in Neural Information Processing Systems36, 44776–44791 (2023)

    Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y., Stone, P.: Libero: Benchmark- ing knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems36, 44776–44791 (2023)

    2023

  32. [32]

    In: Conference on robot learning

    Lynch, C., Khansari, M., Xiao, T., Kumar, V., Tompson, J., Levine, S., Sermanet, P.: Learning latent plans from play. In: Conference on robot learning. pp. 1113–

  33. [33]

    Pmlr (2020) 18 Z.Zhao et al

    2020

  34. [34]

    Commu- nications of the ACM65(1), 99–106 (2021)

    Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Commu- nications of the ACM65(1), 99–106 (2021)

    2021

  35. [35]

    NVIDIA: Isaac sim (2024),https://docs.isaacsim.omniverse.nvidia.com/ latest/index.html

    2024

  36. [36]

    arXiv preprint arXiv:2501.09747 (2025)

    Pertsch, K., Stachowicz, K., Ichter, B., Driess, D., Nair, S., Vuong, Q., Mees, O., Finn, C., Levine, S.: Fast: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747 (2025)

    Pith/arXiv arXiv 2025

  37. [37]

    arXiv preprint arXiv:2506.00613 (2025)

    Quevedo, J., Sharma, A.K., Sun, Y., Suryavanshi, V., Liang, P., Yang, S.: Worldgym: World model as an environment for policy evaluation. arXiv preprint arXiv:2506.00613 (2025)

    arXiv 2025

  38. [38]

    In: International Conference on Machine Learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748–8763. PMLR (2021)

    2021

  39. [39]

    arXiv preprint arXiv:2205.06175 (2022)

    Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S.G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J.T., et al.: A generalist agent. arXiv preprint arXiv:2205.06175 (2022)

    Pith/arXiv arXiv 2022

  40. [40]

    Robotics: Science and Systems XX

    Reuss, M., Yağmurlu, Ö.E., Wenzel, F., Lioutikov, R.: Multimodal diffusion trans- former: Learning versatile behavior from multimodal goals. Robotics: Science and Systems XX. Ed.: D. Kulic (2024)

    2024

  41. [41]

    arXiv preprint arXiv:1511.05952 (2015)

    Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. arXiv preprint arXiv:1511.05952 (2015)

    Pith/arXiv arXiv 2015

  42. [42]

    arXiv preprint arXiv:1511.09249 (2015)

    Schmidhuber, J.: On learning to think: Algorithmic information theory for novel combinations of reinforcement learning controllers and recurrent neural world mod- els. arXiv preprint arXiv:1511.09249 (2015)

    Pith/arXiv arXiv 2015

  43. [43]

    arXiv preprint arXiv:2506.01844 (2025)

    Shukor, M., Aubakirova, D., Capuano, F., Kooijmans, P., Palma, S., Zoui- tine, A., Aractingi, M., Pascal, C., Russi, M., Marafioti, A., et al.: Smolvla: A vision-language-action model for affordable and efficient robotics. arXiv preprint arXiv:2506.01844 (2025)

    Pith/arXiv arXiv 2025

  44. [44]

    In: Forty-second International Conference on Machine Learning (2025)

    Song, K., Chen, B., Simchowitz, M., Du, Y., Tedrake, R., Sitzmann, V.: History- guided video diffusion. In: Forty-second International Conference on Machine Learning (2025)

    2025

  45. [45]

    Song, Z., Gong, X., Liu, B., Zhao, Z.: Mv-s2v: Multi-view subject-consistent video generation (2026),https://arxiv.org/abs/2601.17756

    Pith/arXiv arXiv 2026

  46. [46]

    In: The Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 (2024)

    Sun, Q., Yu, Q., Cui, Y., Zhang, F., Zhang, X., Wang, Y., Gao, H., Liu, J., Huang, T., Wang, X.: Emu: Generative pretraining in multimodality. In: The Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 (2024)

    2024

  47. [47]

    Advances in neural informa- tion processing systems33, 7537–7547 (2020)

    Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Sing- hal, U., Ramamoorthi, R., Barron, J., Ng, R.: Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural informa- tion processing systems33, 7537–7547 (2020)

    2020

  48. [48]

    Team, C.: Chameleon: Mixed-modal early-fusion foundation models (2025),https: //arxiv.org/abs/2405.09818

    Pith/arXiv arXiv 2025

  49. [49]

    arXiv preprint arXiv:2507.21809 (2025)

    Team, H., Wang, Z., Liu, Y., Wu, J., Gu, Z., Wang, H., Zuo, X., Huang, T., Li, W., Zhang, S., et al.: Hunyuanworld 1.0: Generating immersive, explorable, and interactive 3d worlds from words or pixels. arXiv preprint arXiv:2507.21809 (2025)

    Pith/arXiv arXiv 2025

  50. [50]

    arXiv preprint arXiv:2405.12213 (2024) WorldBagel19

    Team, O.M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al.: Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213 (2024) WorldBagel19

    Pith/arXiv arXiv 2024

  51. [51]

    arXiv preprint arXiv:2503.20314 (2025)

    Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.W., Chen, D., Yu, F., Zhao, H., Yang, J., et al.: Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 (2025)

    Pith/arXiv arXiv 2025

  52. [52]

    Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., Luo, P.: Janus: Decoupling visual encoding for unified multimodal understand- ing and generation (2024),https://arxiv.org/abs/2410.13848

    Pith/arXiv arXiv 2024

  53. [53]

    In: The Thirteenth International Conference on Learning Repre- sentations (2025)

    Wu, Y., Zhang, Z., Chen, J., Tang, H., Li, D., Fang, Y., Zhu, L., Xie, E., Yin, H., Yi, L., et al.: Vila-u: a unified foundation model integrating visual understanding and generation. In: The Thirteenth International Conference on Learning Repre- sentations (2025)

    2025

  54. [54]

    arXiv preprint arXiv:2408.12528 (2024)

    Xie, J., Mao, W., Bai, Z., Zhang, D.J., Wang, W., Lin, K.Q., Gu, Y., Chen, Z., Yang, Z., Shou, M.Z.: Show-o: One single transformer to unify multimodal under- standing and generation. arXiv preprint arXiv:2408.12528 (2024)

    Pith/arXiv arXiv 2024

  55. [55]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Xie, R., Du, C., Song, P., Liu, C.: Muse-vl: Modeling unified vlm through semantic discrete encoding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 24135–24146 (2025)

    2025

  56. [56]

    Xue, H., Chen, Y., Ma, L., Zhao, Z., Moukheiber, L., Zhu, Y., Chen, Y.: Acwm- phys: Investigating generalized physical interaction in action-conditioned video world models (2026),https://arxiv.org/abs/2605.08567

    Pith/arXiv arXiv 2026

  57. [57]

    Yan, Z., Lin, K., Li, Z., Ye, J., Han, H., Wang, H., Wang, Z., Lin, B., Li, H., Xiao, X., Wang, J., Wang, H., Yuan, L.: Unified multimodal models as auto-encoders (2026),https://arxiv.org/abs/2509.09666

    arXiv 2026

  58. [58]

    Yuan, B., Zhao, Z., Molodyk, P., Hu, B., Chen, Y.: Clarify before you draw: Proac- tive agents for robust text-to-cad generation (2026),https://arxiv.org/abs/ 2602.03045

    Pith/arXiv arXiv 2026

  59. [59]

    In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024),https://openreview.net/forum? id=nBjmMF2IZU

    Zhai, Y., Bai, H., Lin, Z., Pan, J., Tong, S., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., Levine, S.: Fine-tuning large vision-language models as decision-making agents via reinforcement learning. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024),https://openreview.net/forum? id=nBjmMF2IZU

    2024

  60. [60]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhao, Z., Fan, F., Liao, W., Yan, J.: Grounding and enhancing grid-based models for neural fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19425–19435 (2024)

    2024

  61. [61]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Zhao, Z., Gan, C., Wu, J., Guo, X., Tenenbaum, J.B.: Augmenting policy learning with routines discovered from a single demonstration. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 11024–11032 (2021)

    2021

  62. [62]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings

    Zhao, Z., Gong, X., Liu, B., Song, Z., Zhang, J., Wu, S., Chen, Y., Zhang, H.: Cetcam: Camera-controllable video generation via consistent and extensible tok- enization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings. pp. 4749–4758 (June 2026)

    2026

  63. [63]

    Zhao, Z., Li, Z., Hassibi, K., Azizzadenesheli, K., Yan, J., Bae, H.J., Zhou, D., Anandkumar, A.: Physics-informed neural-operator predictive control for drag re- duction in turbulent flows (2025),https://arxiv.org/abs/2510.03360

    arXiv 2025

  64. [64]

    In: The Fourteenth International Confer- ence on Learning Representations (2026),https://openreview.net/forum?id= kdrc4o6okz

    Zhao, Z., Molodyk, P., Xue, H., Chen, Y.: Lapflow: Laplacian multi-scale flow matching for generative modeling. In: The Fourteenth International Confer- ence on Learning Representations (2026),https://openreview.net/forum?id= kdrc4o6okz

    2026

  65. [65]

    In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W

    Zhao, Z., Samel, K., Chen, B., song, l.: Proto: Program-guided transformer for program-guided tasks. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems. vol. 34, 20 Z.Zhao et al. pp. 17021–17036. Curran Associates, Inc. (2021),https://proceedings.neurips. cc / paper _ files / pa...

    2021

  66. [66]

    In: European Conference on Computer Vision

    Zhao, Z., Wu, Z., Zhuang, Y., Li, B., Jia, J.: Tracking objects as pixel-wise distri- butions. In: European Conference on Computer Vision. pp. 76–94. Springer (2022)

    2022

  67. [67]

    Zhao, Z., Yuan, B., Choi, J., Chen, Y.: Rma: an agentic system for research-level mathematical problems (2026),https://arxiv.org/abs/2605.22875

    Pith/arXiv arXiv 2026

  68. [68]

    In: The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 (2025)

    Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., Levy, O.: Transfusion: Predict the next token and diffuse images with one multi-modal model. In: The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 (2025)

    2025

  69. [69]

    In: International Conference on Machine Learning

    Zhou, G., Pan, H., Lecun, Y., Pinto, L.: Dino-wm: World models on pre-trained visual features enable zero-shot planning. In: International Conference on Machine Learning. pp. 79115–79135. PMLR (2025)

    2025

  70. [70]

    In: Conference on Robot Learning

    Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al.: Rt-2: Vision-language-action models transfer web knowledge to robotic control. In: Conference on Robot Learning. pp. 2165–2183. PMLR (2023) WorldBagel21 Appendix A FF AT and FF AD Mathematical Properties In this section we provide theoretical ana...

    2023

  71. [71]

    stability of the Fourier action representation,

  72. [72]

    information preservation of the Fourier embedding,

  73. [73]

    Throughout this section we assume actions are bounded in a compact interval a∈ A ⊂[−1,1] d which is standard for robot control signals

    consistency of the Fourier-based reconstruction used in FFAD. Throughout this section we assume actions are bounded in a compact interval a∈ A ⊂[−1,1] d which is standard for robot control signals. A.1 Stability of the Fourier Action Representation We first show that the Fourier embedding used by FFAT is Lipschitz continuous, ensuring robustness to small ...

  74. [74]

    A.2 Information Preservation of Fourier Features Next we show that the Fourier embedding retains sufficient information about the action

    (13) Taking square roots completes the proof. A.2 Information Preservation of Fourier Features Next we show that the Fourier embedding retains sufficient information about the action. Theorem 2 (Injectivity on Bounded Domains). Assumea∈(−1,1)andK≥1. Then the Fourier embeddingϕ(a)is injective almost everywhere onA. Proof.Suppose ϕ(a1) =ϕ(a 2). Then for eve...