Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

Dreamweaver: Learning Compositional World Models from Pixels

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Dreamweaver argues that an unsupervised, pixel-only model can learn a library of reusable concept blocks—static and dynamic—from raw video and recombine them into futures never seen in training.

desk verdict A solid extension of SysBinder to video—the RBSU and predictive objective are genuinely useful—but the compositional imagination demos lean on supervised block selection, so the 'unsupervised recombination' claim overreaches. read the letter →

arxiv 2501.14174 v5 pith:PVRCCZYZ submitted 2025-01-24 cs.CV cs.AIcs.LG

Junyeob Baek , Yi-Fu Wu , Gautam Singh , Sungjin Ahn This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords Dreamweavercompositionalworldmodelsobject-centriclearningblock-slotrepresentationvideopredictionimaginationdisentangledunsupervised
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dreamweaver sets out to show that a model can learn a compositional world model from raw pixels alone, without text, masks, or bounding boxes. Its central claim is that splitting each object representation into independent 'blocks' and training the model to predict several future frames makes reusable concepts emerge for both static attributes (color, shape) and dynamic attributes (direction, speed, dance pattern). If the claim is right, generative world models no longer need language to supply compositionality: the model builds its own concept library from video and can recombine those concepts to imagine futures that never appeared in training. The paper reports that this architecture outperforms current object-centric baselines under the DCI framework (which scores how cleanly learned dimensions track ground-truth factors) across five synthetic datasets, and it demonstrates compositional imagination by swapping blocks between objects. A sympathetic reading treats the result as a proof-of-concept that predictive objectives plus block-slot bottlenecks can make dynamic concept abstraction emerge without supervision.

What carries the argument

The Recurrent Block-Slot Unit (RBSU) is the central object: a recurrent state made of slots, each slot a concatenation of blocks, where each block is updated by its own GRU and residual MLP and then attends to a per-block learned prototype memory that acts as an attractor, pulling each block's state toward one of a finite set of reusable values. The final block interaction runs through a single transformer layer. Around this unit, Dreamweaver couples an image encoder to an autoregressive transformer decoder over discrete dVAE tokens; the training objective is cross-entropy over multiple future-frame token sequences, which is what forces dynamic concepts into the blocks.

What would settle it

Intervene on a single learned block in a pre-trained Dreamweaver—swap the block that probes identify as 'color' between two objects—and decode the rollout; if more than the color factor changes (shape, motion, or background also shift), then the block factorization is not compositional and the central claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that dynamic concepts such as motion direction and speed can emerge from pixels, without text or masks, if object representations are organized into independent blocks and the model is trained to predict future frames rather than reconstruct the present one. Each object is carried by a slot; each slot is made of several blocks; each block is updated independently and then pulled toward one of a learned set of prototype vectors via attention, and a transformer decoder synthesizes future frames from the final block-slot state. Measured with DCI, Dreamweaver reports higher disentanglement, completeness, and informativeness than RSSM, STEVE, and a recurrent SysBinder on all five datasets, with dynamic-factor informativeness more than twice that of the baselines. Manipulating block values lets the model generate out-of-distribution compositions, such as a color seen only with one shape applied to another shape, or a dance pattern transferred across objects. The paper also claims the learned representations support downstream reasoning about future object states, including frame offsets beyond the training prediction length.

Load-bearing premise

The load-bearing assumption is that the combination of a block-slot bottleneck, per-block attractor dynamics, and a multi-frame predictive objective is enough to make each learned block converge to a semantically meaningful reusable concept aligned with the dataset's ground-truth factors, with no auxiliary supervision.

Editorial extensions

If this is right

  • Future frames can be rolled out far beyond the training horizon by feeding each predicted frame back through the encoder, turning the learned representation into a self-sustaining simulator.
  • Recombining blocks from seen objects yields videos with factor combinations withheld during training, which is the paper's demonstration of out-of-distribution compositional imagination.
  • Freezing Dreamweaver's representations and probing them with a transformer supports accurate future-state reasoning, including offsets beyond the training prediction length.
  • The ablation results imply that both the prototype concept memory and the predictive objective are necessary components: removing either one lowers disentanglement or dynamic-factor informativeness.
  • The model's own limitation statement says it is not yet applicable to highly complex scene images, so the practical scope of the claim is currently synthetic and semi-synthetic scenes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same prototype-memory mechanism could be used in model-based reinforcement learning and planning, where imagined rollouts from recombined blocks offer a way to test counterfactual actions without language.
  • Editorial inference: the current pipeline identifies which block controls which factor only after training, using ground-truth labels and feature-importance probes, so a fully autonomous system would still need an unsupervised way to discover block-factor correspondence.
  • Editorial inference: for real-world videos the ground-truth factor list is not predefined, so the concept library would need to be evaluated behaviorally—for instance by intervening on one block and checking that only one aspect of the rendered future changes—rather than with DCI-style scores.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces Dreamweaver, an object-centric world model that encodes video frames into 'block-slot' representations, where each slot corresponds to an object and each block to a reusable concept (e.g., color, shape, motion direction, dance pattern). The model uses a Recurrent Block-Slot Unit (RBSU) with per-block prototype memory and a multi-frame predictive objective, trained end-to-end with a dVAE tokenizer and an autoregressive transformer decoder. The authors claim state-of-the-art DCI disentanglement/informativeness scores on five synthetic video datasets, demonstrate compositional imagination by swapping or changing blocks to generate novel future videos, and show improved downstream prediction/reasoning, including out-of-distribution settings. Ablations support the importance of the predictive objective and the concept memory.

Significance. If the results hold, Dreamweaver would be a meaningful advance in unsupervised object-centric world modeling: it is the first to demonstrate learning of both static and dynamic concept-level (block) representations from raw video, with consistent DCI improvements over strong baselines and a convincing ablation study. The predictive objective's role in capturing dynamic concepts is well supported, and the paper includes a clear architectural innovation (RBSU) plus a thorough dataset suite. However, the central 'unsupervised compositional imagination' claim is weakened because block selection for recombination relies on ground-truth labels and masks (Appendix C.5), and the quantitative evaluation lacks error bars and released code. The evaluation is also limited to synthetic datasets, a limitation the authors acknowledge.

major comments (3)
  1. [5.2, Appendix C.5] The compositional imagination protocol is not unsupervised: Appendix C.5 specifies that block-factor correspondence is obtained by Hungarian matching slots to objects using ground-truth masks, training a supervised probe (LASSO/decision trees) to predict ground-truth factor labels from block representations, and manually inspecting the resulting feature importances. Thus the claim in the abstract that Dreamweaver enables compositional imagination 'without auxiliary data, such as text, masks, or bounding boxes' is not established for the recombination step; an external oracle identifies which blocks to swap. To support the claim, the authors should either demonstrate that block identities can be recovered from the model itself (e.g., via unsupervised clustering of block codes, as in Appendix D.1) or explicitly reframe the contribution as externally guided recombination.
  2. [5.1, Figure 3; Table 2] The central quantitative claim that Dreamweaver 'consistently surpasses' all baselines across five datasets is reported without error bars, multiple seeds, or significance tests. DCI scores are known to be sensitive to seed and initialization in object-centric models; without variance estimates the claim of consistent improvement is not statistically supported. Please report means and standard deviations over at least three random seeds (ideally five) for the DCI metrics and for Table 2's MSE/LPIPS/PSNR, and state whether the improvements are consistent across seeds.
  3. [5.1, Appendix C] The DCI evaluation protocol for slot-based representations is underspecified. The paper does not state how slots are matched to ground-truth objects (e.g., Hungarian matching on masks), which slot is used for each factor, or how the probe is trained. Since DCI is computed with ground-truth factors and the representation is a set of slots, these details are necessary to interpret the scores and to reproduce the experiments. Please provide a precise description of the DCI computation, including any use of masks.
minor comments (7)
  1. [Abstract, Section 5.2] The abstract states the method works 'without auxiliary data, such as text, masks, or bounding boxes,' but the compositional imagination protocol in Appendix C.5 uses masks for Hungarian matching and ground-truth factor labels. Please clarify that the 'no auxiliary data' claim refers only to the training of the encoder/decoder, and not to evaluation or the block-selection procedure.
  2. [Section 5.2, Appendix D.2] The out-of-distribution compositional imagination results are presented qualitatively (Figure 9); please add quantitative OOD metrics, or explicitly state that the quantitative results in Table 2 are in-distribution only.
  3. [Table 2] The quantitative imagination metrics (MSE, LPIPS, PSNR) are reported as single numbers without confidence intervals or number of seeds; also clarify how the 'ground truth videos' for swapped-factor manipulations are generated, especially for combinations that may be out-of-distribution.
  4. [Section 1, paragraph 1] There is a typo: 'Specificially' should be 'Specifically'.
  5. [References] The reference list contains duplicated Schölkopf et al. entries (2021a,b,c) with identical titles; please consolidate and ensure the citation keys match the in-text citations.
  6. [Section 2.1, Eq. (1)] The notation 'softmax_N' in the attention equation is unclear; please define the softmax dimension and the renormalization step explicitly.
  7. [Reproducibility Statement] The reproducibility statement says code 'intend to make publicly available,' but no code or data is provided with the paper. For a journal submission, please release the code or provide a detailed implementation appendix sufficient for reimplementation.

Circularity Check

1 steps flagged · score 3.0 of 10

Compositional imagination relies on a supervised block-factor probe (App. C.5), so the 'unsupervised recombination' claim is partially label-driven; DCI and predictive training are otherwise external.

  1. fitted input called prediction [Section 5.2 (Setup) and Appendix C.5]
    "We can perform a factor swap by taking two slots corresponding to distinct objects, selecting the blocks that correspond to a specific factor (e.g., color), and swapping them. ... Details about how we ascertain the correspondence between a ground truth factor and its representative block are provided in Appendix C.5. ... Take a large batch of videos with per-object factor labels given. ... Train a probe to predict ground truth factor labels from the block-slot representation. Use probing methods that provide feature importance (e.g., LASSO or Decision Trees)."

    The compositional-imagination demonstration is centrally about swapping or changing a named factor (color, shape, direction, dance). To know which block index carries that factor, the authors fit a supervised probe on ground-truth factor labels and masks (Hungarian matching, LASSO/decision trees, manual inspection). The factor-swap target is defined by those same labels, so the 'novel recombined future' is orchestrated by an external oracle rather than by the model's own unsupervised concept library. This is a partial fitted-input problem: the block-factor correspondence is fitted to the labels and then presented as a discovered modular concept.

full rationale

The core training loop is not circular: Dreamweaver is trained with a multi-frame cross-entropy prediction objective, and DCI is computed from ground-truth factors after training, so the disentanglement scores are an external evaluation rather than a fitted target. The self-citations to SysBinder (Singh et al., 2023; Wu et al., 2024) provide the block-slot backbone and attractor-memory idea, but the paper's ablations (No-CM, No-Predictive) independently support those design choices, so the self-citations are not load-bearing in a circular way. The main caveat is the compositional imagination protocol: selecting which block to swap or change requires ground-truth factor labels, mask overlap, and a supervised feature-importance probe. This means the claim that the model can 'recombine attributes from previously seen objects' without auxiliary data is only partially supported; the model can execute a recombination once an oracle identifies the relevant blocks, but the unsupervised discovery of that block-factor mapping is not demonstrated. Weighing this supervision leak against the otherwise external DCI evaluation and predictive-objective training gives a mild-to-moderate circularity score of 3.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The model is an engineering system with many hand-chosen hyperparameters, so the ledger lists the most consequential ones. The axioms are the domain assumptions of object-centric learning and the paper-specific hypotheses about the predictive objective and prototype attractors. No new physical or conceptual entities are postulated beyond the learned prototype memory, which is part of the model.

free parameters (5)
  • context_length_T = 2 or 3 (Dancing-Sprites uses 3, others 2)
    Hand-chosen per dataset; ablations show sensitivity (Fig 6d).
  • prediction_length_K = 2 or 3
    Hand-chosen; larger K degrades I-D (Fig 6e).
  • number_of_blocks_M = 8
    Hand-chosen; ablations show saturating performance once large enough (Fig 6b).
  • prototype_count = 64 or 128 for CLEVRTex
    Hand-chosen; performance sensitive to memory size (Fig 6c).
  • block_size_D = 96
    Architectural choice inherited from SysBinder.
assumptions (5)
  • domain assumption Scenes can be decomposed into a fixed number of objects, each with a fixed number of independent attribute blocks.
    Slot-based object-centric learning assumption, stated in Section 2 and dataset design.
  • domain assumption DCI scores computed from ground-truth factor labels are a valid measure of compositional representation quality.
    Section 5.1 uses DCI as the primary evaluation metric.
  • ad hoc to paper A temporally predictive loss is necessary and sufficient for emergence of dynamic concepts.
    Stated as key hypothesis in Section 1 and tested in ablation, but not proven in general.
  • ad hoc to paper Attractor dynamics with a learned prototype memory cause blocks to converge to reusable concepts.
    Section 2.2; relies on prior SysBinder findings.
  • domain assumption The autoregressive transformer decoder with a dVAE can faithfully render future frames from block-slots.
    Adopted from prior work; required for the training signal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dreamweaver: Learning Compositional World Models from Pixels." pith.science (2026). https://pith.science/paper/PVRCCZYZ

@misc{pith2026250114174,
  author       = {Pith},
  title        = {Pith review of: Dreamweaver: Learning Compositional World Models from Pixels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PVRCCZYZ}},
  note         = {Machine review of arXiv:2501.14174}
}
read the original abstract

Humans have an innate ability to decompose their perceptions of the world into objects and their attributes, such as colors, shapes, and movement patterns. This cognitive process enables us to imagine novel futures by recombining familiar concepts. However, replicating this ability in artificial intelligence systems has proven challenging, particularly when it comes to modeling videos into compositional concepts and generating unseen, recomposed futures without relying on auxiliary data, such as text, masks, or bounding boxes. In this paper, we propose Dreamweaver, a neural architecture designed to discover hierarchical and compositional representations from raw videos and generate compositional future simulations. Our approach leverages a novel Recurrent Block-Slot Unit (RBSU) to decompose videos into their constituent objects and attributes. In addition, Dreamweaver uses a multi-future-frame prediction objective to capture disentangled representations for dynamic concepts more effectively as well as static concepts. In experiments, we demonstrate our model outperforms current state-of-the-art baselines for world modeling when evaluated under the DCI framework across multiple datasets. Furthermore, we show how the modularized concept representations of our model enable compositional imagination, allowing the generation of novel videos by recombining attributes from previously seen objects. cun-bjy.github.io/dreamweaver-website

Figures

Figures reproduced from arXiv: 2501.14174 by the authors.

Figure 1
Figure 1. Overview of the Dreamweaver Framework. Our aim is to take a sequential unstructured sensory stream and bind the low-level information into abstract modular concepts to build a memory of reusable concepts, called concept library—all without text and in an unsupervised way. These concepts include both static factors such as color and shape as well as dynamic factors such as direction and speed of motion. Finally, we s… view at source ↗
Figure 2
Figure 2. Model Architecture. Left: The Recurrent Block-Slot Unit (RBSU) is a recurrent unit designed for processing sequences where each item is a set of vectors. RBSU maintains and updates Block-Slots, which represent compositional and semantic concepts such as shape, color, and motion direction. Right: The Dreamweaver model encodes video inputs into Block-Slot representations, which pass through a series of RBSUs with a re… view at source ↗
Figure 3
Figure 3. DCI Performance. We compare our model with the baselines in terms of Disentanglement (D), Completeness (C), Informativeness (I), and Informativeness-Dynamic (I-D). I-D is the informativeness score for dynamic concepts only (e.g., the direction of motion or dance pattern, etc.) to evaluate how effectively the models capture such dynamic concepts. 5.1 UNSUPERVISED MODULAR CONCEPT DISCOVERY FROM VIDEOS Metrics. For qua… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Compositional Imagination. We show compositionally novel videos generated by Dreamweaver. In this visualization, we (1) infer the block-slot representation given an initial context video, (2) perform manipulations on the inferred block-slot representation, and (3) perf…
Figure 5
Figure 5. Figure 5: Compositional Scene Prediction and Reasoning. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Ablation Study Results. (a) Architectural ablations for the predictive objective and concept memory on the Dancing-CLEVR dataset. (b-c) Varying number of blocks and concept memory size on the CLEVR-Hard dataset. (d-e) Varying context length and prediction length on the…
Figure 7
Figure 7. Figure 7: Visualization of Captured Concept in Moving-Sprites dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Visualization of Captured Concept in Dancing-CLEVR dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Out-of-Distribution Compositional Imagination Example. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Compositional imagination examples on Moving-CLEVRTex. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Visualization of Unseen Dance Patterns in Dancing-Sprites (OOD Dynamics) Experiments [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Downstream Performance with Entirely Unseen Concepts [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Overview of Designed Datasets E.1 SIMPLE DYNAMIC DATASETS For the datasets with simple dynamics, we incorporate sliding movements into the existing static datasets. Moving-Sprites features three fixed-sized objects with 12 shapes and 7 colors sliding in 4 directions a…
Figure 14
Figure 14. Figure 14: Visualization of Dance Patterns in Dancing-Sprites. [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Visualization of Color-changing Patterns in Dancing-CLEVR. [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Dyn-O: Building Structured World Models with Object-Centric Representations

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Dyn-O learns object-centric world models directly from pixels in complex Procgen games, using SAM2-guided slot attention and Mamba state-space dynamics, and reports better rollout prediction than DreamerV3.

Reference graph

Works this paper leans on

47 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [7]

    Video language planning

    Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, et al. Video language planning. arXiv preprint arXiv:2310.10625,

    arXiv

  2. [8]

    Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems

    Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, and Michael Mozer. Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. arXiv preprint arXiv:2006.16225,

    arXiv 2006

  3. [9]

    Neural production systems

    Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael C Mozer, and Yoshua Bengio. Neural production systems. Advances in Neural Information Processing Systems, 34:25673–25687, 2021a. Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurr...

    arXiv 1909

  4. [10]

    In this visualization, we demon- strate the generation of compositionally novel videos on visually more complex and textured scenes than the previously tested datasets

    Figure 10: Compositional imagination examples on Moving-CLEVRTex. In this visualization, we demon- strate the generation of compositionally novel videos on visually more complex and textured scenes than the previously tested datasets. Top-Left: We show two original videos from the dataset. Top-Right: We swap the block representations of object texture bet...

    work page 2025

  5. [11]

    On the binding problem in artificial neural networks

    Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208,

    arXiv 2012

  6. [12]

    World models

    David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122,

    arXiv

  7. [13]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551,

    arXiv

  8. [14]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603,

    arXiv 1912

Show all 47 references
  1. [15]

    Mastering atari with discrete world models

    Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193,

    2010 arXiv

  2. [16]

    Image quality metrics: Psnr vs

    Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pp. 2366–2369. IEEE,

    2010

  3. [18]

    Object-centric slot diffusion

    Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. arXiv preprint arXiv:2303.10834,

    arXiv

  4. [19]

    Slot state space models

    Jindong Jiang, Fei Deng, Gautam Singh, Minseung Lee, and Sungjin Ahn. Slot state space models. arXiv preprint arXiv:2406.12272,

    arXiv

  5. [20]

    Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation

    Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation. arXiv preprint arXiv:2111.10265,

    arXiv

  6. [21]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

  7. [23]

    Vitgan: Training gans with vision transformers

    Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Training gans with vision transformers. arXiv preprint arXiv:2107.04589,

    arXiv

  8. [24]

    Rudolph, and Max Welling

    Sindy Lowe, Phillip Lippe, Maja R. Rudolph, and Max Welling. Complex-valued autoencoders for object discovery. arXiv preprint arXiv:2204.02075,

    arXiv

  9. [25]

    Fast and slow learning of recurrent independent mechanisms

    Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, and Yoshua Bengio. Fast and slow learning of recurrent independent mechanisms. arXiv preprint arXiv:2105.08710,

    arXiv

  10. [26]

    Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen

    13 Published as a conference paper at ICLR 2025 A. Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06125,

    2025 arXiv

  11. [27]

    Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij

    URL https://proceedings.neurips.cc/paper_files/paper/ 2022/file/3dc83fcfa4d13e30070bd4b230c38cfe-Paper-Conference.pdf. Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arXiv:1206.6471,

    2022 arXiv

  12. [28]

    URL https://openreview.net/forum?id=4KZpDGD4Nh. Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Scholkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-cen...

    arXiv

  13. [29]

    Illiterate dall-e learns to compose

    Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In International Conference on Learning Representations, 2022a. URL https://openreview.net/forum? id=h0OYV0We3oh. Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning ...

    arXiv

  14. [30]

    Parallelized spatiotemporal binding

    Gautam Singh, Yue Wang, Jiawei Yang, Boris Ivanovic, Sungjin Ahn, Marco Pavone, and Tong Che. Parallelized spatiotemporal binding. arXiv preprint arXiv:2402.17077,

    arXiv

  15. [31]

    Neurocompositional computing in human and machine intelligence: A tutorial.Microsoft Technical Report MSR-TR-2022,

    Paul Smolensky, R Thomas McCoy, Roland Fernandez, Matthew Goldrick, and Jianfeng Gao. Neurocompositional computing in human and machine intelligence: A tutorial.Microsoft Technical Report MSR-TR-2022,

    2022

  16. [32]

    Relational neural expectation maximization: Unsupervised discovery of objects and their interactions

    Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802.10353,

    arXiv

  17. [33]

    Attention is all you need

    14 Published as a conference paper at ICLR 2025 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008,

    2025

  18. [34]

    Entity abstraction in visual model-based reinforcement learning

    Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua B Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. arXiv preprint arXiv:1910.12827,

    1910 arXiv

  19. [35]

    net/forum?id=HYyRwm367m

    URL https://openreview. net/forum?id=HYyRwm367m. Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05861,

    arXiv

  20. [36]

    Slotdiffusion: Object-centric generative modeling with diffusion models

    Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Object-centric generative modeling with diffusion models. arXiv preprint arXiv:2305.11281, 2023b. Wei Yu, Wenxin Chen, Songheng Yin, Steve Easterbrook, and Animesh Garg. Modular action concept g...

    arXiv

  21. [37]

    Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan

    URL https://proceedings.mlr.press/v162/ zhao22b.html. Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404.12377,

    arXiv

  22. [39]

    decomposes state space representations into separate recurrent components that operate independently, with sparse interactions among them. These models are designed to learn a set of reusable mechanisms that are selectively activated through sparse communication, enabling them...

    2022

  23. [40]

    Du et al

    is a framework for object-centric world modeling that leverages a frozen pretrained vision-language foundation model to decompose objects into symbolic attributes. Du et al. (2023); Zhou et al. (2024); Yu et al. (2022); Cho et al. (2024) similarly relies on language to enable ...

    2023

  24. [41]

    B A DDITIONAL MODEL DETAILS B.1 I MAGE TOKENIZATION VIA DISCRETE VAE Since we leverage an autoregressive image transformer to predict the frames xt+1,

    is also a related model that learns disentangled modes of interaction between objects, but does not learn attribute-level representations of the objects. B A DDITIONAL MODEL DETAILS B.1 I MAGE TOKENIZATION VIA DISCRETE VAE Since we leverage an autoregressive image transformer ...

    2019

  25. [42]

    Moving-" or

    with β1 set to 0.9 and β2 set to 0.999. Furthermore, we utilized the architecture and hyperparameters of the backbone image encoder as specified in Singh et al. (2023). Table 1: Hyperparameters of our model used in our experiments. We use a shortened version of the dataset nam...

    2023

  26. [45]

    Moving-Sprites features three fixed-sized objects with 12 shapes and 7 colors sliding in 4 directions at 4 different speeds within a 2D scene (detailed in Table 3)

    21 Published as a conference paper at ICLR 2025 0 1 2 30 20 40 60 80 100 Frame Offset Accuracy (in %) Dancing-Sprites (OOD Shapes) Our Model STEVE SysBinder RSSM 0 1 2 30 20 40 60 80 100 Frame Offset Dancing-Sprites (OOD Dynamics) Figure 12: Downstream Performance with Entirel...

    2025

  27. [46]

    code (link). Shape Materials Size Moving Direction Cube whitemarble 1.6 Forward Sphere polyhaven_leather_red_02 2.0 Backward Cylinder polyhaven_factory_wall Leftside Torus polyhaven_cracked_concrete_wall Rightside poly_haven_stony_dirt_path polyhaven_painted_metal_shutter poly...

    2025

  28. [51]

    The dataset includes four distinct color-changing patterns, as illustrated in (b)

    Color Pattern: 3-hop CCW Lift Up Lift Down 24 Published as a conference paper at ICLR 2025 Figure 15: Visualization of Color-changing Patterns in Dancing-CLEVR. The dataset includes four distinct color-changing patterns, as illustrated in (b). Each sequence is generated by fol...

    2025

  29. [128]

    For RSSM, we use an MLP to make the prediction

    We use the output of a learned class token to predict the target label. For RSSM, we use an MLP to make the prediction. While we experimented with matching the number of parameters with the transformer model, we found better performance using a smaller network with 4 layers an...

    2023

  30. [2010]

    Scalor: Generative world mod- els with scalable object representations

    12 Published as a conference paper at ICLR 2025 Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world mod- els with scalable object representations. In International Conference on Learning Representations ,

    2025

  31. [2014]

    Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff

    Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. arXiv preprint arXiv:2111.12594,

    arXiv

  32. [2017]

    Multi-object representation learning with iterative variational inference

    Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv:1903.00450,

    1903 arXiv

  33. [2018]

    Monet: Unsupervised scene decomposition and representation

    Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390,

    1901 arXiv

  34. [2019]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey K Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML 2020: 37th International Conference on Machine Learning, volume 1, pp. 1691–1703,

    2020

  35. [2020]

    Sora as an agi world model? a complete survey on text-to-video generation

    Joseph Cho, Fachrina Dewi Puspitasari, Sheng Zheng, Jingyao Zheng, Lik-Hang Lee, Tae-Ho Kim, Choong Seon Hong, and Chaoning Zhang. Sora as an agi world model? a complete survey on text-to-video generation. arXiv preprint arXiv:2403.05131,

  36. [2021]

    15 Published as a conference paper at ICLR 2025 A A DDITIONAL RELATED WORKS Learning Compositional Mechanisms. Unlike our model is built on slot-based models, there is another line of work for learning compositional representations motivated by the Independent Causal Mechanism...

    2025

  37. [2022]

    Constructing future behaviour in the hippocampal formation through composition and replay

    Jacob JW Bakermans, Joseph Warren, James CR Whittington, and Timothy EJ Behrens. Constructing future behaviour in the hippocampal formation through composition and replay. bioRxiv, pp. 2023–04,

    2023

  38. [2023]

    Object discovery from motion-guided tokens

    Zhipeng Bao, Pavel Tokmakov, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Object discovery from motion-guided tokens. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22972–22981,

    2023

  39. [2024]

    Exploiting spatial invariance for scalable unsupervised object tracking

    Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. arXiv preprint arXiv:1911.09033, 2019a. Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedin...

    1911 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.