{"id":"4f0b94dd-40d7-482d-86d6-503b05d713e6","arxiv_id":"2501.14174","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An unsupervised recurrent block-slot model that discovers static and dynamic concept blocks from raw video and recombines them to imagine novel future videos.","lead":"Dreamweaver learns object and attribute representations from raw videos, then recombines those attributes to generate new videos without text, masks, or bounding boxes. It is the first object-centric model to capture both static and dynamic concepts from video in an unsupervised way, a step toward compositional imagination in world models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Compositional imagination uses ground-truth labels to pick which block to swap (App. C.5), so the paper does not yet show that the learned concept library supports unsupervised recombination.","rationale":"I read the paper as making two separable claims: (i) the RBSU predictive model yields better DCI scores than prior object-centric baselines, and (ii) the resulting blocks are reusable concepts that can be recombined to imagine novel videos. Claim (i) is reasonably supported by Figure 3 and the ablations, modulo the absence of error bars and code. Claim (ii) is the headline 'compositional world model' contribution. The concern above targets (ii) specifically. The authors are transparent about the block-factor correspondence procedure in Appendix C.5, but that transparency reveals that the recombination experiments are not fully unsupervised: human or supervised selection of the block to swap is doing part of the compositional work. This is not an accusation of dishonesty; the underlying representations may genuinely be factor-aligned. However, the evidence as presented cannot distinguish 'the model learned an indexed concept library' from 'a supervised probe found some readout dimensions that, when edited, produce coherent renderings on synthetic data.' The proposed concrete test would settle this. Because the paper's broader verdict has already been CONDITIONAL, and this concern supports that verdict rather than overturning the quantitative DCI results, I keep the verdict unchanged.","tokens_in":20470,"tokens_out":8238,"duration_ms":84115,"concrete_test":"Re-run the Moving-Sprites Sec. 5.2 factor-swap protocol with block indices chosen by an unsupervised procedure, e.g., cluster each block index's prototype assignments across training videos and use cluster identity, without GT labels or masks; then check on held-out GT whether the intended color or direction change occurs in the generated frames and compare success to the Appendix C.5 supervised probe selection. If unsupervised block selection fails to produce the intended factor change at near the probe-guided rate, the compositional imagination claim is oracle-dependent and should be scaled back; if it succeeds, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's unique, load-bearing capability claim is compositional imagination: generating novel futures by recombining discovered concepts (Sec. 5.2, Fig. 4, App. D.2–D.3). To run a factor swap or change, the experimenter must know which block index encodes color, shape, direction, or dance pattern. Appendix C.5 specifies that this correspondence is obtained by (1) taking videos with per-object ground-truth factor labels, (2) matching slots to ground-truth objects via Hungarian matching on masks, (3) training a supervised probe (LASSO or decision trees) to predict GT factor labels from block-slot representations, and (4) manually inspecting feature importances to choose representative blocks. Thus every recombination demo is mediated by auxiliary labels and masks, even though the abstract claims the method works 'without auxiliary data, such as text, masks, or bounding boxes.' The training objective is unsupervised, but the claimed compositional imagination protocol is not: an external oracle identifies which blocks to exchange. DCI also uses GT labels by design, so that part is standard; the problem is that the central capability demonstration depends on oracle block selection. Without showing that block identities can be recovered from the model itself, for example from prototype or codebook assignments or unsupervised clustering, the conclusion that 'modularized concept representations enable compositional imagination' is not established. Table 2 also reports only self-scores for manipulated generations, with no baseline or random-manipulation control.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Dreamweaver, an object-centric world model that encodes video frames into 'block-slot' representations, where each slot corresponds to an object and each block to a reusable concept (e.g., color, shape, motion direction, dance pattern). The model uses a Recurrent Block-Slot Unit (RBSU) with per-block prototype memory and a multi-frame predictive objective, trained end-to-end with a dVAE tokenizer and an autoregressive transformer decoder. The authors claim state-of-the-art DCI disentanglement/informativeness scores on five synthetic video datasets, demonstrate compositional imagination by swapping or changing blocks to generate novel future videos, and show improved downstream prediction/reasoning, including out-of-distribution settings. Ablations support the importance of the predictive objective and the concept memory.","tokens_in":20775,"tokens_out":8028,"duration_ms":71582,"significance":"If the results hold, Dreamweaver would be a meaningful advance in unsupervised object-centric world modeling: it is the first to demonstrate learning of both static and dynamic concept-level (block) representations from raw video, with consistent DCI improvements over strong baselines and a convincing ablation study. The predictive objective's role in capturing dynamic concepts is well supported, and the paper includes a clear architectural innovation (RBSU) plus a thorough dataset suite. However, the central 'unsupervised compositional imagination' claim is weakened because block selection for recombination relies on ground-truth labels and masks (Appendix C.5), and the quantitative evaluation lacks error bars and released code. The evaluation is also limited to synthetic datasets, a limitation the authors acknowledge.","major_comments":[{"comment":"The compositional imagination protocol is not unsupervised: Appendix C.5 specifies that block-factor correspondence is obtained by Hungarian matching slots to objects using ground-truth masks, training a supervised probe (LASSO/decision trees) to predict ground-truth factor labels from block representations, and manually inspecting the resulting feature importances. Thus the claim in the abstract that Dreamweaver enables compositional imagination 'without auxiliary data, such as text, masks, or bounding boxes' is not established for the recombination step; an external oracle identifies which blocks to swap. To support the claim, the authors should either demonstrate that block identities can be recovered from the model itself (e.g., via unsupervised clustering of block codes, as in Appendix D.1) or explicitly reframe the contribution as externally guided recombination.","section":"5.2, Appendix C.5"},{"comment":"The central quantitative claim that Dreamweaver 'consistently surpasses' all baselines across five datasets is reported without error bars, multiple seeds, or significance tests. DCI scores are known to be sensitive to seed and initialization in object-centric models; without variance estimates the claim of consistent improvement is not statistically supported. Please report means and standard deviations over at least three random seeds (ideally five) for the DCI metrics and for Table 2's MSE/LPIPS/PSNR, and state whether the improvements are consistent across seeds.","section":"5.1, Figure 3; Table 2"},{"comment":"The DCI evaluation protocol for slot-based representations is underspecified. The paper does not state how slots are matched to ground-truth objects (e.g., Hungarian matching on masks), which slot is used for each factor, or how the probe is trained. Since DCI is computed with ground-truth factors and the representation is a set of slots, these details are necessary to interpret the scores and to reproduce the experiments. Please provide a precise description of the DCI computation, including any use of masks.","section":"5.1, Appendix C"}],"minor_comments":[{"comment":"The abstract states the method works 'without auxiliary data, such as text, masks, or bounding boxes,' but the compositional imagination protocol in Appendix C.5 uses masks for Hungarian matching and ground-truth factor labels. Please clarify that the 'no auxiliary data' claim refers only to the training of the encoder/decoder, and not to evaluation or the block-selection procedure.","section":"Abstract, Section 5.2"},{"comment":"The out-of-distribution compositional imagination results are presented qualitatively (Figure 9); please add quantitative OOD metrics, or explicitly state that the quantitative results in Table 2 are in-distribution only.","section":"Section 5.2, Appendix D.2"},{"comment":"The quantitative imagination metrics (MSE, LPIPS, PSNR) are reported as single numbers without confidence intervals or number of seeds; also clarify how the 'ground truth videos' for swapped-factor manipulations are generated, especially for combinations that may be out-of-distribution.","section":"Table 2"},{"comment":"There is a typo: 'Specificially' should be 'Specifically'.","section":"Section 1, paragraph 1"},{"comment":"The reference list contains duplicated Schölkopf et al. entries (2021a,b,c) with identical titles; please consolidate and ensure the citation keys match the in-text citations.","section":"References"},{"comment":"The notation 'softmax_N' in the attention equation is unclear; please define the softmax dimension and the renormalization step explicitly.","section":"Section 2.1, Eq. (1)"},{"comment":"The reproducibility statement says code 'intend to make publicly available,' but no code or data is provided with the paper. For a journal submission, please release the code or provide a detailed implementation appendix sufficient for reimplementation.","section":"Reproducibility Statement"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong submission that has already been through ICLR review; the core architectural contribution and DCI improvements are credible. However, the compositional imagination claim is oversold relative to the actual protocol (which requires supervised block selection), and the lack of statistical rigor and code release is concerning for a journal version. I recommend major revision with either a revised claim or an unsupervised block-selection demonstration, together with error bars and a precise DCI protocol description."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. Dreamweaver is a genuine step forward for object-centric world models: it takes SysBinder's block-slot representation, makes it recurrent, and adds a multi-frame prediction objective. The result is the first unsupervised method to learn both static and dynamic concepts as block slots from raw video. The DCI numbers are consistently better than RSSM, STEVE, and a recurrent SysBinder baseline across five datasets, and the ablations make a credible case that both the concept memory and the predictive objective are doing real work. The downstream reasoning results—especially on the OOD splits—are a nice bonus.\n\nThe soft spot the stress-test flags is real. The compositional imagination section (5.2) depends on knowing which block index encodes color, shape, direction, or dance pattern, and Appendix C.5 shows that this correspondence is established by a supervised procedure: ground-truth factor labels, Hungarian matching on masks, and a probe (LASSO/decision trees) whose feature importances are manually inspected. So the model's training is unsupervised, but the recombination demo is not—an oracle tells you which blocks to swap. That weakens the abstract's claim that the method works 'without auxiliary data, such as text, masks, or bounding boxes' (masks are used in the matching, and factor labels in the probe). The paper would be stronger if it showed that block identities can be recovered from the model itself, e.g., via prototype assignment or unsupervised clustering.\n\nOther issues are more minor: no error bars or multiple seeds reported, code not released (the reproducibility statement just says they intend to), Table 2 reports only self-scores for manipulated generations with no baseline or random-manipulation control, and all experiments are on synthetic datasets. The five datasets are carefully designed, though, and the limitations section is honest about the scope.\n\nIn sum: the core architecture and the empirical DCI story are solid and this deserves a serious referee. The main fix is to either make the block selection fully unsupervised or to explicitly frame the recombination as a supervised oracle-guided demonstration, and to add the missing error bars and code. I'd bring it to reading group and cite it for RBSU.","headline":"A solid extension of SysBinder to video—the RBSU and predictive objective are genuinely useful—but the compositional imagination demos lean on supervised block selection, so the 'unsupervised recombination' claim overreaches.","tokens_in":21266,"tokens_out":2828,"would_cite":true,"duration_ms":23915,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dreamweaver argues that an unsupervised, pixel-only model can learn a library of reusable concept blocks—static and dynamic—from raw video and recombine them into futures never seen in training.","keywords":["Dreamweaver","compositional world models","object-centric learning","block-slot representation","video prediction","compositional imagination","disentangled representation learning","unsupervised learning"],"falsifier":"Intervene on a single learned block in a pre-trained Dreamweaver—swap the block that probes identify as 'color' between two objects—and decode the rollout; if more than the color factor changes (shape, motion, or background also shift), then the block factorization is not compositional and the central claim fails.","tokens_in":20288,"feed_emoji":"🎬","tokens_out":10795,"duration_ms":86045,"temperature":0.7,"pith_summary":"Dreamweaver sets out to show that a model can learn a compositional world model from raw pixels alone, without text, masks, or bounding boxes. Its central claim is that splitting each object representation into independent 'blocks' and training the model to predict several future frames makes reusable concepts emerge for both static attributes (color, shape) and dynamic attributes (direction, speed, dance pattern). If the claim is right, generative world models no longer need language to supply compositionality: the model builds its own concept library from video and can recombine those concepts to imagine futures that never appeared in training. The paper reports that this architecture outperforms current object-centric baselines under the DCI framework (which scores how cleanly learned dimensions track ground-truth factors) across five synthetic datasets, and it demonstrates compositional imagination by swapping blocks between objects. A sympathetic reading treats the result as a proof-of-concept that predictive objectives plus block-slot bottlenecks can make dynamic concept abstraction emerge without supervision.","feed_headline":"Pixel-only model recombines concept blocks into unseen videos","feed_subtitle":"Dreamweaver's block-slot architecture plus future-frame prediction extracts color, shape, and motion from raw video.","key_machinery":"The Recurrent Block-Slot Unit (RBSU) is the central object: a recurrent state made of slots, each slot a concatenation of blocks, where each block is updated by its own GRU and residual MLP and then attends to a per-block learned prototype memory that acts as an attractor, pulling each block's state toward one of a finite set of reusable values. The final block interaction runs through a single transformer layer. Around this unit, Dreamweaver couples an image encoder to an autoregressive transformer decoder over discrete dVAE tokens; the training objective is cross-entropy over multiple future-frame token sequences, which is what forces dynamic concepts into the blocks.","core_discovery":"On its own terms, the central discovery is that dynamic concepts such as motion direction and speed can emerge from pixels, without text or masks, if object representations are organized into independent blocks and the model is trained to predict future frames rather than reconstruct the present one. Each object is carried by a slot; each slot is made of several blocks; each block is updated independently and then pulled toward one of a learned set of prototype vectors via attention, and a transformer decoder synthesizes future frames from the final block-slot state. Measured with DCI, Dreamweaver reports higher disentanglement, completeness, and informativeness than RSSM, STEVE, and a recurrent SysBinder on all five datasets, with dynamic-factor informativeness more than twice that of the baselines. Manipulating block values lets the model generate out-of-distribution compositions, such as a color seen only with one shape applied to another shape, or a dance pattern transferred across objects. The paper also claims the learned representations support downstream reasoning about future object states, including frame offsets beyond the training prediction length.","pith_inferences":["Editorial inference: the same prototype-memory mechanism could be used in model-based reinforcement learning and planning, where imagined rollouts from recombined blocks offer a way to test counterfactual actions without language.","Editorial inference: the current pipeline identifies which block controls which factor only after training, using ground-truth labels and feature-importance probes, so a fully autonomous system would still need an unsupervised way to discover block-factor correspondence.","Editorial inference: for real-world videos the ground-truth factor list is not predefined, so the concept library would need to be evaluated behaviorally—for instance by intervening on one block and checking that only one aspect of the rendered future changes—rather than with DCI-style scores."],"forward_implications":["Future frames can be rolled out far beyond the training horizon by feeding each predicted frame back through the encoder, turning the learned representation into a self-sustaining simulator.","Recombining blocks from seen objects yields videos with factor combinations withheld during training, which is the paper's demonstration of out-of-distribution compositional imagination.","Freezing Dreamweaver's representations and probing them with a transformer supports accurate future-state reasoning, including offsets beyond the training prediction length.","The ablation results imply that both the prototype concept memory and the predictive objective are necessary components: removing either one lowers disentanglement or dynamic-factor informativeness.","The model's own limitation statement says it is not yet applicable to highly complex scene images, so the practical scope of the claim is currently synthetic and semi-synthetic scenes."],"supporting_citations":[{"why":"Introduces the Slot Attention mechanism that Dreamweaver's bottom-up attention and slot structure build on.","marker":"(Locatello et al., 2020)"},{"why":"Introduces block-slot representations and a prototype concept memory, the static-image machinery RBSU extends to video.","marker":"(Singh et al., 2023)"},{"why":"Shows block-slot recombination for static concepts in images, the result Dreamweaver extends to dynamic video concepts.","marker":"(Wu et al., 2024)"},{"why":"Provides the recurrent conditioning formulation used to give the SysBinder baseline temporal context.","marker":"(Kipf et al., 2021)"},{"why":"Supplies the STEVE baseline, a reconstruction-trained slot-based video model Dreamweaver compares against.","marker":"(Singh et al., 2022b)"},{"why":"Supplies the RSSM baseline, a single-vector world model Dreamweaver compares against.","marker":"(Hafner et al., 2018)"},{"why":"Defines the DCI disentanglement metrics used for all quantitative representation-quality comparisons.","marker":"(Eastwood & Williams, 2018)"},{"why":"Supplies the autoregressive discrete-token image transformer architecture used as Dreamweaver's future-frame decoder.","marker":"(Singh et al., 2022a)"}],"fun_headline_variants":["Pixel-only Dreamweaver recombines object attributes to imagine new videos","No text, no boxes: Dreamweaver builds concepts from raw video","Future-frame training lets Dreamweaver invent novel video compositions","Dreamweaver discovers object blocks from pixels and recombines them","Unsupervised model learns to reassemble attributes into unseen futures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the combination of a block-slot bottleneck, per-block attractor dynamics, and a multi-frame predictive objective is enough to make each learned block converge to a semantically meaningful reusable concept aligned with the dataset's ground-truth factors, with no auxiliary supervision.","fun_headline_variants_meta":{"raw":{"variants":["Pixel-only Dreamweaver recombines object attributes to imagine new videos","No text, no boxes: Dreamweaver builds concepts from raw video","Future-frame training lets Dreamweaver invent novel video compositions","Dreamweaver discovers object blocks from pixels and recombines them","Unsupervised model learns to reassemble attributes into unseen futures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000984,"raw_usage":{"total_tokens":4190,"prompt_tokens":977,"completion_tokens":3213,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":3123}},"tokens_in":593,"tokens_out":3213,"duration_ms":22396,"temperature":1.0,"reasoning_tokens":3123,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:18:06.830991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Intervene on a single learned block in a pre-trained Dreamweaver—swap the block that probes identify as 'color' between two objects—and decode the rollout; if more than the color factor changes (shape, motion, or background also shift), then the block factorization is not compositional and the central claim fails.","supporting_citations":[],"review_version":1}