Pith. sign in

REVIEW 3 major objections 6 minor 55 references

This paper claims that slot-attention object segmentation can be improved by running slot attention over several feature layers of a pre-trained vision transformer and fusing the resulting slot sets into one representation, with gains large

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:32 UTC pith:IPOPPPB4

load-bearing objection MUFASA's multi-layer slot attention is a real, well-ablated contribution; the SPOT-M SOTA claim is muddied by a simultaneous distillation-target change that the paper itself admits but never controls for. the 3 major comments →

arxiv 2602.07544 v2 pith:IPOPPPB4 submitted 2026-02-07 cs.CV

MUFASA: A Multi-Layer Framework for Slot Attention

Sebastian Bock , Leonie Sch\"u{\ss}ler , Krishnakant Singh , Simone Schaub-Meyer , Stefan Roth This is my paper
classification cs.CV
keywords unsupervised object segmentationslot attentionobject-centric learningmulti-layer ViT featuresfeature fusionknowledge distillationstate of the art
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MUFASA is a plug-and-play extension for slot-attention models that use a pre-trained DINO ViT encoder. Instead of feeding only the last encoder layer into slot attention, MUFASA runs an independent slot-attention module on each of the last four layers, aligns the resulting slot sets with Hungarian matching, and fuses them into a single set of slots via an MLP module called M-Fusion. The paper demonstrates that this multi-layer design improves unsupervised object segmentation for both DINOSAUR and SPOT across PASCAL VOC, COCO, and MOVi-C, with SPOT-M setting a new state of the art. MUFASA also converges faster to baseline-level accuracy, cutting training time by up to about 94%, with only modest parameter and inference overhead. The claim matters because it says that semantically rich information already present in intermediate ViT layers, currently discarded, is a cheap and effective resource for object-centric learning.

Core claim

The central claim is that layer-wise segmentation quality does not saturate at the final ViT layer. Independent slot-attention modules on the last four DINO layers produce masks that are individually competitive but complementary; aligning the slot sets by Hungarian matching and fusing them through M-Fusion yields masks that are more accurate than any single layer's, because layers compensate for each other's noise. Concretely, the paper reports that SPOT-M reaches 59.8 mBOc on PASCAL VOC versus 55.3 for SPOT, 45.5 versus 44.3 mBOc on COCO, and 49.2 versus 47.0 mBOi on MOVi-C, and that DINOSAUR-M improves on DINOSAUR on nearly every metric and dataset. These gains come with faster convergenc

What carries the argument

The central object is M-Fusion, a slot-fusion module. For each of M selected encoder layers, an independent slot-attention module produces its own set of K slots and an attention mask over image patches. Hungarian matching — an assignment algorithm that pairs up slots across layers so that same-index slots bind to the same object — reorders the slot sets. The aligned slots are combined in adjacent pairs, a design choice that encodes local-layer interactions as an inductive bias, and the concatenated pairs are projected through an MLP into a single fused slot set. The attention masks are fused by a learned weighted sum when teacher–student distillation is present (SPOT-M) and by a uniform ave

Load-bearing premise

The attribution of SPOT-M's state-of-the-art gains to multi-layer fusion is load-bearing; because the same model also switches the distillation target from decoder masks to slot-attention masks (Sec. 4.1, Implementation), the gains may in part or whole come from that switch, and no ablation separates the two.

What would settle it

Train a single-layer SPOT variant (last encoder layer only, all other MUFASA machinery removed) but with the same slot-attention-mask distillation target and hyperparameters as SPOT-M, and compare on PASCAL VOC; if that variant matches SPOT-M's 59.8 mBOc, the multi-layer fusion is not the source of the gain.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Integrating MUFASA into SPOT improves segmentation across VOC, COCO, and MOVi-C, setting a new state of the art (e.g., 59.8 vs 55.3 mBOc on VOC).
  • Integrating MUFASA into DINOSAUR improves it on essentially all metrics, surpassing the previous SPOT state of the art on VOC and MOVi-C despite having fewer parameters.
  • MUFASA reaches baseline-level accuracy substantially faster, reducing time-to-baseline by about 90% or more on VOC and COCO, and reaches peak results earlier on all datasets.
  • The gains hold across encoder backbones and pretraining schemes (DINO, MAE, DINOv2; ViT-B/16, ViT-B/14, ViT-S/8), and across decoder choices.
  • The method adds no auxiliary losses to either base model; it is trained with the base training signal and a modest parameter overhead (e.g., +12% for SPOT-M).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An implication left implicit is that because SPOT-M simultaneously changes the distillation target (from SPOT's decoder attention masks to slot-attention masks), the reported SPOT-M gains are not cleanly attributable to multi-layer fusion alone; a single-layer SPOT variant with the same distillation target would isolate the contribution.
  • The paper's layer-selection ablation shows performance peaks at four consecutive layers, with five slightly worse; this suggests the benefit is not simply 'more layers' but something like a sweet spot where information is semantically rich but not yet over-fit to class-level grouping — a testable hypothesis for other feature hierarchies.
  • If the multi-layer fusion genuinely helps slot binding, the same recipe (independent slot attention per layer + Hungarian alignment + pairwise fusion) could transfer to other slot-attention-based tasks such as compositional generation or world models, where richer slot representations are also desirable.
  • The complementary-information story predicts a specific failure mode: fused masks should help most when individual layers disagree, and help least when layers already agree; a disagreement-weighted analysis across images would test this directly.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes MUFASA, a plug-and-play module for slot-attention-based unsupervised object segmentation. Instead of running slot attention only on the last DINO ViT layer, MUFASA runs independent slot-attention modules on several (typically the last four) encoder layers, aligns the resulting slots with Hungarian matching on binarized attention masks, and fuses them with a learned MLP-based M-Fusion plus a weighted mask fusion. The module is integrated into DINOSAUR and SPOT, yielding DINOSAUR-M and SPOT-M. The main empirical claims are that both variants improve over their baselines on PASCAL VOC, COCO, and MOVi-C, that SPOT-M sets a new state of the art (Table 1: SPOT-M reaches 59.8 mBOc on VOC and 45.5 mBOc on COCO vs. SPOT's 55.3 and 44.3), and that training converges considerably faster with only modest parameter/inference overhead. The paper also includes ablations on layer choice, fusion strategy, encoder backbone, pre-training scheme, and decoder type.

Significance. The multi-layer slot-attention idea is well motivated by the known layer-wise semantics of DINO ViTs, and the paper provides a systematic experimental study: three datasets, multiple backbones and pre-training schemes, decoder and fusion ablations, and training-time analysis. The DINOSAUR-M results are strong across all three datasets and, because DINOSAUR-M changes only the multi-layer mechanism relative to DINOSAUR, they provide genuine support for the central multi-layer claim. The claimed SPOT-M state of the art is, however, not cleanly attributable to the multi-layer mechanism because SPOT-M simultaneously changes the distillation target (decoder masks to slot-attention masks), and the COCO SOTA is fragile on several metrics. The paper is therefore a solid contribution whose headline claim needs additional controls before it can be accepted at face value.

major comments (3)
  1. [Sec. 4.1 (Implementation) and Table 1] SPOT-M differs from SPOT in at least two ways: the multi-layer SA/fusion mechanism, and the distillation target. The text states: 'Contrary to SPOT, we use segmentation masks of the SA module to distill knowledge of the teacher to the student.' The supplement discusses slot vs. decoder masks but does not report a single-layer SPOT variant with SA-mask distillation. Therefore the reported SOTA gains on VOC (+4.5 mBOc), COCO (+1.2 mBOc), and MOVi-C (+2.2 mBOi) cannot be attributed to MUFASA's multi-layer mechanism. Please add an ablation: SPOT with a single layer (L12) but with SA-mask distillation (and the same distillation settings), and ideally also SPOT-M with decoder-mask distillation. Without this control, the paper should not claim that the SPOT-M gains come from multi-layer fusion.
  2. [Table 1 (COCO) and Sec. 4.1] The COCO SOTA claim is fragile. SPOT-M improves mBOc by 1.2 and mBOi by 0.1, but the mBOi difference is within the reported standard deviations (34.8±0.2 vs. 34.7±0.1), while mIoU drops by 0.2 and FG-ARI drops by 2.2 (35.6 vs. 37.8). The text says 'only one marginal exception in mIoU on COCO,' which overlooks the FG-ARI decline. If 'new state of the art' on COCO is claimed, it should be based on metrics with meaningful differences, or supported by paired significance testing, or the claim should be qualified.
  3. [Sec. 3.2 / Eq. (6) and Sec. 4.3] The layer index set I and the learned fusion weights w_m are design choices made on the same benchmarks where the SOTA numbers are reported. The layer set (last four layers) is selected from the VOC ablation in Fig. 6, and SPOT-M's fusion weights are learned during self-training on the evaluation datasets. The manuscript does not state that a held-out validation split was used for model selection. Given the small margins on COCO, this selection procedure could inflate the reported gains. Please clarify the validation protocol, or report results on a held-out split.
minor comments (6)
  1. [Supplementary A] The sentence 'We assign a greater weight to this loss as opposed to [26] with λ= 0.01' is ambiguous, and Table 7 lists 'Distillation λ 0.01'. Please clarify whether SPOT-M uses λ=0.01 or a larger value, and whether λ was tuned per dataset.
  2. [Sec. 3.2 (Hungarian matching)] The binarization of the slot-attention masks used for Hungarian matching is not described. Please specify the threshold/procedure.
  3. [Sec. 4.2 / Table 2] The '94.4% reduction in training time' for SPOT-M on VOC uses EBase (time to reach baseline-level results), not EPeak. Please state this explicitly, since the peak-time reduction is much smaller.
  4. [Table 2] Minor formatting: '12.18 h' has an inconsistent number of decimals compared with the other entries.
  5. [Sec. 3.1] Equation (2) writes softmax over K without a subscript in the displayed formula; please clarify that the softmax is along the slot dimension.
  6. [Sec. 4.3 / Table 3] The differences among Concat-Fusion, T-Fusion, and M-Fusion are small (59.0 vs. 59.8 mBOc) and are reported without standard deviations or significance tests. Please add error bars or note that the comparison is directional.

Circularity Check

0 steps flagged

No circularity found: MUFASA's gains are measured benchmark results, not quantities forced by construction or by a self-citation chain.

full rationale

The paper is an empirical architecture-and-benchmark contribution, not a derivation, so the main circularity patterns do not apply. The fused slots and masks are defined by the new equations (4)-(6) and are trained/evaluated on the datasets; no reported number is the same quantity as an input parameter by construction. The learned mask-fusion weights w in Eq. (6) are not 'predictions'—they are fit on the benchmarks where results are reported, so model-selection/overfitting risk exists, but that is not circularity. The one genuine attribution confound is flagged in Sec. 4 (Implementation): 'Contrary to SPOT, we use segmentation masks of the SA module to distill knowledge of the teacher to the student'; no single-layer SPOT with that distillation target is ablated, so the SPOT-M SOTA gain is not cleanly separable from the multi-layer fusion claim. That weakens the causal attribution but does not make the result equivalent to its input. Likewise, Supplementary Material Sec. C says 'decoder metrics do not reliably reflect the quality of the slot-object binding itself' and the paper reports the max of slot/decoder metrics; that is a measurement-convention concern, not circularity. The cited prior work ([26], [45], [1], [53], [41]) is external, and no load-bearing self-citation or imported uniqueness/ansatz from the authors' own unverified work appears. Therefore no circular step is established.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The central empirical claims rest on the semantic complementarity of DINO layers, on slot alignment via mask-based Hungarian matching, and on tuned hyperparameters (layer set, slot count, fusion/distillation weights). No new physical entities are introduced.

free parameters (4)
  • Layer index set / number of layers M = 4 consecutive layers: {9, 10, 11, 12}
    Selected via ablation on VOC (Fig. 6) as the best-performing set; used for all datasets and backbones.
  • Mask fusion weights w_m (SPOT-M) = learned, softmax-normalized (not reported)
    Equation 6 weights are learned during self-training distillation on the target datasets; DINOSAUR-M uses uniform weights.
  • Number of slots K = 6 (VOC), 7 (COCO), 11 (MOVi-C)
    Taken from SPOT/DINOSAUR; determines slot-attention capacity and is tuned per dataset.
  • Distillation weight lambda = 0.01 (per Tab. 7)
    Chosen empirically for SPOT-M; the text says 'greater weight' than [26] but the table lists 0.01.
axioms (3)
  • domain assumption DINO ViT layers at different depths provide semantically complementary, object-relevant features, with deeper layers most useful.
    Core motivation; supported by [1], PCA visualizations, and the layer-choice ablation. If false, multi-layer fusion would not help.
  • domain assumption After Hungarian matching by mask mIoU, corresponding slots across layers bind to the same object, so slot-wise summation is meaningful.
    M-Fusion (Eq. 5) sums slot vectors across adjacent layers; this requires alignment. The matching objective is segmentation masks, not slot identity.
  • domain assumption Slot attention modules trained independently on each layer with only the base model's reconstruction/distillation losses learn compatible object-centric slots.
    No auxiliary correspondence loss is used; alignment relies on emergent consistency across independent modules.

pith-pipeline@v1.3.0-alltime-deepseek · 18002 in / 12160 out tokens · 108568 ms · 2026-08-03T03:32:36.660708+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of MUFASA: A Multi-Layer Framework for Slot Attention." pith.science (2026). https://pith.science/paper/IPOPPPB4

@misc{pith2026260207544,
  author       = {Pith},
  title        = {Pith review of: MUFASA: A Multi-Layer Framework for Slot Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IPOPPPB4}},
  note         = {Machine review of arXiv:2602.07544}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Unsupervised object-centric learning (OCL) decomposes visual scenes into distinct entities. Slot attention is a popular approach that represents individual objects as latent vectors, called slots. Current methods obtain these slot representations solely from the last layer of a pre-trained vision transformer (ViT), ignoring valuable, semantically rich information encoded across the other layers. To better utilize this latent semantic information, we introduce MUFASA, a lightweight plug-and-play framework for slot-attention-based approaches to unsupervised object segmentation. Our model computes slot attention across multiple feature layers of the ViT encoder, fully leveraging their semantic richness. We propose a fusion strategy to aggregate slots obtained on multiple layers into a unified object-centric representation. Integrating MUFASA into existing OCL methods improves their segmentation results across multiple datasets, setting a new state of the art while simultaneously improving training convergence with only minor inference overhead.

Figures

Figures reproduced from arXiv: 2602.07544 by Krishnakant Singh, Leonie Sch\"u{\ss}ler, Sebastian Bock, Simone Schaub-Meyer, Stefan Roth.

Figure 1
Figure 1. Figure 1: MUFASA. Our novel framework for slot-based methods leverages multiple feature layers of vision transformers for object￾centric learning. Integrated into the current best model, SPOT [26], we achieve a new state of the art in unsupervised object segmentation on PASCAL VOC, COCO, and MOVi-C, producing high-quality segmentation masks while requiring less time to train. attention-based methods to real-world da… view at source ↗
Figure 2
Figure 2. Figure 2: Complementarity of DINO layers. (a) PCA visualiza￾tion for features from layers 4 and 10–12, each encoding varying semantics. (b) Corresponding attention masks from slot attention on these layers, showing different segmentations. (c) Segmentation mask of the single-layer SPOT. (d) The fused slot-attention mask of our SPOT-M captures the person and the dog in a single slot each and follows their boundaries … view at source ↗
Figure 3
Figure 3. Figure 3: MUFASA architecture. (a) For an input image, features from multiple layers of a DINO encoder are processed by multiple slot attention (SA) modules, each producing slots Sm and corresponding attention masks A Slot m . After Hungarian matching, a fusion module merges slots and masks. A ViT decoder reconstructs the last encoder layer’s features from fused slots, yielding the decoder attention mask A Dec. The … view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of segmentations. Exemplary segmentation masks on nine different images for SPOT-M (ours), SPOT, DINOSAUR￾M (ours), and DINOSAUR. The first three images are from VOC, the next three from COCO, and the last three from MOVi-C. Integrating MUFASA results in segmentations that follow the object boundaries more closely compared to the baselines. In [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Segmentation per layer. Layer-wise SA masks and the fused mask on COCO. Each layer contributes complementary information (e.g., row 1: the plaque and bench edges in AˆSlot 3 vs. coarse segments in AˆSlot 2 ); the fused masks appear refined. SPOT 9-10-11-12 6-8-10-12 3-6-9-12 45 50 55 60 (a) SPOT 3 4 5 45 50 55 60 (b) [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Ablations on layers. (a) Results of SPOT-M on VOC in terms of mBOi , mBOc , and mIoU for different selections of layers compared to SPOT. (b) Results of SPOT-M on VOC in terms of mBOi , mBOc , and mIoU for an increasing number of the last feature layers compared to the single-layer baseline SPOT [26]. Moreover, our models converge to a solution earlier on all datasets, suggesting that the multi-layer appro… view at source ↗
Figure 7
Figure 7. Figure 7: PCA of DINO ViT features. Layerwise visualization of the DINO ViT features at different layers via principal component analysis (PCA) for four different images. The first three principal components yield red, green, and blue channels. Semantically meaningful information is absent in earlier layers and begins to emerge in intermediate ones, while becoming increasingly rich in deeper layers. formation about … view at source ↗
Figure 8
Figure 8. Figure 8: PASCAL VOC segmentation masks. Images taken from PASCAL VOC, segmented by SPOT-M (top row), SPOT (second row), DINOSAUR-M (third row), and DINOSAUR (fourth row) compared against the ground truth (bottom row). For SPOT and DINOSAUR, segmentation masks derived from the decoder are shown, while for their respective MUFASA variant, segmentation masks from the slot attention module are depicted. 4 [PITH_FULL_I… view at source ↗
Figure 9
Figure 9. Figure 9: COCO segmentation masks. Images taken from COCO, segmented by SPOT-M (top row), SPOT (second row), DINOSAUR-M (third row), and DINOSAUR (fourth row) compared against the ground truth (bottom row). For SPOT and DINOSAUR, segmentation masks derived from the decoder are shown, while for their respective MUFASA variant, segmentation masks from the slot attention module are depicted. 5 [PITH_FULL_IMAGE:figures… view at source ↗
Figure 10
Figure 10. Figure 10: MOVi-C segmentation masks. Images taken from MOVi-C, segmented by SPOT-M (top row), SPOT (second row), DINOSAUR￾M (third row), and DINOSAUR (fourth row) compared against the ground truth (bottom row). For SPOT and DINOSAUR, segmentation masks derived from the decoder are shown, while for their respective MUFASA variant, segmentation masks from the slot attention module are depicted. 6 [PITH_FULL_IMAGE:fi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 6 linked inside Pith

  1. [1]

    On the effectiveness of ViT features as local semantic descrip- tors

    Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. On the effectiveness of ViT features as local semantic descrip- tors. InECCV Workshop on What is Motion For, 2022. 1, 2, 3, 8

    2022

  2. [2]

    Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner

    Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representation.arXiv:1901.11390 [cs.CV], 2019. 1, 2

    Pith/arXiv arXiv 1901

  3. [3]

    MMFuser: Multimodal multi-layer feature fuser for fine-grained vision-language un- derstanding.arXiv:2410.11829 [cs.CV], 2024

    Yue Cao, Yangzhou Liu, Zhe Chen, Guangchen Shi, Wenhai Wang, Danhuai Zhao, and Tong Lu. MMFuser: Multimodal multi-layer feature fuser for fine-grained vision-language un- derstanding.arXiv:2410.11829 [cs.CV], 2024. 2

    Pith/arXiv arXiv 2024

  4. [4]

    Unsupervised learning of visual features by contrasting cluster assignments.NeurIPS, pages 9912–9924, 2020

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments.NeurIPS, pages 9912–9924, 2020. 1, 2

    2020

  5. [5]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. InICCV, pages 9650–9660, 2021. 1, 2, 5, 8

    2021

  6. [6]

    Learning phrase representations using RNN encoder–decoder for statistical machine translation

    Kyunghyun Cho, Bart van Merri ¨enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In EMNLP, pages 1724–1734, 2014. 3

    2014

  7. [7]

    Moerland

    Jonathan Collu, Riccardo Majellaro, Aske Plaat, and Thomas M. Moerland. Slot structured world models. arXiv:2402.03326 [cs.CV], 2024. 1

    Pith/arXiv arXiv 2024

  8. [8]

    Zero-shot object-centric representation learning

    Aniket Didolkar, Andrii Zadaianchuk, Anirudh Goyal, Mike Mozer, Yoshua Bengio, Georg Martius, and Maximilian Seitzer. Zero-shot object-centric representation learning. In ICLR, 2025. 1

    2025

  9. [9]

    General- ization and robustness implications in object-centric learning

    Andrea Dittadi, Samuele Papa, Michele De Vita, Bernhard Sch¨olkopf, Ole Winther, and Francesco Locatello. General- ization and robustness implications in object-centric learning. InICML, pages 5221–5285, 2021. 1

    2021

  10. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InICLR, 2021. 2

    2021

  11. [11]

    Kosiorek, Oiwi Parker Jones, and Ingmar Posner

    Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative scene inference and sampling with object-centric latent representations. InICLR,

  12. [12]

    S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. InNIPS, 2016. 2

    2016

  13. [13]

    Williams, John Winn, and Andrew Zisserman

    Mark Everingham, Luc Van Gool, Christopher K.I. Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge.IJCV, pages 303–338, 2010. 2, 5

    2010

  14. [14]

    Vision meets robotics: The KITTI dataset.IJRR,

    Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI dataset.IJRR,

  15. [15]

    Multi-object representation learning with iterative variational inference

    Klaus Greff, Rapha¨el Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In ICML, pages 2424–2433, 2019. 1, 2

    2019

  16. [16]

    Kubric: A scalable dataset generator

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, et al. Kubric: A scalable dataset generator. InCVPR, pages 3749–3761, 2022. 5

    2022

  17. [17]

    Fuchs, Ingmar Posner, and Andrea Vedaldi

    Oliver Groth, Fabian B. Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision-based physical intu- ition for generalised object stacking. InECCV, pages 702– 717, 2018. 1, 2

    2018

  18. [18]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InCVPR, pages 16000–16009, 2022. 8, 1

    2022

  19. [19]

    Bridging nonlinearities and stochastic regularizers with Gaussian error linear units

    Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with Gaussian error linear units. arXiv:1606.08415 [cs.LG], 2016. 5, 2

    Pith/arXiv arXiv 2016

  20. [20]

    Visuomotor control in multi-object scenes using object-aware representations

    Negin Heravi, Ayzaan Wahid, Corey Lynch, Pete Florence, Travis Armstrong, Jonathan Tompson, Pierre Sermanet, Jean- nette Bohg, and Debidatta Dwibedi. Visuomotor control in multi-object scenes using object-aware representations. In ICRA, pages 9515–9522, 2023. 1

    2023

  21. [21]

    Improving object- centric learning with query optimization

    Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object- centric learning with query optimization. InICLR, 2022. 1

    2022

  22. [22]

    Scalor: Generative world models with scalable object representations

    Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. InICLR, 2019. 2

    2019

  23. [23]

    Object-centric slot diffusion

    Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. InNeurIPS, 2023. 1, 2

    2023

  24. [24]

    Denoising criterion for variational auto- encoding framework

    Daniel Jiwoong Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio. Denoising criterion for variational auto- encoding framework. InAAAI, 2017. 2

    2017

  25. [25]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elemen- tary visual reasoning. InCVPR, pages 2901–2910, 2017. 1, 2

    2017

  26. [26]

    SPOT: Self-training with patch-order permutation for object-centric learning with autoregressive transformers

    Ioannis Kakogeorgiou, Spyros Gidaris, Konstantinos Karantzalos, and Nikos Komodakis. SPOT: Self-training with patch-order permutation for object-centric learning with autoregressive transformers. InCVPR, pages 22776–22786,

  27. [27]

    Clevr- Tex: A texture-rich benchmark for unsupervised multi-object segmentation

    Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevr- Tex: A texture-rich benchmark for unsupervised multi-object segmentation. InNeurIPS Datasets and Benchmarks Track,

  28. [28]

    Bootstrapping top-down information for self-modulating slot attention

    Dongwon Kim, Seoyeon Kim, and Suha Kwak. Bootstrapping top-down information for self-modulating slot attention. In NeurIPS, pages 103751–103773, 2024. 1 9

    2024

  29. [29]

    The perception of hierarchical structure

    Ruth Kimchi. The perception of hierarchical structure. In The Oxford Handbook of Perceptual Organization. Oxford University Press, 2015. 1

    2015

  30. [30]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InICLR, 2014. 1

    2014

  31. [31]

    Elsayed, Aravindh Mahen- dran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jon- schkowski, Alexey Dosovitskiy, and Klaus Greff

    Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahen- dran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jon- schkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. InICLR, 2022. 2

    2022

  32. [32]

    Sequential attend, infer, repeat: Generative modelling of moving objects.NeurIPS, 31, 2018

    Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects.NeurIPS, 31, 2018. 2

    2018

  33. [33]

    Harold W. Kuhn. The Hungarian method for the assignment problem.Naval Research Logistics Quarterly, pages 83–97,

  34. [34]

    Scouter: Slot attention- based classifier for explainable image recognition

    Liangzhi Li, Bowen Wang, Manisha Verma, Yuta Nakashima, Ryo Kawasaki, and Hajime Nagahara. Scouter: Slot attention- based classifier for explainable image recognition. InICCV, pages 1046–1055, 2021. 1

    2021

  35. [35]

    Learning object- centric representations of multi-object scenes from multiple views

    Nanbo Li, Cian Eastwood, and Robert Fisher. Learning object- centric representations of multi-object scenes from multiple views. InNeurIPS, pages 5656–5666, 2020. 2

    2020

  36. [36]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, pages 740–755, 2014. 2, 5

    2014

  37. [37]

    Improving generative imagination in object-centric world models

    Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. InICML, pages 6140–6149,

  38. [38]

    Space: Unsupervised object-oriented scene representation via spatial attention and decomposition

    Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. InICLR, 2020. 2

    2020

  39. [39]

    StructDiffusion: Language-guided creation of physically-valid structures using unseen objects

    Weiyu Liu, Yilun Du, Tucker Hermans, Sonia Chernova, and Chris Paxton. StructDiffusion: Language-guided creation of physically-valid structures using unseen objects. InRSS,

  40. [40]

    Object-centric learn- ing with slot attention

    Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learn- ing with slot attention. InNeurIPS, pages 11525–11538, 2020. 1, 2, 6

    2020

  41. [41]

    Diffusion hyperfeatures: Searching through time and space for semantic correspondence

    Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holyn- ski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. In NeurIPS, pages 47500–47510, 2023. 7

    2023

  42. [42]

    DI- NOv2: Learning robust visual features without supervision

    Maxime Oquab, Timoth´ee Darcet, Th´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DI- NOv2: Learning robust visual features without supervision. arXiv:2304.07193 [cs.CV], 2023. 2, 8, 1

    Pith/arXiv arXiv 2023

  43. [43]

    Barron, Ferran Marques, and Jitendra Malik

    Jordi Pont-Tuset, Pablo Arbelaez, Jonathan T. Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial group- ing for image segmentation and object proposal generation. TPAMI, pages 128–140, 2016. 5

    2016

  44. [44]

    Do vision trans- formers see like convolutional neural networks? InNeurIPS, pages 12116–12128, 2021

    Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision trans- formers see like convolutional neural networks? InNeurIPS, pages 12116–12128, 2021. 2

    2021

  45. [45]

    Bridging the gap to real-world object- centric learning

    Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Sch ¨olkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object- centric learning. InICLR, 2022. 1, 2, 3, 5, 8

    2022

  46. [46]

    Illiterate DALL-E learns to compose

    Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-E learns to compose. InICLR, 2021. 2, 3, 6

    2021

  47. [47]

    Simple unsu- pervised object-centric learning for complex and naturalistic videos.NeurIPS, pages 18181–18196, 2022

    Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsu- pervised object-centric learning for complex and naturalistic videos.NeurIPS, pages 18181–18196, 2022. 2

    2022

  48. [48]

    Analyz- ing local representations of self-supervised vision transform- ers.arXiv:2401.00463 [cs.CV], 2024

    Ani Vanyan, Alvard Barseghyan, Hakob Tamazyan, Vahan Huroyan, Hrant Khachatrian, and Martin Danelljan. Analyz- ing local representations of self-supervised vision transform- ers.arXiv:2401.00463 [cs.CV], 2024. 2

    Pith/arXiv arXiv 2024

  49. [49]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.NIPS, 2017. 2, 3

    2017

  50. [50]

    Burgess, and Alexander Lerchner

    Nick Watters, Loic Matthey, Chris P. Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for disentangled representations in V AEs. InICLR Workshop on Learning from Limited Labeled Data, 2019. 2, 1

    2019

  51. [51]

    SlotFormer: Unsupervised visual dynamics simulation with object-centric models

    Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. SlotFormer: Unsupervised visual dynamics simulation with object-centric models. InICLR, 2023. 1

    2023

  52. [52]

    SlotDiffusion: Object-centric generative model- ing with diffusion models

    Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Ani- mesh Garg. SlotDiffusion: Object-centric generative model- ing with diffusion models. InNeurIPS, 2023. 1, 2

    2023

  53. [53]

    Dense connector for MLLMs

    Huanjin Yao, Wenhao Wu, Taojiannan Yang, YuXin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang. Dense connector for MLLMs. InNeurIPS, 2024. 1, 2, 4

    2024

  54. [54]

    Zeiler and Rob Fergus

    Matthew D. Zeiler and Rob Fergus. Visualizing and under- standing convolutional networks. InECCV, pages 818–833,

  55. [2014]

    Implementation Details In this section, we provide a more detailed overview of the training and implementation details for DINOSAUR-M and SPOT-M

    2 10 MUFASA: A Multi-Layer Framework for Slot Attention Supplementary Material Sebastian Bock ∗1,2 Leonie Sch¨ußler ∗1,2 Krishnakant Singh 1 Simone Schaub-Meyer 1,3 Stefan Roth 1,2,3 1TU Darmstadt 2Zuse School ELIZA 3hessian.AI ∗equal contribution A. Implementation Details In this section, we provide a more detailed overview of the training and implementa...