REVIEW 14 cited by
Conditional Object-Centric Learning from Video
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Object-centric representations are a promising path toward more systematic generalization by providing flexible abstractions upon which compositional world models can be built. Recent work on simple 2D and 3D datasets has shown that models with object-centric inductive biases can learn to segment and represent meaningful objects from the statistical structure of the data alone without the need for any supervision. However, such fully-unsupervised methods still fail to scale to diverse realistic data, despite the use of increasingly complex inductive biases such as priors for the size of objects or the 3D geometry of the scene. In this paper, we instead take a weakly-supervised approach and focus on how 1) using the temporal dynamics of video data in the form of optical flow and 2) conditioning the model on simple object location cues can be used to enable segmenting and tracking objects in significantly more realistic synthetic data. We introduce a sequential extension to Slot Attention which we train to predict optical flow for realistic looking synthetic scenes and show that conditioning the initial state of this model on a small set of hints, such as center of mass of objects in the first frame, is sufficient to significantly improve instance segmentation. These benefits generalize beyond the training distribution to novel objects, novel backgrounds, and to longer video sequences. We also find that such initial-state-conditioning can be used during inference as a flexible interface to query the model for specific objects or parts of objects, which could pave the way for a range of weakly-supervised approaches and allow more effective interaction with trained models.
Forward citations
Cited by 14 Pith papers
-
From an Image to a Scene: Learning to Imagine the World from a Million 360 Videos
A diffusion model trained on 1 million 360-degree videos synthesizes novel views with camera translation and enables 3D reconstruction from a single image.
-
Factored Latent Action World Models
FLAM splits a scene into separate factors, each with its own latent action, and reports better video prediction and downstream policy learning than monolithic latent-action models.
-
Spotlighting Task-Relevant Features: Object-Centric Representations for Better Generalization in Robotic Manipulation
Slot-based object-centric visual representations, especially with robot-video pretraining, improve out-of-distribution generalization of robotic manipulation policies compared to global and dense pre-trained features.
-
Learning Object-Centric Representations in SAR Images with Multi-Level Feature Fusion
SlotSAR fuses wavelet scattering features with a SAR foundation model's semantic features to make slot attention separate targets from clutter in SAR images, improving segmentation metrics on ATRNet-STAR.
-
Discovering and using Spelke segments
SpelkeNet, a self-supervised video world model, discovers Spelke segments in static images by aggregating motion correlations across imagined pokes.
-
Dyn-O: Building Structured World Models with Object-Centric Representations
Dyn-O learns object-centric world models directly from pixels in complex Procgen games, using SAM2-guided slot attention and Mamba state-space dynamics, and reports better rollout prediction than DreamerV3.
-
Identifiable Object Representations under Spatial Ambiguities
VISA learns view-invariant object representations by aggregating probabilistic slots across multiple unlabeled viewpoints, with an identifiability analysis up to affine and permutation equivalence.
-
Object-Centric Representations Improve Policy Generalization in Robot Manipulation
Slot-based object-centric representations, especially a video model pretrained on robot data, improve policy generalization under visual distribution shifts in simulated and real-world manipulation tasks.
-
Dreamweaver: Learning Compositional World Models from Pixels
An unsupervised recurrent block-slot model that discovers static and dynamic concept blocks from raw video and recombines them to imagine novel future videos.
-
Leveraging Color Channel Independence for Improved Unsupervised Object Detection
Adding the HSV saturation channel to the RGB reconstruction target improves Slot Attention object discovery and disentanglement across several multi-object datasets.
-
ES-Merging: Biological MLLM Merging via Embedding Space Signals
ES-Merging estimates layer-wise and element-wise merge coefficients from coarse- and fine-grained embedding signals and claims better cross-modal reasoning and single-modal knowledge preservation than parameter-space merging.
-
Efficient Object-centric Representation Learning with Pre-trained Geometric Prior
Using CroCo's geometric features as both the encoder and reconstruction target improves object discovery in slot-attention video models, with a one-pass attentional decoder.
-
Is an object-centric representation beneficial for robotic manipulation ?
Evaluating the object-centric SAVi encoder against the global DINO and R3M representations on three simulated manipulation tasks, the authors find SAVi is the only model to solve the pick task and is more robust to un...
-
On the Benefits of Instance Decomposition in Video Prediction Models
Explicit instance decomposition with per-class shared weights improves latent-transformer video prediction in the paper's experiments, but the claimed advantage is weakened by mismatched parameter counts and test-set ...
Discussion (0). Continue with ORCID to comment.