Pith. sign in

REVIEW 2 major objections 5 minor 18 references

Object-centric slots beat dense vision features for robot pick-and-place by 22 points with the same policy and no encoder fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 15:15 UTC pith:YYCTIMDD

load-bearing objection Clean matched sim result: frozen SPOT slots beat DINO global/4×4 by ~22 points under held-out seeds; the structure-not-capacity slogan is real but rests on Global vs 4×4 vs SPOT, not the handicapped 14×14 row. the 2 major comments →

arxiv 2607.09825 v1 pith:YYCTIMDD submitted 2026-07-10 cs.RO cs.AI

More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning

Yi Li (TU Darmstadt) , Alexandre Chapin (LIRIS) , Liming Chen (LIRIS) , Jan Peters (TU Darmstadt) , Alap Kshirsagar (IIT Delhi , ADU) This is my paper
classification cs.RO cs.AI
keywords object-centric representationsslot attentionvisuomotor imitationfrozen vision encodersrobotic manipulationfailure taxonomyspatial grounding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Robotic pick-and-place policies only ever see what their visual encoder keeps, so the structure of that encoder decides whether the policy can act on object and goal positions it never saw in training. This paper holds the policy, goal token, rendering, and calibration fixed and changes only the frozen visual encoder. An object-centric slot representation that compresses each camera view into seven competing per-object slots reaches 55 percent held-out success, 22 points above a dense global DINO feature, while a 16-times denser patch grid does no better than the global feature. Adding an explicit 2D spatial goal and native-resolution rendering lifts the full system to 68.7 percent, just below a privileged 3D oracle. An automated kinematic failure taxonomy then shows that spatial grounding fixes near-miss placement errors but leaves object-tracking failures untouched, and the same taxonomy on a harder stacking task points to occlusion as the remaining bottleneck.

Core claim

Under matched conditions on ManiSkill3 PickCube-v1, a frozen object-centric SPOT representation (DINO ViT-B/16 plus Slot Attention) yields 55.0 ± 2.9 percent held-out success, 22.4 points above a dense DINO global-feature baseline, while a dense patch grid with far more tokens performs no better than the global feature. The gain therefore comes from object-level structure, not capacity or token count. With an explicit 2D spatial goal and native-resolution rendering the system reaches 68.7 percent, just below a 3D-oracle upper bound of 71.7 percent.

What carries the argument

Slot Attention (as implemented in the frozen SPOT encoder): a competition-based bottleneck that routes 196 dense patch tokens into seven slots per camera so that each patch commits to at most one slot, forcing the limited capacity onto coherent object regions rather than scene-level statistics.

Load-bearing premise

The publicly released, COCO-pretrained slot encoder already produces stable, task-relevant object bindings on the simulator images without any task-specific adaptation or temporal memory, so measured success differences can be blamed only on structure.

What would settle it

Re-run the identical matched-policy comparison after replacing the frozen COCO-pretrained slots with a version that is either fine-tuned on ManiSkill images or given temporal slot memory; if the 22-point gap over dense DINO disappears or reverses, the claim that structure alone drives the gain collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies whether object-centric structure improves visuomotor imitation under matched conditions. On ManiSkill3 PickCube-v1, with a frozen encoder, fixed policy, fixed 2D goal token, fixed rendering, and held-out seeds ≥10000, a SPOT (DINO ViT-B/16 + Slot Attention) representation reaches 55.0±2.9% success versus 32.6±1.5% for a DINO global [CLS] baseline and 31.7±3.0% for a 4×4 patch grid; a 14×14 dense grid collapses after being forced to T=1, H=1 by memory. Adding exact calibration and native 224² rendering raises the SPOT system to 68.7±4.2%, near a privileged 3D-oracle bound (71.7±4.1%). An automated kinematic taxonomy (Idle / No-Grasp / Drop / Near-Miss / Unknown) shows that spatial grounding mainly reduces Near-Miss failures while No-Grasp stays flat, and the same taxonomy on StackCube-v1 attributes the low success rate to occlusion-driven No-Grasp failures.

Significance. If the matched-condition result holds, the paper supplies a clean, task-level demonstration that a small competition-based bottleneck on frozen foundation features can outperform denser global and patch representations for placement generalization, without encoder fine-tuning. Strengths include the held-out-seed protocol, three independent 300-episode stability runs with mean±std, explicit isolation of encoder structure from policy and goal conditioning, and an encoder-agnostic kinematic failure taxonomy that transfers across tasks and separates spatial-precision from object-tracking failures. These are concrete, reproducible contributions for practitioners who must choose frozen visual front-ends under a single-GPU budget. The work is complementary to Chapin et al. (visual distribution shifts) by focusing on novel object/goal placements and by adding the grounding and taxonomy analyses.

major comments (2)
  1. [Table I, §V-A, Fig. 2] Table I upper block and §V-A / Fig. 2: the slogan “more tokens alone do not help” and the structure-not-capacity claim rest partly on the DINO 14×14 dense row (392 tokens, 1.0% SR). That row is not a matched capacity control: GPU memory forces T=1, H=1, stripping the temporal window and action chunking (T=5, H=10) that every other configuration enjoys. The paper notes this, yet still presents the collapse as part of the token-count argument. The clean matched evidence is Global (2 tokens, 32.6%) vs 4×4 (32 tokens, 31.7%) vs SPOT (14 tokens, 55.0%). A memory-feasible intermediate dense baseline (e.g. 7×7 or 8×8 patches kept at T=5, H=10) is needed to fully support the capacity claim; without it the 14×14 result should be demoted to a stress-test footnote rather than a primary control.
  2. [§III-A, Fig. 3, Table I] §III-A and the matched comparison: the central attribution of the 22.4-point gap to “object-level structure” assumes that the frozen COCO-pretrained SPOT checkpoint (K=7, fixed) produces sufficiently stable, task-relevant bindings on ManiSkill RGB without task adaptation or temporal memory. Fig. 3 shows one clean episode, but there is no quantitative binding analysis across the held-out evaluation set (e.g. fraction of episodes in which a slot consistently covers the cube/goal, or correlation of binding quality with success/failure). Residual binding failures or viewpoint-specific artifacts could inflate the measured gap. A short quantitative binding audit on the evaluation seeds would make the structure attribution load-bearing rather than illustrative.
minor comments (5)
  1. [Abstract, §V-A] Abstract and §V-A: “22.4% above” is a percentage-point gap (55.0 − 32.6), not a relative 22.4% improvement. Use “22.4 percentage points” consistently.
  2. [Table II] Table II: training success rates are described as approximate and “read off during training on seen initializations.” Either report them under the same held-out protocol or clearly mark them as qualitative only in the table caption.
  3. [§IV] §IV failure taxonomy: the Near-Miss distance threshold (0.1 m) and Idle variance threshold are free parameters. State them explicitly and, if possible, note sensitivity.
  4. [Fig. 4, Fig. 5] Fig. 4 and Fig. 5 report raw episode counts from single 200-episode runs; adding percentages (as in the StackCube funnel) would make cross-condition comparison easier.
  5. [§V-C, §VI] §VI: the fusion comparison was run only without a goal token. A one-sentence caveat in the main results (not only Discussion) would prevent over-generalization of the concatenation preference.

Circularity Check

0 steps flagged

No circularity: purely empirical matched-encoder comparison with held-out-seed success rates measured on ManiSkill3; self-citations are background infrastructure only.

full rationale

The paper's load-bearing claims are measured success rates (Table I, Fig. 2) under a frozen-encoder, held-out-seed protocol that changes only the visual representation while holding policy, goal token, rendering and calibration fixed. These numbers are obtained by training a decoder on 1 000 expert demos and evaluating on disjoint seeds ≥ 10 000; they are not derived from any fitted free parameter that reappears as a 'prediction', nor from any equation that reduces by construction to its inputs. The SPOT checkpoint and the related-work citation to Chapin et al. (overlapping authors) supply the encoder and prior context, but the 22.4-point gap, the failure taxonomy counts, and the StackCube transfer are new empirical observations that stand or fall on the simulator runs themselves. No uniqueness theorem, ansatz, or self-definitional identity is invoked. The experimental-design caveat that the 14 imes14 dense row is memory-handicapped (T=1, H=1) is a validity concern, not circularity. Hence score 0 with empty steps.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The central claim is an empirical comparison that rests on standard simulator and architecture assumptions plus a small set of fixed design choices inherited from public checkpoints and prior codebases. No new physical entities are postulated. Free parameters are ordinary hyperparameters and the fixed K=7 of the released SPOT model; the axioms are domain assumptions of behavior cloning and frozen-feature probing rather than ad-hoc postulates invented to force the result.

free parameters (4)
  • K (number of slots) = 7
    Fixed at 7 by the public SPOT checkpoint; not tuned on the task. The capacity bottleneck and binding competition depend on this choice.
  • Temporal window T and action horizon H = T=5, H=10 (main); T=1,H=1 (dense)
    T=5, H=10 chosen for the main policy; dense 14×14 forced to T=1,H=1 by memory. Affects fairness of the token-count comparison.
  • Policy architecture sizes = 8L/8H/256
    8-layer 8-head causal Transformer, hidden dim 256, 3-layer action MLP; standard but hand-chosen and shared across all encoder conditions.
  • Near-Miss distance threshold = 0.1 m
    0.1 m TCP-to-goal cutoff used by the automated failure taxonomy; directly partitions Near-Miss from other failures.
axioms (5)
  • domain assumption Frozen COCO-pretrained SPOT (DINO ViT-B/16 + Slot Attention) produces object-level bindings that remain useful for ManiSkill pick-and-place without task fine-tuning.
    Invoked in §III-A and throughout the matched comparison; the entire structure-vs-capacity claim treats the released checkpoint as a fixed, adequate feature extractor.
  • domain assumption Held-out random seeds (≥10 000) with randomized object/goal poses constitute a valid test of generalization to novel placements rather than memorization.
    Stated in §IV evaluation protocol; success rates are interpreted as generalization under this seed split.
  • domain assumption Behavior cloning of 1 000 motion-planner demonstrations with a causal Transformer action-chunking policy is a fair decoder probe of frozen visual representations.
    Method §III-C and experimental setup; differences are attributed to the encoder under this fixed policy class.
  • domain assumption Simulator ground-truth goal locations projected to 2D (or given as 3D) are legitimate conditioning signals for measuring the value of spatial grounding.
    §III-B explicitly notes the 2D projection is not self-supervised; the spatial-grounding ladder and oracle comparison rest on this privileged signal.
  • standard math Standard Slot Attention competition (softmax over slots) and causal Transformer self-attention behave as described in the cited literature.
    Equations (1)–(2) and the GPT-style decoder; used as black-box modules.
invented entities (1)
  • Automated kinematic failure taxonomy (Idle / No-Grasp / Drop / Near-Miss / Unknown) no independent evidence
    purpose: Partition failed episodes into spatial-precision vs object-tracking modes without using visual features, enabling encoder-agnostic diagnosis and cross-task transfer.
    Defined in §IV and applied in §V-D/E; useful diagnostic but a paper-specific labeling scheme rather than a new physical or mathematical object. Independent evidence is limited to internal consistency across two ManiSkill tasks.

pith-pipeline@v1.1.0-grok45 · 15203 in / 3749 out tokens · 41086 ms · 2026-07-14T15:15:34.472921+00:00 · methodology

0 comments
read the original abstract

Robotic manipulation policies rely on pre-trained vision models that give either a global scene embedding or a dense patch grid. Both mix task-relevant and task-irrelevant features. Object-centric slot representations are a structured alternative: they group features into a few per-object slots. We test what this structure buys on ManiSkill3 PickCube-v1, with a frozen encoder and a held-out-seed evaluation. Holding the policy, goal token, rendering, and calibration fixed and changing only the encoder, a frozen object-centric SPOT representation (DINO ViT-B/16 + Slot Attention) reaches 55.0$\pm$2.9% success, 22.4% above a dense DINO global-feature baseline (32.6 $\pm$ 1.5%), with the same trainable policy and no encoder fine-tuning. More tokens alone do not help: a dense patch grid with 16x the tokens performs no better than the global feature. Adding an explicit 2D spatial goal and native-resolution rendering raises the full system to 68.7$\pm$4.2%, just below a privileged 3D-oracle upper bound (71.7$\pm$4.1%). An automated kinematic failure taxonomy then separates spatial-precision (Near-Miss) failures from object-tracking (No-Grasp) failures: spatial grounding reduces Near-Miss while leaving No- Grasp unchanged. The same taxonomy transfers to the harder StackCube-v1 and points to occlusion as the main bottleneck.

Figures

Figures reproduced from arXiv: 2607.09825 by ADU), Alap Kshirsagar (IIT Delhi, Alexandre Chapin (LIRIS), Jan Peters (TU Darmstadt), Liming Chen (LIRIS), Yi Li (TU Darmstadt).

Figure 1
Figure 1. Figure 1: System overview. RGB images from two fixed cameras are encoded by the frozen SPOT encoder (DINO ViT-B/16 + Slot Attention) into 7 object slots [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Success rate across representation types under matched conditions; the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Failure breakdown across goal-conditioning conditions. Near-Miss [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 2 linked inside Pith

  1. [1]

    Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, Ian Reid, and John J. Leonard. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age.IEEE Transactions on Robotics, 32(6):1309–1332, 2016

    2016

  2. [2]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), 2021

    2021

  3. [3]

    Spotlighting task-relevant features: Object-centric representations for better gen- eralization in robotic manipulation

    Alexandre Chapin, Bruno Machado, Emmanuel Del- landréa, and Liming Chen. Spotlighting task-relevant features: Object-centric representations for better gen- eralization in robotic manipulation. arXiv:2601.21416, 2026

    Pith/arXiv arXiv 2026

  4. [4]

    Dif- fusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Dif- fusion policy: Visuomotor policy learning via action diffusion. InRobotics: Science and Systems (RSS), 2023

    2023

  5. [5]

    Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Georg Heigold, and Thomas Kipf

    Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Georg Heigold, and Thomas Kipf. SA Vi++: Towards end-to-end object-centric learn- ing from real-world videos. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  6. [6]

    SPOT: Self-training with patch-order permutation for object-centric learning with autoregressive transformers

    Ioannis Kakogeorgiou, Spyros Gidaris, Konstantinos Karantzalos, and Nikos Komodakis. SPOT: Self-training with patch-order permutation for object-centric learning with autoregressive transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  7. [7]

    Elsayed, Aravindh Ma- hendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Gr- eff

    Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Ma- hendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Gr- eff. Conditional object-centric learning from video. In International Conference on Learning Representations (ICLR), 2022

    2022

  8. [8]

    Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto

    Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. InInternational Conference on Machine Learning (ICML), 2024

    2024

  9. [9]

    Object-centric learning with slot attention

    Francesco Locatello, Dirk Weissenborn, Thomas Un- terthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. InAdvances in Neural Information Processing Systems (NeurIPS), volume 33, 2020

    2020

  10. [10]

    R3M: A universal visual representation for robot manipulation.arXiv preprint arXiv:2203.12601, 2022

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation.arXiv preprint arXiv:2203.12601, 2022

    Pith/arXiv arXiv 2022

  11. [11]

    UniGaze: Towards universal gaze estimation via large- scale pre-training

    Jiawei Qin, Xucong Zhang, and Yusuke Sugano. UniGaze: Towards universal gaze estimation via large- scale pre-training. InIEEE Winter Conference on Appli- cations of Computer Vision (WACV), 2025

    2025

  12. [12]

    Real-world robot learning with masked visual pre-training

    Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. InCon- ference on Robot Learning (CoRL), 2022

    2022

  13. [13]

    Bridging the gap to real-world object-centric learning

    Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Martin Engelcke. Bridging the gap to real-world object-centric learning. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  14. [14]

    Vision trans- formers need more than registers

    Cheng Shi, Yizhou Yu, and Sibei Yang. Vision trans- formers need more than registers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

    2026

  15. [15]

    ManiSkill3: GPU parallelized robotics simulation and rendering for generalizable embodied AI.Robotics: Science and Systems, 2025

    Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Na- gaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. ManiSkill3: GPU parallelized robotics simulation and...

    2025

  16. [16]

    SlotDiffusion: Object-centric generative modeling with diffusion models

    Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. SlotDiffusion: Object-centric generative modeling with diffusion models. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  17. [17]

    Utonia: Toward one encoder for all point clouds

    Yujia Zhang, Xiaoyang Wu, Yunhan Yang, Xianzhe Fan, Han Li, Yuechen Zhang, Zehao Huang, Naiyan Wang, and Hengshuang Zhao. Utonia: Toward one encoder for all point clouds. InInternational Conference on Machine Learning (ICML), 2026

    2026

  18. [18]

    Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn

    Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipu- lation with low-cost hardware. InRobotics: Science and Systems (RSS), 2023

    2023