REVIEW 4 major objections 4 minor 31 references
Looking Locally: Object-Centric Vision Transformers as Foundation Models for Efficient Segmentation
T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read FLIP, a fovea-like object-centric vision transformer, claims that sampling multi-resolution patches around an object's 2D Gaussian prompt segments that object without encoding the full image, beating SAM-family models with orders of…
desk verdict A clever foveal-sampling architecture, but the headline SAM comparison is confounded by prompt asymmetry — worth review, not yet proven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the fovea-like input patching module: it turns the object prompt into a 2D Gaussian N(µ,Σ), then draws K patch sizes with counts proportional to the Gaussian integral, sampling continuous-coordinate patches with bilinear interpolation and a spatial hash grid that caps overlap. Because sampling is driven only by µ and Σ, the model is independent of image resolution; the same token budget is spent around the object whether the object is tiny or huge. The encoder and pixel-predictor then use positional embeddings computed from coordinates shifted, rotated and scaled by this Gaussian, which lets the transformer reason about where each patch and pixel sits relative to the object centre in a scale-free frame.
What would settle it
Evaluate FLIP-Tiny using an isotropic Gaussian fitted to the same bounding-box prompt given to SAM (no orientation, sigma from box size) and compare mean IoU on COCO and ObjaScale; if the margin over SAM-H collapses, the reported advantage is carried by the extra orientation and center information in the ground-truth Gaussian prompt.
Extended reading notes
Core claim
The paper's central claim is that segmentation quality and parameter efficiency can both improve by replacing whole-image encoding with fovea-like, object-centred input patching. Concretely, FLIP samples N multi-resolution patches from a 2D Gaussian N(µ,Σ) derived from the target object, embeds each patch with a resolution-specific MLP, processes the token set with a ViT whose attention uses layer-wise relative positional embeddings, and predicts masks by attention over pixel-coordinate queries. Trained end-to-end on SA-1B, FLIP variants surpass all tested SAM, EfficientSAM, MobileSAM and FastSAM models in mean IoU across Hypersim, KITTI-360, OpenImages, COCO, LVIS and the new ObjaScale dataset, while FLIP-Large runs about 6x faster than SAM-H. On ObjaScale, which spans object sizes from 0.0001% to 25% of image area, FLIP keeps 87-89% IoU where SAM-H achieves 73.76%, and the paper attributes this to scale invariance governed by absolute pixel coverage rather than relative object size.
Load-bearing premise
FLIP's evaluation always gives it a 2D Gaussian prompt computed from the ground-truth mask, which encodes the object's centre, size and orientation, while SAM receives only a bounding box; if this extra prompt information is what drives the gap, the parameter-efficiency advantage could vanish when prompts must be predicted rather than taken from ground truth.
Editorial extensions
If this is right
- A 0.51M-parameter segmentation model with 78.24% mean IoU makes real-time, on-device segmentation plausible without a large image encoder.
- Small-object segmentation no longer degrades with image resolution; FLIP's accuracy on ObjaScale remains high down to roughly 10x10 pixel objects, which matters for high-resolution remote sensing, medical imaging and autonomous driving.
- The same object-centred token representation can be reused for tracking, so slot-based scene segmentation architectures can build on FLIP features rather than full-image encodings.
- Because FLIP was trained only on SA-1B and not fine-tuned on the evaluation datasets, the reported cross-dataset gains support the claim that a foveal input strategy generalises as a foundation-model design.
Reading between the lines
- A testable extension is to feed FLIP a Gaussian estimated from a predicted bounding box instead of the ground-truth mask; the paper does not report this setting, and it is needed to know how much of the advantage is architectural rather than prompt-borne.
- The orientation information in the Gaussian may be worth several IoU points on elongated objects; comparing FLIP against a box-prompted SAM under an orientation-free isotropic Gaussian would isolate this effect.
- The paper's ObjaScale results suggest a broader principle: for object-centric models, sampling resolution should scale with absolute object size, not with image size; this principle could be carried into video object segmentation benchmarks to test temporal consistency.
- A practical consequence the author leaves implicit: for deployment, FLIP still needs a separate mechanism to propose Gaussian prompts, so a complete system pairs FLIP with a detector or slot-based proposer.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FLIP, a transformer-based segmentation model that replaces full-image encoding with fovea-like multi-resolution patch sampling centered on a 2D Gaussian prompt. The Gaussian is derived from the ground-truth mask, and the sampled patches are encoded by a small ViT whose per-layer positional embeddings are normalized by the same Gaussian; a pixel-predictor then produces mask logits via attention over query pixel coordinates. FLIP is trained on SA-1B and evaluated on Hypersim, KITTI-360, OpenImages, COCO, LVIS, and a new Blender-based ObjaScale dataset. The central claim is that FLIP variants outperform the SAM family while using orders of magnitude fewer parameters, e.g., FLIP-Tiny at 0.51M parameters reaching 78.24% mean IoU versus SAM-H at 75.41% with 641.1M parameters.
Significance. The core architectural idea is interesting and the engineering is nontrivial: off-grid continuous patch coordinates, resolution-specific embedding modules, per-layer positional embeddings, sparse boundary-focused supervision, and hierarchical inference are combined into a very small model. The paper also introduces ObjaScale, a scale-stress benchmark, and reports standard deviations and ablation studies, which are useful for assessing robustness. If the efficiency and accuracy comparisons were prompt-matched, this would be a valuable contribution to efficient object-centric segmentation. As it stands, however, the headline comparison is confounded by an asymmetry between the information provided to FLIP and to the SAM baselines, and the strongest claims are not supported by the reported tables.
major comments (4)
- [§3.1, §3.5, §4.1]
- [Abstract and §4.2]
- [Abstract (opening paragraph) and §4]
- [§4.4 and Table 3]
minor comments (4)
- [§4.1]
- [Table 1]
- [§3.4]
- [§6]
Circularity Check
No circular derivation: FLIP's headline numbers are empirical benchmark measurements; the GT-derived Gaussian prompt is a prompt-fairness concern, not a reasoning loop.
full rationale
FLIP's claimed contributions are evaluated empirically rather than derived. Sections 3.1-3.4 specify an architecture and a supervised loss, and the performance claims rest on IoU measurements in Tables 1 and 3 against five external benchmarks (Hypersim, KITTI-360, OpenImages, COCO, LVIS) plus the authors' ObjaScale dataset. No equation in the paper converts a model definition into a score, and no fitted parameter is relabeled as a prediction. The self-citations (Traub et al., 2023; 2024a;b) occur only as contextual references to prior object-centric models and are not load-bearing for the FLIP-vs-SAM efficiency claim. The genuine concern is an evaluation-protocol asymmetry: Section 3.1 derives the 2D Gaussian prompt from the ground-truth mask, Section 4.1 states SAM received only bounding-box prompts while FLIP used Gaussian prompts by design, and Section 3.5 restricts FLIP's inference queries to a 5-sigma region computed from that same GT-derived prompt. This could bias the headline comparison, especially on ObjaScale, but it is a benchmark-fairness issue rather than circular reasoning: the output mask is not defined to equal the Gaussian input, and the five established-benchmark results are independent external measurements. Under the requirement to exhibit a specific reduction of a prediction to its inputs by construction, no circularity step is established.
Assumptions & free parameters
free parameters (6)
- Patch count N =
512 (varied around mean during training)
- Patch sizes p_i =
1x1, 2x2, 4x4, 8x8, 16x16 (from Figure 2)
- Coverage parameter c =
range (0.1, 2)
- Overlap threshold tau =
range [0,4]
- 5-sigma bounding box factor =
5
- Hierarchical refinement parameters =
tau=0.01, alpha=4
assumptions (4)
- domain assumption A 2D Gaussian adequately approximates the object's spatial extent and orientation.
- domain assumption Training on SA-1B alone transfers to the six evaluation datasets.
- ad hoc to paper Sparse boundary-focused pixel supervision (2048 pixels per image) is sufficient to learn accurate dense masks.
- domain assumption Mean IoU across six datasets is a meaningful proxy for general segmentation performance.
Cite this review
Pith. "Pith review of Looking Locally: Object-Centric Vision Transformers as Foundation Models for Efficient Segmentation." pith.science (2026). https://pith.science/paper/AFRXODE4
@misc{pith2026250202763,
author = {Pith},
title = {Pith review of: Looking Locally: Object-Centric Vision Transformers as Foundation Models for Efficient Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/AFRXODE4}},
note = {Machine review of arXiv:2502.02763}
}
abstract
Current state-of-the-art segmentation models encode entire images before focusing on specific objects. This wastes computational resources. We introduce FLIP (Fovea-Like Input Patching), a parameter-efficient vision model that realizes object segmentation through biologically-inspired top-down attention. FLIP selectively samples multi-resolution patches centered on objects of interest from the input. As a result, it allocates high-resolution processing to object centers while maintaining coarser peripheral context. This off-grid, scale-invariant design enables FLIP to outperform META's Segment Anything models (SAM, SAM2 and fast variants) by large margins: With more than 440$\times$ fewer parameters, FLIP-Tiny (0.51M parameters) reaches a mean IoU of 79.90% while SAM2-L reaches 75.87% IoU (224.45M parameters). FLIP-Large even achieves 83.26% mean IoU (96.6M parameters), still running about $2\times$ faster than SAM2-L. We evaluate on six benchmarks in total. In five established benchmarks (Hypersim, KITTI-360, OpenImages, COCO, LVIS) FLIP consistently outperforms SAM and various variants of it. In our novel ObjaScale dataset, which stress-tests scale invariance with objects ranging from 0.0001% up to 25% of the image area, we show that FLIP segments even very small objects accurately, where existing models fail severely. FLIP opens new possibilities for real-time, object-centric vision and offers much higher energy efficiency. We believe that FLIP can act as a powerful foundation model, as it is very well-suited to track objects over time, for example, when being integrated into slot-based scene segmentation architectures.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Focal sparse convolutional networks for 3d object detection
Chen, Y., Li, Y., Zhang, X., Sun, J., and Jia, J. Focal sparse convolutional networks for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 5428--5437, 2022
work page 2022
-
[2]
Deformable convolutional networks
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., and Wei, Y. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp.\ 764--773, 2017
work page 2017
-
[3]
Objaverse: A universe of annotated 3d objects
Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., and Farhadi, A. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 13142--13153, June 2023
work page 2023
-
[4]
An image is worth 16x16 words: Transformers for image recognition at scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020
arXiv 2010
-
[5]
Elsayed, G., Mahendran, A., van Steenkiste, S., Greff, K., Mozer, M. C., and Kipf, T. Savi++: Towards end-to-end object-centric learning from real-world videos. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 28940--28954. Curran Associates, Inc., 2022
work page 2022
-
[6]
Lvis: A dataset for large vocabulary instance segmentation
Gupta, A., Dollar, P., and Girshick, R. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 5356--5364, 2019
2019
-
[7]
Perceiver io: A general architecture for structured inputs & outputs
Jaegle, A., Borgeaud, S., Alayrac, J.-B., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021
arXiv 2021
-
[8]
S., Sochenov, A., Leimk \"u hler, T., Okunev, M., Goodall, T., and Rufo, G
Kaplanyan, A. S., Sochenov, A., Leimk \"u hler, T., Okunev, M., Goodall, T., and Rufo, G. Deepfovea: Neural reconstruction for foveated rendering and video compression using learned statistics of natural videos. ACM Transactions on Graphics (TOG), 38 0 (6): 0 1--13, 2019
work page 2019
Show all 31 references
-
[9]
C., Lo, W.-Y., et al
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4015--4026, 2023
2023
-
[10]
The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International jour...
1956
-
[11]
Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d
Liao, Y., Xie, J., and Geiger, A. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (3): 0 3292--3310, 2022
2022
-
[12]
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer vision--ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pp....
2014
-
[13]
A convnet for the 2020s
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 11976--11986, 2022
2022
-
[14]
Object-centric learning with slot attention
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33: 0 11525--11538, 2020
2020
-
[15]
Biologically inspired deep learning model for efficient foveal-peripheral vision
Lukanov, H., K \"o nig, P., and Pipa, G. Biologically inspired deep learning model for efficient foveal-peripheral vision. Frontiers in Computational Neuroscience, 15: 0 746204, 2021
2021
-
[16]
A., Paczan, N., Webb, R., and Susskind, J
Roberts, M., Ramapuram, J., Ranjan, A., Kumar, A., Bautista, M. A., Paczan, N., Webb, R., and Susskind, J. M. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)...
2021
-
[17]
Simple unsupervised object-centric learning for complex and naturalistic videos
Singh, G., Wu, Y.-F., and Ahn, S. Simple unsupervised object-centric learning for complex and naturalistic videos. Advances in Neural Information Processing Systems, 35: 0 18181--18196, 2022
2022
-
[18]
Fovea: Foveated image magnification for autonomous navigation
Thavamani, C., Li, M., Cebron, N., and Ramanan, D. Fovea: Foveated image magnification for autonomous navigation. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 15539--15548, 2021
2021
-
[19]
Traub, M., Otte, S., Menge, T., Karlbauer, M., Thuemmel, J., and Butz, M. V. Learning what and where: Disentangling location and identity tracking without supervision. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?...
2023
-
[20]
Traub, M., Becker, F., Otte, S., and Butz, M. V. Learning object permanence from videos via latent imaginations. In International Conference on Artificial Neural Networks, pp.\ 223--240. Springer, 2024 a
2024
-
[21]
Traub, M., Becker, F., Sauter, A., Otte, S., and Butz, M. V. Loci-segmented: improving scene segmentation learning. In International Conference on Artificial Neural Networks, pp.\ 45--61. Springer, 2024 b
2024
-
[22]
Tinyvit: Fast pretraining distillation for small vision transformers
Wu, K., Zhang, J., Peng, H., Liu, M., Xiao, B., Fu, J., and Yuan, L. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference on computer vision, pp.\ 68--85. Springer, 2022
2022
-
[23]
Efficientsam: Leveraged masked image pretraining for efficient segment anything
Xiong, Y., Varadarajan, B., Wu, L., Xiang, X., Xiao, F., Zhu, C., Dai, X., Wang, D., Sun, F., Iandola, F., Krishnamoorthi, R., and Chandra, V. Efficientsam: Leveraged masked image pretraining for efficient segment anything. arXiv preprint arXiv:2312.00863, 2023
2023 arXiv
-
[24]
Efficient deformable convnets: Rethinking dynamic and sparse operator for vision applications
Xiong, Y., Li, Z., Chen, Y., Wang, F., Zhu, X., Luo, J., Wang, W., Lu, T., Li, H., Qiao, Y., et al. Efficient deformable convnets: Rethinking dynamic and sparse operator for vision applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...
2024
-
[25]
Lape: Layer-adaptive position embedding for vision transformers with independent layer normalization
Yu, R., Wang, Z., Wang, Y., Li, K., Liu, C., Duan, H., Ji, X., and Chen, J. Lape: Layer-adaptive position embedding for vision transformers with independent layer normalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 5886--5896, 2023
2023
-
[26]
Hdri haven, 2016
Zaal, Greg. Hdri haven, 2016. URL https://hdri-haven.com/. Accessed: 2024-11-20
2016
-
[27]
Object-centric learning for real-world videos by predicting temporal feature similarities
Zadaianchuk, A., Seitzer, M., and Martius, G. Object-centric learning for real-world videos by predicting temporal feature similarities. Advances in Neural Information Processing Systems, 36, 2024
2024
-
[28]
U., Bae, S.-H., Lee, S., and Hong, C
Zhang, C., Han, D., Qiao, Y., Kim, J. U., Bae, S.-H., Lee, S., and Hong, C. S. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289, 2023
2023 arXiv
-
[29]
Fast segment anything, 2023
Zhao, X., Ding, W., An, Y., Du, Y., Yu, T., Li, M., Tang, M., and Wang, J. Fast segment anything, 2023
2023
-
[30]
Deformable convnets v2: More deformable, better results
Zhu, X., Hu, H., Lin, S., and Dai, J. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 9308--9316, 2019
2019
-
[31]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.