Pith. sign in

REVIEW 5 major objections 4 minor 58 references

Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Segment This Thing shows that for a single point prompt, an image can be foveated around that point—full resolution only in the center, downsampled rings outward—cutting token count 24x and inference latency roughly 42x versus SAM-H while…

desk verdict Foveated tokenization is a genuinely novel and well-tested efficiency lever, but 'competitive' holds only if you ignore the fixed-crop penalty on large objects. read the letter →

arxiv 2506.11131 v1 pith:U3LDTO6S submitted 2025-06-10 cs.CV eess.IV

Tanner Schmidt , Richard Newcombe This is my paper
classification cs.CVeess.IV
keywords foveatedtokenizationpoint-promptedsegmentationSegmentAnythingefficientvisiontransformervariable-resolutionpatchesgaze-basedpromptingstreamingvideomaskedautoencoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Point-prompted segmentation normally processes the whole image at uniform resolution. Segment This Thing (STT) instead crops a 1280x1280 region centered on the prompt and tokenizes it with full-resolution 16x16 patches in the center and progressively larger patches toward the periphery, downsampled by strides 2, 4, 6, and 8. This foveation reduces the token count from SAM's 4096 to 172, which cuts inference latency to 7.3-26.2 ms on an RTX 3080—roughly 42x faster than SAM-H and 5.6x faster than EfficientSAM. Across nine datasets, STT's segmentation accuracy in mean intersection-over-union is competitive with SAM-H and EfficientSAM on several benchmarks, and its higher central output resolution lets it pick out very small objects. The authors argue that efficiency can come from input foveation instead of shrinking the model, making large-capacity transformers affordable for streaming AR and robotics applications.

What carries the argument

The load-bearing object is the foveation pattern: a set of concentric square grids nested without gaps, with a 4x4 grid of full-resolution 16x16 patches at the center and rings of larger patches downsampled by integer strides 2, 4, 6, and 8, totaling 172 tokens over a 1280x1280 crop. Each patch is reduced to 16x16 pixels with an integral-image box filter and flattened into a token; because the tokens no longer lie on a regular 2D grid, the encoder is a plain Transformer with global self-attention, a register token, and a mask that excludes out-of-bounds tokens. The mechanism that carries the argument is the quadratic dependence of self-attention cost on token count: cutting tokens from 4096 to 172 reduces attention cost by roughly 576x, which more than makes up for using a larger model, and the decoder's four stacked deconvolutions give full-resolution output in the central region where it matters.

What would settle it

Run STT under the standard single-point valid-mask protocol on a dataset dominated by objects that extend beyond the 1280x1280 crop or require fine boundary detail in the periphery; if its mIoU falls well below SAM-H by a margin larger than the TimberSeg gap (0.434 vs 0.674), the claim that foveation preserves enough information for arbitrary segment sizes is falsified.

Watch

Extended reading notes

Core claim

The central claim is that for a single point prompt, most of the image serves as context and can be aggressively downsampled without hurting segmentation accuracy, so long as the region around the prompt stays at full resolution. STT encodes this inductive bias into the input itself: a foveation pattern of nested rings, giving 172 tokens instead of SAM's 4096, and a standard Vision Transformer with global attention processes those tokens in 7.3-26.2 ms. The result is that a larger model with fewer tokens and fewer FLOPs can run faster than smaller models that process uniform full-resolution inputs, while still matching or nearly matching their mean intersection-over-union on datasets such as ADE20K, Cityscapes, and EgoHOS. The authors also show that this depends on training: the encoder must be pre-trained with masked autoencoding on foveated images, and the decoder is modified to output full-resolution masks at the center, which even lets STT segment tiny objects that SAM misses.

Load-bearing premise

The load-bearing premise is that a fixed 1280x1280 crop centered on the prompt, with peripheral regions downsampled up to 8x, preserves enough information to segment objects of any size; large or boundary-crossing segments pay a visible penalty, as on TimberSeg where STT-H scores 0.434 mIoU against SAM-H's 0.674.

Editorial extensions

If this is right

  • On an RTX 3080, STT processes a single point prompt in 7.3-26.2 ms depending on model size, making per-frame segmentation practical for streaming AR and robotics workloads.
  • STT-L runs at 13.7 ms and 108 GFLOPs, faster than MobileSAM's 20.7 ms and 124.4 GFLOPs despite a larger parameter count, showing token reduction can beat model shrinking on efficiency.
  • The foveated input contains only about 44K pixels (roughly a 210x210 image), so the tokenization also cuts sensor-to-compute bandwidth and is orthogonal to further compression like JPEG.
  • Because the decoder produces full-resolution labels in the fovea, STT can segment objects spanning only a few pixels that SAM's quarter-resolution output misses.
  • Each new prompt requires a fresh foveation and a fresh encoder pass, but the per-prompt cost is low enough that STT remains cheaper than SAM until many prompts are issued on the same image.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: foveated tokenization is a general input-side efficiency lever for any task where a location specifies the focus, so the recipe could transfer to point-prompted detection, interactive editing, or gaze-anchored dense prediction.
  • Beyond the paper: since foveation happens before the network and is just box-filter resampling, it could be implemented in the sensor or on a nearby accelerator, allowing edge devices to transmit only the 172-token representation.
  • Beyond the paper: the token-count ablation (100, 172, 268 tokens) shows a smooth accuracy-latency trade-off, so a deployment could tune the pattern; a natural extension is making crop size or peripheral stride adaptive to a coarse first guess of object size.
  • Beyond the paper: the fixed crop assumes the prompt is near the object's center; prompts near a boundary would waste resolution on background, so a variant that shifts the fovea center toward the predicted object centroid is a testable improvement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces Segment This Thing (STT), a point-prompted segmentation model that replaces uniform patch tokenization with a foveated, prompt-centered tokenization: a 1280x1280 crop is divided into a variable-resolution patch pattern (172 tokens vs. 4096 for SAM), with full-resolution central patches and up to 8x downsampling in the periphery. The image encoder is a plain ViT without windowed attention, MAE-pretrained on foveated inputs and fine-tuned on SA-1B with a mask decoder that outputs higher-resolution central maps. Experiments report latency and FLOPs on an RTX 3080, mIoU on nine datasets, and qualitative gaze-prompted results. The central claim is that foveated tokenization makes large-capacity encoders cheaper to run than smaller uniform-resolution models while maintaining competitive accuracy.

Significance. If the results hold, the foveated-tokenization idea is a useful and largely orthogonal efficiency axis: instead of shrinking the model, it reduces the input token count by roughly 24x and concentrates compute near the prompt. The paper's inclusion of a foveated-input control for EfficientSAM (Table 6) is a genuine strength: it shows that a uniform ViT trained on full-resolution grids degrades substantially when given the same foveated input, whereas STT is trained for that input distribution. The reported efficiency gains are large and the architecture is simple enough to be practically interesting for streaming and robotics use cases. The main qualifications are that the accuracy claim depends on object size relative to the fixed crop, and the efficiency claim would be stronger if the unique foveation preprocessing were measured end-to-end rather than excluded.

major comments (5)
  1. [Abstract / Table 5 / Supp. §11.2] The abstract claims STT remains 'competitive' without scope restriction, but the fixed 1280x1280 crop centered on the prompt truncates segments larger than the crop. This is not a corner case: Table 5 shows STT-H at 0.434 mIoU on TimberSeg versus 0.674 for SAM-H and 0.507 for EfficientSAM-S, and the paper's own Supp. §11.2 states that out-of-crop ground-truth pixels 'simply pay the penalty.' Since TimberSeg (log grasping) is directly relevant to the stated robotics motivation, the manuscript should report mIoU stratified by object size or by fraction of the object inside the crop, and should either restrict the abstract's 'competitive' claim to objects that fit within the crop or provide quantitative evidence that the degradation is acceptable for the target applications.
  2. [Supp. §11.2 / Table 6] The cropped-input control only partially resolves the crop confound. When EfficientSAM-S receives the same 1280x1280 crop and the evaluation is restricted to the crop, it reaches 0.444 on Cityscapes while STT-L reaches 0.417; STT overtakes EfficientSAM only in the foveated-input mode (0.417 vs 0.410). The control covers only three datasets, so it does not establish that STT is competitive with a uniform-resolution model given the same receptive field. I recommend adding the same-crop comparison on the full nine-dataset suite and discussing the Cityscapes gap explicitly.
  3. [Supp. Table 5] All mIoU numbers come from a single evaluation with no error bars or repeated runs. Several decisive comparisons are within 0.01-0.02 mIoU (e.g., STT-H 0.552 vs SAM-H 0.543 on ADE20K; STT-B 0.541 vs SAM-B 0.547), so statements such as 'STT-L is competitive with EfficientSAM-Ti' are not statistically supported. Please report variance (e.g., across evaluation subsets or training seeds) or at least state that the observed differences are not significant.
  4. [Table 2 / §12] The reported latency excludes preprocessing for all models, but the foveation and cropping stage is unique to STT and is part of its per-prompt pipeline. The conclusion's 42x/5.6x statements are model-only compute ratios; the end-to-end speedup depends on the cost of integral-image resampling, which the paper asserts but does not measure. I ask for an end-to-end latency table that includes the foveation preprocessing, or at least a measurement of its wall-clock cost.
  5. [Conclusion / Table 2] The conclusion says latency is reduced 'by roughly 42x compared to SAM-H or 5.6x relative to EfficientSAM,' but Table 2 gives STT-H 26.2 ms vs SAM-H 572.7 ms, a 21.9x reduction; the 42x figure corresponds to STT-L (13.7 ms) and the 5.6x figure to STT-L vs EfficientSAM-S (78.6 ms). The text should state which STT variant is being compared and should not imply that the full STT family achieves the largest factor.
minor comments (4)
  1. [Eq. (1)] The denominator contains '1−(1−p i)(1−p q)'; the second factor should use q_i. Also, since IoU is a ratio, Eq. (1) is an approximation of expected IoU rather than the expectation itself; please state this explicitly.
  2. [Throughout] There are several typos: 'Futhermore', 'almost certaintly', 'qualitatitve', 'coarsity', 'appraoch', 'visualizated', and 'hardward'. A proofread pass is needed.
  3. [§4.2] The statement that 'STT-L is competitive with EfficientSAM-Ti in terms of accuracy' should cite the relevant rows of Table 5; as written, the reader must cross-reference the supplementary table without a pointer.
  4. [§4.1.1] The Gaussian prompt-noise offset standard deviation is not reported; please include the value in the supplementary training details so that the noise-augmented gaze-prompting experiments are reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; efficiency and accuracy claims are empirical and benchmarked externally.

full rationale

The paper's central claims are architectural and empirical rather than derived from a fitted parameter or a self-citation chain. The token-count reduction (4096 to 172) is a direct consequence of the foveation pattern definition and Eq. (2), which merely counts non-redundant patches in a prescribed nesting scheme; this is a definitional computation, not a prediction that reduces to its inputs. The latency and FLOP gains follow from standard transformer cost accounting on the reduced token set, and the accuracy claims are measured against external human-annotated benchmarks (ADE20K, Cityscapes, EgoHOS, TimberSeg, etc.) rather than against quantities fitted during training. The model is trained on SA-1B, whose masks were themselves generated by SAM, but the paper discloses this lineage and evaluates on independently annotated datasets; this is a data-lineage caveat, not a circular derivation. There are no load-bearing self-citations, no uniqueness theorems imported from the authors' prior work, and no ansatz smuggled in via citation. The supplementary limitation about out-of-crop ground-truth pixels is an honest scoping caveat, not circularity. Overall, the derivation chain is self-contained: the efficiency gain is computed from the architecture, and the accuracy claims are externally benchmarked.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities; foveated tokenization is a new input representation. The main unstated premises are that peripheral downsampling preserves enough information, that the fixed crop size is adequate, and that SA-1B's SAM-generated masks are suitable training targets.

free parameters (3)
  • Foveation pattern strides and grid sizes = s = {1,2,4,6,8}, g = {4,4,6,8,10}
    Chosen by hand to give 172 tokens in a 1280x1280 crop. The paper reports an ablation varying token count (Table 4) but not a full search over pattern geometry.
  • Token count 172 = 172
    Determined by the foveation pattern; the paper notes that performance increases with token count (Table 4), so this is a deliberate efficiency-accuracy tradeoff.
  • Gaussian prompt-noise offset standard deviation = not reported
    Used in gaze-prompting training ('a small Gaussian-distributed offset vector'), but the noise level is not specified, making replication of this variant difficult.
assumptions (5)
  • domain assumption SA-1B masks are valid segmentation supervision despite being generated by SAM.
    Training and part of evaluation rely on SA-1B, whose masks were produced by SAM. This may bias the model toward SAM-like mask geometry, though external datasets are used for evaluation.
  • domain assumption Box-filter downsampling with integer strides preserves enough information for segmentation.
    The method uses an integral-image box filter, which is aliasing-prone. The paper argues that peripheral downsampling is acceptable for large objects but does not prove it, and the TimberSeg results show a significant drop.
  • domain assumption The fixed 1280x1280 crop is sufficient for the target use cases.
    Objects larger than the crop cannot be fully segmented. The paper acknowledges this and it is a structural limitation.
  • standard math Quadratic self-attention scaling justifies the efficiency claim.
    Standard transformer complexity, used to argue that token reduction lowers FLOPs.
  • domain assumption The single farthest-point prompt evaluation is representative of interactive use.
    Adopted from SAM's zero-shot protocol, but real interactive prompts may be noisy or off-center. The gaze-prompting experiments partially address this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation." pith.science (2026). https://pith.science/paper/U3LDTO6S

@misc{pith2026250611131,
  author       = {Pith},
  title        = {Pith review of: Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U3LDTO6S}},
  note         = {Machine review of arXiv:2506.11131}
}
read the original abstract

This paper presents Segment This Thing (STT), a new efficient image segmentation model designed to produce a single segment given a single point prompt. Instead of following prior work and increasing efficiency by decreasing model size, we gain efficiency by foveating input images. Given an image and a point prompt, we extract a crop centered on the prompt and apply a novel variable-resolution patch tokenization in which patches are downsampled at a rate that increases with increased distance from the prompt. This approach yields far fewer image tokens than uniform patch tokenization. As a result we can drastically reduce the computational cost of segmentation without reducing model size. Furthermore, the foveation focuses the model on the region of interest, a potentially useful inductive bias. We show that our Segment This Thing model is more efficient than prior work while remaining competitive on segmentation benchmarks. It can easily run at interactive frame rates on consumer hardware and is thus a promising tool for augmented reality or robotics applications.

Figures

Figures reproduced from arXiv: 2506.11131 by the authors.

Figure 1
Figure 1. A visualization of the foveated tokenization employed [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (left) The foveation pattern used by our trained models. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The foveated tokenization process. It begins by cropping a square region around the point prompt with a fixed size determined [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Segment This Thing has a precision advantage for ex [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Despite the high downsampling rate in the periphery, the Segment This Thing model is capable of segmenting objects with a [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: A visualization of a training example for MAE pre [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Results of evaluating our models and baselines on publicly available segmentation datasets. Models of the same family are [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: A side-by-side comparison of our Segment This Thing model on a set of images from the SA-1B dataset [ [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: A visualization of prompting Segment This Thing with gaze. In a sequence from the Nymeria dataset [ [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: The foveated tokenization pattern as used in our ex [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: The stride of the segmentation maps produced by STT [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Training curves for STT-B models as a function of [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: The top three rows show the pixel-wise precision, recall, and accuracy of SAM and STT, respectively, as a function of the [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 33 canonical work pages

  1. [1]

    Zerowaste dataset: To- wards deformable object segmentation in cluttered scenes

    Dina Bashkirova, Mohamed Abdelfattah, Ziliang Zhu, James Akl, Fadi Alladkani, Ping Hu, Vitaly Ablavsky, Berk Calli, Sarah Adel Bargal, and Kate Saenko. Zerowaste dataset: To- wards deformable object segmentation in cluttered scenes. In CVPR, pages 21147–21157, 2022. 6

    work page 2022

  2. [2]

    To- ken merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. To- ken merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022. 3

    arXiv 2022

  3. [3]

    The eccentricity effect: Target eccentricity af- fects performance on conjunction searches.Perception & psychophysics, 57:1241–1261, 1995

    Marisa Carrasco, Denise L Evert, Irene Chang, and Svet- lana M Katz. The eccentricity effect: Target eccentricity af- fects performance on conjunction searches.Perception & psychophysics, 57:1241–1261, 1995. 1

    work page 1995

  4. [4]

    Pelk: Parameter-efficient large kernel con- vnets with peripheral convolution

    Honghao Chen, Xiangxiang Chu, Yongjian Ren, Xin Zhao, and Kaiqi Huang. Pelk: Parameter-efficient large kernel con- vnets with peripheral convolution. InCVPR, pages 5557– 5567, 2024. 3

    work page 2024

  5. [5]

    Diffrate: Differentiable compression rate for efficient vision transformers

    Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. InICCV, pages 17164–17174, 2023. 3

    work page 2023

  6. [6]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InCVPR,

  7. [7]

    Rescaling egocentric vision: Collection, pipeline and chal- lenges for epic-kitchens-100.International Journal of Com- puter Vision, pages 1–23, 2022

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and chal- lenges for epic-kitchens-100.International Journal of Com- puter Vision, pages 1–23, 2022. 6

    work page 2022

  8. [8]

    Vision transformers need registers.arXiv preprint arXiv:2309.16588, 2023

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Pi- otr Bojanowski. Vision transformers need registers.arXiv preprint arXiv:2309.16588, 2023. 4

    arXiv 2023

Show all 58 references
  1. [9]

    Epic-kitchens visor benchmark: Video segmenta- tions and object relations

    Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmenta- tions and object relations. InNeurIPS, 2022. 6

    2022

  2. [10]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InCVPR, pages 248–255. Ieee, 2009. 3

    2009

  3. [11]

    An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint ar...

    2010 arXiv

  4. [12]

    Project aria: A new tool for egocentric multi-modal ai research.arXiv preprint arXiv:2308.13561, 2023

    Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattof, Arnie Yuan, Bilal Souti, Brighid Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research.arXiv preprint arXiv:2308.13561, 2023. 8

    2023 arXiv

  5. [13]

    Adaptive token sampling for efficient vision transformers

    Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and J¨urgen Gall. Adaptive token sampling for efficient vision transformers. InECCV, pages 396–414. Springer, 2022. 3

    2022

  6. [14]

    Instance segmen- tation for autonomous log grasping in forestry operations

    Jean-Michel Fortin, Olivier Gamache, Vincent Grondin, Franc ¸ois Pomerleau, and Philippe Gigu`ere. Instance segmen- tation for autonomous log grasping in forestry operations. In2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6064–6071. IE...

    2022

  7. [15]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InICCV, pages 770–778, 2016. 2

    2016

  8. [16]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InCVPR, pages 16000–16009, 2022. 5

    2022

  9. [17]

    Bytes are all you need: Transformers operat- ing directly on file bytes.arXiv preprint arXiv:2306.00238,

    Maxwell Horton, Sachin Mehta, Ali Farhadi, and Moham- mad Rastegari. Bytes are all you need: Transformers operat- ing directly on file bytes.arXiv preprint arXiv:2306.00238,

    arXiv

  10. [18]

    Foveater: Foveated transformer for image classification.arXiv preprint arXiv:2105.14173,

    Aditya Jonnalagadda, William Yang Wang, BS Manjunath, and Miguel P Eckstein. Foveater: Foveated transformer for image classification.arXiv preprint arXiv:2105.14173,

    arXiv

  11. [19]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    2001 arXiv

  12. [20]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InCVPR, pages 4015–4026, 2023. 1, 2, 5, 6, 7

    2023

  13. [21]

    Spvit: Enabling faster vision transformers via latency-aware soft token pruning

    Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. InECCV, pages 620–640. Springer, 2022. 3

    2022

  14. [22]

    Gazegpt: Augment- ing human capabilities using gaze-contingent contextual ai for smart eyewear.arXiv preprint arXiv:2401.17217, 2024

    Robert Konrad, Nitish Padmanaban, J Gabriel Buckmaster, Kevin C Boyle, and Gordon Wetzstein. Gazegpt: Augment- ing human capabilities using gaze-contingent contextual ai for smart eyewear.arXiv preprint arXiv:2401.17217, 2024. 3

    2024 arXiv

  15. [23]

    Imagenet classification with deep convolutional neural net- works.NeurIPS, 25, 2012

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works.NeurIPS, 25, 2012. 2

    2012

  16. [24]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer, 2014. 3

    2014

  17. [25]

    Efficientvit: Memory efficient vision transformer with cascaded group attention

    Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, and Yixuan Yuan. Efficientvit: Memory efficient vision transformer with cascaded group attention. InCVPR, pages 14420–14430, 2023. 2

    2023

  18. [26]

    Nymeria: A massive collection of multimodal egocentric daily motion in the wild.arXiv preprint arXiv:2406.09905, 2024

    Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, et al. Nymeria: A massive collection of multimodal egocentric daily motion in the wild.arXiv preprint arXiv:2406.09905, 2024. 8

    2024 arXiv

  19. [27]

    Token pooling in vision transformers.arXiv preprint arXiv:2110.03860, 2021

    Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, Anish Prabhu, Mohammad Rastegari, and Oncel Tuzel. Token pooling in vision transformers.arXiv preprint arXiv:2110.03860, 2021. 3

    2021 arXiv

  20. [28]

    Adavit: Adaptive vision transformers for efficient image recognition

    Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. InCVPR, pages 12309–12318, 2022. 3

    2022

  21. [29]

    Peripheral vision transformer.NeurIPS, 35:32097–32111,

    Juhong Min, Yucheng Zhao, Chong Luo, and Minsu Cho. Peripheral vision transformer.NeurIPS, 35:32097–32111,

  22. [30]

    Finely-grained annotated datasets for image-based plant phenotyping.Pattern recognition letters, 81:80–89, 2016

    Massimo Minervini, Andreas Fischbach, Hanno Scharr, and Sotirios A Tsaftaris. Finely-grained annotated datasets for image-based plant phenotyping.Pattern recognition letters, 81:80–89, 2016. 6

    2016

  23. [31]

    Rgb no more: Minimally- decoded jpeg vision transformers

    Jeongsoo Park and Justin Johnson. Rgb no more: Minimally- decoded jpeg vision transformers. InCVPR, pages 22334– 22346, 2023. 2

    2023

  24. [32]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification.NeurIPS, 34:13937–13949, 2021

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification.NeurIPS, 34:13937–13949, 2021. 3

    2021

  25. [33]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024. 2

    2024 arXiv

  26. [34]

    Learn- ing to merge tokens in vision transformers.arXiv preprint arXiv:2202.12015, 2022

    Cedric Renggli, Andr ´e Susano Pinto, Neil Houlsby, Basil Mustafa, Joan Puigcerver, and Carlos Riquelme. Learn- ing to merge tokens in vision transformers.arXiv preprint arXiv:2202.12015, 2022. 3

    2022 arXiv

  27. [35]

    Cp-vit: Cascade vision transformer pruning via progressive sparsity prediction.arXiv preprint arXiv:2203.04570, 2022

    Zhuoran Song, Yihong Xu, Zhezhi He, Li Jiang, Naifeng Jing, and Xiaoyao Liang. Cp-vit: Cascade vision transformer pruning via progressive sparsity prediction.arXiv preprint arXiv:2203.04570, 2022. 3

    2022 arXiv

  28. [36]

    On efficient variants of segment anything model: A survey.arXiv preprint arXiv:2410.04960, 2024

    Xiaorui Sun, Jun Liu, Heng Tao Shen, Xiaofeng Zhu, and Ping Hu. On efficient variants of segment anything model: A survey.arXiv preprint arXiv:2410.04960, 2024. 2

    2024 arXiv

  29. [37]

    Ndd20: A large-scale few-shot dolphin dataset for coarse and fine-grained categorisation

    Cameron Trotter, Georgia Atkinson, Matt Sharpe, Kirsten Richardson, A Stephen McGough, Nick Wright, Ben Burville, and Per Berggren. Ndd20: A large-scale few-shot dolphin dataset for coarse and fine-grained categorisation. arXiv preprint arXiv:2005.13359, 2020. 6

    2005 arXiv

  30. [38]

    Neural discrete representation learning.NeurIPS, 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.NeurIPS, 30, 2017. 2

    2017

  31. [39]

    Squeeze- sam: User friendly mobile interactive segmentation.arXiv preprint arXiv:2312.06736, 2023

    Balakrishnan Varadarajan, Bilge Soran, Forrest Iandola, Xi- aoyu Xiang, Yunyang Xiong, Lemeng Wu, Chenchen Zhu, Raghuraman Krishnamoorthi, and Vikas Chandra. Squeeze- sam: User friendly mobile interactive segmentation.arXiv preprint arXiv:2312.06736, 2023. 2

    2023 arXiv

  32. [40]

    Efficientsam: Leveraged masked image pretraining for efficient segment anything

    Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xi- ang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, et al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. In CVPR, pages 16111–16121, 2024. 2, 6, 7

    2024

  33. [41]

    Elastictok: Adap- tive tokenization for image and video.arXiv preprint arXiv:2410.08368, 2024

    Wilson Yan, Matei Zaharia, V olodymyr Mnih, Pieter Abbeel, Aleksandra Faust, and Hao Liu. Elastictok: Adap- tive tokenization for image and video.arXiv preprint arXiv:2410.08368, 2024. 3

    2024 arXiv

  34. [42]

    A-vit: Adaptive tokens for efficient vision transformer

    Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. InCVPR, pages 10809–10818,

  35. [43]

    Woodscape: A multi-task, multi-camera fisheye dataset for autonomous driving

    Senthil Yogamani, Ciar ´an Hughes, Jonathan Horgan, Ganesh Sistu, Padraig Varley, Derek O’Dea, Michal Uric ´ar, Ste- fan Milz, Martin Simon, Karl Amende, et al. Woodscape: A multi-task, multi-camera fisheye dataset for autonomous driving. InICCV, pages 9308–9318, 2019. 6

    2019

  36. [44]

    Language model beats diffusion–tokenizer is key to visual generation.arXiv preprint arXiv:2310.05737, 2023

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation.arXiv preprint arXiv:2310.05737, 2023. 2

    arXiv

  37. [45]

    An image is worth 32 tokens for reconstruction and generation.arXiv preprint arXiv:2406.07550, 2024

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation.arXiv preprint arXiv:2406.07550, 2024. 3

    2024 arXiv

  38. [46]

    Faster segment anything: Towards lightweight sam for mo- bile applications.arXiv preprint arXiv:2306.14289, 2023

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mo- bile applications.arXiv preprint arXiv:2306.14289, 2023. 2, 6, 7

    2023 arXiv

  39. [47]

    Fine-grained egocentric hand-object segmentation: Dataset, model, and applications

    Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications. InECCV, pages 127–145. Springer, 2022. 6

    2022

  40. [48]

    Efficientvit-sam: Accelerated segment anything model without performance loss

    Zhuoyang Zhang, Han Cai, and Song Han. Efficientvit-sam: Accelerated segment anything model without performance loss. InCVPR, pages 7859–7863, 2024. 2

    2024

  41. [49]

    Fast segment any- thing.arXiv preprint arXiv:2306.12156, 2023

    Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment any- thing.arXiv preprint arXiv:2306.12156, 2023. 2

    2023 arXiv

  42. [50]

    Semantic under- standing of scenes through the ade20k dataset.IJCV, 127: 302–321, 2019

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset.IJCV, 127: 302–321, 2019. 3, 6

    2019

  43. [51]

    Edgesam: Prompt-in-the-loop distillation for on-device de- ployment of sam.arXiv preprint arXiv:2312.06660, 2023

    Chong Zhou, Xiangtai Li, Chen Change Loy, and Bo Dai. Edgesam: Prompt-in-the-loop distillation for on-device de- ployment of sam.arXiv preprint arXiv:2312.06660, 2023. 2 Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation Supplementary Material

    2023 arXiv

  44. [52]

    MAE Pre-training We pre-trained our foveated image encoders using MAE pre-training for 500K iterations

    Training Details 6.1. MAE Pre-training We pre-trained our foveated image encoders using MAE pre-training for 500K iterations. We use an AdamW op- timizer with a learning rate of2 −13 and weight decay of 0.001. There is a 10K step linear warm-up of the learning rate, after whic...

  45. [53]

    Here we give a more formal definition of the parameterization of such a pattern

    Foveation Pattern As described in Section 3.1, our foveation patterns consist of a series of nested rings of patches with a dense grid in the center. Here we give a more formal definition of the parameterization of such a pattern. A pattern withNlayers must specify a strides i...

  46. [54]

    We plot the training loss curves in Figure 12

    MAE Pre-training To evaluate the effectiveness of MAE pretraining on foveated tokenizations, we trained STT-B models for 100K iterations with various initial weights: random initializa- tion, a series of MAE checkpoints, and pre-trained publicly available ViT weights. We plot ...

  47. [55]

    Our foveation patterns exists in a high-dimensional design space, and each new pattern requires its own MAE pre-training

    Varying the Token Count We also ran a small experiment to evaluate the effect of the foveation pattern on segmentation accuracy. Our foveation patterns exists in a high-dimensional design space, and each new pattern requires its own MAE pre-training. We thus focused on the tok...

  48. [56]

    Full Evaluation Results We list the full evaluation results in tabular form in Table 5

  49. [57]

    STT Performance Analysis In this section we describe two investigations into the per- formance of STT relative to the baselines to deepen the un- derstanding of the model. 11.1. Breakdown by Distance from Prompt To gain further insight into the relative performance of SAM and ...

  50. [58]

    in computing FLOP counts for transformer architectures (c.f

    Computing FLOPs We follow Kaplanet al. in computing FLOP counts for transformer architectures (c.f. [19], Table 1). Specifi- cally, we omit non-linearities, biases, normalizations, and other such operations with negligible contributions relative to the FLOPS counts incurred by...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.