Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read M2N2 shows that Stable Diffusion self-attention, reinterpreted as a Markov transition operator, can drive training-free unsupervised interactive segmentation that beats trained unsupervised baselines on most benchmarks.

desk verdict A clever Markov-map representation of Stable Diffusion attention, but the SOTA comparison rests on hyperparameters selected on the test sets. read the letter →

arxiv 2411.10411 v2 pith:3PDEZO6Z submitted 2024-11-15 cs.CV cs.AI

Markus Karmann , Onay Urfalioglu This is my paper · ORCID
classification cs.CVcs.AI
keywords interactivesegmentationpointpromptsStableDiffusionself-attentionMarkovchaintraining-freeunsupervisedNumberofClicksMarkov-map
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a frozen text-to-image model's self-attention already contains enough semantic structure for interactive point-prompt segmentation, without any training or pseudo-labels. The route is to read the aggregated self-attention tensor as a Markov transition operator and count, for every pixel, how many iterations a probability mass starting at the clicked point needs to saturate a relative threshold; that count is the Markov-map. The paper claims Markov-maps are less noisy, have sharper semantic boundaries, and are more uniform inside semantically similar regions than raw attention maps. Because of that, a truncated nearest-neighbor rule over the maps reaches 85% IoU in 1.62 average clicks on GrabCut, 2.45 on Berkeley, and 4.60 on DAVIS, improving on the trained unsupervised baseline MIS, which needs 1.94, 3.09, and 6.33 (on SBD the method is second). The reason a reader should care is that the result suggests promptable segmentation need not be a trained module at all; a generative backbone's internal attention, refined by a Markov process, can serve as the distance function.

What carries the argument

The Markov-map is the central object. For a prompt point $i$, it is the matrix $M_i \in \mathbb{R}^{H \times W}$ whose entry at pixel $q$ is the smallest iteration count $t$ for which the Markov-chain probability $p_t[q]/\max p_t$ exceeds the relative threshold $\tau$, with $p_t = p_0 A^t$ and $p_0$ the one-hot start state at the click. Iterative proportional fitting makes the transition matrix $A$ doubly stochastic so every start state converges to the uniform distribution, and a temperature $T$ controls the convergence rate. A modified flood fill then stores the minimum threshold needed to reach each pixel, suppressing local minima so instances become separable, and joint bilateral upsampling raises the map to image resolution. The final distance $d(x_i, x_q) = M_i[x_q]/\lambda_i$ feeds a truncated nearest neighbor, with $\lambda_i$ chosen by a product of four score functions.

What would settle it

Build a set of natural images whose intended foreground object covers more than 40% of the frame and run M2N2 with a single foreground prompt and no background prompt. The score function $s_{i,\mathrm{prior}}(\lambda)$ assigns zero to every threshold that produces a segment above 40% of the image, so the selected threshold must cut inside the object; if the resulting segment is consistently a proper subset of the target, the adaptive threshold heuristic is the load-bearing failure point.

Watch

Extended reading notes

Core claim

At the core, the paper argues that convergence time in a Markov chain built from Stable Diffusion's self-attention is a better semantic distance than the attention values themselves. Starting from the one-hot location of a clicked point, the probability distribution $p_t$ evolves under the doubly-stochastic self-attention matrix, and each pixel's Markov-map value is the minimum $t$ at which its relative probability first exceeds the threshold $\tau$. The paper claims this map separates instances after a flood-fill step and that a truncated nearest-neighbor decision on the rescaled maps--assigning background to query pixels more than one unit away from their nearest prompt--yields the reported NoC improvements, outperforming the pseudo-label-trained MIS on three of four datasets in both NoC85 and NoC90 despite using no training at all.

Load-bearing premise

The method's success rests on a hand-crafted rule that guesses, for each click, how far the Markov-map should extend; if the rule guesses wrong, that click's segment is wrong no matter how good the map is.

Editorial extensions

If this is right

  • Promptable segmentation can run on a frozen generative model at inference time, eliminating the pseudo-label generation and training loop used by previous unsupervised methods.
  • On the three natural-image benchmarks in the paper, the training-free method closes the gap with trained unsupervised methods and surpasses them, so training may be unnecessary for this task in this regime.
  • Attention resolution and input resolution are the main dials: performance improves up to 128x128 attention maps and degrades when the input image is pushed beyond the backbone's native training resolution.
  • Per-click latency around 0.6 seconds on an RTX 4090 at 854x480, with caching of attention and Markov-maps, makes the approach usable in an interactive loop.
  • Thin structures, fine details, and overlapping or obstructed instances remain the failure mode, requiring additional prompt points and sometimes failing at the 20-click cap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the Markov-map construction is a generic post-processing of an attention tensor, it could be dropped into any attention-based backbone, not just Stable Diffusion; the paper's own ViT-B and SD1.1 results are partial evidence, and other diffusion architectures are a natural next test.
  • The hand-crafted threshold selector is the most replaceable part: the edge-score ablation shows boundary alignment is the largest single factor, so a closed-form or learned boundary-alignment criterion would likely reduce the remaining clicks.
  • The 40% size prior implies a systematic blind spot for very large objects; a test set of images whose target fills most of the frame should expose a failure mode that the paper's averaged benchmarks understate.
  • The paper reports a domain bias toward natural images, with lower IoU on medical datasets; extending the method or adapting the Markov-map temperature to such domains is a concrete testable direction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes M2N2, a training-free interactive point-prompt segmentation framework built on Stable Diffusion 2 self-attention. The self-attention tensor is treated as a Markov transition operator; after iterative proportional fitting to make it doubly stochastic, a Markov chain from each prompt point yields a per-pixel hitting-time map (the Markov-map). A modified flood fill and a truncated nearest-neighbor rule combine multiple prompt points into a segmentation. The method is evaluated on GrabCut, Berkeley, SBD, and DAVIS with Number of Clicks (NoC85/NoC90) under the standard click-simulation protocol. The main claim is that M2N2, despite being training-free, outperforms the trained unsupervised baseline MIS on three of the four datasets (GrabCut, Berkeley, DAVIS) while losing only on SBD.

Significance. If the empirical claims survive a properly controlled evaluation, this is a valuable contribution. The core idea is clean and novel: interpreting SD self-attention as a Markov operator and using hitting times as a semantic distance yields a simple, deterministic, training-free segmentation method. The mathematical construction is sound and well explained, including the use of IPF to enforce a uniform stationary distribution. The paper includes extensive ablations (attention blocks, temperature, threshold, resolution, score functions) and releases code, which strengthens reproducibility. The reported gains over the trained unsupervised method MIS on GrabCut (1.62 vs 1.94 NoC85), Berkeley (2.45 vs 3.09), and DAVIS (4.60 vs 6.33) are large, but they rest on hyperparameters selected on the same test benchmarks and on a threshold heuristic whose quantitative reliability is not established. These issues are fixable within the manuscript's scope, so the result is promising but requires revision.

major comments (3)
  1. [Sec. 4.1–4.2, Fig. 4, Table 3] The final configuration (attention resolution 128, SD time step 100, temperature T=0.65, relative threshold τ=0.3, aggregation weights 0.5/0.5) is chosen from the NoC curves in Fig. 4, which are computed on the same evaluation datasets used for the headline comparison in Table 3 (GrabCut, Berkeley, DAVIS, and a 500-image SBD subset). No validation split or cross-validation is described. Because NoC computation requires ground-truth masks to simulate clicks, sweeping hyperparameters to minimize NoC on these datasets is direct test-set fitting. This biases the claimed margins over MIS. Please add a validation protocol: select hyperparameters on a hold-out set or via cross-validation, then report the resulting test-set NoC, and also report sensitivity of the final numbers to a plausible range of hyperparameter choices. Without this, the central empirical claim is not fully supported.
  2. [Supplementary Sec. 7, Algorithm 1] Algorithm 1, line 8 computes λ ← max(λ′, abs(M[y] − M[x])). Here x is the input prompt point while x′ is the currently popped pixel, so the update measures the Markov-map difference from the prompt point rather than from the current pixel. This contradicts the text's description that the flood fill stores the minimum flood threshold required to reach each pixel from the starting point and would, as written, produce a different map than the one used in the experiments. The correct update should reference M[x′] (or equivalently propagate the maximum of the threshold along the path and the absolute difference between adjacent pixels). Please correct the pseudocode and specify the exact update rule, including whether diagonal neighbors are considered, so that the 'with flood fill' results in Table 3 are reproducible.
  3. [Sec. 3.4, Eqs. (8)–(11), Table 2] The adaptive threshold λ_i = arg max_λ s_i(λ) is load-bearing: a wrong threshold produces a wrong segment for that prompt point regardless of Markov-map quality. The score functions are a hand-crafted product of four terms, and their support is mainly an ablation on DAVIS (Table 2) plus qualitative examples. No quantitative evidence is given for how often the heuristic selects a threshold close to the oracle threshold, nor how its failures vary across datasets and images. Because the ablations are aggregated, failures on a subset of images could be masked. Please provide a per-dataset analysis comparing the heuristic threshold with an oracle threshold (e.g., the threshold that would maximize IoU or minimize click count), or a per-image success rate of the score-function selection. This would place the central NoC result on firmer ground.
minor comments (4)
  1. [Sec. 3.3, IPF description] The paper states that IPF converts the attention matrix to a doubly stochastic matrix, but it does not specify the IPF convergence tolerance, maximum number of iterations, or the exact stopping criterion. Since the supplementary notes that the number of IPF iterations depends on temperature, please include these implementation details for reproducibility.
  2. [Fig. 4 and Table 3] The SBD curves in Fig. 4 are based on a randomly sampled subset of 500 images, while Table 3 reports NoC on the full SBD validation set. Please clarify whether the hyperparameters used for SBD in Table 3 were chosen on the subset or on the full set, and whether the same configuration is used for all datasets.
  3. [Sec. 4.3, SD2 Domain Bias] The paragraph reporting BraTS and OAIZIB IoU values appears in the experiments section but is not connected to the rest of the evaluation; please place it in a dedicated limitations/discussion paragraph or expand it into a clear domain-bias analysis, since it is a relevant caveat for the method's general applicability.
  4. [Throughout] There are several typographical errors, including 'DA VIS' with a space in multiple places, 'slected' in the Fig. 5 caption, 'T otal Score' in Fig. 10, and 'spriorsedgespossneg' in Fig. 11. These should be corrected before publication.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline DAVIS SOTA number is the argmin of the paper's own weight ablation on DAVIS, and the global hyperparameters are selected from NoC curves on the same four benchmarks, so the reported 'prediction' is partly forced by construction.

  1. fitted input called prediction [Section 4.2 / Table 1, and Table 3 DAVIS row; Section 4.1 and Figure 4]
    "0 0 0.5 0.5 0 4.60 6.72 ... Table 1. Ablation study of the attention blocks on DA VIS. The blocks up0 and up1 achieve the lowest individual NoC. ... Aggregating up0 and up1 results in the best NoC. ... M2N2 (Ours) SD2 1.62 1.90 2.45 3.88 7.72 10.94 4.60 6.72"

    The attention-aggregation weights (w_up0=0.5, w_up1=0.5) are selected because that configuration gives NoC85=4.60 and NoC90=6.72 on DAVIS in Table 1. The identical numbers are then re-reported in Table 3 as M2N2's DAVIS performance and are used to claim superiority over MIS (6.33 vs 4.60). The reported DAVIS result is therefore the selected minimum of the paper's own DAVIS sweep, not an independent prediction from a fixed configuration. Section 4.1 and Figure 4 show the same pattern for the other global hyperparameters (attention resolution 128, SD timestep 100, temperature T=0.65, threshold tau=0.3), all chosen from NoC curves evaluated on the same four datasets used in the final comparison.

full rationale

Most of M2N2's pipeline is not circular. The Markov-map is a deterministic transformation of Stable Diffusion self-attention via exponentiation, iterative proportional fitting, Markov iteration counting, flood fill, and joint bilateral upsampling; no target segmentation label or ground-truth mask enters the construction. The adaptive threshold lambda_i in Eqs. (8)-(11) uses only the prompt points, a 40% segment-size prior, boundary edge scores, and the user-supplied click labels, so it is not defined in terms of the final benchmark IoU. The paper also cites only standard external facts (Sinkhorn, DiffSeg, Markov chain convergence), with no load-bearing self-citation chain or imported uniqueness theorem. The circularity that exists is confined to the evaluation protocol: the DAVIS NoC used to beat MIS is the very NoC on which the attention-block weights were selected in Table 1, and Figure 4 sweeps the remaining hyperparameters on all four datasets that appear in Table 3. The reported numbers are therefore not predictions from an independently fixed configuration; they are partly the result of fitting the configuration to the evaluation benchmarks. The method's internal derivation remains self-contained, but the central SOTA claim is partially statistically forced, which warrants a score of 6 rather than 0-2. The paper's own disclosures of domain bias on BraTS/OAIZIB and failure cases on SBD are honest limitations and do not themselves constitute circularity.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. The Markov-map is a derived representation, not a postulated entity. The free parameters are numerous and are tuned through ablations on the same benchmarks used for evaluation, which is the main source of circularity burden. The axioms are a mix of standard math (Markov chains, Sinkhorn scaling) and domain assumptions specific to using Stable Diffusion attention as a semantic distance.

free parameters (8)
  • Temperature T = 0.65
    Controls the sharpness of the transition matrix in Eq. (5); chosen via the ablation in Fig. 4.
  • Relative probability threshold tau = 0.3
    Defines the saturation criterion in Eq. (6); the paper notes low sensitivity, but it is a tuned value.
  • Attention resolution = 128x128
    Selected from the ablation in Fig. 4 and Sec. 14; requires an input image size of 1024x1024, beyond SD2's training resolution.
  • SD time step = 100
    Single denoising step uses this time step; chosen to avoid distribution shift, as time steps >200 increase NoC.
  • Attention aggregation weights = w_up0=0.5, w_up1=0.5, others=0
    From Table 1 ablation; combining the up0 and up1 attention blocks yields the lowest NoC.
  • Segment-size prior cap = 40% of image
    Parameter of the s_prior score function in Eq. (9); prevents oversized segments.
  • Maximum Markov iterations = 1000
    Cap for numerical stability; lower temperatures require more iterations.
  • JBU upsampling parameters = sigma_spatial=1, sigma_range=0.1
    Parameters for the joint bilateral upsampling to reach full image resolution.
assumptions (5)
  • domain assumption Self-attention maps of Stable Diffusion 2, after aggregation and IPF, form a semantically meaningful transition matrix for pixel-level Markov chains.
    The core assumption underlying Markov-maps (Sec. 3.2-3.3). If attention does not reflect semantic similarity, the hitting times are meaningless.
  • domain assumption The modified flood fill approach correctly handles instances when they do not overlap.
    Stated limitation in Sec. 3.3: the approach requires instances not to overlap; overlapping instances need multiple prompts.
  • ad hoc to paper The product of score functions s_i(lambda) in Eq. (9) selects a threshold that yields a semantically correct segment.
    This heuristic is introduced without derivation and its reliability is only shown empirically through ablations and qualitative examples.
  • domain assumption A single denoising step with no added noise and an empty text prompt at time step 100 produces attention maps suitable for segmentation.
    Implementation detail in Sec. 4.1; the paper does not analyze sensitivity to this choice beyond a time-step ablation.
  • standard math Sinkhorn's theorem guarantees that IPF converges to a doubly stochastic matrix for positive matrices, and this scaling preserves semantic structure.
    Invoked in Sec. 3.3 to make the transition matrix doubly stochastic so that the stationary distribution is uniform. The semantic-preservation part is an assumption beyond the theorem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation." pith.science (2026). https://pith.science/paper/3PDEZO6Z

@misc{pith2026241110411,
  author       = {Pith},
  title        = {Pith review of: Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3PDEZO6Z}},
  note         = {Machine review of arXiv:2411.10411}
}
read the original abstract

Recent progress in interactive point prompt based Image Segmentation allows to significantly reduce the manual effort to obtain high quality semantic labels. State-of-the-art unsupervised methods use self-supervised pre-trained models to obtain pseudo-labels which are used in training a prompt-based segmentation model. In this paper, we propose a novel unsupervised and training-free approach based solely on the self-attention of Stable Diffusion. We interpret the self-attention tensor as a Markov transition operator, which enables us to iteratively construct a Markov chain. Pixel-wise counting of the required number of iterations along the Markov chain to reach a relative probability threshold yields a Markov-iteration-map, which we simply call a Markov-map. Compared to the raw attention maps, we show that our proposed Markov-map has less noise, sharper semantic boundaries and more uniform values within semantically similar regions. We integrate the Markov-map in a simple yet effective truncated nearest neighbor framework to obtain interactive point prompt based segmentation. Despite being training-free, we experimentally show that our approach yields excellent results in terms of Number of Clicks (NoC), even outperforming state-of-the-art training based unsupervised methods in most of the datasets. Code is available at https://github.com/mkarmann/m2n2.

Figures

Figures reproduced from arXiv: 2411.10411 by the authors.

Figure 1
Figure 1. We introduce M2N2, an unsupervised training-free point prompt based segmentation framework. We enhance the semantic information present in the self-attention of Stable Diffusion 2 by using a Markov process to generate semantically rich Markov-maps. We then perform a truncated nearest neighbor of each point’s Markov-map to obtain a final segmentation. Abstract Recent progress in interactive point prompt based Image S… view at source ↗
Figure 2
Figure 2. M2N2 framework overview. We perform a single denoising step of the input image with Stable Diffusion 2 to obtain attention tensors. The tensors are aggregated and utilized to obtain a Markov-map Mi for each prompt point. The final segmentation is the result of a truncated nearest neighbor of scaled Markov-maps Mi as a measure of semantic distance for each prompt point. The green and red areas in the scaled Markov-ma… view at source ↗
Figure 3
Figure 3. Comparison of semantic maps. Each map is generated from a single prompt point. For better comparison, Markov-maps are inverted such that the lowest value is white and the highest value is black. chain, exceeding the maximum number of iterations and causing numerical instabilities. Different settings of the relative probability threshold τ prove to have a relatively low impact on the NoC. Truncated Nearest Neighbor. … view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Impact of the hyperparameters of SD and Markov-map, respectively, on all four datasets, each represented by a single color. Dashed lines correspond to NoC85, continues lines to NoC90. The graph of SBD is based on a randomly sampled subset of 500 images. NoC90 = 1 NoC90…
Figure 5
Figure 5. Figure 5: Segmentation examples on DAVIS [33]. Each column shows examples slected on the NoC90 value, ranging from easy cases NoC90 = 1 on the left to difficult cases NoC90 = 10 and failure cases NoC90 = 20 on the right. Foreground points are shown in green and background points…
Figure 6
Figure 6. Figure 6: Generation process of a Markov-map. Each column shows the current state of the probability distribution pt and the corre￾sponding Markov-map M for a given number of iterations t. The first row contains the input image and prompt point. The second and third row show the…
Figure 7
Figure 7. Figure 7: mIoU per NoC. For each dataset we show the mIoU at a given number of clicks. tain a doubly stochastic matrix, and the Markov chain to converge to the uniform distribution. 10. Additional Results [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Distribution of the NoC for each dataset. The maximum number of clicks is set to 20. only a few clicks for both NoC85 and NoC90. For DAVIS and especially SBD, we find a significantly higher number of failure cases NoC = 20. As discussed in Sec. 4.3, the high failure ra…
Figure 9
Figure 9. Figure 9: 42 randomly sampled failure cases on SBD. All examples here have NoC90 = 20. 4 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Visualization of individual score functions. For each prompt point i, we show individual score functions si,prior(λ) and the total score function si(λ). The x-axis of each graph is the threshold and the respective score is on the y-axis. The last row shows the segment…
Figure 11
Figure 11. Figure 11: Complex example of score functions. The example segmentation utilizes 7 prompt points. We note that the each prompt point’s segment has a clear semantic meaning (starting from left): jacket, head, hand, shirt, bottle, hand, trousers. It also shows that individual Mark…
Figure 12
Figure 12. Figure 12: Qualitative examples of score functions si,·(λ). The first column fixes the threshold to a constant λi = 0.5, effectively using no score functions si,·(λ), while the other four columns show the impact of disabling one of the four functions. The second and third row di…
Figure 13
Figure 13. Figure 13: Impact of attention resolution on various back￾bones. We evaluate attention maps starting from the native train￾ing resolution up to the highest possible resolution on our hard￾ware. 13. Additional Backbones in Detail ViT-B: We use the pre-trained weights of DinoV2 pr…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SDMatte: Grafting Diffusion Models for Interactive Matting

    cs.CV 2025-08 conditional novelty 6.0 of 10

    SDMatte adapts Stable Diffusion to interactive matting via visual-prompt cross-attention, opacity/coordinate embeddings, and masked self-attention, reporting SOTA results on multiple benchmarks.

Reference graph

Works this paper leans on

64 extracted references · 58 canonical work pages · cited by 1 Pith paper

  1. [1]

    Automated segmentation of knee bone and car- tilage combining statistical shape knowledge and convolu- tional neural networks: Data from the osteoarthritis initia- tive

    Felix Ambellan, Alexander Tack, Moritz Ehlke, and Stefan Zachow. Automated segmentation of knee bone and car- tilage combining statistical shape knowledge and convolu- tional neural networks: Data from the osteoarthritis initia- tive. In Medical Imaging with Deep Learning, 2018. 6

    work page 2018

  2. [2]

    Geodesic matting: A frame- work for fast interactive image and video segmentation and matting

    Xue Bai and Guillermo Sapiro. Geodesic matting: A frame- work for fast interactive image and video segmentation and matting. Int. J. Comput. Vision, 82(2):113–132, 2009. 2, 6

    work page 2009

  3. [3]

    The RSNA- ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification

    Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, et al. The RSNA- ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification. arXiv e- prints, art. arXiv:2107.02314, 2021. 6

    arXiv 2021

  4. [4]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv’e J’egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630–9640, 2021. 1, 2

    work page 2021

  5. [5]

    Diffu- siondet: Diffusion model for object detection

    Shoufa Chen, Pei Sun, Yibing Song, and Ping Luo. Diffu- siondet: Diffusion model for object detection. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 2

    work page 2023

  6. [6]

    FocalClick: Towards Practical Interactive Image Segmentation

    Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. FocalClick: Towards Practical Interactive Image Segmentation . In2022 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 1290–1299, Los Alamitos, CA, USA, 2022. IEEE Computer Society. 2, 6

    work page 2022

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Rep- resentations, ICLR 20...

    2021

  8. [8]

    Random walks for image segmentation

    Leo Grady. Random walks for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 28(11):1768–1783, 2006. 2, 6

    work page 2006

Show all 64 references
  1. [9]

    Geodesic star convexity for interactive image segmentation

    Varun Gulshan, Carsten Rother, Antonio Criminisi, Andrew Blake, and Andrew Zisserman. Geodesic star convexity for interactive image segmentation. In 2010 IEEE Computer So- ciety Conference on Computer Vision and Pattern Recogni- tion, pages 3129–3136, 2010. 2, 6

    2010

  2. [10]

    Lvis: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 6

    2019

  3. [11]

    Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Unsupervised semantic segmentation by distilling feature correspondences. In In- ternational Conference on Learning Representations (ICLR),

  4. [12]

    Semantic contours from in- verse detectors

    Bharath Hariharan, Lubomir Bourdev, Pablo Arbelaez, Jiten- dra Malik, and Subhransu Maji. Semantic contours from in- verse detectors . In 2011 IEEE International Conference on Computer Vision (ICCV 2011), pages 991–998, Los Alami- tos, CA, USA, 2011. IEEE Computer Society. 5, 6

    2011

  5. [13]

    Girshick

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Doll’ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15979–15988, 2021. 1

    2022

  6. [14]

    Interformer real-time interactive image segmentation

    You Huang, Hao Yang, Ke Sun, Shengchuan Zhang, Liu- juan Cao, Guannan Jiang, and Rongrong Ji. Interformer real-time interactive image segmentation. In ICCV, pages 22244–22254. IEEE, 2023. 2, 6

    2023

  7. [15]

    Interactive image segmentation via backpropagating refinement scheme

    Won-Dong Jang and Chang-Su Kim. Interactive image segmentation via backpropagating refinement scheme. In 2019 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 5292–5301, 2019. 2

    2019

  8. [16]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9492–9502, 2023. 2

    2024

  9. [17]

    Segment anything in high quality

    Lei Ke, Mingqiao Ye, Martin Danelljan, Yifan liu, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Segment anything in high quality. In Advances in Neural Information Processing Systems, pages 29914–29934. Curran Associates, Inc., 2023. 2, 6

    2023

  10. [18]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything. In 2023 IEEE/CVF In- ternational Conference on Computer Vision (ICCV) ,...

    2023

  11. [19]

    Cohen, Dani Lischinski, and Matthew Uyttendaele

    Johannes Kopf, Michael F. Cohen, Dani Lischinski, and Matthew Uyttendaele. Joint bilateral upsampling. ACM SIG- GRAPH 2007 papers, 2007. 4, 7

    2007

  12. [20]

    Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak

    Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your Diffusion Model is Se- cretly a Zero-Shot Classifier . In 2023 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 2206– 2217, Los Alamitos, CA, USA, 2023. IEEE Computer So...

    2023

  13. [21]

    Multi- granularity interaction simulation for unsupervised interac- tive segmentation

    Kehan Li, Yian Zhao, Zhennan Wang, Zesen Cheng, Peng Jin, Xiang Ji, Li ming Yuan, Chang Liu, and Jie Chen. Multi- granularity interaction simulation for unsupervised interac- tive segmentation. 2023 IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 666–676, ...

    2023

  14. [22]

    Paintseg: training-free segmentation via painting

    Xiang Li, Chung-Ching Lin, Yinpeng Chen, Zicheng Liu, Jinglu Wang, Rita Singh, and Bhiksha Raj. Paintseg: training-free segmentation via painting. In Proceedings of the 37th International Conference on Neural Information Processing Systems, Red Hook, NY , USA, 2024. Curran As-...

    2024

  15. [23]

    Belongie, Lubomir D

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll’a r, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312,

    arXiv

  16. [24]

    Interactive image segmentation with first click attention

    Zheng Lin, Zhao Zhang, Lin-Zhuo Chen, Ming-Ming Cheng, and Shao-Ping Lu. Interactive image segmentation with first click attention. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13336–13345, 2020. 2

    2020

  17. [25]

    Focuscut: Diving into a focus view in interactive segmentation

    Zheng Lin, Zheng-Peng Duan, Zhao Zhang, Chun-Le Guo, and Ming-Ming Cheng. Focuscut: Diving into a focus view in interactive segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2637–2646, 2022. 2

    2022

  18. [26]

    Click prompt learning with optimal trans- port for interactive segmentation

    Jie Liu, Haochen Wang, Wenzhe Yin, Jan-Jakob Sonke, and Efstratios Gavves. Click prompt learning with optimal trans- port for interactive segmentation. In European Conference on Computer Vision (ECCV), 2024. 2, 6

    2024

  19. [27]

    Simpleclick: Interactive image segmentation with sim- ple vision transformers

    Qin Liu, Zhenlin Xu, Gedas Bertasius, and Marc Nietham- mer. Simpleclick: Interactive image segmentation with sim- ple vision transformers. In Proceedings. IEEE International Conference on Computer Vision, pages 22233–22243, 2023. 2, 3, 5, 6

    2023

  20. [28]

    Rethinking interactive image segmentation with low latency, high quality, and diverse prompts

    Qin Liu, Jaemin Cho, Mohit Bansal, and Marc Niethammer. Rethinking interactive image segmentation with low latency, high quality, and diverse prompts. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024 , pages 3773–

    2024

  21. [29]

    Fully convolutional networks for semantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In 2015 IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 3431–3440, 2015. 2

    2015

  22. [30]

    Martin, C

    D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecologi- cal statistics. In Computer Vision, 2001. ICCV 2001. Pro- ceedings. Eighth IEEE International Conference o...

    2001

  23. [31]

    Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization

    Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In 2022 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 6

    2022

  24. [32]

    Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael ...

    2024

  25. [33]

    A benchmark dataset and evaluation methodology for video object segmentation

    Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR...

    2016

  26. [34]

    DynaMITe : Dynamic Query Boot- strapping for Multi-object Interactive Segmentation Trans- former

    Amit Kumar Rana, Sabarinath Mahadevan, Alexander Her- mans, and Bastian Leibe. DynaMITe : Dynamic Query Boot- strapping for Multi-object Interactive Segmentation Trans- former. In 2023 IEEE/CVF International Conference on Computer Vision workshops : ICCVW 2023 : Paris, France,...

    2023

  27. [35]

    Sam 2: Segment anything in images and videos,

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

  28. [36]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. 2, 6

    2022

  29. [37]

    Sheldon M. Ross. Introduction to Probability Models. Aca- demic Press, San Diego, CA, USA, sixth edition, 1997. 4

    1997

  30. [38]

    ”grabcut”: interactive foreground extraction using iterated graph cuts

    Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. ”grabcut”: interactive foreground extraction using iterated graph cuts. In ACM SIGGRAPH 2004 Papers, page 309–314, New York, NY , USA, 2004. Association for Computing Ma- chinery. 2, 5, 6

    2004

  31. [39]

    Jianbo Shi and J. Malik. Normalized cuts and image seg- mentation. IEEE Trans. on Pattern Analysis and Machine Intelligence, 22(8):888–905, 2000. 2

    2000

  32. [40]

    V o, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick P’erez, Renaud Mar- let, and Jean Ponce

    Oriane Sim ´eoni, Gilles Puy, Huy V . V o, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick P’erez, Renaud Mar- let, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. ArXiv, abs/2109.14279, 2021. 2

    2021 arXiv

  33. [41]

    A Relationship Between Arbitrary Posi- tive Matrices and Doubly Stochastic Matrices

    Richard Sinkhorn. A Relationship Between Arbitrary Posi- tive Matrices and Doubly Stochastic Matrices. The Annals of Mathematical Statistics, 35(2):876 – 879, 1964. 4

    1964

  34. [42]

    Petrov, Olga Barinova, and Anton Konushin

    Konstantin Sofiiuk, Ilya A. Petrov, Olga Barinova, and Anton Konushin. F-brs: Rethinking backpropagating refinement for interactive segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020. 2

    2020

  35. [43]

    Re- viving iterative training with mask guidance for interactive segmentation

    Konstantin Sofiiuk, Ilya A Petrov, and Anton Konushin. Re- viving iterative training with mask guidance for interactive segmentation. In 2022 IEEE International Conference on Image Processing (ICIP), pages 3141–3145. IEEE, 2022. 2

    2022

  36. [44]

    Cfr-icl: Cascade-forward refinement with iterative click loss for interactive image segmentation

    Shoukun Sun, Min Xian, Fei Xu, Luca Capriotti, and Tiankai Yao. Cfr-icl: Cascade-forward refinement with iterative click loss for interactive image segmentation. In Technical Tracks 14, number 5 in Proceedings of the AAAI Conference on Artificial Intelligence, pages 5017–5024....

    2024

  37. [45]

    Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion

    Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonz ´alez-Franco. Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion. 2024 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 3554–3563, 2023. 2, 3, 4

    2024

  38. [46]

    Order-aware interactive segmen- tation, 2024

    Bin Wang, Anwesa Choudhuri, Meng Zheng, Zhongpai Gao, Benjamin Planche, Andong Deng, Qin Liu, Terrence Chen, Ulas Bagci, and Ziyan Wu. Order-aware interactive segmen- tation, 2024. 2, 6

    2024

  39. [47]

    Dense contrastive learning for self-supervised visual pre-training

    Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2021. 2

    2021

  40. [48]

    Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M. Alvarez. Freesolo: Learning to segment objects without annota- tions. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 14176– ...

    2022

  41. [49]

    Seg- ment anything without supervision

    Xudong Wang, Jingfeng Yang, and Trevor Darrell. Seg- ment anything without supervision. In The Thirty-eighth An- nual Conference on Neural Information Processing Systems,

  42. [50]

    Crowley, and Dominique Vaufreydaz

    Yangtao Wang, XI Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufreydaz. Self-supervised trans- formers for unsupervised object discovery using normalized cut. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14523–14533, 2022. 2

    2022

  43. [51]

    Crowley, and Dominique Vaufreydaz

    Yangtao Wang, Xiaoke Shen, Yuan Yuan, Yuming Du, Mao- mao Li, Shell Xu Hu, James L. Crowley, and Dominique Vaufreydaz. Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. IEEE Transactions on Pattern Analysis and Machine Intel...

    2022

  44. [52]

    Deep interactive object selection

    Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas Huang. Deep interactive object selection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 373–381, 2016. 2

    2016

  45. [53]

    Freemask: synthetic images with dense annotations make stronger segmentation models

    Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao. Freemask: synthetic images with dense annotations make stronger segmentation models. In Pro- ceedings of the 37th International Conference on Neural In- formation Processing Systems, Red Hook, NY , USA, 20...

    2024

  46. [54]

    Interac- tive segmentation as gaussian process classification

    Minghao Zhou, Hong Wang, Qian Zhao, Yuexiang Li, Yawen Huang, Deyu Meng, and Yefeng Zheng. Interac- tive segmentation as gaussian process classification. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19488–19497, 2023. 2 11 Repurposing Stabl...

    2023

  47. [56]

    In our approach, the aggregated attention tensor contains the output probabilities of the self-attention softmax operations

    Adjusting Temperature after the Softmax Operation The softmax of logits v ∈ RN and temperature T ∈ R>0 is given by the following equation: pi = e 1 T vi P j e 1 T vj (12) where p ∈ RN is the resulting probability distribution. In our approach, the aggregated attention tensor c...

  48. [57]

    3.3, our modified flood fill approach does not store a desired color in the output map, but instead stores the minimum required flood fill threshold to reach each pixel

    Pseudocode for Flood Fill Approach As described in Sec. 3.3, our modified flood fill approach does not store a desired color in the output map, but instead stores the minimum required flood fill threshold to reach each pixel. We show a possible implementation of this al- gorit...

  49. [58]

    3 we compared our M2N2, which utilizes Markov- maps, with the raw attention maps in Attention-NN and the KL-Divergence in KL-NN

    Hyperparameters of the Baselines In Tab. 3 we compared our M2N2, which utilizes Markov- maps, with the raw attention maps in Attention-NN and the KL-Divergence in KL-NN. We choose different hyperpa- rameters for each of these two baselines to improve their NoC. For Attention-N...

  50. [59]

    3.3 use the attention tensor A as a Markov transition operator to create a Markov chain

    Visualization of the Markov chain Our Markov-maps introduced in Sec. 3.3 use the attention tensor A as a Markov transition operator to create a Markov chain. In Fig. 6 we show an example of the probability distribution pt and Markov-map M over time. The first row shows the dis...

  51. [60]

    7 shows the convergence of M2N2 on the GrabCut, Berkeley, SBD and DA VIS datasets, respectively

    Additional Results Fig. 7 shows the convergence of M2N2 on the GrabCut, Berkeley, SBD and DA VIS datasets, respectively. On all four datasets, M2N2 converges faster than Attention-NN, KL-NN and M2N2 method without flood fill. We observe the fastest convergence on the GrabCut d...

  52. [61]

    10 shows the segmentation of an im- age from the GrabCut dataset with three prompt points

    Visualization of Score Functions The example in Fig. 10 shows the segmentation of an im- age from the GrabCut dataset with three prompt points. For each prompt point, we display the corresponding Markov- map, the score functions si,·(λ) evaluations of potential thresholds λ an...

  53. [62]

    11 shows segmentation examples of each score func- tion

    Score Function Scenarios Fig. 11 shows segmentation examples of each score func- tion. Starting in the first columns on going from left to right: Column 1, No score function : The threshold is con- stant λi = 0 .5. We observe in the first row that the segmentation of the wolf ...

  54. [63]

    During extraction of the attention maps, we remove the CLS token to obtain the image self-attention

    Additional Backbones in Detail ViT-B: We use the pre-trained weights of DinoV2 pro- vided by Hugging Face’s transformers package. During extraction of the attention maps, we remove the CLS token to obtain the image self-attention. We aggregate only the last attention layer as ...

  55. [64]

    Choice of Backbone M2N2’s NoC is highly dependent on the attention map res- olution. Fig. 13 shows a comparison of various backbones on the DA VIS and Berkeley NoC90 values. Each backbone is evaluated on a range of attention map resolutions, start- ing with the lowest resoluti...

  56. [2021]

    OpenReview.net, 2021. 1

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.