{"id":"e05e5337-7a4c-40e5-be4a-e1a69042107a","arxiv_id":"2411.10411","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"By counting Markov-chain hitting times in Stable Diffusion attention, M2N2 performs training-free interactive segmentation that outperforms trained unsupervised methods on three of four benchmarks.","lead":"This paper turns the self-attention maps inside Stable Diffusion into a Markov chain, then counts how many steps each pixel needs to become confident to build a segmentation map. The result is a training-free interactive segmenter that matches or beats trained unsupervised methods on standard benchmarks with fewer clicks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central SOTA claim rests on hyperparameters selected directly on the evaluation benchmarks; without a validation split, the reported margins over MIS may reflect test-set overfitting rather than method quality.","rationale":"The reader's verdict is CONDITIONAL, and the weakest assumption identified is the adaptive threshold heuristic. I agree that heuristic is ad hoc and load-bearing, but I see a more fundamental problem: the central empirical claim is not secured because the method's hyperparameters are selected using the test datasets themselves. Fig. 4 and Tab. 1 present sweeps over attention resolution, temperature, and relative threshold, and the chosen values (128, 0.65, 0.3) are the ones that minimize NoC on the four evaluation benchmarks. The paper does not mention a validation split or any procedure for fixing hyperparameters independently of the test sets. This makes the reported improvements over MIS, and the claim of state-of-the-art unsupervised performance, vulnerable to test-set overfitting. The threshold heuristic is a real concern, but it is at least supported by the ablations in Tab. 2 and the qualitative examples; the hyperparameter selection has no such protection. If the authors were to tune on a separate split, the margins could shrink. The proposed concrete test is feasible with the provided code and would settle whether the central claim generalizes. I therefore recommend keeping the CONDITIONAL verdict, with the additional requirement that the authors disclose and justify their hyperparameter selection procedure. The paper's strengths, including the Markov-map formulation, the flood-fill construction, and the released code, are acknowledged; the concern is about the empirical evaluation methodology, not the internal math.","tokens_in":17825,"tokens_out":8010,"duration_ms":82949,"concrete_test":"Using the released code, fix a validation protocol: tune the main hyperparameters (attention resolution, T, tau, block weights) on DAVIS only, then freeze them and evaluate on GrabCut, Berkeley, and SBD. Compare the resulting NoC85/NoC90 to MIS. If M2N2 does not beat MIS on at least two of the three datasets, the reported SOTA margins are likely test-set artifacts. Also report the chosen hyperparameter values for the validation configuration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To establish that M2N2 outperforms trained unsupervised methods, the paper reports NoC numbers on GrabCut, Berkeley, DAVIS, and SBD. These numbers depend on several free hyperparameters: attention resolution (128), SD timestep (100), Markov temperature T=0.65, relative threshold tau=0.3, aggregation weights w_up0=0.5 and w_up1=0.5, and JBU parameters. Section 4.2 and Fig. 4 show these values were chosen by sweeping each hyperparameter and picking the point with lowest NoC on the same four datasets used for the final comparison. No validation split or cross-validation is described. This is direct test-set fitting: the reported improvement over MIS is at least partly the result of selecting the best configuration from the test-set performance curves. The claim 'training-free' does not protect against this, because the hyperparameters are still tuned on the evaluation data. The adaptive threshold heuristic (Eq. 8-9) is also load-bearing, but it is validated by ablations; the missing validation protocol affects the headline comparison itself. If a fixed configuration chosen on a holdout split were used, the margins over MIS could shrink or disappear, which would invalidate the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes M2N2, a training-free interactive point-prompt segmentation framework built on Stable Diffusion 2 self-attention. The self-attention tensor is treated as a Markov transition operator; after iterative proportional fitting to make it doubly stochastic, a Markov chain from each prompt point yields a per-pixel hitting-time map (the Markov-map). A modified flood fill and a truncated nearest-neighbor rule combine multiple prompt points into a segmentation. The method is evaluated on GrabCut, Berkeley, SBD, and DAVIS with Number of Clicks (NoC85/NoC90) under the standard click-simulation protocol. The main claim is that M2N2, despite being training-free, outperforms the trained unsupervised baseline MIS on three of the four datasets (GrabCut, Berkeley, DAVIS) while losing only on SBD.","tokens_in":18048,"tokens_out":5952,"duration_ms":61411,"significance":"If the empirical claims survive a properly controlled evaluation, this is a valuable contribution. The core idea is clean and novel: interpreting SD self-attention as a Markov operator and using hitting times as a semantic distance yields a simple, deterministic, training-free segmentation method. The mathematical construction is sound and well explained, including the use of IPF to enforce a uniform stationary distribution. The paper includes extensive ablations (attention blocks, temperature, threshold, resolution, score functions) and releases code, which strengthens reproducibility. The reported gains over the trained unsupervised method MIS on GrabCut (1.62 vs 1.94 NoC85), Berkeley (2.45 vs 3.09), and DAVIS (4.60 vs 6.33) are large, but they rest on hyperparameters selected on the same test benchmarks and on a threshold heuristic whose quantitative reliability is not established. These issues are fixable within the manuscript's scope, so the result is promising but requires revision.","major_comments":[{"comment":"The final configuration (attention resolution 128, SD time step 100, temperature T=0.65, relative threshold τ=0.3, aggregation weights 0.5/0.5) is chosen from the NoC curves in Fig. 4, which are computed on the same evaluation datasets used for the headline comparison in Table 3 (GrabCut, Berkeley, DAVIS, and a 500-image SBD subset). No validation split or cross-validation is described. Because NoC computation requires ground-truth masks to simulate clicks, sweeping hyperparameters to minimize NoC on these datasets is direct test-set fitting. This biases the claimed margins over MIS. Please add a validation protocol: select hyperparameters on a hold-out set or via cross-validation, then report the resulting test-set NoC, and also report sensitivity of the final numbers to a plausible range of hyperparameter choices. Without this, the central empirical claim is not fully supported.","section":"Sec. 4.1–4.2, Fig. 4, Table 3"},{"comment":"Algorithm 1, line 8 computes λ ← max(λ′, abs(M[y] − M[x])). Here x is the input prompt point while x′ is the currently popped pixel, so the update measures the Markov-map difference from the prompt point rather than from the current pixel. This contradicts the text's description that the flood fill stores the minimum flood threshold required to reach each pixel from the starting point and would, as written, produce a different map than the one used in the experiments. The correct update should reference M[x′] (or equivalently propagate the maximum of the threshold along the path and the absolute difference between adjacent pixels). Please correct the pseudocode and specify the exact update rule, including whether diagonal neighbors are considered, so that the 'with flood fill' results in Table 3 are reproducible.","section":"Supplementary Sec. 7, Algorithm 1"},{"comment":"The adaptive threshold λ_i = arg max_λ s_i(λ) is load-bearing: a wrong threshold produces a wrong segment for that prompt point regardless of Markov-map quality. The score functions are a hand-crafted product of four terms, and their support is mainly an ablation on DAVIS (Table 2) plus qualitative examples. No quantitative evidence is given for how often the heuristic selects a threshold close to the oracle threshold, nor how its failures vary across datasets and images. Because the ablations are aggregated, failures on a subset of images could be masked. Please provide a per-dataset analysis comparing the heuristic threshold with an oracle threshold (e.g., the threshold that would maximize IoU or minimize click count), or a per-image success rate of the score-function selection. This would place the central NoC result on firmer ground.","section":"Sec. 3.4, Eqs. (8)–(11), Table 2"}],"minor_comments":[{"comment":"The paper states that IPF converts the attention matrix to a doubly stochastic matrix, but it does not specify the IPF convergence tolerance, maximum number of iterations, or the exact stopping criterion. Since the supplementary notes that the number of IPF iterations depends on temperature, please include these implementation details for reproducibility.","section":"Sec. 3.3, IPF description"},{"comment":"The SBD curves in Fig. 4 are based on a randomly sampled subset of 500 images, while Table 3 reports NoC on the full SBD validation set. Please clarify whether the hyperparameters used for SBD in Table 3 were chosen on the subset or on the full set, and whether the same configuration is used for all datasets.","section":"Fig. 4 and Table 3"},{"comment":"The paragraph reporting BraTS and OAIZIB IoU values appears in the experiments section but is not connected to the rest of the evaluation; please place it in a dedicated limitations/discussion paragraph or expand it into a clear domain-bias analysis, since it is a relevant caveat for the method's general applicability.","section":"Sec. 4.3, SD2 Domain Bias"},{"comment":"There are several typographical errors, including 'DA VIS' with a space in multiple places, 'slected' in the Fig. 5 caption, 'T otal Score' in Fig. 10, and 'spriorsedgespossneg' in Fig. 11. These should be corrected before publication.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the hyperparameter-selection protocol: the paper tunes on the test benchmarks and then claims superiority over MIS. If a proper validation split substantially shrinks the margin, the headline claim would weaken, but the method could still be interesting. The flood-fill pseudocode bug should be fixed regardless. I do not see a novelty or integrity concern beyond the tuning issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the Markov-map idea is genuinely new and the method is well-engineered, but the headline comparison to trained unsupervised methods is on shakier ground than the paper suggests, because the hyperparameters appear to be chosen on the very datasets used in the final comparison.\n\nThe core contribution is elegant. The authors treat the aggregated self-attention tensor of Stable Diffusion as a Markov transition operator, use iterative proportional fitting to make it doubly stochastic, and then count how many iterations each pixel needs to reach a relative probability threshold. The result is a Markov-map that is noticeably less noisy and has sharper boundaries than raw attention maps. The flood-fill modification gives instance-level maps. The math is sound: the chain converges to uniform after IPF, and temperature controls the rate of convergence. The ablations are thorough—attention blocks, resolution, timestep, temperature, threshold, and each score function—and the code is available.\n\nThe soft spot is real. Section 4.2 and Fig. 4 show sweeps over the key hyperparameters on the same four datasets that appear in the final comparison (GrabCut, Berkeley, DAVIS, SBD). There is no validation split. So the reported margins over MIS—1.62 vs 1.94, 2.45 vs 3.09, 4.60 vs 6.33—are at least partly the product of selecting the best-looking configuration from the test curves. The margins are big enough that the method is probably still competitive, but the exact SOTA claim is not established. The adaptive threshold heuristic (Eq. 8-9) is also hand-crafted and load-bearing; a wrong threshold yields a wrong segment regardless of Markov-map quality. That is a secondary issue, but worth probing in a revision.\n\nCredit where due: the authors openly report a domain bias on medical data and show failure cases on SBD. That kind of honesty is rare.\n\nWho is this for? Anyone working on training-free segmentation, attention representations, or interactive annotation tools. It deserves a serious referee. The contribution is the representation and the engineering, not just the benchmark table. My recommendation: send it to review, and ask the authors to fix the validation protocol and analyze the threshold heuristic's failure modes.\n\nThe paper is worth engaging with, but the empirical claim needs tightening before I'd quote the numbers.","headline":"A clever Markov-map representation of Stable Diffusion attention, but the SOTA comparison rests on hyperparameters selected on the test sets.","tokens_in":18571,"tokens_out":4419,"would_cite":true,"duration_ms":40218,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"M2N2 shows that Stable Diffusion self-attention, reinterpreted as a Markov transition operator, can drive training-free unsupervised interactive segmentation that beats trained unsupervised baselines on most benchmarks.","keywords":["interactive segmentation","point prompts","Stable Diffusion","self-attention","Markov chain","training-free unsupervised segmentation","Number of Clicks","Markov-map"],"falsifier":"Build a set of natural images whose intended foreground object covers more than 40% of the frame and run M2N2 with a single foreground prompt and no background prompt. The score function $s_{i,\\mathrm{prior}}(\\lambda)$ assigns zero to every threshold that produces a segment above 40% of the image, so the selected threshold must cut inside the object; if the resulting segment is consistently a proper subset of the target, the adaptive threshold heuristic is the load-bearing failure point.","tokens_in":17575,"feed_emoji":"🖱️","tokens_out":8173,"duration_ms":74268,"temperature":0.7,"pith_summary":"This paper tries to establish that a frozen text-to-image model's self-attention already contains enough semantic structure for interactive point-prompt segmentation, without any training or pseudo-labels. The route is to read the aggregated self-attention tensor as a Markov transition operator and count, for every pixel, how many iterations a probability mass starting at the clicked point needs to saturate a relative threshold; that count is the Markov-map. The paper claims Markov-maps are less noisy, have sharper semantic boundaries, and are more uniform inside semantically similar regions than raw attention maps. Because of that, a truncated nearest-neighbor rule over the maps reaches 85% IoU in 1.62 average clicks on GrabCut, 2.45 on Berkeley, and 4.60 on DAVIS, improving on the trained unsupervised baseline MIS, which needs 1.94, 3.09, and 6.33 (on SBD the method is second). The reason a reader should care is that the result suggests promptable segmentation need not be a trained module at all; a generative backbone's internal attention, refined by a Markov process, can serve as the distance function.","feed_headline":"No labels, no training: attention Markov maps cut segmentation clicks","feed_subtitle":"Stable Diffusion's self-attention, read as a Markov chain, hits 85% IoU in 1.62 clicks on GrabCut.","key_machinery":"The Markov-map is the central object. For a prompt point $i$, it is the matrix $M_i \\in \\mathbb{R}^{H \\times W}$ whose entry at pixel $q$ is the smallest iteration count $t$ for which the Markov-chain probability $p_t[q]/\\max p_t$ exceeds the relative threshold $\\tau$, with $p_t = p_0 A^t$ and $p_0$ the one-hot start state at the click. Iterative proportional fitting makes the transition matrix $A$ doubly stochastic so every start state converges to the uniform distribution, and a temperature $T$ controls the convergence rate. A modified flood fill then stores the minimum threshold needed to reach each pixel, suppressing local minima so instances become separable, and joint bilateral upsampling raises the map to image resolution. The final distance $d(x_i, x_q) = M_i[x_q]/\\lambda_i$ feeds a truncated nearest neighbor, with $\\lambda_i$ chosen by a product of four score functions.","core_discovery":"At the core, the paper argues that convergence time in a Markov chain built from Stable Diffusion's self-attention is a better semantic distance than the attention values themselves. Starting from the one-hot location of a clicked point, the probability distribution $p_t$ evolves under the doubly-stochastic self-attention matrix, and each pixel's Markov-map value is the minimum $t$ at which its relative probability first exceeds the threshold $\\tau$. The paper claims this map separates instances after a flood-fill step and that a truncated nearest-neighbor decision on the rescaled maps--assigning background to query pixels more than one unit away from their nearest prompt--yields the reported NoC improvements, outperforming the pseudo-label-trained MIS on three of four datasets in both NoC85 and NoC90 despite using no training at all.","pith_inferences":["Because the Markov-map construction is a generic post-processing of an attention tensor, it could be dropped into any attention-based backbone, not just Stable Diffusion; the paper's own ViT-B and SD1.1 results are partial evidence, and other diffusion architectures are a natural next test.","The hand-crafted threshold selector is the most replaceable part: the edge-score ablation shows boundary alignment is the largest single factor, so a closed-form or learned boundary-alignment criterion would likely reduce the remaining clicks.","The 40% size prior implies a systematic blind spot for very large objects; a test set of images whose target fills most of the frame should expose a failure mode that the paper's averaged benchmarks understate.","The paper reports a domain bias toward natural images, with lower IoU on medical datasets; extending the method or adapting the Markov-map temperature to such domains is a concrete testable direction."],"forward_implications":["Promptable segmentation can run on a frozen generative model at inference time, eliminating the pseudo-label generation and training loop used by previous unsupervised methods.","On the three natural-image benchmarks in the paper, the training-free method closes the gap with trained unsupervised methods and surpasses them, so training may be unnecessary for this task in this regime.","Attention resolution and input resolution are the main dials: performance improves up to 128x128 attention maps and degrades when the input image is pushed beyond the backbone's native training resolution.","Per-click latency around 0.6 seconds on an RTX 4090 at 854x480, with caching of attention and Markov-maps, makes the approach usable in an interactive loop.","Thin structures, fine details, and overlapping or obstructed instances remain the failure mode, requiring additional prompt points and sometimes failing at the 20-click cap."],"supporting_citations":[{"why":"Supplies the Stable Diffusion backbone whose self-attention tensors are reinterpreted as a Markov transition operator.","marker":"[36]"},{"why":"Establishes that Stable Diffusion self-attention can be aggregated and used for zero-shot segmentation, which motivates the attention tensor chosen here.","marker":"[45]"},{"why":"Defines the MIS pseudo-label-trained unsupervised baseline and the click-simulation protocol, and is the main comparison the paper claims to outperform on three datasets.","marker":"[21]"},{"why":"Supplies the iterative proportional fitting result that the paper uses to make the attention matrix doubly stochastic, so every start state converges to the uniform distribution.","marker":"[41]"},{"why":"Provides the interactive-segmentation protocol and click simulation used to evaluate Number of Clicks in the experiments.","marker":"[27]"},{"why":"Provides joint bilateral upsampling, which the paper modifies to raise Markov-maps from attention resolution to image resolution.","marker":"[19]"},{"why":"Provides the GrabCut dataset used for evaluation and a classical unsupervised baseline (GraphCut) that M2N2 outperforms.","marker":"[38]"}],"fun_headline_variants":["Training-free segmentation: Markov maps from Stable Diffusion attention","No training needed: Stable Diffusion attention becomes Markov maps","From attention to Markov maps: fewer clicks for segmentation","Stable Diffusion attention reinterpreted as Markov chains for segmentation","Markov maps from Stable Diffusion: training-free interactive segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's success rests on a hand-crafted rule that guesses, for each click, how far the Markov-map should extend; if the rule guesses wrong, that click's segment is wrong no matter how good the map is.","fun_headline_variants_meta":{"raw":{"variants":["Training-free segmentation: Markov maps from Stable Diffusion attention","No training needed: Stable Diffusion attention becomes Markov maps","From attention to Markov maps: fewer clicks for segmentation","Stable Diffusion attention reinterpreted as Markov chains for segmentation","Markov maps from Stable Diffusion: training-free interactive segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000965,"raw_usage":{"total_tokens":4101,"prompt_tokens":932,"completion_tokens":3169,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":3091}},"tokens_in":548,"tokens_out":3169,"duration_ms":21940,"temperature":1.0,"reasoning_tokens":3091,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:38:12.578783+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a set of natural images whose intended foreground object covers more than 40% of the frame and run M2N2 with a single foreground prompt and no background prompt. The score function $s_{i,\\mathrm{prior}}(\\lambda)$ assigns zero to every threshold that produces a segment above 40% of the image, so the selected threshold must cut inside the object; if the resulting segment is consistently a proper subset of the target, the adaptive threshold heuristic is the load-bearing failure point.","supporting_citations":[{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the Stable Diffusion backbone whose self-attention tensors are reinterpreted as a Markov transition operator."},{"cited_title":"Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion","cited_arxiv_id":null,"evidence_quote":"Establishes that Stable Diffusion self-attention can be aggregated and used for zero-shot segmentation, which motivates the attention tensor chosen here."},{"cited_title":"Multi- granularity interaction simulation for unsupervised interac- tive segmentation","cited_arxiv_id":null,"evidence_quote":"Defines the MIS pseudo-label-trained unsupervised baseline and the click-simulation protocol, and is the main comparison the paper claims to outperform on three datasets."},{"cited_title":"A Relationship Between Arbitrary Posi- tive Matrices and Doubly Stochastic Matrices","cited_arxiv_id":null,"evidence_quote":"Supplies the iterative proportional fitting result that the paper uses to make the attention matrix doubly stochastic, so every start state converges to the uniform distribution."},{"cited_title":"Simpleclick: Interactive image segmentation with sim- ple vision transformers","cited_arxiv_id":null,"evidence_quote":"Provides the interactive-segmentation protocol and click simulation used to evaluate Number of Clicks in the experiments."},{"cited_title":"Cohen, Dani Lischinski, and Matthew Uyttendaele","cited_arxiv_id":null,"evidence_quote":"Provides joint bilateral upsampling, which the paper modifies to raise Markov-maps from attention resolution to image resolution."},{"cited_title":"”grabcut”: interactive foreground extraction using iterated graph cuts","cited_arxiv_id":null,"evidence_quote":"Provides the GrabCut dataset used for evaluation and a classical unsupervised baseline (GraphCut) that M2N2 outperforms."}],"review_version":1}