{"id":"bb061f25-2e0c-443e-bf0e-f065ef5ecac0","arxiv_id":"2506.21835","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ProSAM improves SAM-based visual reference segmentation by learning a Student-t prompt distribution through noise injection, beating VRP-SAM by about one mIoU point on Pascal-5^i and COCO-20^i.","lead":"ProSAM, a probabilistic prompt encoder for SAM-based visual reference segmentation, trains with noisy prompt samples drawn from a Student-t distribution and uses only the mean prompt at test time. It reports consistent mIoU improvements over VRP-SAM on Pascal-5^i and COCO-20^i, though the gains are modest and baseline reproducibility is not fully resolved.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed Laplacian-regularization mechanism can vanish because the learned noise scale sigma_z is unconstrained: the optimizer can drive sigma_z to zero, reducing ProSAM to deterministic VRP-SAM.","rationale":"The reader correctly identified that the theoretical argument rests on smoothness and small-noise assumptions that are unverified. My concern is more specific and more load-bearing: in addition to those assumptions, the objective has no mechanism that keeps the noise variance nonzero. Because sigma_z is learned and unregularized, the optimizer can collapse it to zero, which would make the Laplacian penalty term vanish exactly and reduce ProSAM to deterministic VRP-SAM. This is an internal gap between Proposition 1 and the optimization in Eq. 8-10, not merely a question of whether the loss is smooth. The empirical results may still be valid, especially if the fixed seed and Monte Carlo sampling keep sigma_z alive, but without reporting learned variances or ablating the variance prediction, the paper's explanation for its own success is unsupported. This reinforces the reader's CONDITIONAL verdict rather than overturning it, so the verdict is unchanged.","tokens_in":20397,"tokens_out":5383,"duration_ms":65307,"concrete_test":"Retrain ProSAM on PASCAL-5^0 with logging of the predicted log sigma_z on the validation set each epoch. If the mean log sigma_z falls by more than 90% from its initialization or reaches a negligible magnitude at convergence, the Laplacian penalty is inactive. Then run two controls: one with sigma_z fixed to a positive constant, and one with sigma_z fixed to zero. If the fixed-positive-sigma control matches the reported F-0 mIoU of 75.26, the benefit comes from noise injection rather than the learned variational distribution; if the sigma_z=0 control matches, the central robustness mechanism is not what drives the gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 1 in Appendix 7.1 is mathematically correct as a Taylor expansion, but the paper's mapping of it to the ProSAM objective is not justified. In the actual optimization (Eq. 8-10), the noise scale sigma_z is a learned output of the prompt encoder, and the objective contains no prior, KL term, or constraint that keeps sigma_z away from zero. Under the paper's own assumption in Section 7.2 that the loss is locally convex with lambda_i >= 0 near a local minimum, the Laplacian term is nonnegative, so for a fixed mean mu_z the expected loss E_z[L(z)] is locally minimized at sigma_z = 0. At sigma_z = 0 the proposed Laplacian penalty disappears and ProSAM becomes a deterministic prompt encoder indistinguishable from VRP-SAM plus two linear layers. The finite Monte Carlo estimate (K=10) may prevent exact collapse through stochastic gradient noise, but that is an optimization accident, not part of the theoretical argument. Because the paper never reports the learned sigma_z values or any diagnostic of variance collapse, the central claim that ProSAM's distribution actively pushes prompts toward flat, stable regions is not empirically established. If sigma_z is small at convergence, the reported mIoU gains must be attributed to something other than the advertised robustness mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ProSAM, a modification of the SAM-based visual reference segmentation method VRP-SAM. ProSAM adds a variational prompt encoder that predicts a multivariate Student-t distribution over prompt embeddings instead of a single deterministic embedding. During training, prompts are sampled from this distribution using the reparameterization trick and the expected segmentation loss is minimized; at inference, only the predicted mean prompt is used. The authors justify the method through Proposition 1, which states that minimizing the expected loss under small zero-mean noise is equivalent to minimizing the loss plus a Laplacian regularization term, thereby encouraging flatter, more robust regions of the prompt-loss landscape. Experiments on Pascal-5^i and COCO-20^i report consistent mIoU improvements over VRP-SAM, along with ablations, a domain-shift study, and qualitative analyses.","tokens_in":20646,"tokens_out":5004,"duration_ms":56687,"significance":"If the proposed mechanism holds, the paper makes a simple and useful contribution: adding a learned noise distribution and using the mean at inference yields robustness gains without architectural complexity or additional inference cost. The empirical study is unusually careful in two respects: the authors re-run VRP-SAM under identical settings after failing to reproduce its published numbers, and they include a parameter-matched ablation (VRP-SAM with two linear layers) in Appendix 10.2. The theoretical result in Proposition 1 is mathematically correct as a standalone statement. However, the central claim that ProSAM's learned distribution actively pushes prompts toward flat, stable regions is not yet established because the learned noise scale can collapse, and the assumptions connecting flatness to the center of the target prompt region are unverified.","major_comments":[{"comment":"The training objective contains no KL term, prior, or constraint on the learned noise scale sigma_z. Under the paper's own local-convexity assumption in Section 7.2, the Laplacian term is nonnegative, so for a fixed mean the expected loss is locally minimized at sigma_z = 0; at that point the implicit Laplacian penalty in Eq. (12) vanishes and ProSAM degenerates to deterministic VRP-SAM with two extra linear layers. The paper never reports learned sigma_z values or any diagnostic of variance collapse, so the advertised mechanism is not empirically supported. Please report sigma_z statistics at convergence, or add a prior, KL term, or explicit lower bound that keeps the distribution nondegenerate, and ablate with a fixed nonzero sigma_z to confirm that the gains come from the noise-regularization mechanism rather than from the extra parameters or Monte Carlo gradient noise.","section":"Section 4.3, Eqs. (8)-(10), and Section 7.2"},{"comment":"Proposition 1 requires f to be twice continuously differentiable and the noise variance to be small; the paper does not verify either condition for the frozen SAM mask decoder or for the Student-t noise with ν=5. More importantly, the mapping from 'flat loss region' to 'center of the target prompt region R_{Ir,Mr,It}' is assumed rather than demonstrated. A flat region need not coincide with the robust center, and the verification in Section 5.3 approximates the center using the prompt embedding of a center point in the ground-truth mask, which is not necessarily the center of R for mask, box, or scribble prompts. The reported cosine similarities between predicted prompts and those center-point embeddings are near zero (0.007 to 0.042), so this evidence does not strongly support the claim that ProSAM prompts align with the target-region center. Please either validate the flatness-to-center assumption with a direct measurement or soften the claim accordingly.","section":"Sections 7.1-7.2 and 4.2"},{"comment":"All experimental results are from a single random seed, while training includes stochastic Monte Carlo sampling. The margins over VRP-SAM are modest (for example, +0.90 mIoU on PASCAL-5^i mask and +1.35 on COCO-20^i mask), and without variance estimates or significance testing the claim that ProSAM 'consistently surpasses' VRP-SAM is not statistically supported. Please report mean and standard deviation over at least three seeds, or provide paired significance tests for the main tables.","section":"Section 5.1 and Tables 1-3"},{"comment":"The gradient-descent verification study is presented as evidence that ProSAM's prompts cluster at the center of the target prompt region, but it analyzes a single sample image and relies on a qualitative t-SNE projection. More importantly, the 'center' in that study is defined relative to the variational distribution's own samples, so the conclusion that the samples are centered is partly circular. Please provide a quantitative, multi-image evaluation of the distance to a well-defined robust-prompt reference, or clearly state the limitations of this illustrative study.","section":"Appendix 9"}],"minor_comments":[{"comment":"The phrase 'heavy-tailless' should be 'heavy-tailedness'.","section":"Section 4.2"},{"comment":"The evaluation metric is written inconsistently as both 'mIoU' and 'mIOU'; please standardize.","section":"Section 5.2 and Table 2"},{"comment":"The table header is duplicated ('MethodParameters PASCAL-5i' appears twice), which obscures the ablation layout.","section":"Table 5"},{"comment":"References [42], [43], and [44] appear to be duplicates of the same paper (Few-shot segmentation via cycle-consistent transformer); please consolidate them.","section":"References"},{"comment":"The statement 'ϵ ∈ Rn is an i.i.d. distributed random noise vector' is redundant; 'i.i.d.' should apply to the components, not to the vector itself.","section":"Appendix 7.1"},{"comment":"The noise-perturbation robustness study is reported only for PASCAL-5^i F-0 with no error bars or per-image variability; please clarify whether this result is representative across folds and datasets.","section":"Section 5.3 and Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's framing as 'variational inference' is loose, since the objective is simply an expected loss under a learned noise distribution without a KL term or posterior matching. The main technical risk is learned-variance collapse, which would invalidate the proposed mechanism; this is fixable in revision by reporting sigma_z and adding a constraint, but it is currently load-bearing. The single-seed results and the near-zero cosine-similarity verification also need strengthening before the empirical claims can be fully trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an empirical paper with a small but consistently reproduced win. ProSAM trains a prompt encoder to output a Student-t distribution over SAM prompt embeddings, samples during training, and uses the mean at inference. On Pascal-5^i and COCO-20^i it beats their self-rerun VRP-SAM by roughly 0.9–1.3 mIoU across point/scribble/box/mask prompts, with a matched setup. The gain is modest but real. What's new is the application to SAM prompts; the theory, Proposition 1, is a textbook Taylor expansion (noise injection ~ Laplacian penalty), proved correctly but not deep.\n\nThe paper does several things right. They couldn't reproduce the published VRP-SAM numbers, so they retrained both methods under matched settings and said so. They control for the extra two linear layers by adding the same layers to VRP-SAM. The domain-shift test and the robustness-to-noise perturbation study support the idea that ProSAM prompts are more stable. The ablations (Gaussian vs t, K, nu) are informative.\n\nSoft spots: single seed, no error bars, no code. The 'variational' framing is a stretch—no prior, no KL term, it's just learned noise injection. More importantly, the learned sigma_z is unconstrained; the objective could in principle push it to zero, which would delete the Laplacian penalty and reduce ProSAM to deterministic VRP-SAM plus two layers. The stress-test note raises this, and I think it's a fair concern. The paper never reports the learned sigma values. The empirical gains might come from early-training noise acting as a curriculum or from stochastic gradient noise interacting with the MC estimator, rather than the advertised flat-minimum mechanism. The theory also assumes smoothness and small sigma, which are unverified for the frozen SAM decoder. These are fixable: report sigma statistics, run a fixed-sigma variant, add multi-seed results.\n\nOverall the central empirical claim holds up for me: the method works, and the improvements are consistent. The theoretical story needs to match what the optimizer actually does. I'd send this to peer review; a good referee would ask for the sigma diagnostics and code, but the paper deserves the time. I'd cite it if I worked on SAM prompting. Bring it to reading group, maybe—it's a good example of a simple trick with honest evaluation.","headline":"A simple, plausible robustness trick for SAM prompt generation; the empirical gains are consistent, but the theory is overclaimed and the learned variance could collapse—still worth peer review.","tokens_in":21160,"tokens_out":3995,"would_cite":true,"duration_ms":44345,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ProSAM learns a distribution over SAM prompt embeddings and uses its mean to make visual reference segmentation more stable and accurate.","keywords":["visual reference segmentation","Segment Anything Model","probabilistic prompts","variational prompt encoder","prompt robustness","few-shot semantic segmentation","Laplacian regularization","Student-t distribution"],"falsifier":"A concrete check: compute the average trace of the Hessian of SAM's mask loss at the prompt embeddings produced by ProSAM and by the deterministic baseline on the same images; if ProSAM's prompts do not show lower curvature, the flatness mechanism is not what drives the reported gains. An even simpler check is to inject the same Gaussian noise into the deterministic baseline's prompts during training and see whether its mIoU rises by ProSAM's margin; if it does, no distributional prompt encoder is needed.","tokens_in":20176,"feed_emoji":"🎯","tokens_out":9255,"duration_ms":96210,"temperature":0.7,"pith_summary":"This paper tries to establish that the instability of SAM-based visual reference segmentation comes from prompts that land near the boundary of the set of prompt embeddings that yield acceptable masks, and that this can be fixed by treating prompts as samples from a learned probability distribution rather than as single deterministic vectors. The proposed ProSAM trains a variational prompt encoder that predicts a multivariate Student-t distribution over prompt embeddings, injects noise during training via the reparameterization trick, and at inference uses only the distribution's mean. The paper proves (Proposition 1) that minimizing the expected loss under small noise is equivalent to minimizing the original loss plus a Laplacian (curvature) penalty, so the mean prompt is pushed into flatter, more central regions of the loss landscape. Across Pascal-5^i and COCO-20^i, ProSAM consistently outperforms the state-of-the-art deterministic baseline under all annotation types and under domain shift, with fewer learnable parameters than leading few-shot segmentation alternatives.","feed_headline":"Prompt distributions push SAM prompts off unstable boundaries","feed_subtitle":"Noise-injected training flattens the prompt loss landscape and beats the deterministic baseline on Pascal-5i and COCO-20i.","key_machinery":"The load-bearing machinery is the variational prompt encoder $q_\\phi(z|I_r, M_r, I_t)$, which turns a visual reference and a target image into a multivariate Student-t distribution over prompt embeddings, parameterized by a mean $\\hat{\\mu}_z$ and a diagonal scale $\\hat{\\sigma}_z^2$ and sampled through the reparameterization trick. The load-bearing identity is Proposition 1: if the mask loss $L$ is twice continuously differentiable and the prompt noise $\\epsilon$ has mean zero and covariance $\\sigma^2 I$, then $\\mathbb{E}_\\epsilon[L(z+\\epsilon)] = L(z) + \\frac{\\sigma^2}{2}\\Delta L(z) + O(\\sigma^3)$. This identity converts the intractable expected-loss objective into the standard mask loss plus an implicit penalty on the Laplacian (trace of the Hessian) of the loss, which is what pushes the mean prompt into flatter, more central regions of the target prompt region. Choosing a Student-t instead of a Gaussian additionally enlarges the fourth-order curvature penalty, giving an extra push toward flat regions.","core_discovery":"The central discovery claim is that prompt robustness, not prompt accuracy alone, is the bottleneck for SAM-based visual reference segmentation. There is a target prompt region whose prompts all produce acceptable masks, but existing trained prompt encoders tend to produce prompts near its boundary, where small perturbations cause large mask changes. ProSAM's variational prompt encoder models the full prompt distribution as a multivariate Student-t with diagonal covariance and reparameterized sampling. Because noise injection during training is mathematically equivalent to penalizing the Laplacian of the segmentation loss (Proposition 1), optimization favors flat regions of the loss landscape, which are claimed to be closer to the center of the target prompt region. The authors demonstrate that ProSAM's mean prompt is less sensitive to injected Gaussian noise, is more cosine-similar to center-point prompt embeddings, and achieves consistently higher mIoU than the deterministic baseline on Pascal-5^i and COCO-20^i across point, scribble, box, and mask references.","pith_inferences":["Editorial extension: the same noise-injection-as-Laplacian-penalty argument should transfer to other prompt types (points, boxes, text) and to other frozen prompt-based segmenters, because Proposition 1 only assumes a smooth loss over embeddings.","Editorial extension: the paper reports cosine similarity to center-point prompts as a proxy, but a direct measure of flatness — the trace of the Hessian of the mask loss at learned prompts — would test the claimed mechanism more directly.","Editorial extension: because a Student-t's advantage appears first in the fourth-order term, heavier-tailed noise or a noise schedule with larger variance early in training could push mean prompts even deeper into flat regions; the paper does not explore this."],"forward_implications":["Mean-prompt inference costs the same as deterministic prompt inference, so the robustness gain comes without extra computation or memory at test time.","The mIoU gains hold across point, scribble, box, and mask reference annotations on both Pascal-5^i and COCO-20^i, so the mechanism is not specific to one prompt format.","Under COCO-to-PASCAL domain shift, the method reaches 77.65 mean mIoU against 76.44 for the deterministic baseline, supporting stronger zero-shot generalization to unseen classes.","Swapping ResNet-50 for DINOv2 improves both methods while ProSAM keeps its advantage, indicating the robustness mechanism is compatible with stronger feature extractors.","With only 1.73M learnable parameters, ProSAM outperforms few-shot segmentation methods that train tens of millions of parameters on the same benchmarks."],"supporting_citations":[{"why":"Supplies the deterministic training-based prompt-encoder architecture that ProSAM modifies with mean and variance heads, and is the baseline it must beat.","marker":"[32]"},{"why":"Provides the frozen SAM image encoder, prompt encoder, and mask decoder that define the loss landscape the variational prompts are optimized against.","marker":"[14]"},{"why":"Introduces the reparameterization trick that makes sampling from the prompt distribution differentiable for backpropagation.","marker":"[13]"},{"why":"Gives the Student-t reparameterized sampling formulation used to draw prompt embeddings during training.","marker":"[1]"},{"why":"Motivates the heavy-tailed Student-t choice for the prompt distribution because it increases the chance of sampling outlier prompts near boundaries.","marker":"[12]"},{"why":"Frames the variational-inference objective that ProSAM's expected-loss training instantiates.","marker":"[3]"},{"why":"Documents why explicit second-order curvature penalties are impractical in automatic-differentiation frameworks, motivating the implicit Laplacian penalty.","marker":"[2]"},{"why":"Supports the claim that heavier-tailed distributions produce more outlier samples, which is what creates the push away from boundaries.","marker":"[5]"}],"fun_headline_variants":["Probabilistic prompts keep SAM away from unstable boundaries","ProSAM models prompt distributions to boost segmentation robustness","Noise-injected prompt training flattens loss and improves SAM","Prompt distribution learning beats boundary prompts in SAM","ProSAM's variational encoder dodges fragile prompt regions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that SAM's mask-generation loss is smooth enough, and the training noise small enough, for the second-order expansion in Proposition 1 to describe what training actually does, and that flat regions of the loss landscape sit at the center of the usable prompt region; if either condition is false, noise injection would simply add training noise without delivering better prompts.","fun_headline_variants_meta":{"raw":{"variants":["Probabilistic prompts keep SAM away from unstable boundaries","ProSAM models prompt distributions to boost segmentation robustness","Noise-injected prompt training flattens loss and improves SAM","Prompt distribution learning beats boundary prompts in SAM","ProSAM's variational encoder dodges fragile prompt regions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1252,"prompt_tokens":924,"completion_tokens":328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":252}},"tokens_in":540,"tokens_out":328,"duration_ms":3647,"temperature":1.0,"reasoning_tokens":252,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:18:10.452040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: compute the average trace of the Hessian of SAM's mask loss at the prompt embeddings produced by ProSAM and by the deterministic baseline on the same images; if ProSAM's prompts do not show lower curvature, the flatness mechanism is not what drives the reported gains. An even simpler check is to inject the same Gaussian noise into the deterministic baseline's prompts during training and see whether its mIoU rises by ProSAM's margin; if it does, no distributional prompt encoder is needed.","supporting_citations":[{"cited_title":"VRP-SAM: Sam with visual reference prompt","cited_arxiv_id":null,"evidence_quote":"Supplies the deterministic training-based prompt-encoder architecture that ProSAM modifies with mean and variance heads, and is the baseline it must beat."},{"cited_title":"Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick","cited_arxiv_id":null,"evidence_quote":"Provides the frozen SAM image encoder, prompt encoder, and mask decoder that define the loss landscape the variational prompts are optimized against."},{"cited_title":"t3-variational autoencoder: Learning heavy-tailed data with student’s t and power divergence","cited_arxiv_id":null,"evidence_quote":"Motivates the heavy-tailed Student-t choice for the prompt distribution because it increases the chance of sampling outlier prompts near boundaries."},{"cited_title":"Vari- ational inference: A review for statisticians","cited_arxiv_id":null,"evidence_quote":"Frames the variational-inference objective that ProSAM's expected-loss training instantiates."},{"cited_title":"Automatic dif- ferentiation in machine learning: a survey","cited_arxiv_id":null,"evidence_quote":"Documents why explicit second-order curvature penalties are impractical in automatic-differentiation frameworks, motivating the implicit Laplacian penalty."},{"cited_title":"On the conditional distribution of the multivari- ate t distribution","cited_arxiv_id":null,"evidence_quote":"Supports the claim that heavier-tailed distributions produce more outlier samples, which is what creates the push away from boundaries."}],"review_version":1}