{"id":"30272f8d-86a8-4369-a049-5831f70872d4","arxiv_id":"2505.05049","paper_version":4,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"USAM trains small MLPs on SAM's mask and IoU tokens to estimate predictive, prompt, task, and model uncertainty, achieving strong selective-correction results at negligible computational overhead.","lead":"This paper introduces USAM, a lightweight post-hoc method that quantifies uncertainty in the Segment Anything Model (SAM) by estimating expected accuracy gaps for different prompts, model sizes, and task selections. It matters because it offers a fast, practical way to make SAM-based segmentation more reliable and cost-efficient across many applications.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-dataset appendix tables partly address the reader's in-distribution worry, but the absence of a true distribution-shift test leaves the practical UQ claim conditional.","rationale":"The reader's weakest-assumption analysis correctly identifies the in-distribution evaluation as the main limitation, but it understates the appendix: Tables 11-13 include cross-dataset evaluations, and those results are close to the per-dataset-trained results, which is genuine evidence of transfer across the five considered benchmarks. In good faith, I therefore do not treat the reader's exact concern as fully decisive. The remaining soft spot is the jump from these five natural-image datasets, which are all close to SAM's pretraining distribution, to the broad applications advertised in the abstract and introduction. The paper is otherwise technically sound: the MLP design is simple, the Bayesian entropy baselines are clearly defined, the runtime comparison is favorable, the code is promised, and the cross-dataset appendix adds support. The DAVIS caveat in Tables 1-2 is acknowledged in Section 4.2 and weakens only the wording of the abstract, not the core method. The appropriate verdict remains conditional: the method is likely useful and well executed, but the headline claim should be tied to demonstrated distributions unless a genuine out-of-distribution test is added.","tokens_in":27653,"tokens_out":14709,"duration_ms":163654,"concrete_test":"Train one USAM model (all MLPs) on SA-V only and freeze it; run the model, prompt, and task supervision protocols from Tables 1-3 on a held-out medical segmentation benchmark (e.g., CVC-ClinicDB or BUSI) using the paper's point-prompt protocol, and compare relative AUC with (i) a USAM trained on that medical set and (ii) the SamScore, HStd, HTheta, and HXP baselines. If the SA-V-trained model's relative AUC drops more than about 5 points below its SA-V result or falls below the baselines, the abstract's practical-UQ claim should be scoped to per-dataset training on natural-image distributions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that USAM is a practical, state-of-the-art post-hoc UQ method for SAM. The main-text evaluation is in-distribution: Tables 1-4 train on each dataset's training split and evaluate on that dataset's validation split, with MOSE evaluated by SA-V-trained models on MOSE train data. The reader's concern is real but partially mitigated by the appendix: Tables 11-13 report classifiers trained on DAVIS, ADE20k, COCO, or SA-V and evaluated on all five datasets; the relative-AUC differences between training datasets are small (e.g., Table 13, COCO task uncertainty: 70.155 for COCO-trained vs 70.009 for SA-V-trained), suggesting the MLPs are not memorizing dataset-specific cues. However, all five datasets are natural-image segmentation benchmarks sharing SAM's training distribution; this is cross-dataset transfer, not out-of-distribution generalization. The abstract and introduction motivate medical, satellite, and autonomous-driving applications, yet no experiment tests a genuinely different image distribution. Because USAM must be trained with ground-truth masks to define its IoU targets, a user in a new domain either needs to annotate data or trust an unvalidated transfer. The paper's own DAVIS results (Tables 1-2) also show USAM below HStd and HTheta for model and prompt uncertainty, so the 'superior on all five datasets' phrasing in the abstract is stronger than the data. Thus the strongest unguarded part of the claim is the leap from five natural-image datasets to general practical UQ capability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces UncertainSAM (USAM), a post-hoc uncertainty quantification method for the Segment Anything Model (SAM). It first proposes a Bayesian entropy decomposition that separates predictive uncertainty into aleatoric task uncertainty, aleatoric prompt uncertainty, and epistemic model uncertainty, implemented via Monte Carlo sampling over image augmentations, sampled prompts, SAM's mask proposals, and four pretrained model sizes. It then trains lightweight MLPs on SAM's mask and IoU tokens to directly predict expected IoU and gap quantities (prompt gap, task gap, model gap), avoiding the computational cost of MC sampling. Experiments on SA-V, MOSE, ADE20k, DAVIS, and COCO measure ranking quality through correction-AUC for model selection, prompt refinement, task supervision, and overall segmentation-error correction, comparing USAM against SamScore, mask entropy, and the Bayesian entropy baselines. The paper claims that USAM is faster, lighter, and generally more accurate than existing UQ methods for SAM.","tokens_in":28004,"tokens_out":5802,"duration_ms":65971,"significance":"If the central claim holds, the paper would provide a practical, low-cost uncertainty estimator that can be attached to SAM without retraining the foundation model, with a conceptually useful decomposition into prompt, task, and model uncertainty. The strengths of the paper include a publicly available code repository, a broad set of experiments across five datasets, token ablations, runtime measurements, and cross-dataset tables in the appendix that show some robustness of the trained MLPs. The significance is tempered, however, by the fact that the main comparison is between a supervised method (trained on ground-truth IoU targets) and training-free baselines, and by the absence of a genuine out-of-distribution test. The theoretical framing as a Bayesian formulation rests on ad-hoc uniform/proxy distributions rather than a derived posterior. With these caveats, the contribution is a useful empirical study and a lightweight estimator, but the claimed state-of-the-art superiority is not yet fully established.","major_comments":[{"comment":"The main evaluation compares USAM, whose MLPs are trained on ground-truth IoU and gap targets computed on the training split of each evaluated dataset, against SamScore, mask entropy, and the Bayesian entropies, all of which are used without any training or ground-truth supervision on the test distribution. The reported 'superior' relative-AUC values therefore do not separate the quality of the uncertainty representation from the advantage of supervised in-distribution fitting. This is a load-bearing issue for the central claim that USAM 'outperforms existing methods.' The appendix cross-dataset tables (Tables 11–13) partially address memorization concerns, but all five datasets are natural-image segmentation benchmarks close to SAM's training distribution, and the abstract motivates medical, satellite, and autonomous-driving applications that are never tested. I ask the authors to (a) move or summarize the cross-dataset experiment in the main text, (b) add at least one genuinely different image distribution, and (c) explicitly state that the main-text comparisons are supervised in-distribution evaluations against training-free baselines.","section":"§4.1, Tables 1–4"},{"comment":"The Bayesian derivation is not a posterior approximation over network weights: p(θ|D) is set to a uniform distribution over four pretrained checkpoints of different architecture sizes, p(xP|t(xI),a) is a uniform distribution over eight sampled coordinate prompts, and p(â|...) is set proportional to the SamScore. These choices are acknowledged in the appendix, but they are heuristic sampling distributions, not a variational or Monte Carlo approximation of a posterior. In addition, Eq. (2) contains a likelihood term p(y|xI,â,θ), but the computation in Eq. (4) replaces this with deterministic SAM masks and computes the entropy of their weighted average, without defining a likelihood. I therefore find the 'theoretically motivated Bayesian entropy formulation' claim in the abstract stronger than what is actually established. The authors should either provide a concrete link between these sampling distributions and Eq. (1), or rename the approach as a heuristic ensemble-entropy estimator and temper the theoretical claims accordingly.","section":"§3.1 and §A, Eqs. (8)–(10)"},{"comment":"The task-uncertainty gap is trained to predict the difference between the IoU of the ground-truth-best mask and the IoU of the SamScore-selected mask, and the evaluation protocol ranks samples by exactly this gap and then substitutes the ground-truth-best mask. Since the MLPs are trained and evaluated on the same data distribution, the near-94% relative AUC in Table 3 may reflect successful regression of the supervised target rather than discovery of a distinct 'task uncertainty' signal. The comparison with the unsupervised Bayesian HA is therefore not apples-to-apples. Please add an analysis that separates these explanations: for example, train on one dataset and evaluate on a different domain, compare against a supervised baseline that regresses the same target from the same tokens under an identical protocol but without the USAM decomposition, or report calibration/reliability curves for the predicted gaps. The claim in §4.4 that 'USAM is superior' should be conditional on such an analysis.","section":"§4.4, Table 3"}],"minor_comments":[{"comment":"The abstract states 'superior predictive capabilities on the SA-V, MOSE, ADE20k, DAVIS, and COCO datasets,' but Table 1 shows HΘ outperforming both USAM variants on DAVIS (73.46 vs. 66.85/59.08) and matched results on MOSE, and Table 2 shows HXP outperforming USAM on DAVIS (80.75 vs. 75.04/75.53). Please qualify the claim to avoid overstating the results.","section":"Abstract and §4.2"},{"comment":"There is a typo in the caption: 'The third and foruth' should read 'The third and fourth.'","section":"Figure 5 caption"},{"comment":"Table 8 lists 70 hyperparameter configurations in the main text; this level of detail is better placed in supplementary material or summarized as a range, since it distracts from the method description.","section":"Table 8"},{"comment":"The index notation for the mask y is inconsistent between equations: Eq. (4) uses the order (t, xP, θ, â), while Eqs. (5)–(7) use different orderings or omit indices. Please define a single consistent notation, including the explicit dependence of â on xI, xP, and θ.","section":"Eqs. (4)–(7)"},{"comment":"The text says 'the training is performed only using the training data of the respective dataset without access to validation data,' but for MOSE the models are trained on SA-V and evaluated on MOSE train data. This is a deviation from the stated protocol and should be described as a cross-dataset evaluation, not as standard training/validation on 'the respective dataset.'","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the UQ-for-foundation-models community, and the code release is a positive element. My main concern is not novelty but the gap between the claims and what the evaluation actually shows: supervised in-distribution training against training-free baselines, and a 'Bayesian' framing resting on ad-hoc distributions. I believe the authors can address these points by reframing the claims, adding a true cross-domain experiment, and making the comparison fair. If those changes are made, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What is actually new here: the paper splits SAM's predictive uncertainty into epistemic model, aleatoric prompt, and aleatoric task components, and trains lightweight MLPs on SAM's tokens to estimate each gap directly. That decomposition is sensible and the gap-MLP training objectives are a clean, practical idea. The experiments are extensive: five datasets, four SAM sizes, ablations, runtime measurements, and a correlation analysis. The task-uncertainty result in Table 3 is genuinely striking: USAM reaches about 94% relative AUC on all datasets, far above the Bayesian task entropy (43–78%). That is the strongest evidence in the paper, and it is credible because the MLP is directly trained to predict the gap between supervised and SamScore-selected masks, so high in-distribution performance is expected rather than mysterious.\n\nThe soft spots are real but not fatal. The main-text evaluation is in-distribution: MLPs are trained and evaluated on the same datasets. The appendix partially answers this with cross-dataset transfer tables (11–13), where training on one dataset and testing on another gives similar AUCs, so the MLPs are not simply memorizing dataset-specific cues. But all five datasets are natural-image segmentation benchmarks that share SAM's training distribution; there is no test on a genuinely different distribution like medical or satellite images, despite the introduction motivating those domains. That limits the practical claim. The abstract says 'superior predictive capabilities' on all five datasets, but on DAVIS the Bayesian HTheta and HXP beat USAM for model and prompt uncertainty (Tables 1–2), so the phrasing is too strong. The Bayesian baselines also get an unfair advantage in some settings: HTheta and HXP require the known ground-truth task to select the best mask proposal, which is not a realistic post-hoc setup. The theoretical framing is occasionally loose—the 'Bayesian' entropy approximation uses uniform prompt/model probabilities and SamScore as the task probability, assumptions the appendix acknowledges—but for an applied paper that is acceptable if the claims stay within the evidence.\n\nBottom line: this is a solid, useful paper for anyone building on SAM who needs cheap uncertainty estimates. It deserves a serious referee. I would recommend conditional acceptance with two requests: add an out-of-distribution experiment or explicitly limit the claims to the tested setting, and soften the 'new state-of-the-art' wording to match the DAVIS results.","headline":"A useful, well-engineered post-hoc UQ method for SAM with a sensible uncertainty decomposition, but the evaluation is in-distribution and the 'state-of-the-art' claim is stronger than the data support.","tokens_in":28462,"tokens_out":1245,"would_cite":true,"duration_ms":14782,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a deterministic post-hoc MLP can separate SAM's uncertainty into model, prompt, and task components, and that this estimator beats SAM's own confidence score on five datasets.","keywords":["uncertainty quantification","Segment Anything Model","Bayesian entropy","aleatoric uncertainty","epistemic uncertainty","prompt uncertainty","task uncertainty","post-hoc MLP estimator"],"falsifier":"Train USAM's gap MLPs on SA-V only and evaluate them on COCO, ADE20k, DAVIS, and MOSE; if the relative AUC for task uncertainty drops to the level of the Bayesian entropy baseline, the reported superiority is in-distribution memorization rather than a general latent signal.","tokens_in":27451,"feed_emoji":"🎯","tokens_out":5785,"duration_ms":51342,"temperature":0.7,"pith_summary":"The paper argues that uncertainty in the Segment Anything Model (SAM) comes in three distinguishable flavors --- uncertainty caused by the model's size (epistemic), by an insufficient prompt (aleatoric prompt uncertainty), and by genuinely ambiguous tasks (aleatoric task uncertainty) --- and that all three can be read off SAM's own latent tokens by tiny trained MLPs, with no sampling and no retraining of SAM. On five datasets (SA-V, MOSE, ADE20k, COCO, DAVIS), these MLPs, called USAM, rank the samples that most need correction at least as well as a Monte Carlo Bayesian entropy approximation and clearly better than SAM's built-in SamScore confidence and standard mask entropy. The strongest result is for task uncertainty: a gap MLP reaches roughly 94% relative AUC on every dataset, while the Bayesian task-entropy baseline lands between 43% and 78%. If the claim holds, reliable uncertainty estimates for SAM come almost for free, which matters for medical, driving, and interactive segmentation applications where a wrong mask is costly.","feed_headline":"Tiny post-hoc nets beat Bayesian sampling for SAM uncertainty","feed_subtitle":"On five datasets, the paper's MLPs rank which masks need help better than SAM's confidence or MC entropy.","key_machinery":"The load-bearing machinery is a Bayesian entropy decomposition of SAM's predictive uncertainty (their Equation 2), which extends the standard aleatoric/epistemic split by explicitly factoring in the prompt and the task-definition step, plus USAM's three-layer MLP gap predictors. The decomposition writes predictive uncertainty as an integral over task hypotheses, prompt distributions, and model weights; the MLPs approximate the same quantities by regressing on differences of expected IoU --- $\\Delta_{X_P}$ for prompt quality, $\\Delta_A$ for task ambiguity, $\\Delta_\\Theta$ for model scale --- directly from SAM's mask and IoU tokens. This design lets the method run deterministically at inference time, without image or prompt augmentation, without ensembling, and without any change to SAM's weights.","core_discovery":"The paper's central claim is that USAM, a set of deterministic post-hoc MLPs that ingest SAM's 256-dimensional mask and IoU tokens, accurately quantifies where SAM's segmentation uncertainty comes from. Starting from a Bayesian entropy decomposition of predictive uncertainty into epistemic, prompt, and task components, the authors train separate MLPs to predict the expected IoU for a given SAM variant and the expected IoU gaps between a single-point and a refined prompt, between supervised and SamScore mask selection, and between the Tiny and Large backbones. They report that these gap estimators identify the samples that most need a better model, a better prompt, or human supervision across SA-V, ADE20k, COCO, DAVIS, and MOSE, and that the direct \"delta-star\" versions match or surpass a full Bayesian Monte Carlo entropy approximation while adding only a few milliseconds of runtime.","pith_inferences":["Editorial extension: the published experiments train and evaluate USAM on the same dataset distributions, so the method's cross-dataset generalization is untested; the natural next check is to train on SA-V alone and report AUC on COCO, ADE20k, and DAVIS.","Editorial extension: because the gap MLPs depend only on SAM's mask and IoU tokens, the same recipe should transfer to SAM 2 and to other promptable segmenters if their decoders expose analogous tokens, but the paper does not demonstrate this transfer.","Editorial extension: USAM's predicted task uncertainty could be used as a training signal itself, for example to down-weight ambiguous masks during fine-tuning, but the paper does not explore this use."],"forward_implications":["In applications that use SAM Tiny for speed, USAM's $\\Delta_\\Theta$ can flag the images where switching to SAM Large buys the most accuracy, so the large model is only run when it matters.","In interactive segmentation, users can be prompted to add points exactly where $\\Delta_{X_P}$ is high, improving masks without full manual review.","USAM's $\\Delta_A$ can route ambiguous proposals to human supervision: the experiment shows that supervising the most uncertain samples recovers nearly oracle-level mIoU.","Since USAM only reads tokens SAM already computes, it can be attached to existing SAM pipelines at negligible latency (about 0.004 seconds per image in the paper's runtime table)."],"supporting_citations":[{"why":"Supplies the SAM architecture, its mask and IoU tokens, the SamScore confidence, and the three mask proposals that USAM reads.","marker":"Kirillov et al., 2023"},{"why":"Supplies the four pretrained SAM variants (Tiny, Small, Base+, Large) and the SA-V dataset used for training and evaluation.","marker":"Ravi et al., 2024"},{"why":"Supplies the test-time augmentation template for approximating aleatoric uncertainty that the Bayesian entropy baseline extends.","marker":"Wang et al., 2019"},{"why":"Supplies the model-ensemble Monte Carlo idea used to approximate epistemic uncertainty in the Bayesian baseline.","marker":"Lakshminarayanan et al., 2017"},{"why":"Provides an existing SAM-specific prompt-augmentation uncertainty method that the paper compares against and extends.","marker":"Deng et al., 2023"},{"why":"Provides another SAM uncertainty baseline that motivates the need for prompt-aware UQ in SAM.","marker":"Zhang et al., 2023b"},{"why":"Supplies the conceptual template of training a deterministic network to predict epistemic uncertainty directly, which USAM adapts.","marker":"Lahlou et al., 2023"},{"why":"Defines the predictive, aleatoric, and epistemic uncertainty taxonomy that the paper's Bayesian formulation builds on.","marker":"Gawlikowski et al., 2023"}],"fun_headline_variants":["USAM: Small nets match Bayesian UQ for SAM at low cost","Deterministic post-hoc UQ beats Bayesian sampling for SAM masks","Fast SAM uncertainty via MLPs: USAM outperforms MC entropy","Lightweight USAM identifies where SAM needs help in five datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"USAM's MLPs are trained with ground-truth masks on the same datasets they are evaluated on, so the headline results measure in-distribution ranking ability rather than transfer to a new distribution.","fun_headline_variants_meta":{"raw":{"variants":["USAM: Small nets match Bayesian UQ for SAM at low cost","Deterministic post-hoc UQ beats Bayesian sampling for SAM masks","Fast SAM uncertainty via MLPs: USAM outperforms MC entropy","Lightweight USAM identifies where SAM needs help in five datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1408,"prompt_tokens":887,"completion_tokens":521,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":446}},"tokens_in":503,"tokens_out":521,"duration_ms":5016,"temperature":1.0,"reasoning_tokens":446,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:13:13.879616+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train USAM's gap MLPs on SA-V only and evaluate them on COCO, ADE20k, DAVIS, and MOSE; if the relative AUC for task uncertainty drops to the level of the Bayesian entropy baseline, the reported superiority is in-distribution memorization rather than a general latent signal.","supporting_citations":[{"cited_title":"C., Lo, W.-Y., et al","cited_arxiv_id":null,"evidence_quote":"Supplies the SAM architecture, its mask and IoU tokens, the SamScore confidence, and the three mask proposals that USAM reads."},{"cited_title":"V., Carion, N., Wu, C.-Y., Girshick, R., Doll \\'a r, P., and Feichtenhofer, C","cited_arxiv_id":null,"evidence_quote":"Supplies the four pretrained SAM variants (Tiny, Small, Base+, Large) and the SA-V dataset used for training and evaluation."},{"cited_title":"Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the test-time augmentation template for approximating aleatoric uncertainty that the Bayesian entropy baseline extends."},{"cited_title":"Simple and scalable predictive uncertainty estimation using deep ensembles","cited_arxiv_id":null,"evidence_quote":"Supplies the model-ensemble Monte Carlo idea used to approximate epistemic uncertainty in the Bayesian baseline."},{"cited_title":"SAM - U : Multi -box Prompts Triggered Uncertainty Estimation for Reliable SAM in Medical Image","cited_arxiv_id":null,"evidence_quote":"Provides an existing SAM-specific prompt-augmentation uncertainty method that the paper compares against and extends."},{"cited_title":"I., Bertin, P., Rector-Brooks, J., Korablyov, M., and Bengio, Y","cited_arxiv_id":null,"evidence_quote":"Supplies the conceptual template of training a deterministic network to predict epistemic uncertainty directly, which USAM adapts."},{"cited_title":"A survey of uncertainty in deep neural networks","cited_arxiv_id":null,"evidence_quote":"Defines the predictive, aleatoric, and epistemic uncertainty taxonomy that the paper's Bayesian formulation builds on."}],"review_version":1}