Pith. sign in

REVIEW 3 major objections 6 minor 68 references

Phoenix claims that mask refinement is best learned from adversarial noise generated in a frozen SAM decoder's embedding space, paired with a tri-directional contrastive loss, yielding large gains across segmentation tasks and model familie

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 14:23 UTC pith:LF6B4RQB

load-bearing objection Solid refinement method with a real train/eval overlap problem on its headline benchmark; the rest of the evidence holds up. the 3 major comments →

arxiv 2607.29059 v1 pith:LF6B4RQB submitted 2026-07-31 cs.CV

Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation

Beomyoung Kim , Sung Ju Hwang This is my paper
classification cs.CV
keywords mask refinementadversarial mask perturbationcontrastive learningimage segmentationSAM decoderpseudo-label refinementfine-grained segmentationzero-shot transfer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Phoenix makes the case that mask refinement should be learned from adversarial noise rather than geometric distortions. The paper proposes two linked mechanisms: Adversarial Mask Perturbation (AMP), which produces synthetic noisy masks by attacking the frozen SAM decoder's embedding space, and Contrastive Mask Refinement Learning (CMRL), which organizes pixels into true, success, and failure regions and trains the refiner to pull failures toward correct features while separating foreground from background. On the paper's experiments, Phoenix beats earlier refiners across semi-supervised pseudo-labels, weakly supervised pseudo-labels, and fine-grained segmentation, with reported gains as large as +16.1 APmask and +21.0 IoU, and it also improves several modern instance segmentation models. The broader claim is that realistic segmentation-error patterns can be manufactured cheaply and that a refiner trained on them transfers to models and domains it never saw.

Core claim

At the center of the paper is the claim that the type of noise used to train a mask refiner matters more than the refiner architecture. Existing refiners simulate errors with morphological operations—dilation, erosion, random boundary shifts—which the paper shows are semantically uncorrelated with image content. Phoenix instead generates perturbation embeddings, feeds them into a frozen SAM decoder, and updates them with FGSM-style sign gradients against a guidance mask, so the resulting noisy masks concentrate errors where the decoder is least certain. A second contribution, CMRL, treats refinement as a feature-space learning problem: pixels are partitioned into true, success, and failure s

What carries the argument

Adversarial Mask Perturbation (AMP): learnable perturbation embeddings are concatenated with visual prompt embeddings in the frozen SAM decoder and updated by FGSM-style sign gradients of a guidance-mask loss. Expansion, contraction, and inversion guidance masks control the semantic direction of noise, while an IoU-threshold controller adjusts perturbation strength; the paper reports each update costs about 6 ms because only the lightweight decoder runs. Contrastive Mask Refinement Learning (CMRL): the six-region partition (true, success, failure in foreground and background) feeds three InfoNCE-style losses—intra-class consistency, inter-class separation, and self-improvement regularization

Load-bearing premise

The load-bearing premise is that FGSM-style perturbations of the frozen SAM decoder's embedding generate noise whose error statistics substitute for the error patterns of a wide range of real segmentation models; the paper supports this functionally through better downstream refinement and zero-shot transfer rather than by direct distributional measurement.

What would settle it

Measure, on the same images, the spatial and semantic error statistics of AMP noise against those of real models (e.g., error correlation with boundaries/texture, error-region size distribution, class-confusion structure). If AMP noise is no closer to real model errors than morphological noise, or if a refiner trained on real model masks beats Phoenix when both are evaluated on real model outputs at scale, the paper's realism claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Refiners trained on adversarial noise can be swapped into existing pipelines without retraining the base segmentation model, raising mask quality on models as different as Mask R-CNN, SOLO, Mask2Former, ViTDet, and MaskDINO.
  • In label-efficient settings, refinement of pseudo-labels from 1%-supervised models is substantially more effective, with reported gains of up to +16.1 APmask and +17.3 boundary AP.
  • Fine-grained masks with thin structures and intricate boundaries can be recovered much more accurately, with average IoU gains of 11–21 points over prior refiners on the DIS benchmark.
  • A refiner trained on LVIS instance data transfers zero-shot to urban scenes and medical images, suggesting the learned refinement principle is not dataset-specific.
  • Self-supervised refinement is possible when SAM-generated pseudo-targets replace ground-truth masks, retaining most of the supervised performance on instance segmentation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the AMP recipe is not tied to SAM; the same 'attack the frozen decoder to synthesize training noise' principle could be applied to other dense predictors (matting, depth estimation, boundary detection), though the noise distribution would inherit that model's uncertainty patterns.
  • Editorial extension: because the paper ties 'semantic' to model uncertainty rather than human categories, Phoenix's noise is only as realistic as the student decoder's error geography; if the target model fails in ways the decoder does not, transfer could degrade—a direct test is to train on a different frozen decoder and refine masks from many model families.
  • Editorial extension: the reported result that AMP beats training on real model errors suggests that diversity of error patterns, not fidelity to one model, drives refinement; a natural extension is adaptive noise that conditions on the target model's estimated failure modes at inference time.
  • Editorial extension: the failure cases the paper identifies (merged objects, misclassified regions) point to combining Phoenix with open-vocabulary or text-conditioned models, which could convert it from a boundary fixer into a semantic corrector.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Phoenix, a mask refinement model built on a frozen SAM encoder and a fine-tuned lightweight decoder. Two main components are introduced: Adversarial Mask Perturbation (AMP), which generates training noise by applying FGSM-style updates to learnable perturbation embeddings in the decoder's embedding space under a guidance mask, and Contrastive Mask Refinement Learning (CMRL), a tri-directional contrastive loss over true, noisy, and refined masks. The method is evaluated on pseudo-labels from semi-/weakly-supervised instance segmentation, on outputs of several instance segmentation models, on fine-grained DIS segmentation, and in zero-shot settings. The paper reports large gains, including +16.1 APmask on 1%-supervision PointWSSIS pseudo-labels, and consistent improvements over SegRefiner and SAMRefiner across settings.

Significance. If the empirical claims hold, this is a valuable contribution: it repurposes adversarial perturbation as a constructive noise-generation mechanism for training data, introduces a contrastive objective tailored to mask refinement, and provides extensive validation including cross-architecture transfer, 5-seed stability, and zero-shot evaluation. The paper is also commendably transparent about which components are principled vs. empirical design choices, and it includes implementation-level details and stability analysis. However, the headline result on COCO train5K is threatened by a potential train/evaluation image overlap, so the significance is currently conditional. The COCO-validation, DIS, and zero-shot results are substantially less exposed and would support a weaker but still meaningful claim.

major comments (3)
  1. [§3.3, Eq. (1)] I have a load-bearing concern about train/evaluation overlap. Phoenix is trained on LVIS, which is constructed from COCO train2017 images, and evaluated on 'COCO train5K' using LVIS annotations; COCO train5K is also drawn from COCO train2017. The paper never states that images in COCO train5K were excluded from the LVIS training set. If they are included, the refiner can memorize the target masks for those images, and the Table 1 improvements over the training-free SAMRefiner baseline would be inflated by memorization. This affects not only the headline +16.1 APmask number but also the AP1 column of the ablation tables that support the component claims. Please state whether the two sets overlap, and if they do, rerun the train5K evaluation on a disjoint subset or report held-out results (e.g., COCO val with LVIS annotations). This is necessary before the paper's central generalization cl
  2. [§3.3, Eq. (1)] Equation (1) asserts ||∇_{E_p} f_dec(...)||_2 ∝ -log p(y|...) as a 'principled' gradient–uncertainty relationship. For a softmax classifier with cross-entropy loss, the gradient norm with respect to logits is sqrt((1-p_y)^2 + sum_{i≠y} p_i^2), which is not proportional to -log p_y; the relationship to the gradient with respect to the perturbation embeddings E_p is even less direct. Since this equation is used to justify the 'semantic-aware' property of AMP, the authors should either supply a correct derivation or explicitly demote Eq. (1) to an empirical observation supported by measurements in their setting. The later disclaimer in §3.3 and Appendix B.5 helps, but the equation as stated is technically misleading.
  3. [§4.4, Table 4e and Appendix B.5] The claim that AMP noise 'mimics real segmentation errors' is supported only indirectly via functional substitutability (Table 4d/4e) and semantic-correlation analysis (Appendix B.5). The Pearson correlation with edge/texture maps measures alignment with low-level image structures, not similarity to the actual error statistics of PointWSSIS, Mask R-CNN, or other models whose masks Phoenix is asked to refine. I am not requesting a new theory, but the narrative would be more accurate if the paper either added a direct distributional comparison (e.g., error maps from real models vs. AMP on held-out images) or softened the realism claim to 'empirically effective synthetic noise.'
minor comments (6)
  1. [Table 1] The row label 'SegRefine' is missing the final 'r'.
  2. [Table 9(d)] The column header 'IoU2' appears twice; one should be 'IoU1'.
  3. [Algorithm 1] The inner-loop break condition 'if iou < τ+ε then break' can exit immediately when the initial IoU is already below τ+ε. Please clarify the intended behavior in the low-IoU regime.
  4. [§3.4 and §3.3] The symbol P is used both for the number of perturbation embeddings and for the projection feature map p=g(F). This creates ambiguity; consider renaming one of them.
  5. [Appendix A.3] The 'split-then-merge strategy' is cited to SAMRefiner but not described. Please give a one-sentence explanation or a precise reference.
  6. [§4.4] The statement that AMP 'even surpasses training directly on real UNet/ISNet errors' is strong because Table 4e compares against only two specific models. I suggest rewording to 'surpasses training on these two real-noise sources'.

Circularity Check

0 steps flagged

No circularity found: Phoenix's AMP/CMRL components are trained on LVIS targets and tested on external pseudo-labels and real model masks; realism claims are explicitly empirical, not derived from the evaluation.

full rationale

Phoenix's derivation chain is not circular. AMP generates training noise by FGSM-style perturbation of perturbation embeddings fed to a frozen SAM decoder, with an IoU threshold controlling magnitude; CMRL is a contrastive loss defined on pixel regions derived from target/noisy/refined masks. Neither component is defined in terms of the evaluation masks or metrics. The central performance claim is tested on masks from external models and benchmarks (PointWSSIS, NB, Mask R-CNN, Mask2Former, U-Net, ISNet, etc.), which are not produced by the AMP process, so no 'prediction' is forced by construction. The realism claim for AMP is explicitly framed as an empirical claim: the paper states 'we make the scope of this analysis explicit, separating principled foundations from empirically-validated design choices' and 'the resulting realism of AMP is established empirically rather than by formal proof.' The supporting evidence (functional substitutability in Tables 4d/4e, monotonic degradation with morphological noise share, zero-shot transfer) is independent of the training objective. Self-citations such as PointWSSIS [24] are used as benchmarks, not as load-bearing justification for the method. The LVIS training / COCO train5K evaluation overlap is a potential experimental confound, but it is not a circular reduction: the refined masks are pseudo-labels, not the LVIS targets used in training, and no equation or fitted parameter is identical to the reported outcome. No circular step can be exhibited, so the score is 0.

Axiom & Free-Parameter Ledger

9 free parameters · 6 axioms · 1 invented entities

The method's success depends on the assumptions that SAM features are good for refinement, that embedding-space adversarial noise approximates real errors, and that the contrastive losses transfer. These are validated empirically through ablations and cross-domain tests, but none is proven from first principles. The hyperparameters are all tuned on the evaluation benchmarks, which is standard for learning-based methods but means the reported gains include some fitting to validation sets.

free parameters (9)
  • IoU threshold τ range = U(0.3, 0.9)
    Controls noise intensity during AMP; chosen via ablation (Table 4a).
  • Guidance mask selection (M_g) = random among expansion/contraction/inversion
    Determines semantic direction of perturbation; ablation in Table 9e.
  • Perturbation step size α0 = 0.01
    Initial FGSM step size; tuned in Fig. 7b.
  • Maximum inner iterations N = 10
    Tuned in Fig. 7b; stability across N≥5.
  • Number of perturbation embeddings P = 50
    Ablation Fig. 7c; performance plateaus beyond 50.
  • Contrastive loss weights λ_intra, λ_inter, λ_self = 0.4, 0.4, 0.2
    Tuned via ablation; each component contributes (Fig. 7d).
  • Overall CMRL scale λ_CMRL = 1e-3
    Ablation Fig. 7e; larger values degrade segmentation.
  • Number of cascade refinement steps = 5
    Inference strategy; empirically chosen balance in Fig. 7a.
  • Projector architecture = 3-layer MLP
    Ablation Table 9c; 3 layers optimal.
axioms (6)
  • domain assumption SAM's pre-trained features provide a suitable representation for mask refinement.
    The entire pipeline is built on a frozen SAM ViT-H encoder/decoder; if these embeddings are unsuitable for boundary correction, the method fails. Validated only empirically.
  • ad hoc to paper Gradient magnitude w.r.t. perturbation embeddings approximately tracks prediction uncertainty (Eq. 1).
    Eq. (1) asserts a proportionality based on Bayesian deep learning [23]; the paper later softens this to a heuristic. It underpins the semantic-awareness claim of AMP.
  • domain assumption Adversarial noise from a frozen SAM decoder is a good surrogate for real segmentation errors of other models.
    Core to AMP; supported by functional substitutability (Tables 4d/4e) and zero-shot transfer, but not by direct error-process comparison.
  • domain assumption Contrastive losses in feature space, as formulated in Eqs. (2)-(4), improve refinement.
    Standard InfoNCE-based assumption; supported by ablations (Table 4c).
  • domain assumption The six-region decomposition with detached masks provides a stable curriculum for refining.
    CMRL relies on this decomposition; no theoretical guarantee, only empirical validation.
  • domain assumption Evaluation datasets (LVIS, COCO, DIS5K, ThinObject5K) and metrics (AP, APboundary, IoU) are representative of mask refinement performance.
    Standard benchmarks in the field; not derived.
invented entities (1)
  • Learnable perturbation embeddings E_p no independent evidence
    purpose: Injected into the frozen decoder to generate adversarial noisy masks during training; not used at inference.
    Algorithmic construct; no external falsifiable prediction.

pith-pipeline@v1.3.0-daily-deepseek · 28045 in / 13409 out tokens · 141565 ms · 2026-08-03T14:23:54.731635+00:00 · methodology

0 comments
read the original abstract

Despite significant advances in image segmentation, even state-of-the-art models produce masks with imperfect boundaries, semantic inconsistencies, and structural errors. Mask refinement addresses these limitations, yet current approaches rely on simplistic synthetic noise that fails to capture the complex error patterns of real segmentation models. We introduce Phoenix, a novel framework that leverages adversarial learning to generate semantically meaningful noise patterns and contrastive learning to model refinement relationships. Our approach consists of two key innovations: (1) Adversarial Mask Perturbation, which employs embedding attacks to create semantic-aware noise that mimics real segmentation errors, and (2) Contrastive Mask Refinement Learning, which establishes a tri-directional framework that ensures feature consistency within semantic regions while maintaining separation between classes. Experiments demonstrate that Phoenix significantly outperforms existing methods across diverse tasks, while consistently enhancing state-of-the-art segmentation models with substantial improvements. Our code and project page are publicly available at https://phoenix-eccv26.github.io.

Figures

Figures reproduced from arXiv: 2607.29059 by Beomyoung Kim, Sung Ju Hwang.

Figure 1
Figure 1. Figure 1: Phoenix refines coarse segmentation masks into high-quality ones. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative Comparison of noise patterns (a) between morphological and our adversarial noise masks. (b, c) Distribution of the Pearson correlation between noise location and image edge/texture maps on LVIS val. (b) Morphological noise is narrowly concentrated near 0 (semantically uncorrelated), while (c) our adversarial noise spans [−0.6, 0.8], where positive values indicate alignment with semantic struc￾t… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of our Phoenix framework. The pipeline consists of three main com￾ponents: (1) Image Embedding Extraction using SAM’s encoder, (2) Adversarial Mask Perturbation that generates realistic noise patterns through adversarial embedding attacks, and (3) Contrastive Mask Refinement Learning that uses the tri-directional relationships between masks to improve refinement quality. nature of errors in real s… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative Samples of generated noisy masks according to the IoU threshold τ and guid￾ance mask (a) expansion guide, (b) contraction guide, and (c) in￾version guide. Dice or MSE loss). Three primary configurations yield distinct noise patterns, as shown in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative results for semantic segmentation refinement [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative results for human annotation refinement [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Ablation study (continue) using instance segmentation AP1 results. declining from 28.7% AP1 when using purely adversarial noise (pmorph = 0.0) to 23.8% AP1 when relying exclusively on morphological noise (pmorph = 1.0). This demonstrates that traditional morphological operations are insufficient for generating realistic noise patterns. These results underscore the importance of our semantic-aware adversari… view at source ↗
Figure 8
Figure 8. Figure 8: Noise pattern analysis demonstrating the superiority of adversarial mask perturbation over morphological methods. Our approach produces diverse, realistic fail￾ure patterns, including contextual errors, segmentation inconsistencies, and boundary imprecision that closely match real segmentation model failures, leading to improved refinement performance [PITH_FULL_IMAGE:figures/full_fig_p036_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Additional qualitative results for instance segmentation refinement. [PITH_FULL_IMAGE:figures/full_fig_p038_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Additional qualitative results for fine-grained segmentation refine [PITH_FULL_IMAGE:figures/full_fig_p039_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Failure case analysis across different segmentation tasks [PITH_FULL_IMAGE:figures/full_fig_p041_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 6 linked inside Pith

  1. [1]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Arnab, A., Miksik, O., Torr, P.H.: On the robustness of semantic segmentation models to adversarial attacks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 888–897 (2018) 4, 6

    2018

  2. [2]

    arXiv preprint arXiv:1607.06450 (2016) 25

    Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016) 25

    Pith/arXiv arXiv 2016

  3. [3]

    In: Proceedings eighth IEEE international conference on computer vision

    Boykov, Y.Y., Jolly, M.P.: Interactive graph cuts for optimal boundary & region segmentation of objects in nd images. In: Proceedings eighth IEEE international conference on computer vision. ICCV 2001. vol. 1, pp. 105–112. IEEE (2001) 4

    2001

  4. [4]

    IEEE Transactions on pattern analysis and machine intelligence (6), 679–698 (1986) 27

    Canny, J.: A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence (6), 679–698 (1986) 27

    1986

  5. [5]

    Advances in neural information processing systems34, 10919–10930 (2021) 15

    Cha, S., Yoo, Y., Moon, T., et al.: Ssul: Semantic segmentation with unknown la- bel for exemplar-based class-incremental learning. Advances in neural information processing systems34, 10919–10930 (2021) 15

    2021

  6. [6]

    In: Proceedings of the European conference on computer vision (ECCV)

    Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV). pp. 801–818 (2018) 3

    2018

  7. [7]

    In: International conference on machine learning

    Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for con- trastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PmLR (2020) 5, 9 16 B. Kim and S. J. Hwang

    2020

  8. [8]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Cheng, B., Girshick, R., Dollár, P., Berg, A.C., Kirillov, A.: Boundary iou: Im- proving object-centric image segmentation evaluation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15334– 15342 (2021) 11

    2021

  9. [9]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299 (2022) 2, 3, 12

    2022

  10. [10]

    arXiv preprint arXiv:1902.03368 (2019) 24

    Codella, N., Rotemberg, V., Tschandl, P., Celebi, M.E., Dusza, S., Gutman, D., Helba,B.,Kalloo,A.,Liopyris,K., Marchetti,M., et al.:Skinlesionanalysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368 (2019) 24

    Pith/arXiv arXiv 2018

  11. [11]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3213–3223 (2016) 24

    2016

  12. [12]

    In: International Conference on Learning Representations (2021) 11, 25, 34

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021) 11, 25, 34

    2021

  13. [13]

    International journal of computer vision88, 303–338 (2010) 20

    Everingham,M.,VanGool,L.,Williams,C.K.,Winn,J.,Zisserman,A.:Thepascal visual object classes (voc) challenge. International journal of computer vision88, 303–338 (2010) 20

    2010

  14. [14]

    In: European conference on computer vision

    Ghiasi,G.,Gu,X.,Cui,Y.,Lin,T.Y.:Scalingopen-vocabularyimagesegmentation with image-level labels. In: European conference on computer vision. pp. 540–557. Springer (2022) 20, 40

    2022

  15. [15]

    In: International Conference on Learning Representations (ICLR) (2015) 4, 6

    Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: International Conference on Learning Representations (ICLR) (2015) 4, 6

    2015

  16. [16]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Gupta, A., Dollar, P., Girshick, R.: Lvis: A dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5356–5364 (2019) 8, 10, 21, 24, 35

    2019

  17. [17]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9729–9738 (2020) 5, 9

    2020

  18. [18]

    In: Proceedings of the IEEE international conference on computer vision

    He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 2961–2969 (2017) 12, 13, 24, 29

    2017

  19. [19]

    He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016) 29

    2016

  20. [20]

    arXiv preprint arXiv:1606.08415 (2016) 25

    Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016) 25

    Pith/arXiv arXiv 2016

  21. [21]

    In: International Conference on Learning Representations (2020) 4, 6

    Huang, Z., Zhang, T.: Black-box adversarial attack with transferable model-based embedding. In: International Conference on Learning Representations (2020) 4, 6

    2020

  22. [22]

    Advances in Neural Information Processing Systems36, 29914–29934 (2023) 12

    Ke, L., Ye, M., Danelljan, M., Tai, Y.W., Tang, C.K., Yu, F., et al.: Segment anything in high quality. Advances in Neural Information Processing Systems36, 29914–29934 (2023) 12

    2023

  23. [23]

    Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems30(2017) 8, 28 Phoenix 17

    2017

  24. [24]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition

    Kim, B., Jeong, J., Han, D., Hwang, S.J.: The devil is in the points: Weakly semi-supervised instance segmentation via point-guided mask representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition. pp. 11360–11370 (2023) 2, 4, 10, 11, 12, 29

    2023

  25. [25]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Kim, B., Shin, C., Jeong, J., Jung, H., Lee, S.Y., Chun, S., Hwang, D.H., Yu, J.: Zim: Zero-shot image matting for anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 23828–23838 (2025) 15

    2025

  26. [26]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Kim, B., Yoo, Y., Rhee, C.E., Kim, J.: Beyond semantic to instance segmentation: Weakly-supervised instance segmentation via semantic knowledge transfer and self- refinement. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4278–4287 (2022) 2

    2022

  27. [27]

    In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition

    Kim, B., Yu, J., Hwang, S.J.: Eclipse: Efficient continual learning in panoptic segmentation with visual prompt tuning. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. pp. 3346–3356 (2024) 15

    2024

  28. [28]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023) 2, 3, 5, 11, 25

    2023

  29. [29]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Kirillov,A.,Wu,Y.,He,K.,Girshick,R.:Pointrend:Imagesegmentationasrender- ing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9799–9808 (2020) 12

    2020

  30. [30]

    Advances in neural information processing systems24(2011) 4

    Krähenbühl, P., Koltun, V.: Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems24(2011) 4

    2011

  31. [31]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Li, F., Zhang, H., Xu, H., Liu, S., Zhang, L., Ni, L.M., Shum, H.Y.: Mask dino: Towards a unified transformer-based framework for object detection and segmenta- tion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3041–3050 (2023) 12

    2023

  32. [32]

    In: International conference on machine learning

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730–19742. PMLR (2023) 20, 40

    2023

  33. [33]

    In: European conference on computer vision

    Li, Y., Mao, H., Girshick, R., He, K.: Exploring plain vision transformer backbones for object detection. In: European conference on computer vision. pp. 280–296. Springer (2022) 12

    2022

  34. [34]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition

    Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., Mar- culescu, D.: Open-vocabulary semantic segmentation with mask-adapted clip. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 7061–7070 (2023) 20, 40

    2023

  35. [35]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Liew, J.H., Cohen, S., Price, B., Mai, L., Feng, J.: Deep interactive thin object selection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 305–314 (2021) 11

    2021

  36. [36]

    Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection.In:ProceedingsoftheIEEEinternationalconferenceoncomputervision. pp. 2980–2988 (2017) 12, 34

    2017

  37. [37]

    In: Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13. pp. 740–755. Springer (2014) 11, 24

    2014

  38. [38]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Lin, Y., Chen, M., Wang, W., Wu, B., Li, K., Lin, B., Liu, H., He, X.: Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15305–15314 (2023) 21 18 B. Kim and S. J. Hwang

    2023

  39. [39]

    In: The Thirteenth International Conference on Learning Representations (2025) 2, 3, 4, 5, 21, 26, 35, 38

    Lin, Y., Li, H., Shao, W., Yang, Z., Zhao, J., He, X., Luo, P., Zhang, K.: Samrefiner: Taming segment anything model for universal mask refinement. In: The Thirteenth International Conference on Learning Representations (2025) 2, 3, 4, 5, 21, 26, 35, 38

    2025

  40. [40]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3431–3440 (2015) 3

    2015

  41. [41]

    arXiv preprint arXiv:1711.05101 (2017) 11, 26

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017) 11, 26

    Pith/arXiv arXiv 2017

  42. [42]

    In: International Conference on Learning Representations (2018) 4, 6

    Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: International Conference on Learning Representations (2018) 4, 6

    2018

  43. [43]

    Proceedings of the Royal Society of London

    Marr, D., Hildreth, E.: Theory of edge detection. Proceedings of the Royal Society of London. Series B. Biological Sciences207(1167), 187–217 (1980) 27

    1980

  44. [44]

    In: 2016 fourth international confer- ence on 3D vision (3DV)

    Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international confer- ence on 3D vision (3DV). pp. 565–571. Ieee (2016) 34

    2016

  45. [45]

    arXiv preprint arXiv:1807.03748 (2018) 9, 29

    Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748 (2018) 9, 29

    Pith/arXiv arXiv 2018

  46. [46]

    arXiv preprint arXiv:1912.01703 (2019) 26

    Paszke, A.: Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 (2019) 26

    Pith/arXiv arXiv 1912

  47. [47]

    note on regression and inheritance in the case of two parents

    Pearson, K.: Vii. note on regression and inheritance in the case of two parents. proceedings of the royal society of London58(347-352), 240–242 (1895) 8, 27

  48. [48]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine- Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 724–732 (2016) 11

    2016

  49. [49]

    In: European Conference on Computer Vision

    Qin, X., Dai, H., Hu, X., Fan, D.P., Shao, L., Van Gool, L.: Highly accurate di- chotomous image segmentation. In: European Conference on Computer Vision. pp. 38–56. Springer (2022) 11, 12, 29, 37

    2022

  50. [50]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021) 20, 40

    2021

  51. [51]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Rong, S., Tu, B., Wang, Z., Li, J.: Boundary-enhanced co-training for weakly su- pervised semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 19574–19584 (2023) 21

    2023

  52. [52]

    In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18. pp. 234–241. Springer (2015) 3, 12, 13, 25, 29

    2015

  53. [53]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Shen, T., Zhang, Y., Qi, L., Kuen, J., Xie, X., Wu, J., Lin, Z., Jia, J.: High quality segmentation for ultra high-resolution images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1310–1319 (2022) 12, 13, 29

    2022

  54. [54]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Tang, C., Chen, H., Li, X., Li, J., Zhang, Z., Hu, X.: Look closer to segment better: Boundary patch refinement for instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13926– 13935 (2021) 2, 3, 4, 8 Phoenix 19

    2021

  55. [55]

    In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, Au- gust 23–28, 2020, Proceedings, Part I 16

    Tian, Z., Shen, C., Chen, H.: Conditional convolutions for instance segmentation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, Au- gust 23–28, 2020, Proceedings, Part I 16. pp. 282–298. Springer (2020) 12

    2020

  56. [56]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Tian, Z., Shen, C., Wang, X., Chen, H.: Boxinst: High-performance instance seg- mentation with box annotations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5443–5452 (2021) 2

    2021

  57. [57]

    Advances in neural information pro- cessing systems30(2017) 26

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017) 26

    2017

  58. [58]

    IEEE transactions on pattern analysis and machine intelligence43(10), 3349–3364 (2020) 12

    Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al.: Deep high-resolution representation learning for visual recog- nition. IEEE transactions on pattern analysis and machine intelligence43(10), 3349–3364 (2020) 12

    2020

  59. [59]

    Advances in Neural Information Processing Systems36, 79761–79780 (2023) 2, 3, 4, 5, 8, 10, 35, 38

    Wang, M., Ding, H., Liew, J.H., Liu, J., Zhao, Y., Wei, Y.: Segrefiner: Towards model-agnostic segmentation refinement with discrete diffusion process. Advances in Neural Information Processing Systems36, 79761–79780 (2023) 2, 3, 4, 5, 8, 10, 35, 38

    2023

  60. [60]

    In: International conference on ma- chine learning

    Wang, T., Isola, P.: Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In: International conference on ma- chine learning. pp. 9929–9939. PMLR (2020) 5, 9

    2020

  61. [61]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Wang, X., Zhang, R., Shen, C., Kong, T., Li, L.: Dense contrastive learning for self-supervised visual pre-training. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3024–3033 (2021) 5

    2021

  62. [62]

    Wang, Z., Li, Y., Wang, S.: Noisy boundaries: Lemon or lemonade for semi- supervised instance segmentation? In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16826–16835 (2022) 2, 4, 10, 11, 12

    2022

  63. [63]

    Advances in neural information processing systems34, 12077–12090 (2021) 2, 3

    Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems34, 12077–12090 (2021) 2, 3

    2021

  64. [64]

    In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition

    Xie, Z., Lin, Y., Zhang, Z., Cao, Y., Lin, S., Hu, H.: Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16684–16693 (2021) 5

    2021

  65. [65]

    In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16

    Yuan, Y., Xie, J., Chen, X., Wang, J.: Segfix: Model-agnostic boundary refinement for segmentation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16. pp. 489–506. Springer (2020) 2, 3, 4, 5, 8

    2020

  66. [66]

    In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition

    Zhang, Z., Cai, H., Han, S.: Efficientvit-sam: Accelerated segment anything model without performance loss. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. pp. 7859–7863 (2024) 34

    2024

  67. [67]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2881–2890 (2017) 3, 12

    2017

  68. [68]

    In: European Conference on Computer Vision

    Zhou, C., Loy, C.C., Dai, B.: Extract free dense labels from clip. In: European Conference on Computer Vision. pp. 696–712. Springer (2022) 21 20 B. Kim and S. J. Hwang Appendix A Additional Applications of Phoenix A.1 Self-Supervised Mask Refinement Learning. Our intriguing finding is the potential for self-supervised mask refinement with- out ground-tru...

    2022