REVIEW 2 major objections 5 minor 47 references
Automatic image matting can match fully supervised quality using only unlabeled RGB images.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-14 12:02 UTC pith:ELQQZHPT
load-bearing objection First competitive automatic matting from RGB alone, but the win is real mainly for salient opaque objects (portraits/animals) where DINO seeds already work. the 2 major comments →
Self-supervised Automatic Matting
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
SSMatte is the first framework that trains an automatic matting network end-to-end from RGB images alone and still matches the quantitative performance of fully supervised automatic matting models on portrait benchmarks while outperforming prior methods that require at least trimap supervision.
What carries the argument
Semantic anchoring loss L_sea: a training-efficient generalized Rayleigh-quotient objective that propagates sparse class-token seeds across frozen self-supervised ViT patch affinities, producing a coherent soft matting prompt that replaces manual trimaps.
Load-bearing premise
The method assumes that the noisy foreground seeds taken from a frozen self-supervised Vision Transformer’s class token, once cleaned by the Rayleigh-quotient loss, are reliable enough to tell the model which object to extract.
What would settle it
Train and test the identical pipeline on a large collection of images whose main foregrounds are transparent or non-salient; if the resulting mattes stay far below fully supervised baselines while portrait numbers remain high, the claim that the framework solves automatic matting in general is falsified.
If this is right
- Automatic matting models can be improved simply by collecting more unlabeled photographs rather than more expensive alpha annotations.
- Trimap-based weakly supervised training is no longer required to reach competitive accuracy on portrait matting.
- A lightweight, label-free pretraining step on the semantic inducer alone improves generalization across object categories.
- Performance continues to rise with data volume up to tens of thousands of unlabeled images before plateauing.
Where Pith is reading between the lines
- The same two-stage pattern (high-level semantic prompt plus low-level color-alpha consistency) may transfer to other dense tasks that still rely on costly pixel labels, such as soft segmentation or edge-aware depth refinement.
- Remaining gaps on transparent and non-salient objects point to a concrete test for future self-supervised backbones: stronger multi-object or transparency cues should close the gap without reintroducing labels.
- Because the backbone stays frozen, the approach is well-suited to continual or web-scale training on streaming image collections.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SSMatte, a self-supervised framework for automatic image matting trained solely on RGB images with no alpha, trimap, or mask labels. It decomposes the task into semantic anchoring (propagating sparse CLS-token seeds from a frozen DINOv2 ViT via a novel generalized Rayleigh-quotient loss L_sea implemented by a lightweight Semantic Inducer) and detail matting (a fixed-point target loss L_t that regularizes the converged solution of the directional distance consistency iteration, combined with L_DDC and a high-confidence L_sem on the induced prompt). End-to-end training of a ViTMatte-style decoder yields results that surpass the prior trimap-only baseline (AFM) and approach or match several fully-supervised automatic matting models on portrait (P3M) and animal (AM-2K) benchmarks, with additional scaling and cross-category experiments.
Significance. If the empirical claims hold, the work removes the last dense-annotation requirement for competitive automatic matting and thereby opens a path to scaling with web-scale unlabeled images. The two technical ingredients—an efficient spectral-style anchoring loss that turns frozen self-supervised features into a soft matting prompt, and a fixed-point reformulation that mitigates texture accumulation in color-affinity losses—are concrete, reusable contributions. Ablations isolate each term, a scaling curve shows monotonic gains with unlabeled volume, trainable parameters are reduced to ~3 M by freezing the backbone, and code is promised; these are genuine strengths that raise the bar for future label-efficient matting.
major comments (2)
- [Table 1 / §4.2] Table 1 and §4.2: the abstract and introduction claim performance “on par with fully-supervised automatic matting” and “using only RGB images, with no manual annotation at all.” The strongest portrait numbers (SSMatte†) rely on a trimap-guided crop that, while used only for data selection, still requires trimap annotations at training-time preparation. The non-† row remains competitive with several fully-supervised baselines but is no longer clearly “on par” with the best of them (e.g., GFM/P3M). Primary claims and the abstract should be supported by the pure-RGB numbers, or the distinction between the two settings must be stated more prominently so that the “annotation-free” framing is not overstated.
- [§3.1 / Table 5] §3.1 (Eqs. 2, 9) and Table 5: the method explicitly prioritizes “salient opaque foregrounds” whose seeds emerge from the CLS-token attention of a frozen DINOv2. Table 5 confirms that performance on transparent/meticulous and non-salient categories remains substantially weaker than fully-supervised AIM/SMat even after DUTS pre-training of the inducer. The abstract’s unqualified statements about “automatic matting” and “favorable au generalization behaviors” therefore need tighter scoping to the regime the seed already favors; otherwise the central claim of a general annotation-free paradigm is not fully supported by the evidence.
minor comments (5)
- [Abstract] Abstract and §1: “an fresh, fully annotation-free paradigm” contains a grammatical error (“an” → “a”).
- [Table 5] Table 5 caption states “For full metrics please refer to the Supplementary,” yet several key SAD-Type and SAD-Category numbers are already the only numbers shown; either expand the main table or make the supplementary reference more precise.
- [§3.2] §3.2, Eq. (11)–(13): the transition from the iterative map α(t+1)=Wα(t)+b to the fixed-point residual L_t is clear, but a short remark on how the high-confidence mask ˜S is obtained from morphological erosion of the continuous prompt S would help reproducibility without consulting the supplement.
- [Fig. 4] Fig. 4 caption and surrounding text use both L_DDC and L_t; a consistent notation for the fixed-point residual (currently L_t) would avoid momentary confusion with the iteration index t.
- [§4.1] §4.1 lists several free parameters (λ1–4, K, θ1–2) whose values are given but whose sensitivity is deferred to the supplement; a one-sentence statement of the most sensitive parameter in the main text would strengthen the experimental section.
Circularity Check
No load-bearing circularity; losses are standard spectral/nonlocal constructions evaluated on external GT benchmarks, with only routine architectural self-citation.
full rationale
The paper's central claim is empirical (competitive automatic matting from RGB alone via two self-supervised losses). L_sea (Eq. 9) is an explicit generalized Rayleigh quotient on frozen DINOv2 affinities plus a seed map from [CLS] attention (Eq. 2); this is the classic normalized-cut objective (cited to Shi-Malik and TokenCut) applied to produce a soft prompt, not a quantity defined in terms of the final alpha. L_dpm (Eq. 13) combines a fixed-point residual of the known DDC iteration (Eq. 11) with the original DDC term; the fixed-point rewrite is algebraically derived from the iteration map and is not tautological with the target matte. Both losses are optimized end-to-end on unlabeled RGB only; final numbers are measured against held-out ground-truth alphas on P3M/AM-2K/AIM-500 and compared to independent fully- and weakly-supervised baselines. Self-citations (ViTMatte backbone, Smat [CLS] usage) supply only architectural components, not uniqueness theorems or fitted constants that force the reported metrics. The category-specific seed limitation noted by the reader is a correctness/scope issue, not circularity. Hence score 1 (minor non-load-bearing self-citation) rather than 0.
Axiom & Free-Parameter Ledger
free parameters (4)
- λ1, λ2 (semantic anchoring)
- λ3, λ4 (detail matting)
- K (nearest color neighbors)
- θ1, θ2 (prompt binarization thresholds)
axioms (3)
- domain assumption Frozen DINOv2 [CLS]-token attention maps provide usable seeds for the salient opaque foreground of interest.
- domain assumption The classical compositing equation I = αF + (1-α)B together with nonlocal color affinity is a sufficient inductive bias for recovering fine alpha once a coarse semantic prompt is given.
- standard math The second-smallest eigenvector of the modified Laplacian (L − λ1 Tr(Du0 u0ᵀ D)) yields a coherent soft matting prompt.
invented entities (2)
-
Semantic Inducer + L_sea (generalized Rayleigh-quotient anchoring loss)
no independent evidence
-
Fixed-point target loss Lt
no independent evidence
read the original abstract
High-quality alpha mattes are notoriously expensive to annotate, creating a fundamental data bottleneck for deep image matting. While prior work attempts to reduce annotation cost using coarser labels like trimaps or masks, they remain reliant on costly per-pixel supervision, limiting scalability and generalization. In this work, we push the boundary further and ask: can we train an automatic matting model using only RGB images, with no manual annotation at all? We answer this by presenting SSMatte, a self-supervised framework that for the first time achieves performance on par with fully-supervised automatic matting. Our key insight is to decompose the problem into semantic anchoring and detail matting. SSMatte first generates a semantic matting prompt from frozen self-supervised ViT features by propagating class-token seeds via a novel, training-efficient semantic anchoring loss based on a generalized Rayleigh quotient. This prompt then anchors a detail matting network, which is optimized via a fixed-point-based loss that enforces alpha-RGB consistency. Extensive experiments show SSMatte outperforms prior weakly-supervised methods, matches the performance of fully-supervised models on portrait benchmarks, and demonstrates favorable scaling and generalization behaviors with additional data. Our work pushes automatic matting to an fresh, fully annotation-free paradigm. Code will be available.
Figures
Reference graph
Works this paper leans on
-
[1]
ACM TOG37(4), 1–13 (2018)
Aksoy, Y., Oh, T.H., Paris, S., Pollefeys, M., Matusik, W.: Semantic soft segmen- tation. ACM TOG37(4), 1–13 (2018)
2018
-
[2]
In: CVPR
Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., Le- Cun, Y., Ballas, N.: Self-supervised learning from images with a joint-embedding predictive architecture. In: CVPR. pp. 15619–15629 (2023)
2023
-
[3]
arXiv preprint arXiv:2106.08254 (2021)
Bao, H., Dong, L., Piao, S., Wei, F.: Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 (2021)
Pith/arXiv arXiv 2021
-
[4]
In: ICML
Bianchi, F.M., Grattarola, D., Alippi, C.: Spectral clustering with graph neural networks for graph pooling. In: ICML. pp. 874–883. PMLR (2020)
2020
-
[5]
NeurIPS35, 33371–33386 (2022)
Bielski, A., Favaro, P.: Move: Unsupervised movable object segmentation and de- tection. NeurIPS35, 33371–33386 (2022)
2022
-
[6]
In: ICCV
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV. pp. 9650– 9660 (2021)
2021
-
[7]
In: ACM MM
Chen, Q., Ge, T., Xu, Y., Zhang, Z., Yang, X., Gai, K.: Semantic human matting. In: ACM MM. pp. 618–626 (2018)
2018
-
[8]
In: CVPR
Chen, X., He, K.: Exploring simple siamese representation learning. In: CVPR. pp. 15750–15758 (2021)
2021
-
[9]
arXiv preprint arXiv:2103.12337 (2021)
Deora, R., Sharma, R., Raj, D.S.S.: Salient image matting. arXiv preprint arXiv:2103.12337 (2021)
Pith/arXiv arXiv 2021
-
[10]
NeurIPS33, 21271–21284 (2020)
Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Do- ersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.: Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS33, 21271–21284 (2020)
2020
-
[11]
TIP34(12), 2487–2498 (2025)
Guan, H., Lin, J., Lau, R.W.: A contrastive-learning framework for unsupervised salient object detection. TIP34(12), 2487–2498 (2025)
2025
-
[12]
In: CVPR
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: CVPR. pp. 16000–16009 (2022)
2022
-
[13]
In: CVPR
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: CVPR. pp. 9729–9738 (2020)
2020
-
[14]
In: AAAI
Ke, Z., Sun, J., Li, K., Yan, Q., Lau, R.W.: Modnnet: Real-time trimap-free por- trait matting via objective decomposition. In: AAAI. pp. 1140–1147 (2022)
2022
-
[15]
In: ICCV
Kim, B., Shin, C., Jeong, J., Jung, H., Lee, S.Y., Chun, S., Hwang, D.H., Yu, J.: Zim: Zero-shot image matting for anything. In: ICCV. pp. 23828–23838 (2025)
2025
-
[16]
arXiv preprint arXiv:2404.00921 (2024)
Kim, B., Yi, M.Y., Yu, J., Yoo, Y.J., Hwang, S.J.: Towards label-efficient human matting: A simple baseline for weakly semi-supervised trimap-free human matting. arXiv preprint arXiv:2404.00921 (2024)
Pith/arXiv arXiv 2024
-
[17]
Kong, Y., Li, J., Hu, L., Li, X.: Semi-supervised learning matting algorithm based on semantic consistency of trimaps. Appl. Sci.13(15), 8616 (2023)
2023
-
[18]
arXiv preprint arXiv:2304.13509 (2023)
Kvanchiani, K., Petrova, E., Efremyan, K., Sautin, A., Kapitanov, A.: Easyportrait–face parsing and portrait segmentation dataset. arXiv preprint arXiv:2304.13509 (2023)
Pith/arXiv arXiv 2023
-
[19]
In: ACMMM
Li, J., Ma, S., Zhang, J., Tao, D.: Privacy-preserving portrait matting. In: ACMMM. pp. 3501–3509 (2021)
2021
-
[20]
IJCV130(2), 246–266 (2022)
Li, J., Zhang, J., Maybank, S.J., Tao, D.: Bridging composite and real: Towards end-to-end deep image matting. IJCV130(2), 246–266 (2022)
2022
-
[21]
In: Zhou, Z.H
Li, J., Zhang, J., Tao, D.: Deep automatic natural image matting. In: Zhou, Z.H. (ed.) Proceedings of the Thirtieth International Joint Conference on Artificial In- telligence, IJCAI-21. pp. 800–806. IJCAI (2021) Self-Supervised Automatic Matting 17
2021
-
[22]
In: WACV
Li, Y., Huang, Z., Yu, G., Chen, L., Wei, Y., Jiao, J.: Disentangled pre-training for image matting. In: WACV. pp. 169–178 (2024)
2024
-
[23]
Li, Y., Salehi, S., Ungar, L., Kording, K.P.: Does object binding naturally emerge in large pretrained vision transformers? arXiv preprint arXiv:2510.24709 (2025)
arXiv 2025
-
[24]
In: CVPR
Lin, S., Ryabtsev, A., Sengupta, S., Curless, B.L., Seitz, S.M., Kemelmacher- Shlizerman, I.: Real-time high-resolution background matting. In: CVPR. pp. 8762–8771 (2021)
2021
-
[25]
In: CVPR
Liu, J., Yao, Y., Hou, W., Cui, M., Xie, X., Zhang, C., Hua, X.s.: Boosting semantic human matting with coarse annotations. In: CVPR. pp. 8563–8572 (2020)
2020
-
[26]
In: AAAI
Liu, W., Ye, Z., Lu, H., Cao, Z., Yue, X.: Training matting models without alpha labels. In: AAAI. pp. 5604–5612 (2025)
2025
-
[27]
In: CVPR
Melas-Kyriazi, L., Rupprecht, C., Laina, I., Vedaldi, A.: Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and local- ization. In: CVPR. pp. 8364–8375 (2022)
2022
-
[28]
Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez,P.,Haziza,D.,Massa,F.,El-Nouby,A.,Howes,R.,Huang,P.Y.,Xu,H., Sharma, V., Li, S.W., Galuba, W., Rabbat, M., Assran, M., Ballas, N., Synnaeve, G., Misra, I., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: Dinov2: Learning robust visual features without ...
2023
-
[29]
In: CVPR
Park, K., Woo, S., Oh, S.W., Kweon, I.S., Lee, J.Y.: Mask-guided matting in the wild. In: CVPR. pp. 1992–2001 (2023)
1992
-
[30]
In: CVPR
Qiao, Y., Liu, Y., Yang, X., Zhou, D., Xu, M., Zhang, Q., Wei, X.: Attention-guided hierarchical structure aggregation for image matting. In: CVPR. pp. 13676–13685 (2020)
2020
-
[31]
In: CVPR
Rhemann,C.,Rother,C.,Wang,J.,Gelautz,M.,Kohli,P.,Rott,P.:Aperceptually motivated online benchmark for image matting. In: CVPR. pp. 1826–1833 (2009)
2009
-
[32]
In: CVPR
Sengupta, S., Jayaram, V., Curless, B., Seitz, S.M., Kemelmacher-Shlizerman, I.: Background matting: The world is your green screen. In: CVPR. pp. 2291–2300 (2020)
2020
-
[33]
IEEE TPAMI22(8), 888–905 (2000)
Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE TPAMI22(8), 888–905 (2000)
2000
-
[34]
In: CVPR
Siméoni,O.,Sekkat,C.,Puy,G.,Vobeck` y,A.,Zablocki,É.,Pérez,P.:Unsupervised object localization: Observing the background to discover objects. In: CVPR. pp. 3176–3186 (2023)
2023
-
[35]
Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., Massa, F., Haziza, D., Wehrstedt, L., Wang, J., Darcet, T., Moutakanni, T., Sentana, L., Roberts, C., Vedaldi, A., Tolan, J., Brandt, J., Couprie, C., Mairal, J., Jégou, H., Labatut, P., Bojanowski, P.: DINOv3 (2025)
2025
-
[36]
In: CVPR
Sun, Y., Tang, C.K., Tai, Y.W.: Ultrahigh resolution image/video matting with spatio-temporal sparsity. In: CVPR. pp. 14112–14121 (2023)
2023
-
[37]
In: CVPR
Wang, L., Lu, H., Wang, Y., Feng, M., Wang, D., Yin, B., Ruan, X.: Learning to detect salient objects with image-level supervision. In: CVPR. pp. 136–145 (2017)
2017
-
[38]
IEEE TPAMI45(12), 15790–15801 (2023)
Wang, Y., Shen, X., Yuan, Y., Du, Y., Li, M., Hu, S.X., Crowley, J.L., Vaufrey- daz, D.: Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. IEEE TPAMI45(12), 15790–15801 (2023)
2023
-
[39]
In: CVPR
Wei, T., Chen, D., Zhou, W., Liao, J., Zhao, H., Zhang, W., Yu, N.: Improved image matting via real-time user clicks and uncertainty estimation. In: CVPR. pp. 15374–15383 (2021) 18 F. Author et al
2021
-
[40]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Xu, B., Huang, H., Lu, C., Li, Z., Guo, Y.: Virtual multi-modality self- supervised foreground matting for human-object interaction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 438–447 (2021)
2021
-
[41]
In: CVPR
Xu, N., Price, B., Cohen, S., Huang, T.: Deep image matting. In: CVPR. pp. 2970–2979 (2017)
2017
-
[42]
INFORM FUSION103, 102091 (2024)
Yao, J., Wang, X., Yang, S., Wang, B.: Vitmatte: Boosting image matting with pre-trained plain vision transformers. INFORM FUSION103, 102091 (2024)
2024
-
[43]
In: CVPR
Ye, Z., Liu, W., Guo, H., Liang, Y., Hong, C., Lu, H., Cao, Z.: Unifying automatic and interactive matting with pretrained vits. In: CVPR. pp. 25585–25594 (2024)
2024
-
[44]
In: CVPR
Yu, Q., Zhang, J., Zhang, H., Wang, Y., Lin, Z., Xu, N., Bai, Y., Yuille, A.: Mask guided matting via progressive refinement network. In: CVPR. pp. 1154– 1163 (2021)
2021
-
[45]
DIGIT SIGNAL PROCESS133, 103849 (2023)
Zhang, X., Gao, C., Wang, G., Sang, N., Dong, H.: Semi-supervised portrait mat- ting using transformer. DIGIT SIGNAL PROCESS133, 103849 (2023)
2023
-
[46]
In: CVPR
Zhang, Y., Gong, L., Fan, L., Ren, P., Huang, Q., Bao, H., Xu, W.: A late fusion cnn for digital matting. In: CVPR. pp. 7469–7478 (2019)
2019
-
[47]
In: ICIG
Zhang, Y., Wang, C., Zhang, Y., Bao, H., Xu, W.: Weakly supervised image mat- ting via patch clustering. In: ICIG. pp. 130–141 (2023)
2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.