REVIEW 3 major objections 3 minor 34 references
Unsupervised Class Generation to Expand Semantic Segmentation Datasets
T0 review · 3 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read By pasting diffusion-generated cutouts into synthetic training sets, the paper's UDA pipeline learns to segment novel classes and improves existing ones.
desk verdict Novel-class addition is a solid, honest result; the existing-class improvement claim is confounded by a missing cutout-only control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the attention-to-mask extraction chain. Aggregated cross-attention maps localize the class token in a Stable Diffusion image; thresholding plus DenseCRF produces a coarse binary region; a bounding box around that region with five sample points prompts SAM to segment the object; and three geometric filter metrics (Polsby-Popper compactness, smoothed-perimeter ratio, and contour angular energy) discard masks that would inject noisy labels. The integration mechanism is a MixUP-style paste: a cutout and its binary mask are inserted into a random location of a synthetic source image and the label map is updated with the new class id, so the UDA algorithm trains on C+1 classes without any change to the algorithm.
What would settle it
Take a class with known ground truth, such as objects from a simulator or a real segmentation dataset, run the proposed attention-plus-SAM mask extraction, and compare each generated mask to the true mask by IoU; if the masks that pass the geometric filters are not substantially more accurate than those rejected, or if passing masks still have low IoU, then the curation stage is not doing the work the paper attributes to it.
Extended reading notes
Core claim
The central claim is that the closed-set limitation of synthetic segmentation datasets can be bypassed by generating class examples with a diffusion model and segmenting them automatically. For each desired class, the method writes varied text prompts, aggregates Stable Diffusion's cross-attention maps for the class token across layers, timesteps and heads, thresholds them at 0.5, applies DenseCRF to get a box, and prompts SAM with five points to obtain a binary mask. A curation stage keeps only compact, smooth, low-energy masks, discarding close-ups, interiors, and noisy edges; 2000 filtered cutouts per class are then pasted into source images at random locations following a MixUP-style scheme with label Q. In experiments with the DAFormer UDA pipeline on synthetic-to-Cityscapes adaptation, novel classes reach an average of 51% IoU, and adding the missing classes to Synthia or CARLA-4AGT raises mean IoU, often by reducing confusions such as trucks predicted as cars or buses and trains predicted as trucks. Thus the paper claims both effective novel-class learning and a positive transfer to existing classes.
Load-bearing premise
The method assumes the automatically produced object cutouts have masks accurate enough to serve as training labels, a trust calibrated only by thresholds chosen after looking at 20 bus images.
Editorial extensions
If this is right
- A fixed simulator-based dataset can be extended with novel object classes without re-running or modifying the simulator, so long as the novel class can be described in text.
- Pasting generated cutouts can reduce confusion between visually similar existing classes, since the added class absorbs predictions that previously fell onto neighbors, for example trucks no longer predicted as cars.
- The mix probability pm is a dataset-dependent hyperparameter: too low and the model under-sees the class, too high and it overfits, so it should be tuned to image and crop size.
- Mask filtering is necessary: unfiltered cutouts degrade novel-class IoU and can lower overall mIoU below the baseline.
Reading between the lines
- Beyond the paper, the same cutout-pasting recipe should transfer to any UDA algorithm that trains with source labels, not just DAFormer, because the integration only touches the training data.
- A natural stress test is to apply the pipeline to classes with thin or articulated structures such as poles, riders, or fences, where the geometric smoothness filters may systematically discard valid masks; the paper's threshold tuning on 20 bus images would likely need re-calibration per class.
- The method could be inverted into an evaluation tool: by pasting cutouts with known ground-truth masks into a controlled scene, one could measure how much label noise from imperfect SAM masks degrades segmentation, which the paper only observes indirectly through filtering ablations.
- The reported optimal mix rate varying with dataset resolution suggests scaling the mix probability with crop-to-image size could make the method more portable across source domains.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an unsupervised, training-free pipeline that uses Stable Diffusion cross-attention maps (following the authors' earlier OVAM method) to localize a novel class in generated images, prompts SAM to obtain a binary mask, filters masks with three hand-tuned shape metrics, and extracts RGB cutouts with pseudo-ground-truth masks. These cutouts are pasted into synthetic source images at random locations with probability p_m, thereby adding new classes to existing closed-set segmentation datasets without modifying the simulator or the UDA algorithm. The method is evaluated with the DAFormer UDA pipeline on Synthia and CARLA-4AGT as sources and Cityscapes as target, adding bus, truck, and/or train classes. The paper reports novel-class IoUs and improved existing-class mIoU, plus an ablation showing that mask filtering matters. The central claim is that models can learn novel classes (about 51% average IoU) while also improving performance on existing classes.
Significance. If the claims are substantiated, the contribution is practically useful: it provides a way to extend closed-set synthetic datasets with new semantic classes using only text prompts and off-the-shelf generative models, without retraining the simulator or modifying the UDA algorithm. The pipeline is conceptually simple, uses public components (Stable Diffusion, SAM, DAFormer), and is evaluated on two standard synthetic-to-real UDA settings with multiple new classes. The filtering ablation and per-class confusion matrices are informative. The main weaknesses are experimental: the existing-class improvement claim lacks a control for the augmentation confound, the mixing probability is selected on the evaluation set, and there are no repeated-seed statistics. These issues are addressable, so the paper merits a major revision rather than rejection.
major comments (3)
- [§3.3, Table 1] The central claim that adding novel-class cutouts 'reduce[s] errors for other, already existing classes' is not yet supported because the only comparison is against a baseline with no cutouts. The mix operation itself changes the source distribution and could improve mIoU through augmentation, regularization, or label-smoothing-like effects even if the pasted label were an existing class. I request a control condition that pastes the same number of cutouts (same positions, sizes, and mixing probability) carrying an existing-class label or a label-flipped version; without this, the existing-class gains in Table 1 (e.g., Synthia 54.9→57.0/58.0/59.1 and 4AGT 55.2→61.0/61.1/63.2) could be due to augmentation alone. The novel-class IoU numbers are less affected by this concern, but the abstract's claim about reducing errors for existing classes requires the control.
- [§4.2, Figure 6] The protocol for choosing the mixing probability p_m is not specified, and if the best value is selected on the same Cityscapes evaluation reported in Table 1, the final numbers are subject to selection bias. Moreover, all tables report a single training run with no error bars, so differences of 2–3 mIoU points may be within run-to-run noise. I ask for results for all p_m values, a pre-specified selection protocol (e.g., a validation split or a fixed per-dataset policy), and at least three random seeds with mean and standard deviation reported for the main configurations.
- [§3.2, Eqs. (1)–(3)] The mask-curation thresholds (T_th=0.5, max attention ratio 40%, Polsby-Popper >0.6, smoothness threshold, and contour energy <50) were set after manually inspecting 20 bus images and then applied to all classes. Because the filtering ablation in Table 2 demonstrates that filtering materially affects performance, the lack of any sensitivity analysis for these thresholds leaves open whether the results depend on hand-tuned values that may not transfer to new classes or image resolutions. I request at least a threshold sensitivity experiment (e.g., varying each threshold over a reasonable range for one class) or an automatic/self-tuned selection procedure.
minor comments (3)
- [§4] In the experiments section, 'ground-truth labels pf Cityscapes' should read 'ground-truth labels of Cityscapes'.
- [Tables 1 and 2] The tables are dense and the meaning of the new-class column headers is not fully clear from the captions; please make explicit which classes are added in each row and use consistent decimal notation (some rows use commas as decimal separators).
- [§2.1 and §4] The evaluation would be strengthened by a comparison against existing generative-data methods such as DatasetDM or DGInStyle, at least in terms of mask quality or downstream IoU, since the current results do not establish whether the proposed pipeline is preferable to those alternatives.
Circularity Check
No significant circularity: the reported IoU numbers are measured against external Cityscapes labels, and the generation pipeline's fitted thresholds and self-cited attention-localization component do not encode those IoU values.
full rationale
The paper's derivation chain is empirical rather than formal. Synthetic cutouts are generated with Stable Diffusion, localized via attention maps using the authors' prior CVPR work [13], masked by SAM, filtered by hand-set thresholds, and pasted into Synthia or CARLA-4AGT source images; DAFormer is then trained and evaluated with per-class IoU against the external Cityscapes validation set. None of the reported metrics are recomputed from the pipeline's own masks or from the filtering thresholds: the thresholds in Section 3.2 (PP>0.6, S<1.0, EP<50, selected from 20 bus images) affect only which training cutouts are kept, and the final IoU values come from Cityscapes ground truth, so there is no fitted parameter renamed as a prediction. The citation to Marcos-Manchon et al. [13] is a self-citation (shared author Juan C. SanMiguel), but it supplies an externally published attention-localization component and is not used to assert the paper's central claim or to rule out alternatives. The main experimental weakness, noted in the skeptic review, is the absence of a mixup-control condition proving that the existing-class mIoU gains are caused by the novel-class labels rather than by the paste augmentation itself; that is an internal-validity and attribution concern, not a circularity of definition or fitted-input-as-prediction. Under the hard rules, no quoted equation or construction reduces the paper's output to its input, so the circularity score is 0.
Assumptions & free parameters
free parameters (7)
- Attention threshold T_th =
0.5
- Max attention pixel ratio =
40% of image
- Polsby-Popper minimum =
0.6
- Smoothness ratio threshold =
S < 1.0 discarded
- Contour energy threshold =
E_P < 50
- Mix probability p_m =
per-dataset optimal, not stated numerically for Table 1
- Cutout count per class =
2000
assumptions (5)
- domain assumption Class-token cross-attention in Stable Diffusion localizes the target object.
- domain assumption SAM produces object-accurate masks when prompted with a bounding box derived from attention.
- domain assumption Generated synthetic cutouts are sufficiently close to the real target domain to transfer during UDA.
- domain assumption DAFormer and its pseudo-label mechanism can exploit added source classes without modification.
- domain assumption Cutout pasting into source images does not introduce harmful artifacts beyond what DACS/MixUp already handles.
Cite this review
Pith. "Pith review of Unsupervised Class Generation to Expand Semantic Segmentation Datasets." pith.science (2026). https://pith.science/paper/UJDHBXT7
@misc{pith2026250102264,
author = {Pith},
title = {Pith review of: Unsupervised Class Generation to Expand Semantic Segmentation Datasets},
year = {2026},
howpublished = {\url{https://pith.science/paper/UJDHBXT7}},
note = {Machine review of arXiv:2501.02264}
}
read the original abstract
Semantic segmentation is a computer vision task where classification is performed at a pixel level. Due to this, the process of labeling images for semantic segmentation is time-consuming and expensive. To mitigate this cost there has been a surge in the use of synthetically generated data -- usually created using simulators or videogames -- which, in combination with domain adaptation methods, can effectively learn how to segment real data. Still, these datasets have a particular limitation: due to their closed-set nature, it is not possible to include novel classes without modifying the tool used to generate them, which is often not public. Concurrently, generative models have made remarkable progress, particularly with the introduction of diffusion models, enabling the creation of high-quality images from text prompts without additional supervision. In this work, we propose an unsupervised pipeline that leverages Stable Diffusion and Segment Anything Module to generate class examples with an associated segmentation mask, and a method to integrate generated cutouts for novel classes in semantic segmentation datasets, all with minimal user input. Our approach aims to improve the performance of unsupervised domain adaptation methods by introducing novel samples into the training data without modifications to the underlying algorithms. With our methods, we show how models can not only effectively learn how to segment novel classes, with an average performance of 51% IoU, but also reduce errors for other, already existing classes, reaching a higher performance level overall.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Visual domain adaptation through lo- cality information
Devika A.K., Rakesh Kumar Sanodiya, Babita Roslind Jose, and Jimson Mathew. Visual domain adaptation through lo- cality information. Engineering Applications of Artificial In- telligence, 123:106172, 2023. 2
work page 2023
-
[2]
The cityscapes dataset for semantic urban scene understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 3212– 3223, 2016. 1, 5
work page 2016
-
[3]
A method of assigning numerical and percentage values to the degree of roundness of sand grains
EP Cox. A method of assigning numerical and percentage values to the degree of roundness of sand grains. Journal of paleontology, 1(3):179–183, 1927. 4
work page 1927
-
[4]
CARLA: An open urban driving simulator
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017. 2
work page 2017
-
[5]
Mark Everingham, Luc Van Gool, C. K. I. Williams, J. Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge, 2010. 5
work page 2010
-
[6]
Ot- clda: Optimal transport and contrastive learning for domain adaptive semantic segmentation
Qizhe Fan, Xiaoqin Shen, Shihui Ying, and Shaoyi Du. Ot- clda: Optimal transport and contrastive learning for domain adaptive semantic segmentation. IEEE Transactions on In- telligent Transportation Systems, pages 1–13, 2024. 2
work page 2024
-
[7]
Dsp: Dual soft-paste for unsupervised domain adaptive semantic segmentation
Li Gao, Jing Zhang, Lefei Zhang, and Dacheng Tao. Dsp: Dual soft-paste for unsupervised domain adaptive semantic segmentation. In ACM Int. Conf. Multimedia (MM) , page 2825–2833, 2021. 2
work page 2021
-
[8]
Lukas Hoyer, Dengxin Dai, and Luc Van Gool. DAFormer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 9924–9935,
Show all 34 references
-
[9]
HRDA: Context-aware high-resolution domain-adaptive semantic segmentation
Lukas Hoyer, Dengxin Dai, and Luc Van Gool. HRDA: Context-aware high-resolution domain-adaptive semantic segmentation. In IEEE Eur. Conf. Comput. Vis. (ECCV) , page 372–391, 2022. 2, 4
2022
-
[10]
MIC: Masked image consistency for context- enhanced domain adaptation
Lukas Hoyer, Dengxin Dai, Haoran Wang, and Luc Van Gool. MIC: Masked image consistency for context- enhanced domain adaptation. In IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023. 2, 4
2023
-
[11]
Segment any- thing
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 1, 3
2023
-
[12]
Efficient inference in fully connected crfs with gaussian edge potentials
Philipp Kr ¨ahenb¨uhl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. Ad- vances in neural information processing systems , 24, 2011. 3
2011
-
[13]
Open-vocabulary atten- tion maps with token optimization for semantic segmentation in diffusion models
Pablo Marcos-Manch ´on, Roberto Alcover-Couso, Juan C SanMiguel, and Jose M Mart ´ınez. Open-vocabulary atten- tion maps with token optimization for semantic segmentation in diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognitio...
2024
-
[14]
Nguyen, Trinh V
Kieu Dang Nam, Tu M. Nguyen, Trinh V . Dieu, Muriel Visani, Thi-Oanh Nguyen, and Dinh Viet Sang. A novel unsupervised domain adaption method for depth-guided se- mantic segmentation using coarse-to-fine alignment. IEEE Access, 10:101248–101262, 2022. 2
2022
-
[15]
The third criterion: Compactness as a procedural safeguard against partisan ger- rymandering
Daniel D Polsby and Robert D Popper. The third criterion: Compactness as a procedural safeguard against partisan ger- rymandering. Yale L. & Pol’y Rev., 9:301, 1991. 4
1991
-
[16]
Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun
Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In IEEE Eur. Conf. Comput. Vis. (ECCV), pages 102– 118, 2016. 1, 2
2016
-
[17]
High-Resolution Im- age Synthesis with Latent Diffusion Models, Apr
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-Resolution Im- age Synthesis with Latent Diffusion Models, Apr. 2022. arXiv:2112.10752 [cs]. 1, 3
2022 arXiv
-
[18]
Lgsvl simulator: A high fidelity simulator for autonomous driving
Guodong Rong, Byung Hyun Shin, Hadi Tabatabaee, Qiang Lu, Steve Lemke, M ¯artin ¸ˇs Mo ˇzeiko, Eric Boise, Geehoon Uhm, Mark Gerow, Shalin Mehta, et al. Lgsvl simulator: A high fidelity simulator for autonomous driving. In2020 IEEE 23rd International conference on intelligent ...
2020
-
[19]
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez. The synthia dataset: A large collection of synthetic images for semantic segmenta- tion of urban scenes. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 3234–3243, 2016. 1, 2, 4, 5
2016
-
[20]
sch ¨afer, Nico M
Manuel Schwonberg, Joshua Niemeijer, Jan-Aike Term¨ohlen, J ¨org P. sch ¨afer, Nico M. Schmidt, Hanno Gottschalk, and Tim Fingscheidt. Survey on unsuper- vised domain adaptation for semantic segmentation for visual perception in automated driving. IEEE Access , 11:54296–54336,...
2023
-
[21]
Airsim: High-fidelity visual and physical simulation for autonomous vehicles
Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics: Re- sults of the 11th International Conference , pages 621–635. Springer, 2018. 2
2018
-
[22]
What the daam: Interpreting stable diffu- sion using cross attention
Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture. What the daam: Interpreting stable diffu- sion using cross attention. arXiv preprint arXiv:2210.04885,
-
[23]
Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 5
2023 arXiv
-
[24]
Dacs: Domain adaptation via cross- domain mixed sampling
Wilhelm Tranheden, Viktor Olsson, Juliano Pinto, and Lennart Svensson. Dacs: Domain adaptation via cross- domain mixed sampling. IEEE Winter Conf. App. Comp. Vis. (WACV), pages 1378–1388, 2020. 2, 4
2020
-
[25]
Learning to adapt structured output space for semantic seg- mentation
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Ki- hyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic seg- mentation. In IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 7472–7481, 2018. 2
2018
-
[26]
Advent: Adversarial entropy mini- mization for domain adaptation in semantic segmentation
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick P´erez. Advent: Adversarial entropy mini- mization for domain adaptation in semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 2512–2521, 2019. 2
2019
-
[27]
Pseudo-label assisted optimization of multi-branch net- work for cross-domain person re-identification
Zhengyang Wang, Shuxiang Guo, Xue Shang, and Xiufen Ye. Pseudo-label assisted optimization of multi-branch net- work for cross-domain person re-identification. In IEEE International Conference on Mechatronics and Automation (ICMA), pages 13–18, 2023. 4
2023
-
[28]
Datasetdm: Synthesizing data with perception annota- tions using diffusion models
Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annota- tions using diffusion models. Advances in Neural Informa- tion Processing Systems, 36:54683–54695, 2023. 2
2023
-
[29]
Self-supervised adversarial learn- ing for domain adaptation of pavement distress classifica- tion
Yanwen Wu, Mingjian Hong, Ao Li, Sheng Huang, Hui- jun Liu, and Yongxin Ge. Self-supervised adversarial learn- ing for domain adaptation of pavement distress classifica- tion. IEEE Transactions on Intelligent Transportation Sys- tems, 25(2):1966–1977, 2024. 2
1966
-
[30]
Transfer learning from synthetic to real lidar point cloud for semantic segmentation
Aoran Xiao, Jiaxing Huang, Dayan Guan, Fangneng Zhan, and Shijian Lu. Transfer learning from synthetic to real lidar point cloud for semantic segmentation. InProceedings of the AAAI conference on artificial intelligence, volume 36, pages 2795–2803, 2022. 2
2022
-
[31]
H. Yan, Y . Ding, P. Li, Q. Wang, Y . Xu, and W. Zuo. Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 945–954, 2017. 2
2017
-
[32]
Weighted and class-specific maxi- mum mean discrepancy for unsupervised domain adaptation
Hongliang Yan, Zhetao Li, Qilong Wang, Peihua Li, Yong Xu, and Wangmeng Zuo. Weighted and class-specific maxi- mum mean discrepancy for unsupervised domain adaptation. IEEE Transactions on Multimedia, PP:1–1, 2019. 2
2019
-
[33]
Dgin- style: Domain-generalizable semantic segmentation with image diffusion models and stylized semantic control
Yuru Jia, Lukas Hoyer, Shengyu Huang, Tianfu Wang, Luc Van Gool, Konrad Schindler, and Anton Obukhov. Dgin- style: Domain-generalizable semantic segmentation with image diffusion models and stylized semantic control. arXiv preprint arXiv:2312.03048, 2023. 2
2023 arXiv
-
[34]
Parauda: Invariant feature learning with auxil- iary synthetic samples for unsupervised domain adaptation
Wenwen Zhang, Jiangong Wang, Yutong Wang, and Fei- Yue Wang. Parauda: Invariant feature learning with auxil- iary synthetic samples for unsupervised domain adaptation. IEEE Transactions on Intelligent Transportation Systems , 23(11):20217–20229, 2022. 2
2022
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.