Pith. sign in

REVIEW 9 cited by

Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.02773 v3 pith:SWIKC7TB submitted 2023-09-06 cs.CV

Jinglong Wang , Xiawei Li , Jing Zhang , Qingyuan Xu , Qin Zhou , Qian Yu , Lu Sheng , Dong Xu This is my paper
classification cs.CV
keywords segmentationsemanticmodelsdiffusionresultsgenerativeobjectopen-vocabulary
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The pre-trained text-image discriminative models, such as CLIP, has been explored for open-vocabulary semantic segmentation with unsatisfactory results due to the loss of crucial localization information and awareness of object shapes. Recently, there has been a growing interest in expanding the application of generative models from generation tasks to semantic segmentation. These approaches utilize generative models either for generating annotated data or extracting features to facilitate semantic segmentation. This typically involves generating a considerable amount of synthetic data or requiring additional mask annotations. To this end, we uncover the potential of generative text-to-image diffusion models (e.g., Stable Diffusion) as highly efficient open-vocabulary semantic segmenters, and introduce a novel training-free approach named DiffSegmenter. The insight is that to generate realistic objects that are semantically faithful to the input text, both the complete object shapes and the corresponding semantics are implicitly learned by diffusion models. We discover that the object shapes are characterized by the self-attention maps while the semantics are indicated through the cross-attention maps produced by the denoising U-Net, forming the basis of our segmentation results.Additionally, we carefully design effective textual prompts and a category filtering mechanism to further enhance the segmentation results. Extensive experiments on three benchmark datasets show that the proposed DiffSegmenter achieves impressive results for open-vocabulary semantic segmentation.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Noise2Map: End-to-End Diffusion Model for Semantic Segmentation and Change Detection

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    Noise2Map repurposes diffusion model denoising into a direct predictor for semantic segmentation and change detection tasks in remote sensing, achieving top average ranks on benchmark datasets.

  2. DINO Soars: DINOv3 for Open-Vocabulary Semantic Segmentation of Remote Sensing Imagery

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    CAFe-DINO achieves SOTA open-vocabulary semantic segmentation on remote sensing datasets by leveraging DINOv3 features with cost aggregation and upsampling, fine-tuned solely on an RS-targeted COCO-Stuff subset.

  3. Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Images

    cs.CV 2025-08 conditional novelty 6.0 of 10

    SegEarth-OV performs annotation-free open-vocabulary segmentation of remote-sensing images by upsampling CLIP features, removing global bias, and distilling optical knowledge into a SAR encoder.

  4. Adapting Vehicle Detectors for Aerial Imagery to Unseen Domains with Weak Supervision

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A multi-stage diffusion-based framework that generates labeled synthetic aerial images from weak image-level labels improves cross-domain vehicle detection AP50 over prior adaptation methods.

  5. Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Stable Diffusion features, especially when conditioned on the question, improve vision-centric multimodal question answering when fused with CLIP.

  6. Interpretable Diffusion Models with B-cos Networks

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A Stable Diffusion-style U-Net rebuilt with B-cos layers produces per-token pixel attributions, and those attributions can flag prompts whose concepts are missing from the generated image.

  7. ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    ReME builds a cleaned, synonym-enriched segment-text reference set from real images and shows that simple similarity retrieval on it beats 14 prior training-free open-vocabulary segmentation methods across ten benchmarks.

  8. FA-Seg: A Fast and Accurate Diffusion-Based Method for Open-Vocabulary Segmentation

    cs.CV 2025-06 unverdicted novelty 5.0 of 10

    FA-Seg delivers state-of-the-art training-free open-vocabulary segmentation performance (43.8% mIoU average) on standard benchmarks by extracting and refining attention from a single forward pass of a pretrained diffu...

  9. Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution

    cs.CV 2026-07 reject novelty 4.0 of 10

    DDR-SR routes each real-world low-resolution image to one of two diffusion experts based on a high-frequency-loss difficulty score, using a low-compression VAE for hard images and a high-compression VAE for easy image...

Pith tools