REVIEW 5 cited by
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Text-to-image diffusion models have made significant advances in generating and editing high-quality images. As a result, numerous approaches have explored the ability of diffusion model features to understand and process single images for downstream tasks, e.g., classification, semantic segmentation, and stylization. However, significantly less is known about what these features reveal across multiple, different images and objects. In this work, we exploit Stable Diffusion (SD) features for semantic and dense correspondence and discover that with simple post-processing, SD features can perform quantitatively similar to SOTA representations. Interestingly, the qualitative analysis reveals that SD features have very different properties compared to existing representation learning features, such as the recently released DINOv2: while DINOv2 provides sparse but accurate matches, SD features provide high-quality spatial information but sometimes inaccurate semantic matches. We demonstrate that a simple fusion of these two features works surprisingly well, and a zero-shot evaluation using nearest neighbors on these fused features provides a significant performance gain over state-of-the-art methods on benchmark datasets, e.g., SPair-71k, PF-Pascal, and TSS. We also show that these correspondences can enable interesting applications such as instance swapping in two images.
Forward citations
Cited by 5 Pith papers
-
MOSAIC: Multi-Subject Personalized Generation via Correspondence-Aware Alignment and Disentanglement
MOSAIC improves multi-subject personalized image generation by supervising attention maps with semantic point correspondences and a disentanglement loss, and introduces the SemAlign-MS dataset for training.
-
Hierarchical Concept-to-Appearance Guidance for Multi-Subject Image Generation
A diffusion-transformer framework with VLM-grounded masked attention and VAE dropout improves identity and prompt fidelity for multi-subject image generation.
-
Self-Supervised Spatial Correspondence Across Modalities
Dense pixel-level correspondence across visual modalities (RGB, depth, thermal, sketch, style) can be learned from unlabeled videos via cycle-consistent contrastive random walks.
-
Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution
DDR-SR routes each real-world low-resolution image to one of two diffusion experts based on a high-frequency-loss difficulty score, using a low-compression VAE for hard images and a high-compression VAE for easy image...
-
Towards Robust Semantic Correspondence: A Benchmark and Insights
The abstract promises an adverse-condition benchmark for semantic correspondence, yet the full text is a GRB magnetar analysis, so the claimed benchmark is unverifiable.
Discussion (0). Sign in to comment.