pith. sign in

arxiv: 2502.04320 · v2 · pith:47KEQGKBnew · submitted 2025-02-06 · 💻 cs.CV · cs.LG

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features

Alec Helbling , Tuna Han Salih Meral , Ben Hoover , Pinar Yanardag , Duen Horng Chau This is my paper
classification 💻 cs.CV cs.LG
keywords conceptattentionattentionhighlylayersmapsdiffusionditseven
0
0 comments X
read the original abstract

Do the rich representations of multi-modal diffusion transformers (DiTs) exhibit unique properties that enhance their interpretability? We introduce ConceptAttention, a novel method that leverages the expressive power of DiT attention layers to generate high-quality saliency maps that precisely locate textual concepts within images. Without requiring additional training, ConceptAttention repurposes the parameters of DiT attention layers to produce highly contextualized concept embeddings, contributing the major discovery that performing linear projections in the output space of DiT attention layers yields significantly sharper saliency maps compared to commonly used cross-attention maps. ConceptAttention even achieves state-of-the-art performance on zero-shot image segmentation benchmarks, outperforming 15 other zero-shot interpretability methods on the ImageNet-Segmentation dataset. ConceptAttention works for popular image models and even seamlessly generalizes to video generation. Our work contributes the first evidence that the representations of multi-modal DiTs are highly transferable to vision tasks like segmentation.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. What Concepts Lie Within? Detecting and Suppressing Risky Content in Diffusion Transformers

    cs.CV 2026-05 unverdicted novelty 7.0

    A method using attention head vectors detects and suppresses risky content generation in Diffusion Transformers at inference time.

  2. Unified Safe In-context Image Generation in Multimodal Diffusion Transformers via Restricting Unsafe Information Flows

    cs.CV 2026-06 unverdicted novelty 6.0

    UVR is a training-free framework that uses attention modulation based on identified information flow stages in multimodal DiT attention to erase unsafe semantics in image synthesis and editing at 91% and 77% rates whi...

  3. TS-Attn: Temporal-wise Separable Attention for Multi-Event Video Generation

    cs.CV 2026-04 unverdicted novelty 6.0

    TS-Attn dynamically separates and rearranges attention in existing text-to-video models to improve temporal consistency and prompt adherence for videos with multiple sequential actions.

  4. The Cow of Rembrandt - Analyzing Artistic Prompt Interpretation in Text-to-Image Models

    cs.CV 2025-07 unverdicted novelty 6.0

    Text-to-image diffusion models exhibit varying degrees of emergent content-style separation in art generation, with content tokens primarily influencing object regions and style tokens affecting backgrounds and textures.

  5. Consistency Regularised Gradient Flows for Inverse Problems

    stat.ML 2026-05 unverdicted novelty 5.0

    A consistency-regularized Euclidean-Wasserstein-2 gradient flow performs joint posterior sampling and prompt optimization in latent space for efficient low-NFE inverse problem solving with diffusion models.

  6. FAIR_XAI: Improving Multimodal Foundation Model Fairness via Explainability for Wellbeing Assessment

    cs.AI 2026-04 unverdicted novelty 4.0

    Vision-language models for wellbeing assessment exhibit dataset-dependent performance and demographic biases, with explainability interventions providing inconsistent fairness gains at potential accuracy costs.