CatchPhrase improves audio-to-image generation by enriching weak class labels with LLM- and audio-caption-based prompts, filtering and retrieving the best prompt per clip, and training a mapping adapter with contrastive learning.
VerbDiff: Text-Only Diffusion Models with Enhanced Interaction Awareness
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Recent large-scale text-to-image diffusion models generate photorealistic images but often struggle to accurately depict interactions between humans and objects due to their limited ability to differentiate various interaction words. In this work, we propose VerbDiff to address the challenge of capturing nuanced interactions within text-to-image diffusion models. VerbDiff is a novel text-to-image generation model that weakens the bias between interaction words and objects, enhancing the understanding of interactions. Specifically, we disentangle various interaction words from frequency-based anchor words and leverage localized interaction regions from generated images to help the model better capture semantics in distinctive words without extra conditions. Our approach enables the model to accurately understand the intended interaction between humans and objects, producing high-quality images with accurate interactions aligned with specified verbs. Extensive experiments on the HICO-DET dataset demonstrate the effectiveness of our method compared to previous approaches.
fields
cs.MM 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
CatchPhrase: EXPrompt-Guided Encoder Adaptation for Audio-to-Image Generation
CatchPhrase improves audio-to-image generation by enriching weak class labels with LLM- and audio-caption-based prompts, filtering and retrieving the best prompt per clip, and training a mapping adapter with contrastive learning.