REVIEW 5 cited by
Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulation
read the original abstract
Diffusion models, while increasingly adept at generating realistic images, are notably hindered by hallucinations -- unrealistic or incorrect features inconsistent with the trained data distribution. In this work, we propose Adaptive Attention Modulation (AAM), a novel approach to mitigate hallucinations by analyzing and modulating the self-attention mechanism in diffusion models. We hypothesize that self-attention during early denoising steps may inadvertently amplify or suppress features, contributing to hallucinations. To counter this, AAM introduces a temperature scaling mechanism within the softmax operation of the self-attention layers, dynamically modulating the attention distribution during inference. Additionally, AAM employs a masked perturbation technique to disrupt early-stage noise that may otherwise propagate into later stages as hallucinations. Extensive experiments demonstrate that AAM effectively reduces hallucinatory artifacts, enhancing both the fidelity and reliability of generated images. For instance, the proposed approach improves the FID score by 20.8% and reduces the percentage of hallucinated images by 12.9% (in absolute terms) on the Hands dataset.
Forward citations
Cited by 5 Pith papers
-
IMPLICITSTAINER: Resolution Agnostic Data-Efficient Virtual Staining Using Neural Implicit Functions
Neural implicit functions enable resolution-agnostic, deterministic virtual staining from H&E to IHC images with SOTA results and better low-data performance than patch-based GAN or diffusion methods.
-
Score-Control for Hallucination Reduction in Diffusion Models
VSM modulates the score Jacobian using variance guidance to reduce hallucinations in diffusion models by up to 25% on synthetic and real datasets while preserving fidelity and diversity.
-
Geometric Decoupling: Diagnosing the Structural Instability of Latent
Latent diffusion models exhibit geometric decoupling where curvature in out-of-distribution generation is misallocated to unstable semantic boundaries instead of image details, identifying geometric hotspots as the st...
-
Unified Backbone Refinement for Diffusion Models via Internal-Latent Analysis
Early abrupt deviations in deep diffusion latents track artifacts; EMA detection plus backbone-specific suppression (DUNE) reduces them without retraining.
-
Enhancing Layer Interaction Using Key-Correlated Layer Attention
KCLA is a linear-complexity layer attention mechanism that exploits high key cosine similarity to preserve dynamic updates and long-range cross-layer connections.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.