Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion
read the original abstract
Text-to-image generation is a significant domain in modern computer vision and has achieved substantial improvements through the evolution of generative architectures. Among these, there are diffusion-based models that have demonstrated essential quality enhancements. These models are generally split into two categories: pixel-level and latent-level approaches. We present Kandinsky1, a novel exploration of latent diffusion architecture, combining the principles of the image prior models with latent diffusion techniques. The image prior model is trained separately to map text embeddings to image embeddings of CLIP. Another distinct feature of the proposed model is the modified MoVQ implementation, which serves as the image autoencoder component. Overall, the designed model contains 3.3B parameters. We also deployed a user-friendly demo system that supports diverse generative modes such as text-to-image generation, image fusion, text and image fusion, image variations generation, and text-guided inpainting/outpainting. Additionally, we released the source code and checkpoints for the Kandinsky models. Experimental evaluations demonstrate a FID score of 8.03 on the COCO-30K dataset, marking our model as the top open-source performer in terms of measurable image generation quality.
This paper has not been read by Pith yet.
Forward citations
Cited by 7 Pith papers
-
Manifold-Aligned Guided Integrated Gradients for Reliable Feature Attribution
MA-GIG uses VAE latent space to align Integrated Gradients paths with the data manifold for more faithful feature attributions in deep neural networks.
-
Graph-PiT: Enhancing Structural Coherence in Part-Based Image Synthesis via Graph Priors
Graph-PiT adds graph priors and a hierarchical GNN to part-based image synthesis to enforce relational constraints and improve structural coherence over vanilla PiT.
-
CSF: Black-box Fingerprinting via Compositional Semantics for Text-to-Image Models
CSF is the first black-box method to attribute fine-tuned text-to-image models to original lineages via compositional semantic probes and Bayesian decisions across multiple model families.
-
Manifold-Aligned Guided Integrated Gradients for Reliable Feature Attribution
MA-GIG improves Integrated Gradients by performing path integration in the latent space of a pre-trained VAE so that decoded points remain closer to the learned data manifold and reduce off-manifold gradient noise.
-
TIQA: Human-Aligned Perceptual Text Quality Assessment in Generated Images
TIQA introduces datasets and a model that predict human perceptual quality of rendered text in AI images, achieving PLCC 0.942 on crops and improving selected image text quality by 0.36 MOS.
-
Dissect and Prune: Enhancing Robustness in AI-Generated Image Detection
DEAR prunes channel features whose activations align strongly with inpaint masks, retaining only those capturing genuine generative artifacts to improve robustness against post-processing and unseen generators.
-
SSAFE: Simple and Strong AI-Generated Image Detection via Frozen Vision Encoders
Frozen multimodal encoders enable robust AI-generated image detection via linear classification on a 10K-image curated training set that improves generalization over larger datasets.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.