pith. sign in

arxiv: 2301.07093 · v2 · pith:KY3UCKHSnew · submitted 2023-01-17 · 💻 cs.CV · cs.AI· cs.CL· cs.GR· cs.LG

GLIGEN: Open-Set Grounded Text-to-Image Generation

Yuheng Li , Haotian Liu , Qingyang Wu , Fangzhou Mu , Jianwei Yang , Jianfeng Gao , Chunyuan Li , Yong Jae Lee This is my paper
classification 💻 cs.CV cs.AIcs.CLcs.GRcs.LG
keywords generationgligengroundingtext-to-imagediffusionexistinggroundedinputs
0
0 comments X
read the original abstract

Large-scale text-to-image diffusion models have made amazing advances. However, the status quo is to use text input alone, which can impede controllability. In this work, we propose GLIGEN, Grounded-Language-to-Image Generation, a novel approach that builds upon and extends the functionality of existing pre-trained text-to-image diffusion models by enabling them to also be conditioned on grounding inputs. To preserve the vast concept knowledge of the pre-trained model, we freeze all of its weights and inject the grounding information into new trainable layers via a gated mechanism. Our model achieves open-world grounded text2img generation with caption and bounding box condition inputs, and the grounding ability generalizes well to novel spatial configurations and concepts. GLIGEN's zero-shot performance on COCO and LVIS outperforms that of existing supervised layout-to-image baselines by a large margin.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Visual Instruction Tuning

    cs.CV 2023-04 unverdicted novelty 7.0

    LLaVA is trained on GPT-4 generated visual instruction data to achieve 85.1% relative performance to GPT-4 on synthetic multimodal tasks and 92.53% accuracy on Science QA.

  2. MaskAttn-SDXL: Controllable Region-Level Text-To-Image Generation

    cs.CV 2025-09 unverdicted novelty 6.0

    MaskAttn-SDXL adds token-conditioned spatial gating to SDXL cross-attention to sparsify irrelevant token-to-location bindings and improve region-level controllability without retraining or inference edits.

  3. PhyDrawGen: Physically Grounded Diagram Generation from Natural Language

    cs.AI 2026-05 unverdicted novelty 5.0

    PhyDrawGen is a neuro-symbolic pipeline that extracts typed scene graphs via LLM, converts them to physically constrained PSLGs via deterministic solver, and refines via fine-tuned Qwen-VL, claiming superior performan...