Pith. sign in

REVIEW 10 cited by

Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2211.11138 v1 pith:4CQ2B7MJ submitted 2022-11-21 cs.CV

Ling Yang , Zhilin Huang , Yang Song , Shenda Hong , Guohao Li , Wentao Zhang , Bin Cui , Bernard Ghanem
show 1 more author
Ming-Hsuan Yang
This is my paper · ORCID
classification cs.CV
keywords sceneimagesgraphsembeddingsgraphlossmaskedsgdiff
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Generating images from graph-structured inputs, such as scene graphs, is uniquely challenging due to the difficulty of aligning nodes and connections in graphs with objects and their relations in images. Most existing methods address this challenge by using scene layouts, which are image-like representations of scene graphs designed to capture the coarse structures of scene images. Because scene layouts are manually crafted, the alignment with images may not be fully optimized, causing suboptimal compliance between the generated images and the original scene graphs. To tackle this issue, we propose to learn scene graph embeddings by directly optimizing their alignment with images. Specifically, we pre-train an encoder to extract both global and local information from scene graphs that are predictive of the corresponding images, relying on two loss functions: masked autoencoding loss and contrastive loss. The former trains embeddings by reconstructing randomly masked image regions, while the latter trains embeddings to discriminate between compliant and non-compliant images according to the scene graph. Given these embeddings, we build a latent diffusion model to generate images from scene graphs. The resulting method, called SGDiff, allows for the semantic manipulation of generated images by modifying scene graph nodes and connections. On the Visual Genome and COCO-Stuff datasets, we demonstrate that SGDiff outperforms state-of-the-art methods, as measured by both the Inception Score and Fr\'echet Inception Distance (FID) metrics. We will release our source code and trained models at https://github.com/YangLing0818/SGDiff.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LaRender: Training-Free Occlusion Control in Image Generation via Latent Rendering

    cs.CV 2025-08 conditional novelty 7.0 of 10

    LaRender replaces cross-attention layers in a pretrained diffusion model with a latent alpha-compositing operation that renders object features in occlusion order, giving training-free occlusion control.

  2. FMOPF: Latent Flow Matching with Constraint-Aware Interaction Priors for AC Optimal Power Flow

    cs.LG 2026-07 conditional novelty 6.0 of 10

    FMOPF uses latent flow matching plus a constraint-aware interaction prior to sample feasible near-optimal AC-OPF solutions, and reports the first generative-OPF scaling to 300 buses — but its feasibility claim is not ...

  3. SurGrID: Controllable Surgical Simulation via Scene Graph to Image Diffusion

    cs.CV 2025-02 conditional novelty 6.0 of 10

    SurGrID uses scene graphs to control diffusion-based generation of realistic surgical images, improving fidelity and graph coherence over text- and mask-conditioned diffusion models.

  4. G2PDiffusion: Cross-Species Genotype-to-Phenotype Prediction via Evolutionary Diffusion

    cs.LG 2025-02 conditional novelty 6.0 of 10

    G2PDiffusion is a diffusion model that generates morphological insect images from DNA barcodes, evolutionary alignments, and latitude/longitude, with inference-time guidance toward DNA-image alignment.

  5. Heterogeneous Image GNN: Graph-Conditioned Diffusion for Image Synthesis

    cs.CV 2025-02 conditional novelty 6.0 of 10

    A graph-conditioned diffusion method (HIG) couples image nodes with conditioning nodes via a magnitude-preserving GNN, reporting FID 8.79 on Visual Genome layout-to-image and 11.42 on COCO-stuff mask-to-image at 512x512.

  6. LAION-SG: An Enhanced Large-Scale Dataset for Training Complex Image-Text Models with Structural Annotations

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A 540,005-image dataset with GPT-4o-produced scene graph annotations improves compositional text-to-image generation when used to fine-tune SDXL-based models.

  7. What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generation

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Scene-Bench introduces MegaSG (1M scene graphs), SGScore (LLM-based object/relation recall), and a feedback loop that measurably raises factual consistency in scene graph-to-image generation.

  8. SATURN: Autoregressive Image Generation Guided by Scene Graphs

    cs.CV 2025-08 conditional novelty 5.0 of 10

    SATURN serializes scene graphs into salience-ordered captions, enabling a frozen CLIP encoder and a fine-tuned VAR transformer to generate images from graphs, with reported FID falling from 56.45 to 21.62 on Visual Genome.

  9. Inversion-DPO: Precise and Efficient Post-Training for Diffusion Models

    cs.CV 2025-07 reject novelty 5.0 of 10

    Inversion-DPO uses DDIM inversion to convert winning and losing images into noise trajectories, yielding a simpler DPO loss for diffusion model alignment that trains faster and improves text-to-image and compositional...

  10. From Data to Modeling: Fully Open-vocabulary Scene Graph Generation

    cs.CV 2025-05 conditional novelty 4.0 of 10

    OvSGTR jointly predicts unseen objects and relationships in scene graphs using a DETR-like transformer, relation-aware pre-training, and knowledge distillation, achieving state-of-the-art results on VG150 and GQA200.

Pith tools