InstanceGen: Image Generation with Instance-level Instructions

Etai Sella; Hadar Averbuch-Elor; Yanir Kleiman

REVIEW 4 major objections 5 minor 11 references

InstanceGen: Image Generation with Instance-level Instructions

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read InstanceGen couples attention-derived instance segmentation with LLM-written per-segment instructions to make text-to-image models obey object counts, attributes, and spatial relations without retraining.

desk verdict A well-engineered inference-time method for compositional prompts that is worth a serious referee, but its headline SOTA claim rests on a self-built GPT-4o-judged benchmark with no error bars or human validation. read the letter →

arxiv 2505.05678 v3 pith:5MLNDMTZ submitted 2025-05-08 cs.CV

Etai Sella , Yanir Kleiman , Hadar Averbuch-Elor This is my paper · ORCID

classification cs.CV

keywords text-to-imagegenerationdiffusionmodelsinstance-levelattributesattentionmapslayoutguidancepromptfidelityobjectcountingCompoundPromptsbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a text-to-image model can follow complex prompts—exact object counts, attributes tied to specific instances, and spatial relations—without retraining and without user-supplied bounding boxes, by using its own first draft as the structural guide. The proposed pipeline, InstanceGen, generates an initial image, extracts an instance-level segmentation from the cross-attention maps (local maxima serve as anchor points for each object instance), and asks an LLM to assign each segment an object label and optional attributes. It then regenerates the image while applying attention losses, attention masking, and a background-preserving term so the final image realizes the assignment while staying close to the initial composition. On the introduced CompoundPrompts benchmark, the paper reports 0.60 VQA Accuracy against 0.43 for the underlying Emu model, with the largest gains on prompts that compound attributes and spatial arrangements.

What carries the argument

The load-bearing object is the anchor point: a local maximum of the aggregated cross-attention map for an object word, taken as evidence that one instance of that object occupies that image region. Anchor points convert the base model's own attention signal into segmentation seeds, letting Mask R-CNN and SAM2 masks be filtered, merged, and assigned to object words and attributes. Everything downstream—LLM assignment, attention losses, and attention masking—assumes these anchor points localize every requested instance.

What would settle it

Run the pipeline on prompts with many small, partially occluded instances, such as ten birds on a branch with several behind leaves, and record how often the number of anchor points in the initial image falls below the requested count across many seeds. A setup with a high miss rate would show that the anchor-point premise, and therefore the layout quality, does not hold in the regime the method targets.

Watch

Extended reading notes

Core claim

The central claim is that a fine-grained structural signal with instance-level semantics can be obtained from the base diffusion model itself and fed back into that same model to fix complex-prompt adherence. InstanceGen first renders an initial image and aggregates cross-attention maps; local maxima of those maps serve as anchor points for Mask R-CNN and SAM2 segments, producing a layout of instance masks. An LLM, given the parsed prompt and per-segment attention scores, assigns each segment an object word and any instance attributes, deleting segments that do not fit. A second diffusion pass then optimizes latents with a weighted object-attention binary cross-entropy loss, an attribute cross-entropy loss, attention masking to stop semantic leakage, and a background-preserving regularization, so the output stays close to the initial composition while conforming to the layout instructions. The paper reports that this yields the strongest results among tested baselines on CompoundPrompts, DrawBench, and GenEval, with the largest gains on attribute and spatial tiers.

Load-bearing premise

Every requested object instance must leave at least one detectable peak in the aggregated attention maps of the initial image; if instances overlap heavily or are too small to form a peak, segmentation misses them and the only fallbacks are repeated seeds or copying masks, which reduce layout quality.

Editorial extensions

If this is right

Users can obtain fine-grained multi-object layouts from a single text prompt without drawing boxes, because the base model itself supplies the layout.
Instance-level attributes and spatial relations, the weakest spot of current text-to-image models, become addressable at inference time and can be bolted onto any diffusion model that exposes cross-attention maps.
Attention masking is doing much of the spatial work: removing it drops Tier C accuracy from 0.60 to 0.32 on the paper's ablation subset.
The background-preservation loss trades a little prompt fidelity for much better composition quality, per the ablation's VQA Similarity results.
CompoundPrompts gives the community a three-tier stress test so future methods can be compared separately on counting, attribute binding, and spatial fidelity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because InstanceGen never retrains the base model, its ceiling should rise as base diffusion models improve; better attention maps would make anchor-point detection more reliable and reduce the need for seed search and mask copying.
The anchor-point premise implies a bottleneck at attention resolution: scenes with many small or overlapping instances may exhaust the supply of distinct peaks, so a direct test of that regime would map the method's practical limits.
The paper's front/back failure hints that injecting depth or occlusion ordering into the LLM assignment prompt could resolve spatial instructions that 2D segment coordinates cannot express.
CompoundPrompts-style evaluation with an MLLM judge could become a reusable stress test for compositional fidelity, but its scores will be sensitive to the choice of judge model and question wording.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

InstanceGen is a well-engineered inference-time method for compositional text-to-image prompts. The new part is the close coupling of attention-derived instance masks from an initial generation with LLM assignment of per-segment objects and attributes, followed by attention-based optimization of the latent. That combination is not in CountGen or SLD, and the paper makes a credible case that fine-grained structural anchors beat bounding boxes. The ablations are real and informative, and Section C.5 honestly reports failure modes. Credit where it is due: this is a serious piece of work.

The soft spots are in the evidence for the headline claim. Table 2, the core result, is VQA Accuracy scored by GPT-4o on CompoundPrompts, a benchmark the authors constructed. There are no error bars, no human validation of the judge, and no public release of the benchmark or code. The Tier B/C gaps (0.57 vs 0.40; 0.50 vs 0.30) could plausibly be inflated by a systematic 'yes' bias in GPT-4o for template-consistent or spatially tidy images; the single-point estimates cannot rule that out. GenEval is a genuine independent check, but its categories only partially overlap with the combined attribute-plus-spatial case that motivates the method.

The anchor-point assumption is a real secondary risk, but the paper has fallbacks and the qualitative results show the pipeline often works despite imperfect segmentation. The larger issue is the overclaim: “adhere to all parts of the text prompt” does not follow from a 0.60 VQA Accuracy, which means 40% of prompts still fail some question. The abstract and conclusion should be tempered.

This paper deserves a serious referee: the method is novel enough and the results promising enough to justify referee time. The referee should require release of the benchmark and evaluation harness, human-agreement numbers for the GPT-4o judge, error bars, and a revised claims section. With those changes, it would be a solid contribution. For now, treat the SOTA claim as conditional.

Referee Report

4 major / 5 minor

Summary. InstanceGen is a training-free, inference-time method for text-to-image generation aimed at prompts that combine object counts, instance-level attributes, and spatial relations. The pipeline first generates an initial image with a pretrained diffusion model (Emu), extracts anchor points from aggregated cross-attention maps, and constructs a fine-grained instance segmentation by combining Mask R-CNN with SAM2. A Llama-based LLM parses the prompt and assigns each segment an object label plus optional instance attributes. The final image is produced by a conditioned diffusion stage that optimizes latents against cross-attention losses, applies attention masking, and regularizes the background against the initial image. The paper also introduces CompoundPrompts, a 540-prompt benchmark with three difficulty tiers, and reports experiments on DrawBench, GenEval, and CompoundPrompts, claiming state-of-the-art performance on the new benchmark.

Significance. If the empirical claims hold, InstanceGen is a meaningful step for a genuine failure mode of text-to-image models: it replaces coarse LLM-generated bounding boxes with attention-derived instance masks, uses per-segment text conditioning, and explicitly targets counting, attribute-binding, and spatial errors. The independent GenEval results (Table 6) and DrawBench results (Table 1) are real points in the paper's favor, as are the detailed implementation hyperparameters and the candid limitations section. The central caveat is that the headline comparison on CompoundPrompts is not yet independently verifiable: the benchmark is self-constructed, GPT-4o is both part of the pipeline and the judge, and no human validation or uncertainty quantification is reported. The anchor-point assumption is a secondary but real correctness risk. These issues are addressable with additional experiments and a corrected presentation, so the contribution is potentially solid.

major comments (4)

[Section 4.3, Table 2] The central SOTA claim rests on VQA Accuracy numbers produced by GPT-4o answering yes/no questions on CompoundPrompts, a benchmark constructed by the same authors, with no human validation of the judge, no confidence intervals or significance tests, and no public release of the benchmark or evaluation harness. A systematic evaluator bias, for example toward spatially tidy layouts or toward template-consistent compositions, could inflate the reported 0.60 versus 0.43 gap. Please add a human-validated subset with inter-annotator agreement, report per-prompt variance and paired significance tests for the main comparisons, and release the benchmark and evaluation code; without these, the phrase "significantly surpasses" in Section 4.3 is stronger than the evidence supports.
[Section 3.3.2, Eq. (3)] Equation (3) as printed multiplies the cross-attention value by delta=-1.5 when m_i=1, i.e., inside the segment assigned to the word, and leaves the value unchanged when m_i=0, i.e., outside the segment. This is the opposite of the stated goal of masking attention in segments to which the word is not assigned, and negative attention weights are not explained anywhere in the text. Since the ablation in Table 3 shows that attention masking has a strong effect on Tier C performance, the implementation of this component is load-bearing. Please correct the condition (presumably m_i=0) or, if the negative multiplier is intentional, give a precise account of how negative attention values are used by the diffusion backbone.
[Section 3.1.2 and Supplementary B.1.3] The completeness of the proposed layout generation relies on the empirical claim that "every object instance produces at least one such anchor point" in the aggregated cross-attention map, but no failure-rate statistics are reported. When this assumption fails, the method falls back to seed search and random instance copying, which the paper itself describes as degrading layout quality and visual plausibility. Please report, on a sample of CompoundPrompts prompts, the fraction of object instances that yield no anchor point, the success rate of the seed search, and the frequency with which instance copying is needed, together with its effect on final VQA Accuracy.
[Section 4.4, Table 3] The ablation study is difficult to interpret as reported. The "Ours" row (0.76/0.72/0.60/0.69) does not match the full-benchmark result in Table 2 (0.72/0.57/0.50/0.60), because a "representative subset" is used, but the subset is not defined. In addition, "w/o Seed Search" improves Tier A from 0.76 to 0.84 and leaves VQA Sim unchanged at 0.90, which undermines the text's conclusion that this component is "not the primary factor driving our increased performance." Without confidence intervals or a significance test, the reader cannot distinguish real ablation effects from noise. Please report ablations on the identical full benchmark with variance estimates.

minor comments (5)

[Abstract and Section 1] The phrase "adhere to all parts of the text prompt" overstates the quantitative results, since the full-benchmark VQA Accuracy in Table 2 is 0.60 and Fig. 9 documents failures on each tier; "better adherence" would be more precise.
[Supplementary C.3] The supplementary results folder is described as containing images for "all150 prompts" in CompoundPrompts, while the main text says the benchmark contains 540 prompts; please reconcile this discrepancy.
[Section 4.3, Table 1] The DrawBench evaluation is described only as using "the procedure used in [Phung et al. 2024]"; please specify which prompt subset was used and how the counting and spatial accuracies were computed, so that the numbers in Table 1 can be reproduced.
[Fig. 7 caption] The caption states that the initial images are "post our robust initialization stage," but the seed search stage only selects a seed and does not edit the image; please clarify what post-processing was applied to the displayed initial images.
[Supplementary B.1.3] The json layout summary labels attention scores as "object_probabilities" and "attribute_probabilities," even though they are averaged cross-attention values and not probabilities; please rename these fields to avoid misleading readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: InstanceGen's claims are empirically evaluated, and the self-built benchmark and GPT-4o judge issue affects evaluation confidence, not derivation circularity.

full rationale

InstanceGen is an inference-time engineering pipeline rather than a first-principles derivation. The claimed capability—adherence to complex prompts with counts, instance-level attributes, and spatial relations—is evaluated empirically on rendered images, using GPT-4o yes/no questions on CompoundPrompts plus external benchmarks DrawBench and GenEval. No equation in the paper defines the output in terms of the evaluation metric, and no fitted parameter is renamed as a prediction. The anchor-point assumption in Section 3.1.2 is an empirical observation with explicit fallbacks (seed search and instance copying), so even if the assumption fails, the method does not become circular. The attention maps, layout extraction, LLM assignment, and attention losses are components of the method, not circular definitions: the final VQA check is performed by a separate MLLM on the rendered image, not on the layout or attention maps. The co-authored citation (Patashnik et al. 2023) is used only as contrast in related work and is not load-bearing. The main validity risks—self-built benchmark, no human validation of the judge, no error bars, and no public release—bear on evaluation robustness and reproducibility, not on circularity of the derivation. The paper is self-contained against external benchmarks, so the honest finding is no circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The ledger contains no latent physical or conceptual entities beyond the CompoundPrompts benchmark, which is a dataset contribution rather than an invented entity. All free parameters are manually chosen constants or scheduling choices; the axioms are domain assumptions about attention localization, LLM reliability, segmentation quality, MLLM evaluation validity, and access to Emu internals.

free parameters (6)

Attention-masking constant delta = -1.5
Chosen to invert cross-attention in unassigned regions; no sensitivity analysis reported.
Attribute loss weight lambda_att = 0.8
Fixed in all experiments; no ablation or sensitivity shown.
Background regularization weight lambda_bg = 0.3
Fixed; ablating L_bg changes VQA Similarity, but no tuning analysis is reported.
Object loss foreground pixel weight lambda_i = 1.5 inside mask, 1.0 outside
Hand-set to emphasize foreground objects; no sensitivity study.
Optimization schedule and learning rate = start 0, end 20, 15/15/5 iterations at t=0/5/10, LR 0.015
Empirically chosen; not swept in the paper.
Segmentation thresholds = 0.33% max mask area, 66.6% overlap merge, 30 px min, 2 px margin
Hand-set post-processing thresholds with no reported robustness analysis.

assumptions (5)

domain assumption Cross-attention maps of the base diffusion model localize each object instance with at least one anchor point.
Stated in Section 3.1.2 as an empirical observation without failure statistics; drives segmentation reliability.
domain assumption The LLM (Llama 3.3) can parse prompts and assign segments correctly from JSON layout summaries with attention scores.
Section 3.2 relies on this; assignment errors are corrected by a negative-example loop, which softens but does not remove the assumption.
domain assumption Off-the-shelf Mask R-CNN and SAM2 provide usable segmentations when combined with attention anchors.
Section 3.1.2 and Supplementary B.1.3; merging heuristics are tuned by hand.
domain assumption GPT-4o yes/no judgments of image-prompt alignment are a valid proxy for human fidelity.
Section 4.2 uses GPT-4o for VQA Accuracy without a human correlation study on CompoundPrompts.
domain assumption The base Emu model's latents and attention are accessible and optimizable during inference.
Supplementary B.1.1 and Section 3.3; the method is tied to that checkpoint.

how reviews work

0 comments

Cite this review

Pith. "Pith review of InstanceGen: Image Generation with Instance-level Instructions." pith.science (2026). https://pith.science/paper/5MLNDMTZ

@misc{pith2026250505678,
  author       = {Pith},
  title        = {Pith review of: InstanceGen: Image Generation with Instance-level Instructions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5MLNDMTZ}},
  note         = {Machine review of arXiv:2505.05678}
}

read the original abstract

Despite rapid advancements in the capabilities of generative models, pretrained text-to-image models still struggle in capturing the semantics conveyed by complex prompts that compound multiple objects and instance-level attributes. Consequently, we are witnessing growing interests in integrating additional structural constraints, typically in the form of coarse bounding boxes, to better guide the generation process in such challenging cases. In this work, we take the idea of structural guidance a step further by making the observation that contemporary image generation models can directly provide a plausible fine-grained structural initialization. We propose a technique that couples this image-based structural guidance with LLM-based instance-level instructions, yielding output images that adhere to all parts of the text prompt, including object counts, instance-level attributes, and spatial relations between instances.

Figures

Figures reproduced from arXiv: 2505.05678 by the authors.

**Figure 1.** We present a method that incorporates instance-level instructions in image generation, capable of adhering to prompts with various [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Method Overview. Given a text prompt (possibly) containing object counts, instance-level attributes, and spatial relations, our approach combines image-based and text-based components (visualized in pink and blue boxes above) for generating an output image (illustrated on the right). We first generate an initial image using a pretrained text-to-image diffusion model. Given the image and attention information from th… view at source ↗

**Figure 3.** Instance Layout Generation. (a) The initial image generated by a pretrained text-to-image model. (b) The aggregated cross attention for all foreground words, with anchor points marked by green dots. (c) Initial segmentation after discarding segments without anchor points. (d) Final segmentation after adding segments for each unassigned anchor point. the segment should not be part of the foreground of the final image… view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Instance assignment inputs. The input given to the LLM for instance assignment contains in-context examples, the parsed prompt, and the visual layout of each segment. 3.3 Assignment Conditioned Image Generation Once assignments have been made, we omit the segments mar…

**Figure 5.** Figure 5: Qualitative results from the CompoundPrompts benchmark. We present results for two unique prompts from the CompoundPrompts benchmark, presenting results for all three tiers (A, B and C) for each prompt. performance. In our assessment, this shows that the semantic leak…

**Figure 6.** Figure 6: Extended qualitative comparison. Additional results for prompts sampled from CompoundPrompts, along with VQA Acc. metric evaluation. For each tier, we consider an image correct (✓) if it receives positive answers from an MLLM (GPT4o) when prompted with all questions be…

**Figure 7.** Figure 7: Assorted Tier C prompt results: Initial Images shown here are post our robust initialization stage [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Qualitative ablations. We present compare results produced by our method when ablating three different components: cross attention losses (“w/o L𝑜𝑏 𝑗 & L𝑎𝑡𝑡 ”), attention masking (“w/o Attn. Masking”) and composition preserving regularization (“w/o L𝑏𝑔”) Self-correctin…

**Figure 9.** Figure 9: Limitations. We present three failure cases of our method. In the first example (top row) our method wrongfully creates a gap in the pillow stack. In the second (middle row), two pairs of cats are merged into one. In the final example (bottom row) our wrongfully places…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 5 canonical work pages

[1]

There are 3 total clusters in the layout dict,we are required to assign one cluster as 'porcupine', one clus- ter as 'squirrel' and onecluster as 'raccoon' meaning we do not have to delete any clusters

work page
[2]

The 'squirrel' object also does not require spa- tial arrangment

The 'porcupine' object does not require spatial arra- ngment. The 'squirrel' object also does not require spa- tial arrangment. The 'raccoon' object also does not req- uire spatial arrangment

work page
[3]

arXiv preprint arXiv:2407.21783 (2024)

The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al . 2024. Scal- ing rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Lea...

arXiv 2024
[5]

In Proceedings of the IEEE/CVF International Conference on Computer Vision

Localizing object-level shape variations with text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 23051– 23061. William Peebles and Saining Xie. 2023. Scalable Diffusion Models with Transformers. arXiv:2212.09748 [cs.CV] https://arxiv.org/abs/2212.09748 Quynh Phung, Songwei Ge, and Jia-Bin Huang. 20...

arXiv 2023
[6]

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10684–10695. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022. Photorealistic text-to...

work page 2022
[7]

Six meerkats standing watch in the Savannah

Instancediffusion: Instance-level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 6232–6242. Tsung-Han Wu, Long Lian, Joseph E Gonzalez, Boyi Li, and Trevor Darrell. 2024a. Self- correcting llm-controlled diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and P...

arXiv 2024
[10]

1" with 'raccoon' probability (0.75). This leaves us with cluster

'porcupine' has the lowest desired quantity (1) so we start by assiging it to the cluster with the highest 'porcupine' probability, which is cluster "3" with 'por- cupine' probability (0.71). We then assign the 'raccoon' InstanceGen: Image Generation with Instance-level Instructions • 13 object to the cluster with the highest 'raccoon' probab- ility, whic...

work page
[11]

3") as req- uired. We have set one cluster as 'squirrel' (

We have set one cluster as 'porcupine' ("3") as req- uired. We have set one cluster as 'squirrel' ("2") as required. We have set one cluster as 'raccoon' ("1") as required. We have set one 'squirrel' cluster as 'holdi- ng nut ' ("2") as required. ** ASSIGNMENTS ** { "1": { "object": "raccoon", "attributes": [] }, "2": { "object": "squirrel", "attributes":...

work page

Show all 11 references

[2022]

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 1290–1299. Omer Dahary, Or Patashnik, Kfir Aberman, and Daniel Cohen-Or. 2025. Be your- self: Bounded attention for multi-...

2021 arXiv
[2023]

In Proceedings of the IEEE/CVF International Conference on Computer Vision

Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4015–4026. Black Forest Labs. 2023. FLUX. https://github.com/black-forest-labs/flux. Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yon...

2022 arXiv
[2024]

arXiv preprint arXiv:2406.10210 (2024)

Make It Count: Text-to-Image Generation with an Accurate Number of Objects. arXiv preprint arXiv:2406.10210 (2024). Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. 2023. Training diffusion models with reinforcement learning.arXiv preprint arXiv:2305.1...

2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

There are 3 total clusters in the layout dict,we are required to assign one cluster as 'porcupine', one clus- ter as 'squirrel' and onecluster as 'raccoon' meaning we do not have to delete any clusters

work page

[2] [2]

The 'squirrel' object also does not require spa- tial arrangment

The 'porcupine' object does not require spatial arra- ngment. The 'squirrel' object also does not require spa- tial arrangment. The 'raccoon' object also does not req- uire spatial arrangment

work page

[3] [3]

arXiv preprint arXiv:2407.21783 (2024)

The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al . 2024. Scal- ing rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Lea...

arXiv 2024

[4] [5]

In Proceedings of the IEEE/CVF International Conference on Computer Vision

Localizing object-level shape variations with text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 23051– 23061. William Peebles and Saining Xie. 2023. Scalable Diffusion Models with Transformers. arXiv:2212.09748 [cs.CV] https://arxiv.org/abs/2212.09748 Quynh Phung, Songwei Ge, and Jia-Bin Huang. 20...

arXiv 2023

[5] [6]

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10684–10695. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022. Photorealistic text-to...

work page 2022

[6] [7]

Six meerkats standing watch in the Savannah

Instancediffusion: Instance-level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 6232–6242. Tsung-Han Wu, Long Lian, Joseph E Gonzalez, Boyi Li, and Trevor Darrell. 2024a. Self- correcting llm-controlled diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and P...

arXiv 2024

[7] [10]

1" with 'raccoon' probability (0.75). This leaves us with cluster

'porcupine' has the lowest desired quantity (1) so we start by assiging it to the cluster with the highest 'porcupine' probability, which is cluster "3" with 'por- cupine' probability (0.71). We then assign the 'raccoon' InstanceGen: Image Generation with Instance-level Instructions • 13 object to the cluster with the highest 'raccoon' probab- ility, whic...

work page

[8] [11]

3") as req- uired. We have set one cluster as 'squirrel' (

We have set one cluster as 'porcupine' ("3") as req- uired. We have set one cluster as 'squirrel' ("2") as required. We have set one cluster as 'raccoon' ("1") as required. We have set one 'squirrel' cluster as 'holdi- ng nut ' ("2") as required. ** ASSIGNMENTS ** { "1": { "object": "raccoon", "attributes": [] }, "2": { "object": "squirrel", "attributes":...

work page