Pith. sign in

REVIEW 4 major objections 6 minor 69 references

ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that separating instance formation from semantic assignment fixes multi-object generation in diffusion models without retraining.

desk verdict A genuinely novel instance-first attention-control method with broad empirical gains, held back by an unvalidated K-means clustering step and a self-built benchmark supplying headline numbers. read the letter →

arxiv 2505.20935 v4 pith:QOUUIX2S submitted 2025-05-27 cs.CV

Sanghyun Jo , Wooyeol Lee , Ziseok Lee , Jonghyun Choi , Jaesik Park , Kyungsu Kim This is my paper
classification cs.CV
keywords multi-instancegenerationtext-to-imagediffusionattentioncontrolself-attentioncross-attentiontraining-freeguidanceIntraCompBenchcompositional
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that multi-object text-to-image diffusion fails because instance boundaries are vague, not because semantic tokens are hard to separate. It proposes ISAC, a training-free, model-agnostic guidance that first carves the requested number of instance regions out of self-attention maps and then binds semantics to those regions through cross-attention. This ordering matters because early denoising steps show stable instance structures in self-attention while semantic maps still overlap, so semantic-only controls act on blurred regions. ISAC reports large accuracy gains on standard compositional benchmarks and on a new intra-category benchmark, plus improved layout-to-image control.

What carries the argument

The central object is the hierarchical two-phase loss $L_t(X_t)=\lambda_{\mathrm{ins}}(t)L_{\mathrm{ins}}(X_t)+\lambda_{\mathrm{sem}}(t)L_{\mathrm{sem}}(X_t)$. In Phase 1, an averaged self-attention map is clustered on foreground-gated rows (with optional coordinate concatenation) into $N$ class-agnostic instance masks, and the worst pairwise pixel overlap is penalized by the maximum pixel-wise overlap (MPO), yielding clean instance boundaries. In Phase 2, the stabilized self-attention is multiplied into cross-attention to form instance-aware semantic masks $CA^{\mathrm{ins}}_t = SA_t CA_t$, and a repel-and-bind loss pushes apart tokens that belong to different instances while pulling together tokens that describe the same instance. The schedule $\lambda_{\mathrm{ins}}(t)=t/T$, $\lambda_{\mathrm{sem}}(t)=1-t/T$ makes instance formation dominate early denoising and semantic refinement dominate later.

What would settle it

Take a prompt with N objects, extract ISAC's Phase 1 masks in the earliest denoising steps, and compare them pixel-wise to ground-truth instance segmentation from a scene with known object counts; if the masks frequently merge or split objects before semantic maps become clear, the premise that self-attention can be clustered into clean instance layouts fails and the whole two-phase control inherits that failure.

Watch

Extended reading notes

Core claim

ISAC claims that multi-object generation in diffusion models fails because semantic signals are separated before instance boundaries exist, and that the correct order is instance-first. It shows that self-attention maps already expose class-agnostic instance layouts in early denoising, while cross-attention semantic maps remain overlapping blobs, and builds a two-phase guidance that first clusters self-attention into the requested N instance masks and repels their overlaps, then injects those stabilized masks into cross-attention to repel incompatible tokens and bind attributes within each instance. With the instance-to-semantic schedule $\lambda_{\mathrm{ins}}(t)=t/T$, $\lambda_{\mathrm{sem}}(t)=1-t/T$, ISAC reports consistent gains on T2I-CompBench, HRS-Bench, and its new IntraCompBench across five backbones, including at least 50% relative multi-class improvement on IntraCompBench, and refines overlapping box layouts in layout-to-image models.

Load-bearing premise

The load-bearing premise is that K-means clustering of early self-attention maps, restricted to a semantic foreground gate, reliably yields exactly one clean region per requested instance, because Phase 2 only assigns semantics to the regions Phase 1 creates.

Editorial extensions

If this is right

  • On the authors' IntraCompBench, ISAC raises multi-class accuracy to 36% on SD1.5 (from 20% for the strongest prior training-free baseline) and to 52% on SD3.5-M (from 34% for Self-Cross).
  • ISAC beats count-supervised methods on instance-counting accuracy without fine-tuning: 70% versus 49% on SD1.4 and 76% versus 69% on SDXL.
  • ISAC improves layout-to-image controllers under overlapping boxes by refining coarse box layouts into dense instance masks, rather than only constraining attention inside each box.
  • The instance-to-semantic ordering is essential: reversing the schedule to semantic-to-instance drops multi-class accuracy from 36% to 21%, and removing either phase degrades performance.
  • A latent-selection variant carries the same scoring to very large diffusion backbones without backpropagation, trading a larger sample batch for lower memory overhead.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Implicit in the results is that the method's transfer to very large diffusion transformers depends on self-attention staying class-agnostic early in denoising; a testable extension is measuring cluster-consistency across timesteps on large-scale backbones.
  • Because the foreground gate is built from the very cross-attention maps that Phase 2 must correct, a wrong gate can silently drop an object before clustering; replacing the semantic gate with a purely class-agnostic saliency signal would isolate this risk.
  • MPO's max-over-pixels form means one stubbornly touching pair sets the entire loss, so a top-k or area-weighted overlap penalty is a natural variant that might spread the gradient more evenly across crowded scenes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ISAC, a training-free, model-agnostic guidance for multi-instance text-to-image generation. The method performs a two-phase latent optimization: Phase 1 clusters the averaged self-attention map into N instance masks (after an LLM-based parser extracts N, class tokens, and attributes) and penalizes overlap between masks; Phase 2 injects these instance-aware masks into cross-attention and applies a repel-and-bind loss to separate incompatible tokens and bind attributes within each instance. An instance-to-semantic schedule shifts weight from Phase 1 to Phase 2 over the denoising trajectory. The method is evaluated on T2I-CompBench, HRS-Bench, and a newly introduced IntraCompBench across nine diffusion backbones, with consistent gains over prior training-free methods and competitive or better results than count-supervised approaches. The paper also demonstrates improvements when ISAC is applied to layout-to-image controllers and fine-tuned models.

Significance. If the central claim holds, ISAC is a valuable contribution: it identifies a concrete mechanism (early self-attention instance layouts) and turns it into a simple, training-free objective that improves multi-instance and multi-class generation across many backbones. The new IntraCompBench fills a real gap in evaluating intra-category compositions, and the paper is unusually thorough: ablations for the loss schedule, the overlap metric, the clustering algorithm, the gradient step size, and a best-of-N latent-selection variant, plus extensions to layout-to-image and fine-tuned models. The quantitative gains are large and consistent on external benchmarks (T2I-CompBench, HRS-Bench), which supports the general direction. The main weakness is that the load-bearing premise—that K-means on self-attention rows yields clean, exactly-N instance masks—is never directly validated, and the headline numbers come from a benchmark that the authors constructed and used for design choices.

major comments (4)
  1. [Sec. 3.2, Eq. (5)] The central mechanism of ISAC is Phase 1's clustering of self-attention into N instance masks, but the paper does not directly measure the quality of these masks. The qualitative PCA visualizations and the clustering-algorithm ablation (Tab. 7) show that the choice of clustering algorithm does not affect final accuracy, but they do not establish that the masks correspond to true instances. Since Phase 2 binds semantics to these masks (Eq. 8), any split/merge error propagates, and the reported gains could partly arise from the additional gradient signal rather than from accurate instance layouts. Please add a quantitative mask-quality metric (e.g., IoU between Phase-1 masks and masks from a detector such as Grounding DINO or SAM on a sample of IntraCompBench prompts) and report split/merge statistics as a function of timestep.
  2. [Tables 2, 3, 15] The headline comparisons report single-point accuracies without confidence intervals. Since IntraCompBench uses 10 images per prompt, standard errors are computable and should be reported. This is needed to assess small margins such as Table 2's SD3.5-M #3 row (ISAC 40% vs. Self-Cross 38%) and Table 3's SDXL #2 row (ISAC 96% vs. CountGen 97%). The variance-reduction claim in Appendix C.4 (Tab. 16) covers only T2I-CompBench Numeracy; the IntraCompBench metrics that support the abstract's claims should also include seed variance.
  3. [IntraCompBench and Tabs. 5, 6] The loss components (MPO, the instance-to-semantic schedule, and the schedule balance) appear to have been selected using IntraCompBench, and the same benchmark provides the headline numbers in the abstract. While external benchmarks (T2I-CompBench, HRS-Bench) provide some validation, the paper should explicitly state which design choices were tuned on IntraCompBench and provide a held-out split or an additional external validation for the final configuration. Without this, the magnitude of the claimed gains (e.g., 'at least 50%' in the abstract) cannot be independently interpreted.
  4. [Appendix A, parser robustness] The method depends on the LLM parser for N and token relations, and the authors acknowledge that ambiguous counts are not resolved. However, no experiment quantifies the effect of parser errors on the end-to-end results. Because N determines the number of K-means clusters, a robustness study with perturbed or manually corrected counts would strengthen the practical-utility claim. As a separate point, the paper is explicit in Tab. 1 that instance counts are required, so I do not see a claim of count-independence; the concern about a contradiction with the algorithm does not land.
minor comments (6)
  1. [Abstract] The paper's abstract states 'Code will be released upon publication' while the arXiv abstract states 'Code and IntraCompBench are available at https://shjo-april.github.io/ISAC.' Please harmonize these statements.
  2. [Abstract and throughout] There are many missing spaces in the typeset text (e.g., 'introduceISAC(Instance-to-SemanticAttentionControl)', 'Lins', 'Lsem'), which makes the abstract and some sections hard to read. Please proofread the final version.
  3. [Tables 2 and 15] The latency/VRAM numbers for the same method and backbone differ between the two tables (e.g., SD1.5 + ISAC is 21s/9.7 GB in Table 2 but 30s/9.6 GB in Table 15; SD1.5 + A&E is 17s/9.2 GB in Table 2 but 24s/9.1 GB in Table 15). Please reconcile these values or explain the discrepancy.
  4. [Eq. (12)] The notation for softmax normalization 'CAsoftmax_t = softmax(β·CA_t[1:])' is informal with respect to the token dimension; please clarify that the softmax is applied along the token dimension excluding the SOT token.
  5. [Sec. 4.1 and Appendix A.1] The parser is described as 'GPT-OSS [44]' in the main text and 'GPT-OSS-20B' in the appendix; please use a consistent name.
  6. [Contributions list] The contribution states 'improves multi-object metrics by at least 1.9× over baseline [1]', but it is not clear which metric this refers to; please specify the metric and the comparison basis.

Circularity Check

1 steps flagged · score 4.0 of 10

The central premise that early self-attention rows form instance clusters is anchored to a co-authored DiffEgg preprint; no equation-level reduction appears, and the benchmark evidence is otherwise external.

  1. self citation load bearing [Section 1 (introduction paragraph after Fig. 1); operationalized in Section 3.2, Eq. (5).]
    "In contrast, self-attention maps capture 'where' instance structures emerge, and already exhibit instance-like clustering early in the diffusion process [11]. ... We hypothesize that pixels belonging to the same instance exhibit higher mutual attention, whereas pixels from different instances attend to each other less."

    Reference [11] is DiffEgg (Jo, Lee, Lee, Kim), a preprint by four of the six authors of this paper. It is the only external citation offered for the load-bearing premise that early self-attention rows already encode N instance-like clusters; Phase 1 turns this premise into K-means masks in Eq. (5), M = SAt[I,I] stopgrad(K), and Phase 2 binds semantics to those masks. The paper reports no direct mask-quality metric (no IoU or instance-segmentation agreement with ground truth), so the premise is supported mainly by a same-author preprint plus qualitative PCA visualizations rather than by independent evidence. This makes the self-citation load-bearing, although it does not reduce any equation to its own input.

full rationale

No definitional circularity was found: ISAC's losses in Eqs. (7)-(9) are optimization objectives over attention maps, not fitted parameters, and the headline accuracies come from external open-vocabulary detectors on T2I-CompBench, HRS-Bench, and IntraCompBench, so the numbers are not equal to the objective by construction. The main circularity burden is the early-self-attention instance-clustering hypothesis, which is cited to the authors' own DiffEgg preprint and is never validated with a direct segmentation-quality measurement; this raises the score to 4. Ablations in Tables 5 and 6 select the MPO metric and the instance-to-semantic schedule on IntraCompBench, which is a test-set selection risk rather than a forced reduction. Explicit limitations acknowledged by the paper, such as unresolved ambiguous count expressions (Appendix A) and the lack of 3D-aware reasoning (Appendix C.5), do not create circularity but should bound interpretation of the claimed gains.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method rests on an empirical hypothesis about self-attention structure, an LLM parser that provides counts and relations, and a hand-designed loss schedule. No new physics or external entities are introduced. The benchmark IntraCompBench is newly proposed but is an evaluation artifact, not a theoretical postulate.

free parameters (2)
  • gradient step size eta = 0.01
    Single shared step size for latent optimization, chosen via sensitivity analysis in Appendix A; affects stability and image quality.
  • loss schedule lambda_ins/lambda_sem = t/T and 1-t/T
    Hand-designed linear schedule encoding the instance-first principle; Table 5 shows the ordering matters.
assumptions (4)
  • domain assumption Pixels belonging to the same instance exhibit higher mutual self-attention than pixels from different instances, allowing K-means to recover N instance masks.
    Stated as a hypothesis in Sec 3.2 and used to justify Eq. 5; supported only by qualitative PCA visualizations and the authors' own DiffEgg preprint.
  • domain assumption A foreground gate formed by binarizing CAins columns covers all relevant instance pixels and excludes background.
    Eqs. 3-4; clustering is restricted to foreground, so missing a foreground pixel can drop an instance.
  • domain assumption The GPT-OSS-20B LLM parser extracts correct class tokens, counts, and token relations from arbitrary prompts.
    Sec 4.1 and Fig 11; the parser output directly sets N and Prepel/Pbind, so parsing errors propagate into Phase 1 and Phase 2.
  • domain assumption Averaging attention maps over all layers and heads preserves the instance structure needed for clustering.
    Eq. 1 and Appendix A.1; averaging could wash out instance-level signal in some layers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation." pith.science (2026). https://pith.science/paper/QOUUIX2S

@misc{pith2026250520935,
  author       = {Pith},
  title        = {Pith review of: ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOUUIX2S}},
  note         = {Machine review of arXiv:2505.20935}
}
abstract

Recent open-weight text-to-image (T2I) diffusion models still struggle with multi-instance prompts, often omitting or merging instances and mixing semantics among similar objects. We trace these failures to early denoising steps, before instance boundaries are reliably stabilized. Existing training-free guidance is largely driven by cross-attention or other token-conditioned semantic signals. Such guidance can separate concepts at the token level, but largely assumes that distinct instance regions have already emerged. In early denoising steps, it cannot reliably carve out these regions, so count failures and semantic mixing persist. By contrast, self-attention exposes class-agnostic instance layouts during early denoising. To exploit this asymmetry, we propose $\textbf{ISAC}$ ($\textbf{I}$nstance-to-$\textbf{S}$emantic $\textbf{A}$ttention $\textbf{C}$ontrol), a training-free, model-agnostic objective that first stabilizes self-attention layouts and then binds cross-attention semantics within them, without fine-tuning or external vision models. Across T2I-CompBench, HRS-Bench, and our newly curated IntraCompBench, ISAC consistently outperforms prior training-free methods. Furthermore, ISAC enhances layout-to-image controllers by refining coarse, overlapping bounding boxes into dense instance masks. Code and IntraCompBench are available at https://shjo-april.github.io/ISAC.

Figures

Figures reproduced from arXiv: 2505.20935 by the authors.

Figure 1
Figure 1. Importance of instance-level control for multi-object generation. Existing text-to￾image diffusion models (e.g., SD1.5 [1]) and prior training-free guidance methods [2, 3] still suffer from count failures (missing or merged instances) and semantic mixing (attributes spilling across objects), whereas ISAC’s instance-first design yields the correct number of instances with clearly separated semantics. 1 Introduction T… view at source ↗
Figure 2
Figure 2. Semantic overlap across object pairs. We measure semantic mixing by the Dice coefficient between the two instance-aware semantic masks in Eq. (2) for prompts of the form “A photo of a <object1> and a <object2>” with SD3.5-M [5]. Left: heatmap over object pairs. Within-supercategory pairs (fruits, vehicles, animals; blue boxes) show consistently higher overlap, revealing that semantic masks tend to cover multiple sim… view at source ↗
Figure 3
Figure 3. Dynamics of Text-to-image diffusion models. In early diffusion steps, instance structures actively emerge while semantics underdeveloped. In later diffusion steps, instance structures are stabilized and semantic refinements happen. As detection models (e.g., [19]) rely on strong semantic cues, they are only effective in later steps. We use a prompt of “A photo of a cat and a dog” on SD3.5-M [5]. generation, while ev… view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: Overview of ISAC. Given a multi-object prompt, ISAC steers diffusion in two phases. In Phase 1 (Sec. 3.2), we cluster the self-attention map into N class-agnostic instance masks and apply an instance separation loss that repels overlaps, yielding clean instance layouts…
Figure 5
Figure 5. Figure 5: Qualitative comparison using SD1.5 [1] and SD3.5-M [5] as a backbone and added attention control methods. backbone, ISAC still provides a clear margin with especially large gains in the more crowded #4 and #5 cases. These trends indicate that our instance forming first…
Figure 6
Figure 6. Figure 6: Qualitative examples on SD1.5 [1] and SD3.5-M [5] where prompts specify exact counts. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Example of applying ISAC to layout-to-image generation. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Qualitative comparison of ISAC with various step sizes [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Concatenating normalized coordinates (x, y) ∈ [−1, 1]2 to SA features stabilizes clustering, reducing erroneous merges and ignorance on hidden boundary signals. The image is generated with the prompt, “a photo of two donuts” on SD3.5-M [5]. We averaged all accumulated …
Figure 10
Figure 10. Figure 10: Qualitative comparison of clustering algorithms. The image is generated with the prompt, [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Details on LLM-guided automatic prompt parsing. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Semantic separation of layout guidance methods ( [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Limitation of Bounded Attention Masking. Bounded Attention [38] enforces exclusive pixel ownership among bounding boxes on self-attention maps. Each box can only attend to its owned pixels and the background. We adopt a “smaller-box-first” ownership rule to build self…
Figure 14
Figure 14. Figure 14: Qualitative results of latent selection with ISAC scoring. [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Limitation of ISAC. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: ISAC applied to different subsets of SDXL [4] attention layers. [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Diffusion dynamics visualization with or without ISAC. This is an extended version of [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]
Figure 18
Figure 18. Figure 18: Qualitative comparisons of attention control methods, InitNO [ [PITH_FULL_IMAGE:figures/full_fig_p028_18.png]
Figure 19
Figure 19. Figure 19: Qualitative comparisons of attention control methods, InitNO [ [PITH_FULL_IMAGE:figures/full_fig_p029_19.png]
Figure 20
Figure 20. Figure 20: Qualitative comparisons of attention control methods, InitNO [ [PITH_FULL_IMAGE:figures/full_fig_p030_20.png]
Figure 21
Figure 21. Figure 21: Qualitative comparisons of ISAC witth layout guidance method, Attention Refocusing [PITH_FULL_IMAGE:figures/full_fig_p031_21.png]
Figure 22
Figure 22. Figure 22: Qualitative results across multiple seeds for the prompt [PITH_FULL_IMAGE:figures/full_fig_p032_22.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 47 canonical work pages

  1. [1]

    High-resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684–10695, 2022

    2022

  2. [2]

    Initno: Boosting text-to-image diffusion models via initial noise optimization,

    X. Guo, J. Liu, M. Cui, J. Li, H. Yang, and D. Huang, “Initno: Boosting text-to-image diffusion models via initial noise optimization,” inCVPR, 2024

    work page 2024

  3. [3]

    Self-cross diffusion guidance for text-to-image synthesis of similar subjects,

    W. Qiu, J. Wang, and M. Tang, “Self-cross diffusion guidance for text-to-image synthesis of similar subjects,” inProceedings of the Computer Vision and Pattern Recognition Conference, pp. 23528–23538, 2025

    work page 2025

  4. [4]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis,

    D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rom- bach, “Sdxl: Improving latent diffusion models for high-resolution image synthesis,”arXiv preprint arXiv:2307.01952, 2023

    arXiv 2023

  5. [5]

    Scaling rectified flow transformers for high-resolution image synthesis,

    P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel,et al., “Scaling rectified flow transformers for high-resolution image synthesis,” in Forty-first international conference on machine learning, 2024

    work page 2024

  6. [6]

    Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis,

    J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y. Wu, Z. Wang, J. Kwok, P. Luo, H. Lu, and Z. Li, “Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis,” 2023

    work page 2023

  7. [7]

    B. F. Labs, “Flux.”https://github.com/black-forest-labs/flux, 2024

    work page 2024

  8. [8]

    Token merging for training-free semantic binding in text-to-image synthesis,

    T. Hu, L. Li, J. van de Weijer, H. Gao, F. Shahbaz Khan, J. Yang, M.-M. Cheng, K. Wang, and Y. Wang, “Token merging for training-free semantic binding in text-to-image synthesis,” Advances in Neural Information Processing Systems, vol. 37, pp. 137646–137672, 2024

    work page 2024

Show all 69 references
  1. [9]

    A cat is a cat (not a dog!): Unraveling infor- mation mix-ups in text-to-image encoders through causal analysis and embedding optimization,

    C.-Y. Chen, C. Tseng, L.-W. Tsao, and H.-H. Shuai, “A cat is a cat (not a dog!): Unraveling infor- mation mix-ups in text-to-image encoders through causal analysis and embedding optimization,” Advances in Neural Information Processing Systems, 2024

    2024

  2. [10]

    Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models,

    H. Chefer, Y. Alaluf, Y. Vinker, L. Wolf, and D. Cohen-Or, “Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models,” 2023

    2023

  3. [11]

    Diffegg: Diffusion-driven edge generation as a pixel- annotation-free alternative for instance annotation,

    S. Jo, Z. Lee, W. Lee, and K. Kim, “Diffegg: Diffusion-driven edge generation as a pixel- annotation-free alternative for instance annotation,” 2025

    2025

  4. [12]

    Gligen: Open-set grounded text-to-image generation,

    Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang, J. Gao, C. Li, and Y. J. Lee, “Gligen: Open-set grounded text-to-image generation,”CVPR, 2023

    2023

  5. [13]

    T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation,

    K. Huang, C. Duan, K. Sun, E. Xie, Z. Li, and X. Liu, “T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 5, pp. 3563–3579, 2025

    2025

  6. [14]

    Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models,

    E. M. Bakr, P. Sun, X. Shen, F. F. Khan, L. E. Li, and M. Elhoseiny, “Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models,” inProceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20041–20053, 2023

    2023

  7. [15]

    Make it count: Text- to-image generation with an accurate number of objects,

    L. Binyamin, Y. Tewel, H. Segev, E. Hirsch, R. Rassin, and G. Chechik, “Make it count: Text- to-image generation with an accurate number of objects,”arXiv preprint arXiv:2406.10210, 2024

    2024 arXiv

  8. [16]

    Counting guidance for high fidelity text-to-image synthesis,

    W. Kang, K. Galim, H. I. Koo, and N. I. Cho, “Counting guidance for high fidelity text-to-image synthesis,” in2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 899–908, IEEE, 2025

    2025

  9. [17]

    Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment,

    R. Rassin, E. Hirsch, D. Glickman, S. Ravfogel, Y. Goldberg, and G. Chechik, “Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  10. [18]

    Conform: Contrast is all you need for high-fidelity text-to-image diffusion models,

    T. H. S. Meral, E. Simsar, F. Tombari, and P. Yanardag, “Conform: Contrast is all you need for high-fidelity text-to-image diffusion models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9005–9014, 2024. 33

    2024

  11. [19]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su,et al., “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” in European Conference on Computer Vision, pp. 38–55, Springer, 2024

    2024

  12. [20]

    Prompt-to- prompt image editing with cross attention control,

    A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or, “Prompt-to- prompt image editing with cross attention control,”arXiv preprint arXiv:2208.01626, 2022

    2022 arXiv

  13. [21]

    Training-free structured diffusion guidance for compositional text-to-image synthesis,

    W. Feng, X. He, T.-J. Fu, V. Jampani, A. R. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Y. Wang, “Training-free structured diffusion guidance for compositional text-to-image synthesis,” inThe Eleventh International Conference on Learning Representations, 2023

    2023

  14. [22]

    Llm-grounded diffusion: Enhancing prompt un- derstanding of text-to-image diffusion models with large language models,

    L. Lian, B. Li, A. Yala, and T. Darrell, “Llm-grounded diffusion: Enhancing prompt un- derstanding of text-to-image diffusion models with large language models,”arXiv preprint arXiv:2305.13655, 2023

    2023 arXiv

  15. [23]

    Realcompo: Balancing realism and compositionality improves text-to-image diffusion models,

    X. Zhang, L. Yang, Y. Cai, Z. Yu, K. Wang, J. Xie, Y. Tian, M. Xu, Y. Tang, Y. Yang, and B. Cui, “Realcompo: Balancing realism and compositionality improves text-to-image diffusion models,”Advances in Neural Information Processing Systems, 2024

    2024

  16. [24]

    Migc: Multi-instance generation controller for text-to-image synthesis,

    D. Zhou, Y. Li, F. Ma, X. Zhang, and Y. Yang, “Migc: Multi-instance generation controller for text-to-image synthesis,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6818–6828, 2024

    2024

  17. [25]

    Instancediffusion: Instance-level control for image generation,

    X. Wang, T. Darrell, S. S. Rambhatla, R. Girdhar, and I. Misra, “Instancediffusion: Instance-level control for image generation,” 2024

    2024

  18. [26]

    Hico: Hierarchical controllable diffusion model for layout-to-image generation,

    B. Cheng, Y. Ma, L. Wu, S. Liu, A. Ma, X. Wu, D. Leng, and Y. Yin, “Hico: Hierarchical controllable diffusion model for layout-to-image generation,” 2024

    2024

  19. [27]

    3dis: Depth-driven decoupled instance synthesis for text-to-image generation,

    D. Zhou, J. Xie, Z. Yang, and Y. Yang, “3dis: Depth-driven decoupled instance synthesis for text-to-image generation,”arXiv preprint arXiv:2410.12669, 2024

    2024

  20. [28]

    Creatilayout: Siamese multimodal diffusion transformer for creative layout-to-image generation,

    H. Zhang, D. Hong, Y. Wang, J. Shao, X. Wu, Z. Wu, and Y.-G. Jiang, “Creatilayout: Siamese multimodal diffusion transformer for creative layout-to-image generation,” inProceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18487–18497, 2025

    2025

  21. [29]

    Overlaybench: A benchmark for layout-to-image generation with dense overlaps,

    B. Li, C.-Y. Wang, H. Xu, X. Zhang, E. Armand, D. Srivastava, S. Xiaojun, Z. Chen, J. Xie, and Z. Tu, “Overlaybench: A benchmark for layout-to-image generation with dense overlaps,” in Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS 2025), ...

    2025

  22. [30]

    Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms,

    L. Yang, Z. Yu, C. Meng, M. Xu, S. Ermon, and B. Cui, “Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms,” inInternational Conference on Machine Learning, 2024

    2024

  23. [31]

    Multidiffusion: Fusing diffusion paths for controlled image generation,

    O. Bar-Tal, L. Yariv, Y. Lipman, and T. Dekel, “Multidiffusion: Fusing diffusion paths for controlled image generation,”arXiv preprint arXiv:2302.08113, 2023

    2023 arXiv

  24. [32]

    Noisecollage: A layout-aware text-to-image diffusion model based on noise cropping and merging,

    T. Shirakawa and S. Uchida, “Noisecollage: A layout-aware text-to-image diffusion model based on noise cropping and merging,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  25. [33]

    Boxdiff: Text-to- image synthesis with training-free box-constrained diffusion,

    J. Xie, Y. Li, Y. Huang, H. Liu, W. Zhang, Y. Zheng, and M. Z. Shou, “Boxdiff: Text-to- image synthesis with training-free box-constrained diffusion,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7452–7461, 2023

    2023

  26. [34]

    Training-free layout control with cross-attention guidance,

    M. Chen, I. Laina, and A. Vedaldi, “Training-free layout control with cross-attention guidance,” arXiv preprint arXiv:2304.03373, 2023

    2023 arXiv

  27. [35]

    R&b: Region and boundary aware zero-shot grounded text-to-image generation,

    J. Xiao, L. Li, H. Lv, S. Wang, and Q. Huang, “R&b: Region and boundary aware zero-shot grounded text-to-image generation,” 2023

    2023

  28. [36]

    Groundit: Grounding diffusion transformers via noisy patch transplantation,

    P. Y. Lee, T. Yoon, and M. Sung, “Groundit: Grounding diffusion transformers via noisy patch transplantation,” inAdvances in Neural Information Processing Systems, 2024

    2024

  29. [37]

    Grounded text-to-image synthesis with attention refocusing,

    Q. Phung, S. Ge, and J.-B. Huang, “Grounded text-to-image synthesis with attention refocusing,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7932–7942, 2024. 34

    2024

  30. [38]

    Be yourself: Bounded attention for multi-subject text-to-image generation,

    O. Dahary, O. Patashnik, K. Aberman, and D. Cohen-Or, “Be yourself: Bounded attention for multi-subject text-to-image generation,” 2024

    2024

  31. [39]

    Be decisive: Noise-induced layouts for multi-subject generation,

    O. Dahary, Y. Cohen, O. Patashnik, K. Aberman, and D. Cohen-Or, “Be decisive: Noise-induced layouts for multi-subject generation,” inProceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, pp. 1–12, 2025

    2025

  32. [40]

    Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering,

    Y. Hu, B. Liu, J. Kasai, Y. Wang, M. Ostendorf, R. Krishna, and N. A. Smith, “Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering,” inProceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20406–20417, 2023

    2023

  33. [41]

    Beta sampling is all you need: Efficient image generation strategy for diffusion models using stepwise spectral analysis,

    H. Lee, H. Lee, S. Gye, and J. Kim, “Beta sampling is all you need: Efficient image generation strategy for diffusion models using stepwise spectral analysis,” in2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4215–4224, IEEE, 2025

    2025

  34. [42]

    Yoloe: Real-time seeing anything,

    A. Wang, L. Liu, H. Chen, Z. Lin, J. Han, and G. Ding, “Yoloe: Real-time seeing anything,” 2025

    2025

  35. [43]

    Qwen-image technical report,

    C. Wu, J. Li, J. Zhou, J. Lin, K. Gao, K. Yan, S. ming Yin, S. Bai, X. Xu, Y. Chen, Y. Chen, Z. Tang, Z. Zhang, Z. Wang, A. Yang, B. Yu, C. Cheng, D. Liu, D. Li, H. Zhang, H. Meng, H. Wei, J. Ni, K. Chen, K. Cao, L. Peng, L. Qu, M. Wu, P. Wang, S. Yu, T. Wen, W. Feng, X. Xu, Y...

    2025

  36. [44]

    gpt-oss-120b & gpt-oss-20b model card,

    OpenAI, “gpt-oss-120b & gpt-oss-20b model card,” 2025

    2025

  37. [45]

    Learning to count anything: Reference-less class-agnostic counting with weak supervision,

    M. Hobley and V. Prisacariu, “Learning to count anything: Reference-less class-agnostic counting with weak supervision,”arXiv preprint arXiv:2205.10203, 2022

    2022 arXiv

  38. [46]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” inInternational Conference on Medical image computing and computer-assisted intervention, pp. 234–241, Springer, 2015

    2015

  39. [47]

    Pixart-σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation,

    J. Chen, C. Ge, E. Xie, Y. Wu, L. Yao, X. Ren, Z. Wang, P. Luo, H. Lu, and Z. Li, “Pixart-σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation,” inEuropean Conference on Computer Vision, pp. 74–91, Springer, 2024

    2024

  40. [48]

    Mean shift: A robust approach toward feature space analysis,

    D. Comaniciu and P. Meer, “Mean shift: A robust approach toward feature space analysis,” IEEE Transactions on pattern analysis and machine intelligence, vol. 24, no. 5, pp. 603–619, 2002

    2002

  41. [49]

    Seg- sort: Segmentation by discriminative sorting of segments,

    J.-J. Hwang, S. X. Yu, J. Shi, M. D. Collins, T.-J. Yang, X. Zhang, and L.-C. Chen, “Seg- sort: Segmentation by discriminative sorting of segments,” inProceedings of the IEEE/CVF international conference on computer vision, pp. 7334–7344, 2019

    2019

  42. [50]

    An intriguing failing of convolutional neural networks and the coordconv solution,

    R. Liu, J. Lehman, P. Molino, F. Petroski Such, E. Frank, A. Sergeev, and J. Yosinski, “An intriguing failing of convolutional neural networks and the coordconv solution,”Advances in neural information processing systems, vol. 31, 2018

    2018

  43. [51]

    Tokencompose: Text-to-image diffusion with token-level supervision,

    Z. Wang, Z. Sha, Z. Ding, Y. Wang, and Z. Tu, “Tokencompose: Text-to-image diffusion with token-level supervision,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8553–8564, June 2024

    2024

  44. [52]

    Microsoft COCO: Common objects in context,

    T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” inECCV, pp. 740–755, Springer, 2014

    2014

  45. [53]

    Yolov12: Attention-centric real-time object detectors,

    Y. Tian, Q. Ye, and D. Doermann, “Yolov12: Attention-centric real-time object detectors,” arXiv preprint arXiv:2502.12524, 2025

    2025 arXiv

  46. [54]

    Rethinking the spatial inconsistency in classifier-free diffusion guidance,

    D. Shen, G. Song, Z. Xue, F.-Y. Wang, and Y. Liu, “Rethinking the spatial inconsistency in classifier-free diffusion guidance,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9370–9379, 2024

    2024

  47. [55]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,”arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  48. [56]

    Training-free dense-aligned diffusion guidance for modular conditional image synthesis,

    Z. Wang, D. Peng, F. Chen, Y. Yang, and Y. Lei, “Training-free dense-aligned diffusion guidance for modular conditional image synthesis,” inProceedings of the Computer Vision and Pattern Recognition Conference, pp. 13135–13145, 2025. 35

    2025

  49. [57]

    Universal guidance for diffusion models,

    A. Bansal, H.-M. Chu, A. Schwarzschild, S. Sengupta, M. Goldblum, J. Geiping, and T. Goldstein, “Universal guidance for diffusion models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 843–852, 2023

    2023

  50. [58]

    Noise2score: tweedie’s approach to self-supervised image denoising without clean images,

    K. Kim and J. C. Ye, “Noise2score: tweedie’s approach to self-supervised image denoising without clean images,”Advances in Neural Information Processing Systems, vol. 34, pp. 864–874, 2021

    2021

  51. [59]

    Flow matching for generative modeling,

    Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,”arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  52. [60]

    Rare-to-frequent: Unlocking composi- tional generation power of diffusion models on rare concepts with llm guidance,

    D. Park, S. Kim, T. Moon, M. Kim, K. Lee, and J. Cho, “Rare-to-frequent: Unlocking composi- tional generation power of diffusion models on rare concepts with llm guidance,”The Thirteenth International Conference on Learning Representations, 2025

    2025

  53. [61]

    Comat: Aligning text- to-image diffusion model with image-to-text concept matching,

    D. Jiang, G. Song, X. Wu, R. Zhang, D. Shen, Z. Zong, Y. Liu, and H. Li, “Comat: Aligning text- to-image diffusion model with image-to-text concept matching,”arXiv preprint arXiv:2404.03653, 2024

    2024 arXiv

  54. [62]

    Grounded sam: Assembling open-world models for diverse visual tasks,

    T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F. Yan, et al., “Grounded sam: Assembling open-world models for diverse visual tasks,”arXiv preprint arXiv:2401.14159, 2024

    2024 arXiv

  55. [63]

    Itercomp: Iterative composition-aware feedback learning from model gallery for text-to-image generation,

    X. Zhang, L. Yang, G. Li, Y. Cai, J. Xie, Y. Tang, Y. Yang, M. Wang, and B. Cui, “Itercomp: Iterative composition-aware feedback learning from model gallery for text-to-image generation,” arXiv preprint arXiv:2410.07171, 2024

    arXiv

  56. [64]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,”Advances in neural information processing systems, vol. 30, 2017

    2017

  57. [65]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al., “Learning transferable visual models from natural language supervi- sion,” inInternational conference on machine learning, pp. 8748–8763, PmLR, 2021

    2021

  58. [66]

    Rethinking fid: Towards a better evaluation metric for image generation,

    S. Jayasumana, S. Ramalingam, A. Veit, D. Glasner, A. Chakrabarti, and S. Kumar, “Rethinking fid: Towards a better evaluation metric for image generation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9307–9315, 2024

    2024

  59. [67]

    Imagereward: Learning and evaluating human preferences for text-to-image generation,

    J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong, “Imagereward: Learning and evaluating human preferences for text-to-image generation,”Advances in Neural Information Processing Systems, vol. 36, pp. 15903–15935, 2023

    2023

  60. [68]

    All seeds are not equal: Enhancing compositional text-to-image generation with reliable random seeds,

    S. Li, H. Le, J. Xu, and M. Salzmann, “All seeds are not equal: Enhancing compositional text-to-image generation with reliable random seeds,”arXiv preprint arXiv:2411.18810, 2024

    2024 arXiv

  61. [69]

    Self-rectifying diffusion sampling with perturbed-attention guidance,

    D. Ahn, H. Cho, J. Min, W. Jang, J. Kim, S. Kim, H. H. Park, K. H. Jin, and S. Kim, “Self-rectifying diffusion sampling with perturbed-attention guidance,” inEuropean Conference on Computer Vision, pp. 1–17, Springer, 2024. 36

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.