Pith. sign in

REVIEW 4 major objections 4 minor 39 references

Text-to-image models can be steered toward compositional faithfulness at inference time by reward-aligning the sampling distribution to the model's own concept likelihoods.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:50 UTC pith:DQUCZZNN

load-bearing objection Useful framework, but the 1/K weights in the guidance (Eqs. 16–17) don't match the stated reward (Eq. 13/18), so the rigorous-derivation claim fails as written. the 4 major comments →

arxiv 2607.21606 v1 pith:DQUCZZNN submitted 2026-05-16 cs.AI

TILT: Improving Compositional Generation in Diffusion Models with a Model-Intrinsic Reward

Debottam Dutta , Jaehoon Hahm , Jianchong Chen , Romit Roy Choudhury This is my paper
classification cs.AI
keywords compositional generationdiffusion modelstest-time reward alignmentclassifier-free guidanceTweedie estimatetotal correlationmodel-intrinsic rewardpure-mode sampling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper is trying to establish that the compositional failures of modern text-to-image diffusion models are not random but stem from overlap modes between the joint distribution over a prompt's concepts and the per-concept marginals, and that a reward defined from the model's own densities can tilt the sampling distribution away from those modes at test time. The proposed reward is log pθ(x0|C) − Σ log pθ(x0|ci), the log ratio of the joint conditional to the product of per-concept conditionals. Maximizing this reward under a KL constraint yields a closed-form tilted target p*(x0) ∝ pθ(x0|C) exp(βR(x0)), and the paper derives diffusion-posterior-sampling guidance steps that approximate this target. It further shows that a previous heuristic corrector, CO3, is a special case of the resulting update under two approximations, which would explain its empirical success. If the framework is right, compositional alignment can be improved at inference time without retraining, external reward models, or access to attention maps.

Core claim

The central claim is that pure-mode compositional sampling—samples where all concepts coexist with balanced likelihood—is a constrained reward-alignment problem whose solution is the tilted distribution p*(x0) ∝ pθ(x0|C) exp(β(log pθ(x0|C) − Σ_i log pθ(x0|ci))). Guidance rules follow via diffusion posterior sampling: the intractable reward gradient becomes a Jacobian-vector product through the Tweedie estimate, yielding shared-Jacobian and per-concept updates plus a hybrid schedule. A prior heuristic corrector is recovered from the shared-Jacobian update with identity Jacobian and time-frozen scores, grounding that heuristic in the same objective.

What carries the argument

The central object is the reward-tilted target p*(x0) ∝ pθ(x0|C) exp(βR(x0)), with R(x0)=log pθ(x0|C) − Σ_i log pθ(x0|ci). The load-bearing identity is the Bayes decomposition showing that, up to constants, this reward equals the pointwise conditional total correlation ιC(x)=log pθ(C|x)/∏pθ(ci|x), and that its expectation over a prompt distribution aggregates to the conditional Total Correlation of the concepts given the image. The guidance machinery is DPS-style posterior sampling: a binary observation O=1 for high reward, a likelihood p(O=1|x0,C)∝exp(−L(x0)) with L=−R, and a chain rule that translates the reward gradient into a Jacobian-vector product applied to t=0 score evaluations on th

Load-bearing premise

The argument hinges on treating the score network evaluated at the Tweedie estimate with time index 0 as the true gradient of an image's log-likelihood under the joint and per-concept conditionals (the replacement made in §3.3); if those time-0 scores are biased or off the data manifold, the sampler is no longer targeting the tilted distribution, and the paper's own limitation section acknowledges that such model-likelihood gradient optimization can be unstable.

What would settle it

Run TILT-H and a standard classifier-free guidance baseline on a set of prompts with known concept composition, and measure whether the predicted reward signal sθ(ˆx0,0,C) − (1/K)Σ_i sθ(ˆx0,0,ci) is positive for images that contain all concepts and negative for images missing one. If the sign or magnitude of this t=0 score difference does not correlate with actual concept presence, the reward is not measuring what the paper claims, and the resulting guidance cannot be the one that induces p*.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the paper is right, compositional failure is a mode-overlap problem: samples that honor only one concept live where the joint distribution overlaps a per-concept marginal, and suppressing those modes should reduce concept omission.
  • A training-free, model-intrinsic reward is sufficient: no retraining, no external reward models, and no access to cross-attention internals, so the method transfers across architectures that expose conditional scores.
  • The derived guidance steps, rather than the heuristic corrector, can be credited with the improvements on the Shape and Complex prompt categories, where multi-concept reasoning dominates.
  • The recovery of a prior corrector as a special case explains that method's empirical success and gives future test-time samplers a single objective to optimize.
  • Because the derivation is modality-agnostic and only needs factorizable conditioning variables, the same pure-mode reward alignment applies to text-to-audio, molecular generation, and multi-attribute editing—the future work the paper proposes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because the reward signal is read off the score network at t=0, whether the framework truly induces p* is testable in isolation: replace sθ(ˆx0,0,·) with a clean-image likelihood proxy (e.g., an off-the-shelf concept classifier) and compare guidance quality; a divergence would localize the error to this approximation.
  • Editorial inference: The prompt parser strips adjectives and articles to noun-only sub-prompts; the paper's weaker Color-category results are consistent with this choice, and an attribute-aware decomposition (e.g., 'a black cat' as concept 'black cat') would test whether the marginal definition, not the reward, limits attribute binding.
  • Editorial inference: The total-correlation interpretation implies the reward is only informative when the model couples concepts through shared visual structure; on a model whose per-concept posteriors are nearly independent given an image, the reward collapses and the method should produce no gains—a testable diagnostic of the model's compositional coupling.
  • Editorial inference: The closed-form target suggests that any training-free composition heuristic that can be written as a reweighting of the joint can be compared to p*, offering a way to benchmark correction schemes without running them.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents TILT, a training-free method for compositional text-to-image generation. It formalizes concept dominance as overlap between joint and per-concept conditionals, defines an intrinsic pure-mode reward R(x0)=log pθ(x0|C)−Σ_i log pθ(x0|ci), and proposes to sample from a KL-tilted target p*(x0)∝pθ(x0|C) exp(βR(x0)). From this target it derives DPS-style guidance updates (TILT-S and TILT-C) and a hybrid schedule. Experiments on T2ICompBench with SDXL compare against CFG, Composable Diffusion, R2F, CFG++, and CO3, reporting ImageReward, CLIP, DINO, and BLIP-VQA. The paper also claims that CO3 emerges as a special case of the framework, providing theoretical grounding for that prior method.

Significance. If the derivation were correct, this would be a valuable contribution: a principled, training-free, model-intrinsic reward-alignment framework that unifies existing correction methods and offers a general test-time alignment recipe. The empirical study is broad, uses external baselines, and the method is fully training-free. However, the manuscript contains a systematic mismatch between the stated reward and the implemented guidance, together with an unvalidated score-at-t=0 extrapolation. These issues directly affect the central theoretical claim and must be resolved before the paper's main contribution can be accepted.

major comments (4)
  1. [§3.3, Eqs. 16–17; §3.2, Eq. 13; Appendix C] The stated reward R(x0)=log pθ(x0|C)−Σ_i log pθ(x0|ci) has gradient sθ(·,0,C)−Σ_i sθ(·,0,ci). Substituting this into the DPS-style chain rule gives a guidance term with −Σ_i, not −(β/K)Σ_i. Equations 16 and 17 therefore implement guidance for the different reward R_K(x0)=log pθ(x0|C)−(1/K)Σ_i log pθ(x0|ci), i.e., a geometric-mean inverse-marginal reward, with tilted target p*_K(x0)∝pθ(x0|C) exp(βR_K(x0)). Appendix C makes the same −β/K choice when recovering CO3, so the issue is not an isolated typo. As written, the central claim that TILT-S/TILT-C are rigorously derived from Eq. 13 fails; the paper must either redefine the reward to be R_K (updating Prop. 3.1, Eq. 18, and the interpretation) or change the guidance to use −βΣ_i.
  2. [§3.3, Eq. 16] The derivation replaces ∇_{x0} log pθ(x0|·) with sθ(x̂0,0,·), evaluating the score network at time t=0 on a Tweedie estimate. Score networks are trained for t>0; t=0 is the data manifold where the score is not defined by the training objective, and x̂0 may lie off it. This is not a minor detail: the reward gradient is the entire steering signal, and the claim that the sampler targets p* depends on this being an accurate surrogate. The paper should either provide a formal justification (e.g., the relationship between sθ(x̂0,t,·) and the likelihood gradient) or empirical validation (e.g., comparing the t=0 score with a finite-difference/actual log-likelihood gradient on relevant samples).
  3. [§4.1 / Appendix E] The concept parser strips adjectives and articles, so for a prompt 'a black cat and a brown dog' the per-concept prompts are 'cat' and 'dog'. The reward actually optimized is therefore log pθ(x|C)−log pθ(x|'cat')−log pθ(x|'dog'), not the reward in Eq. 13 for the concept conditionals appearing in C. This mismatch is consequential for attribute-binding categories (Color, Shape, Texture), where the denominator does not contain the attribute information that the joint-conditioned numerator is trying to enforce. The weak Color/BLIP-VQA results in Table 1 are consistent with this. The manuscript should either parse full noun phrases (or otherwise define ci consistently with C) and report the corresponding results, or explicitly reframe the method as a noun-level composition corrector.
  4. [§3.1 / Appendix B] Proposition 3.1 states that the reward decomposes into pointwise conditional Total Correlation 'modulo x-independent terms', but Eq. 20 contains (1−K) log pθ(x), which is x-dependent. Equation 24 then treats this term as 'constant in any optimization that varies x', which is not correct: varying x changes log pθ(x). The equality can only hold approximately, for a sufficiently small KL radius ε (or if pθ(x) is uniform). The proposition and Appendix B should state this approximation explicitly; the current wording overstates the information-theoretic identity.
minor comments (4)
  1. [Throughout] Typos: 'braod' in §3.3, 'mathmatically' in §6, 'promtps' in the Figure 1 caption. Please proofread.
  2. [Table 1 / Abstract] The abstract's claim of 'improves compositional alignment' is stronger than the table supports. TILT does not consistently beat CO3 or CFG++; e.g., Color BLIP-VQA: TILT 0.5770 vs CO3 0.6326; Texture ImageReward: TILT 0.5929 vs CFG++ 0.6053. The text should qualify this claim.
  3. [Algorithm 1] The stopping index M is not defined in the main text; clarify how it is set in the experiments (the text mentions the first 5 denoising steps).
  4. [§4.2] The statement 'our method achieves the best ImageReward score on the Shape and Complex categories' is correct, but the reader may also note that CFG++ has the highest Texture ImageReward; consider reporting all best entries explicitly.

Circularity Check

2 steps flagged

The implemented guidance uses a −1/K per-concept weight absent from the stated reward and imported from the authors' prior CO3, so Algorithm 1 targets a different tilted distribution than Eq. 13.

specific steps
  1. other [§3.3, Eq. 13 vs Eqs. 16–17]
    "p∗(x0) = 1/Z pθ(x0|C) exp(β R(x0)) (13), where R(x0) := log pθ(x0|C)QK i=1 pθ(x0|ci) ... Eq. 16: ≈sθ(ˆx0, t|C) + Jxt(ˆx0)⊤{β sθ(ˆx0,0,C) − β/K ΣK i=1 sθ(ˆx0,0,ci)}"

    Differentiating the stated reward R(x0) = log pθ(x0|C) − Σ_i log pθ(x0|ci) gives ∇R = sθ(·,0,C) − Σ_i sθ(·,0,ci), with weight −1 on each per-concept score. Equations 16–17 instead place weight −β/K on each per-concept score. Thus the implemented guidance does not steer toward the tilted target p* of Eq. 13; it steers toward a different target with reward log pθ(x0|C) − (1/K)Σ_i log pθ(x0|ci). The −1/K is not obtained by differentiating R; it appears only later in Appendix C as CO3's heuristic weight. The derivation from the announced reward therefore breaks at this step by construction.

  2. ansatz smuggled in via citation [§3.3 'Comparison with CO3' and Appendix C]
    "Appendix C shows that CO3’s update is recovered from TILT-S by setting the diffusion Jacobian to identity and freezing score evaluation at (xt, t). Thus, CO3 appears as a special case of the proposed framework. ... CO3 forms a weighted Tweedie-mean composition ... with weights w0 = 1+β and wk =−β/K"

    The −β/K coefficient in Eq. 16 is exactly CO3's wk from Appendix C, so CO3 is not independently recovered from the reward formulation: the coefficient is chosen to match CO3, and then the match is presented as theoretical grounding. Because CO3 is the same group's prior work, the chain 'CO3 heuristic → TILT-S coefficient → CO3 as special case' closes through a self-citation. The claim that this provides a principled justification for CO3's empirical success is therefore not independent evidence; the special-case relationship is baked into the chosen per-concept weight rather than following from the stated reward.

full rationale

The construction up to Eq. 13 is self-contained: the KL-constrained optimization and exponential tilting of pθ(x0|C) are standard, and the evaluation is anchored to external benchmarks (CFG, Comp-Diff, CFG++, etc.). However, the load-bearing step from the stated reward to the implemented Eqs. 16–17 is not a faithful differentiation of R. The per-concept coefficient should be −1 if the target is Eq. 13, but the paper uses −β/K, the same coefficient appearing in CO3's heuristic weights (Appendix C). The paper then advertises CO3's recovery as theoretical grounding. This is partial circularity: the central claim that the guidance objectives are rigorously derived from the reward formulation is weakened because the implemented update is constructed to match the authors' prior method, and the resulting target distribution differs from the announced p*. The empirical results themselves are not circular, but the derivation of the algorithm from the stated objective is.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 1 invented entities

The central claim rests on tuning knobs the paper does not ablate (β_S=0.05, β_C=0.004, unreported τ, 5-step window, 10/5 iterations), an unstated reward-normalization choice (1/K) that makes the guidance match CO3 rather than the stated reward, a t=0 score extrapolation, a postulated pseudo-observation model, the mode-overlap hypothesis, and a noun-only concept decomposition. One invented device (event O=1) is a methodological fiction with no falsifiable handle.

free parameters (6)
  • β (reward alignment strength) = β=0.05 (TILT-S), β=0.004 (TILT-C)
    Appendix E. Controls how far the tilted target deviates from the joint conditional; set per variant with no sensitivity analysis or ablation.
  • Switching threshold τ (TILT-H) = unreported
    Algorithm 1 requires τ; the effective schedule is TILT-S at the first step and TILT-C at steps 2–5 (Appendix E), but τ's numeric value is never given.
  • Correction window and iterations = first 5 denoising steps; 10 iterations at step 1, 5 at steps 2–5
    §4.1 says 'one latent correction step per corrected timestep' while Appendix E says 5 optimization steps per timestep; the window is motivated only by the claim that global composition is determined early.
  • Per-concept weight 1/K in Eqs. 16–17 = 1/K
    The stated reward (Eqs. 13, 18) has weight 1 on each concept term; the implemented guidance uses 1/K. Not derived; matches the CO3 weights in Appendix C. Consistent only with a geometric-mean reward the paper never states.
  • Noun-only concept parsing = sub-prompts with attributes stripped
    Appendix E: 'a black cat and a brown dog' becomes c1='cat', c2='dog'; for attribute-binding prompts this may be the wrong marginal to divide out.
  • CFG / CFG++ guidance scales = 5.0 (CFG), 0.8 (CFG++)
    §4.1: user-chosen but shared by all methods; a fairness control rather than a fitted constant.
axioms (6)
  • domain assumption Score network evaluated at t=0 on the Tweedie estimate equals the gradient of the clean-data log-likelihood: sθ(ˆx0, 0, c) ≈ ∇_{x0} log pθ(x0|c)
    Invoked in Eqs. 16–17 to turn the intractable reward gradient into 'a simple model forward pass' (§3.3). Unvalidated extrapolation: score networks are trained on noisy latents.
  • domain assumption DPS approximation for the reward posterior: ∇xt log p(O=1|xt, C) ≈ −∇xt L(ˆx0(xt)) via the one-step Tweedie estimate
    §3.3, Eq. 14; standard in inverse-problem diffusion literature (Chung et al. 2022) but still an approximation whose error is not discussed.
  • ad hoc to paper Postulated likelihood p(O=1|x0, C) ∝ exp(R(x0)) defines the 'event of observing a high-reward sample'
    §3.3: the Bayes conditioning in Eq. 14 is a construction, not a measurement model; any reward can be packaged this way.
  • domain assumption Mode-overlap hypothesis: compositional failures are dominated by samples from regions where the joint conditional overlaps a single-concept marginal
    §3, Figure 1: support is a 2-D histogram illustration with no N, prompt list, or quantitative overlap test; the entire reward design rests on this premise.
  • domain assumption Noun-only concept marginals are the correct divisors: pθ(x|'cat') should suppress overlap for 'a black cat and a brown dog'
    Appendix E strips articles and adjectives; attributes (color, texture) are excluded from concept prompts, which may mis-model the failure mode for attribute-binding categories.
  • domain assumption KL-smallness implies expectation closeness: |E_{p*}[R] − E_{pθ(·|C)}[R]| = O(ε)
    Eq. 23 / Appendix B.3: requires regularity of R that is not stated; for unbounded rewards a small KL shift need not bound the expectation shift.
invented entities (1)
  • Binary observation event O=1 ('sample has high reward') no independent evidence
    purpose: Converts reward maximization into Bayesian conditioning so that DPS-style guidance applies
    Introduced in §3.3 with p(O=1|x0, C) ∝ exp(−L(x0)); a methodological fiction in the control-as-inference style, not a measurable event. No falsifiable handle outside the paper.

pith-pipeline@v1.3.0-alltime-deepseek · 14422 in / 37026 out tokens · 304105 ms · 2026-08-02T13:50:19.802743+00:00 · methodology

0 comments
read the original abstract

Recent advances in powerful text-to-image generation models have made it increasingly important to develop test-time methods that modify the sampling trajectory to produce images more faithful to complex compositional prompts. We present TILT, a training-free framework for compositional text-to-image generation via test-time reward alignment. We interpret compositional failures as overlap modes between joint and single-concept distributions, and define a reward that favors samples where all concepts are jointly present. This reward is intrinsic to the base model and does not require any external supervision or reward models. This yields a KL-constrained objective with a closed-form tilted target distribution and principled guiding steps for diffusion sampling. The interaction of concept distributions together with the above reward naturally leads to two different guidance strategies while a hybrid approach that balances their respective benefits produces stronger performance. Experiments on prompts from T2ICompBench show that our method improves compositional alignment while preserving image quality compared to previous baselines.

Figures

Figures reproduced from arXiv: 2607.21606 by Debottam Dutta, Jaehoon Hahm, Jianchong Chen, Romit Roy Choudhury.

Figure 1
Figure 1. Figure 1: Histogram of alignment scores (DINOv2) between each sub-prompts c1, c2 and the generated images. The text promtps are in the form of "A [Animal] and a [Object]". Given a pretrained conditional score sθ(xt, t | c), a composi￾tional prompt C = {c1, . . . , cK}, and concept conditionals p θ (x | C) (joint) and p θ (x | ci) (per-concept), we define xˆ0(xt) as the Tweedie posterior mean (with superscripts indic… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of different test-time correction methods. (a) CO3 infers multiple denoised samples xˆ0,C , {xˆ0,ci }, and then heuristically proposes new sample xˆ ′ 0 as a convex combination of them. (b) TILT (Ours) proposes new sample xˆ ′ 0 with the gradi￾ent of a reward function, using a more mathematically principled approach. We observe that this improves the empirical composi￾tionality of the generated … view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison of text-to-image compositional generation methods on T2ICompBench prompts. The prompts, ordered by row, are drawn from the color, shape, texture, and complex categories. Our method shows improved text alignment with better compositional consistency compared to prior baselines. compositional generation by interpolating between frequent￾and rare-concept distributions. Kwon & Ye (2025) … view at source ↗
Figure 4
Figure 4. Figure 4: More qualitative comparison on T2ICompBench prompts. The prompts, ordered by row, are drawn from the color, shape, texture, and complex categories. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative visualization of some failure examples from T2ICompBench. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 17 linked inside Pith

  1. [1]

    Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models

    Chefer, H., Ratzon, O., Paiss, R., and Wolf, L. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. In ACM SIGGRAPH Conference on Computer Graphics and Interactive Techniques, 2023

    2023

  2. [2]

    T., Klasky, M

    Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., and Ye, J. C. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687, 2022

    Pith/arXiv arXiv 2022

  3. [3]

    Y., Nam, H., and Ye, J

    Chung, H., Kim, J., Park, G. Y., Nam, H., and Ye, J. C. Cfg++: Manifold-constrained classifier free guidance for diffusion models. arXiv preprint arXiv:2406.08070, 2024 a

    Pith/arXiv arXiv 2024

  4. [4]

    Chung, H., Lee, S., and Ye, J. C. Decomposed diffusion sampler for accelerating large-scale inverse problems. In The Twelfth International Conference on Learning Representations, 2024 b . URL https://openreview.net/forum?id=DsEhqQtfAG

    2024

  5. [5]

    B., Dieleman, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., and Grathwohl, W

    Du, Y., Durkan, C., Strudel, R., Tenenbaum, J. B., Dieleman, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., and Grathwohl, W. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc, 2024. URL https://arxiv.org/abs/2302.11552

    Pith/arXiv arXiv 2024

  6. [6]

    Dutta, D., Chen, J., Rajagopalan, R., Wei, Y.-L., and Choudhury, R. R. Steer away from mode collisions: Improving composition in diffusion models. In The Fourteenth International Conference on Learning Representations, 2026

    2026

  7. [7]

    E., and Wang, W

    Feng, W., He, X., Fu, T.-J., Jampani, V., Akula, A., Narayana, P., Basu, S., Wang, X. E., and Wang, W. Y. Training-free structured diffusion guidance for compositional text-to-image synthesis. In Advances in Neural Information Processing Systems (NeurIPS) Workshops, 2022. arXiv:2212.05032

    Pith/arXiv arXiv 2022

  8. [8]

    Z., Salakhutdinov, R., and Ermon, S

    He, Y., Murata, N., Lai, C.-H., Takida, Y., Uesaka, T., Kim, D., Liao, W.-H., Mitsufuji, Y., Kolter, J. Z., Salakhutdinov, R., and Ermon, S. Manifold preserving guided diffusion, 2023. URL https://arxiv.org/abs/2311.16424

    Pith/arXiv arXiv 2023

  9. [9]

    and Salimans, T

    Ho, J. and Salimans, T. Classifier-free diffusion guidance. In Advances in Neural Information Processing Systems (NeurIPS) Workshop on Deep Generative Models and Downstream Applications, 2022

    2022

  10. [10]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, 2020. URL https://arxiv.org/abs/2006.11239. v2

    Pith/arXiv arXiv 2006

  11. [11]

    Ella: Equip diffusion models with llm for enhanced semantic alignment

    Hu, X., Wang, R., Fang, Y., Fu, B., Cheng, P., and Yu, G. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024. URL https://arxiv.org/abs/2403.05135

    Pith/arXiv arXiv 2024

  12. [12]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

    Huang, K., Sun, K., Xie, E., Li, Z., and Liu, X. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36: 0 78723--78747, 2023

    2023

  13. [13]

    Dense text-to-image generation with attention modulation

    Kim, Y., Lee, J., Kim, J.-H., Ha, J.-W., and Zhu, J.-Y. Dense text-to-image generation with attention modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. URL https://openaccess.thecvf.com/content/ICCV2023/papers/Kim_Dense_Text-to-Image_Generation_with_Attention_Modulation_ICCV_2023_paper.pdf

    2023

  14. [14]

    and Ye, J

    Kwon, G. and Ye, J. C. Tweediemix: Improving multi-concept fusion for diffusion-based image/video generation, 2025. URL https://arxiv.org/abs/2410.05591

    Pith/arXiv arXiv 2025

  15. [15]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp.\ 12888--12900. PMLR, 2022

    2022

  16. [16]

    Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., and Lee, Y. J. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. URL https://openaccess.thecvf.com/content/CVPR2023/papers/Li_GLIGEN_Open-Set_Grounded_Text-to-Image_Generation_CVPR_2023_paper.pdf

    2023

  17. [17]

    Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. B. Compositional visual generation with composable diffusion models. In Computer Vision -- ECCV 2022, volume 13677 of Lecture Notes in Computer Science, pp.\ 325--343. Springer, 2022. doi:10.1007/978-3-031-19790-1\_26

    doi:10.1007/978-3-031-19790-1 2022

  18. [18]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., Shan, Y., and Qie, X. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023. URL https://arxiv.org/abs/2302.08453

    Pith/arXiv arXiv 2023

  19. [19]

    Dinov2: Learning robust visual features without supervision

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Rare-to-frequent: Unlocking compositional generation power of diffusion models on rare concepts with llm guidance

    Park, D., Kim, S., Moon, T., Kim, M., Lee, K., and Cho, J. Rare-to-frequent: Unlocking compositional generation power of diffusion models on rare concepts with llm guidance. arXiv preprint arXiv:2410.22376, 2024

    arXiv 2024

  21. [21]

    Grounded text-to-image synthesis with attention refocusing

    Phung, Q., Ge, S., and Huang, J.-B. Grounded text-to-image synthesis with attention refocusing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. URL https://openaccess.thecvf.com/content/CVPR2024/papers/Phung_Grounded_Text-to-Image_Synthesis_with_Attention_Refocusing_CVPR_2024_paper.pdf

    2024

  22. [22]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Qi, P., Zhang, Y., Zhang, Y., Bolton, J., and Manning, C. D. S tanza: A Python natural language processing toolkit for many human languages. In Celikyilmaz, A. and Wen, T.-H. (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp.\ 101--108, Online, July 2020. Association for Computationa...

    doi:10.18653/v1/2020.acl-demos.14 2020

  24. [24]

    Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation

    Qu, L., Wu, S., Fei, H., Nie, L., and Chua, T.-S. Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation. arXiv preprint arXiv:2308.05095, 2023. URL https://arxiv.org/abs/2308.05095

    Pith/arXiv arXiv 2023

  25. [25]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PmLR, 2021

    2021

  26. [26]

    Hierarchical text-conditional image generation with clip latents

    Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  27. [27]

    High-resolution image synthesis with latent diffusion models

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 10684--10695, 2022

    2022

  28. [28]

    Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2022 a

    2022

  29. [29]

    K., Karagol Ayan, B., Mahdavi, S

    Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Seyed Ghasemipour, S. K., Karagol Ayan, B., Mahdavi, S. S., Gontijo-Lopes, R., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2022 b

    2022

  30. [30]

    The superposition of diffusion models using the it\^o density estimator

    Skreta, M., Atanackovic, L., Bose, J., Tong, A., and Neklyudov, K. The superposition of diffusion models using the it\^o density estimator. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=2o58Mbqkd2

    2025

  31. [31]

    Denoising diffusion implicit models

    Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021

    2021

  32. [32]

    S., Girdhar, R., Misra, I., and Darrell, T

    Wang, X., Rambhatla, S. S., Girdhar, R., Misra, I., and Darrell, T. Instancediffusion: Instance-level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. URL https://arxiv.org/abs/2402.03290

    Pith/arXiv arXiv 2024

  33. [33]

    Information theoretical analysis of multivariate correlation

    Watanabe, S. Information theoretical analysis of multivariate correlation. IBM J. Res. Dev., 4 0 (1): 0 66–82, January 1960. ISSN 0018-8646. doi:10.1147/rd.41.0066. URL https://doi.org/10.1147/rd.41.0066

    doi:10.1147/rd.41.0066 1960

  34. [34]

    Xie, J., Li, Y., Huang, Y., Liu, H., Zhang, W., Zheng, Y., and Shou, M. Z. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. URL https://arxiv.org/abs/2307.10816

    Pith/arXiv arXiv 2023

  35. [35]

    Imagereward: Learning and evaluating human preferences for text-to-image generation

    Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 0 15903--15935, 2023

    2023

  36. [36]

    Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms

    Yang, L., Yu, Z., Meng, C., Xu, M., Ermon, S., and Cui, B. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024. URL https://arxiv.org/abs/2401.11708

    Pith/arXiv arXiv 2024

  37. [37]

    Training-free diffusion model alignment with sampling demons, 2025

    Yeh, P.-H., Lee, K.-H., and Chen, J.-C. Training-free diffusion model alignment with sampling demons, 2025. URL https://arxiv.org/abs/2410.05760

    Pith/arXiv arXiv 2025

  38. [38]

    Adding conditional control to text-to-image diffusion models

    Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. URL https://openaccess.thecvf.com/content/ICCV2023/papers/Zhang_Adding_Conditional_Control_to_Text-to-Image_Diffusion_Models_ICCV_2023_paper.pdf

    2023

  39. [39]

    Zhao, P., Li, H., Jin, R., and Zhou, S. K. Loco: Locally constrained training-free layout-to-image synthesis. arXiv preprint arXiv:2311.12342, 2023. URL https://arxiv.org/abs/2311.12342

    Pith/arXiv arXiv 2023