Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Unlocking the Potential of Text-to-Image Diffusion with PAC-Bayesian Theory

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a PAC-Bayesian prior over attention maps, applied during denoising, improves attribute-object alignment in text-to-image diffusion models and comes with a generalization guarantee.

desk verdict The reader is right: the PAC-Bayes theory breaks at Eq (6) and the generalization-guarantee claim collapses, though the empirical attention-loss recipe underneath is coherent and worth a serious referee. read the letter →

arxiv 2411.17472 v1 pith:ER6T4OKI submitted 2024-11-25 cs.CV cs.LGstat.ML

Eric Hanchen Jiang , Yasi Zhang , Zhi Zhang , Yixin Wan , Andrew Lizarraga , Shufan Li , Ying Nian Wu This is my paper · ORCID
classification cs.CVcs.LGstat.ML
keywords text-to-imagediffusionPAC-Bayesattentionmapsattribute-objectalignmenttraining-freeguidancecompositionalgenerationCLIPsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that attribute-object misalignment in text-to-image diffusion models can be reduced without retraining by steering cross-attention maps during the denoising loop with priors drawn from PAC-Bayesian theory. The method builds a combined attention distribution, chooses a uniform distribution as the prior, and minimizes the KL divergence between them together with three linguistically motivated losses: separation of different objects, alignment of modifiers to their nouns, and suppression of irrelevant tokens. The authors claim that this procedure inherits a PAC-Bayes generalization guarantee and, as a training-free add-on to Stable Diffusion, achieves the highest CLIP-similarity scores on the AnE, DVMP, and ABC-6K benchmarks. The proposal matters because it turns attention maps into an interpretable object of control rather than requiring a new model, new training data, or fine-tuning.

What carries the argument

The load-bearing object is the factorized attention distribution $A(l)$ over spatial locations, built as a product of per-token attention maps $A_i,A_j,A_k$ raised to indicator-weighted exponents, together with the identity that decomposes $D_{\mathrm{KL}}(A\|U)$ into a weighted sum of per-map KL divergences plus a constant. This identity is what converts a single PAC-Bayes divergence into interpretable guidance: divergence between object maps, similarity between modifier and noun maps, and suppression of 'outside' tokens. The same machinery supplies the regularizer $\mathcal{R}_{\mathrm{PAC}}$, so the whole pipeline is a gradient update on latents with respect to a loss whose terms all come from one distributional prior.

What would settle it

Numerically evaluate both sides of Eq (6) using the actual aggregated attention maps extracted from Stable Diffusion on a multi-object prompt such as 'a yellow bird and a blue lion'; if the identity fails up to a constant, the regularizer is not the KL divergence it claims to be. As a behavioral check, replace $\mathcal{R}_{\mathrm{PAC}}$ with a fixed scalar of the same magnitude and re-run the AnE evaluation to see whether CLIP scores change.

Watch

Extended reading notes

Core claim

The central discovery the paper puts forward is that the attention mechanism of a diffusion model can be treated as a posterior distribution over attention maps, with a uniform prior, so that improving generation becomes a matter of minimizing $D_{\mathrm{KL}}(A \| U)$ plus task-specific terms. Concretely, it defines $A(l) \propto \frac{1}{Z} \prod_{i \in \mathcal{A}_1} (A_i(l))^{\alpha \mathbf{1}_{\Omega_i}(l)} \prod_{j \in \mathcal{A}_2} (A_j(l))^{\beta \mathbf{1}_{\Omega_j}(l)} \prod_{k \in \mathcal{A}_3} (A_k(l))^{\gamma \mathbf{1}_{\Omega_k}(l)}$ and asserts the decomposition $D_{\mathrm{KL}}(A \| U) = \alpha \sum_i D_{\mathrm{KL}}(A_i \| U) + \beta \sum_j D_{\mathrm{KL}}(A_j \| U) + \gamma \sum_k D_{\mathrm{KL}}(A_k \| U) + C$. That identity turns the PAC-Bayes bound into a per-map regularizer $\mathcal{R}_{\mathrm{PAC}}$, and the total loss $\lambda_{\mathrm{div}}\mathcal{L}_{\mathrm{div}} + \lambda_{\mathrm{sim}}\mathcal{L}_{\mathrm{sim}} + \lambda_{\mathrm{out}}\mathcal{L}_{\mathrm{out}} + \lambda_{\mathrm{PAC}}\mathcal{R}_{\mathrm{PAC}}$ is minimized with respect to the latents $z_t$ over the first half of the denoising steps. The paper reports that this recipe improves CLIP Full Similarity, Minimum Similarity, and Text-Caption Similarity over Stable Diffusion, SynGen, Attend-and-Excite, and EBAMA on the AnE benchmark, and over those baselines on DVMP and ABC-6K, with ablations attributing part of the gain to each loss component.

Load-bearing premise

The load-bearing premise is that the divergence between the combined attention distribution and the uniform prior equals the weighted sum of the per-map divergences given in Eq (6), so that minimizing those per-map KL terms really minimizes the quantity the PAC-Bayes bound controls.

Editorial extensions

If this is right

  • Because the intervention is training-free and operates only on cross-attention maps and latents, it can be applied to any Stable Diffusion-style model without re-training or fine-tuning.
  • Attribute binding and object inclusion improve on multi-object prompts, as measured by CLIP-based Full, Minimum, and Text-Caption Similarity on AnE, DVMP, and ABC-6K.
  • The PAC-Bayes regularizer contributes a measurable share of the gain: removing it in the paper's ablation lowers Full Similarity by about 1.74 percent on the ABC dataset.
  • The framework offers a general recipe for injecting linguistic structure, such as parser-identified modifier-noun pairs, into any attention-based generative model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: If the Eq (6) decomposition fails on real attention maps because each map is not the sole active component on its support, the method's empirical gains may remain, but its stated PAC-Bayes generalization guarantee would not hold; the recipe would then stand or fall as a heuristic loss.
  • Inference: The same prior-loss framework could be tested on self-attention layers or on newer base models such as SDXL, a stress test the paper does not run.
  • Inference: The repeated pattern in the paper's ablations, where too little regularization underfits and too much overemphasizes dominant attributes, suggests the confidence and step-size settings are doing much of the work, so a worthwhile extension is to calibrate them per prompt rather than per dataset.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a training-free method for improving attribute-object alignment in text-to-image diffusion models. It treats cross-attention maps as probability distributions, defines a factorized posterior distribution over these maps (Eq. 5), and derives a KL-based regularizer (Eq. 6) that is claimed to follow from PAC-Bayes theory. The method adds three heuristic losses (divergence, similarity, outside) plus the PAC regularizer to update latents during denoising (Algorithm 1). Experiments on AnE, DVMP, and ABC-6K report CLIP-similarity scores that are said to outperform existing baselines. The central claims are that the approach 'provides theoretical generalization guarantees' (Abstract, Section 4.3) and achieves state-of-the-art results.

Significance. If the theoretical claims were valid, the paper would offer a principled PAC-Bayesian justification for attention-map regularizers in generative models, which would be a meaningful contribution to a growing literature on attention guidance. The practical idea of designing task-specific priors over attention maps and updating latents during inference is sensible and aligns with prior work such as SynGen and EBAMA, but it is not novel in itself. The paper also ships no code and the empirical evaluation is thin: no error bars, limited metrics, and small differences from baselines. The main theoretical derivation (Eq. 6 and Appendix C) is algebraically incorrect, and the link to the PAC-Bayes bound is not established. As a result, the paper's core contribution—a theoretical guarantee—collapses. The empirical results, even if reproducible, would not by themselves support the paper's stated contributions.

major comments (4)
  1. [Eq. (6) and Appendix C] The KL decomposition in Eq. (6) does not follow from the factorized distribution in Eq. (5). Eq. (5) defines A(l) as a normalized product (geometric mean) of the per-token attention maps Ai, Aj, Ak. The appendix claims that 'A(l) aligns with Ai(l) when l is in Ωi', which is false because A(l) is a function of all component maps, not equal to any single Ai(l). Consequently, the cross-entropy term Σ_{l∈Ωi} A(l) log Ai(l) is not equal to D_KL(Ai || U) = Σ_{l∈Ωi} Ai(l) log(Ai(l)/U(l)). The constant C in Eq. (6) cannot absorb the mismatch because the discrepancy depends on the values of the attention maps themselves. Therefore, the regularizer RPAC defined in Eq. (12) is not the KL divergence D_KL(A || U) claimed, and the 'PAC-Bayes' regularization term is not derived from the bound.
  2. [Section 4.2 and Section 4.3] The PAC-Bayes bound in Eq. (3) is not connected to the actual procedure. The bound concerns a prior P and posterior Q over a hypothesis class H, with an empirical risk computed on N training samples and a confidence parameter δ. In this paper, there is no hypothesis class, no training set, and no empirical risk estimate. The procedure operates on a single prompt and a single latent trajectory during inference; N and δ are treated as free hyperparameters (Appendix D sets N implicitly and δ = 0.15). The statement in Section 4.3 that 'PAC-Bayes theorem suggests optimizing the training loss along with a regularizer... providing a high confidence guarantee' is therefore unsupported. The bound in Eq. (3) cannot be applied to the proposed algorithm as described.
  3. [Section 4.4 and Appendix D] The signs of the loss components contradict the stated objectives. Eq. (8) defines Ldiv as the negative average symmetric KL divergence, and Eq. (12) defines RPAC as the negative square-root term. Appendix D sets λdiv = -1.25 and λPAC = -0.15. Substituting these into Eq. (7), the total loss includes +1.25 * (average symmetric KL) and +0.15 * sqrt(D_KL(A||U)+...). Minimizing this total loss would therefore reduce the symmetric KL between object attention maps (encouraging similarity, not divergence) and reduce D_KL(A||U) (encouraging uniformity, not 'moving away from uniformity' as stated in Section 4.4.4). The intended behavior would require λdiv and λPAC to have the opposite sign. This suggests the actual implementation, if any, does not match the paper's description, or the loss definitions are mis-stated.
  4. [Section 6, Tables 1-2 and Figure 6] The empirical evaluation lacks error bars and statistical significance testing. Table 1 reports CLIP scores without any variance or confidence intervals, and Table 2 reports percentage changes without uncertainty. Many of the reported improvements over baselines are small (e.g., 0.348 vs 0.340 for Animal-Animal Full Sim. in Table 1), and without error bars it is impossible to tell whether the differences are meaningful. Figure 6 shows bar charts without error bars. The paper does not specify how many seeds or runs were used, nor whether the 64 images per prompt (AnE) and 4 images per prompt (DVMP/ABC-6K) are averaged with any measure of spread. The claim of state-of-the-art performance is therefore not statistically substantiated.
minor comments (5)
  1. [Section 4.3.2] There is a typo: 'uniofrm' should be 'uniform'. Also, the uniform prior U is not defined precisely: it is not clear whether U is uniform over the union of all Ω_i or over the full spatial grid.
  2. [Appendix C, Eq. (16)] The derivation contains redundant factors of 1/Z in the product expansion; the normalization constant Z appears multiple times. This is a notational error that obscures the algebra.
  3. [Table 1 and Figure 3] The paper interchangeably uses 'SG' and 'SyGen' for the same method, and 'EMAMA' in Figure 3 versus 'EBAMA' in the text. The notation should be consistent throughout.
  4. [References] Reference [34] is listed as 'Geometry-free view synthesis' but the text cites it as the Stable Diffusion paper; this is clearly a title mismatch. Also, several references (e.g., [17]) are incomplete or have odd formatting ('To appear, 2017').
  5. [Section 6] The paper says 'we randomly sample 200 prompts from each dataset' for DVMP and ABC-6K, but the previous paragraph says 'we generate 64 images per prompt for the AnE dataset and 4 images per prompt for the DVMP and ABC-6K datasets.' It is unclear whether the 200 prompts are a subset of the full datasets and how the reported scores are aggregated over prompts and images.

Circularity Check

1 steps flagged · score 6.0 of 10

The PAC-Bayes 'generalization guarantee' is a restatement of the paper's own engineered KL losses: Eq. (6) reduces DKL(A||U) to the same weighted per-attention-map terms used to build the losses in Eq. (7), so the theoretical claim is self-definitional rather than an independent first-principles result.

  1. self definitional [Section 4.3.2-4.4.4, Eqs. (5)-(7), (12); Appendix C]
    "We define a multinomial-like structured distribution A over attention maps that encode our desired properties. Specifically, we factorize A as: A(l) ∝ 1/Z ∏_{i∈A1}(Ai(l))^{αΩ_i(l)} ... A1 corresponds to the divergence component ... A2 corresponds to the similarity component ... A3 corresponds to the outside component. ... DKL(A || U) = α∑_{i∈A1} DKL(Ai || U) + β∑_{j∈A2} DKL(Aj || U) + γ∑_{k∈A3} DKL(Ak || U) + C ... Ltotal = λdivLdiv + λsimLsim + λoutLout + λPACRPAC, where λdiv = α, λsim = β, λout = γ, and λPAC = η, corresponding to the exponents in (5)."

    The paper's 'posterior' A is not an independent object: Eq. (5) constructs it by exponentiating the very per-attention-map terms (divergence, similarity, outside) that later define Ldiv, Lsim, Lout, with the same weights α, β, γ. Eq. (6) then equates DKL(A||U) to the weighted sum of those per-map KL terms, and Eq. (12) defines RPAC solely as a function of this DKL. Thus the claimed 'theoretical generalization guarantee' is the same as the chosen KL losses, restated through PAC-Bayes notation; Eq. (3) is never instantiated with an empirical risk, a hypothesis class, or a sample size N. The guarantee therefore adds no independent content beyond the definition of A.

full rationale

The empirical comparisons (Tables 1-2, Figures 3-6) are external and not circular: they evaluate against independent baselines on standard benchmarks. The self-citations ([5], [42], [44], [45]) are related-work or baseline references and are not load-bearing for the central argument. However, the paper's central theoretical claim—that RPAC provides 'theoretical generalization guarantees'—reduces by construction: A in Eq. (5) is built from the same per-attention KL components (divergence, similarity, outside) with the same weights as the losses in Eq. (7); Eq. (6) declares DKL(A||U) equal to that weighted sum; and Eq. (12) defines RPAC as a function of this same DKL. The PAC-Bayes bound in Eq. (3) is never instantiated with an empirical risk, a hypothesis class, or a sample size, so the only content of the 'guarantee' is the definitional KL term. Separately, Appendix C's step claiming 'A(l) aligns with Ai(l) when l ∈ Ωi' is algebraically false for the normalized geometric-mean product in Eq. (5), which makes the claimed reduction invalid as well as circular. Overall, the theoretical claim is partially circular (score 6), while the empirical SOTA claim is independent.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The method rests on a set of hand-chosen loss weights and schedule constants, plus the unproven applicability of the PAC-Bayes bound and KL decomposition to attention maps. No code or data artifacts are provided. The invented-entity count is zero because no new physical objects are claimed; the 'prior' is a modeling device whose content is exactly the losses being minimized.

free parameters (7)
  • λdiv (divergence loss weight) = -1.25
    Hand-set in Appendix D; with Ldiv defined with a leading minus, this sign actually minimizes KL between object attention maps, opposite of the stated separation goal.
  • λsim (similarity loss weight) = 2.0
    Hand-set; no tuning protocol or grid search is reported, and it directly controls the strength of modifier-noun binding.
  • λout (outside loss weight) = 0.15
    Hand-set; no tuning protocol is reported, and this weight multiplies a negative loss to discourage similarity to outside tokens.
  • λPAC (PAC regularizer weight) = -0.15
    Hand-set; since RPAC is defined negative, the product λPAC * RPAC is positive, which minimizes KL to the uniform prior, contradicting Section 4.4.4's claim that the regularizer pushes attention away from uniformity.
  • δ (PAC-Bayes confidence parameter) = 0.15
    Selected via visual ablations on ABC-6K (Appendix F, Figure 7); this is a free parameter fit to image quality on an evaluation benchmark.
  • T′ (number of updated denoising timesteps) = 25
    Selected via visual ablations on ABC-6K (Appendix F, Figure 8); the number of updated denoising timesteps is tuned by visual quality.
  • α (latent update step size) = 20
    Selected via visual ablations on ABC-6K (Appendix F, Figure 9); the latent update step size is tuned by visual quality.
assumptions (4)
  • standard math PAC-Bayes bound (McAllester 1999) as stated in Eq (3) is assumed applicable to attention-map guidance.
    Invoked in Section 4.2, but no hypothesis class, loss, or sample count is defined for the T2I setting, so applicability is an unverified assumption.
  • domain assumption Cross-attention maps are valid probability distributions whose KL divergences reflect semantic alignment.
    Section 3.2 and 4.3 treat normalized attention maps as distributions; the paper provides no evidence that KL geometry corresponds to attribute-object correctness.
  • domain assumption spaCy dependency parser correctly identifies nouns, modifiers, and their associations for arbitrary prompts.
    Section 4.1 relies on parsing to define loss groups; Appendix A limits this by admitting parsing errors can degrade output.
  • ad hoc to paper The factorized attention distribution in Eq (5) and the KL decomposition in Eq (6) hold.
    Appendix C derives Eq (6) by identifying A(l) with Ai(l) on each support, which is not established and is generally false for the geometric-mean form in Eq (5). This is the core of the claimed PAC-Bayes regularizer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unlocking the Potential of Text-to-Image Diffusion with PAC-Bayesian Theory." pith.science (2026). https://pith.science/paper/ER6T4OKI

@misc{pith2026241117472,
  author       = {Pith},
  title        = {Pith review of: Unlocking the Potential of Text-to-Image Diffusion with PAC-Bayesian Theory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ER6T4OKI}},
  note         = {Machine review of arXiv:2411.17472}
}
read the original abstract

Text-to-image (T2I) diffusion models have revolutionized generative modeling by producing high-fidelity, diverse, and visually realistic images from textual prompts. Despite these advances, existing models struggle with complex prompts involving multiple objects and attributes, often misaligning modifiers with their corresponding nouns or neglecting certain elements. Recent attention-based methods have improved object inclusion and linguistic binding, but still face challenges such as attribute misbinding and a lack of robust generalization guarantees. Leveraging the PAC-Bayes framework, we propose a Bayesian approach that designs custom priors over attention distributions to enforce desirable properties, including divergence between objects, alignment between modifiers and their corresponding nouns, minimal attention to irrelevant tokens, and regularization for better generalization. Our approach treats the attention mechanism as an interpretable component, enabling fine-grained control and improved attribute-object alignment. We demonstrate the effectiveness of our method on standard benchmarks, achieving state-of-the-art results across multiple metrics. By integrating custom priors into the denoising process, our method enhances image quality and addresses long-standing challenges in T2I diffusion models, paving the way for more reliable and interpretable generative models.

Figures

Figures reproduced from arXiv: 2411.17472 by the authors.

Figure 1
Figure 1. Diffusion models often struggle to accurately represent multiple objects in the input text. We identify the root causes of these [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of our workflow for optimizing the sta￾ble diffusion model. It includes aggregation of attention maps, computation of object-centric attention loss, and updates to zt. models, paving the way for more reliable and interpretable generative models. 1. Introduction Recent text-to-image (T2I) diffusion models have achieved remarkable success in generating high-fidelity, diverse, and realistic images condition… view at source ↗
Figure 3
Figure 3. Qualitative comparison on the AnE dataset (the left two columns) and the DVMP dataset (the right two columns). We compared our model using the same prompt and random seed as SD [34], SG [32], AnE [6], and EMAMA [44], with each column sharing the same random seed. 4.3.3 KL Divergence PAC-Bayes theory relies on the KL divergence between the posterior and the prior: We have DKL(A ∥ U) = X l A(l) log  A(l) U(l)  Subst… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison on the ABC-6K dataset. We compared our model using the same prompt and random seed as SD [34], SG [32], AnE [6], and EMAMA [44], with each column sharing the same random seed. where P = {(i, j) | i, j ∈ O, i < j} is the set of all unique object p…
Figure 5
Figure 5. Figure 5: Qualitative Results of the Ablation Study. Each col￾umn illustrates the generated image for the prompt ’a baby monkey and a wooden curved crown and an orange guitar,’ showing the ef￾fects of omitting specific loss components. From left to right: the full model output, …
Figure 6
Figure 6. Figure 6: Full Sim. results on DVMP and ABC-6K datasets. We randomly sample 200 prompts from each dataset and generate 4 images for each prompt. Model Score Change (%) Ours 0.344 – - RP AC 0.338 -1.74% ↓ - Lout 0.336 -0.59% ↓ - Lsim 0.331 -1.48% ↓ [PITH_FULL_IMAGE:figures/full_…
Figure 7
Figure 7. Figure 7: Ablation demonstration for PAC Regularizer’s δ (a) a man in red shirt eating something that is green; (b) A green cat with orange eyes is laying over a paper; (c) A white bus with a painting of a blue flower on the front stopped on a street near a snow-covered sidewalk…
Figure 8
Figure 8. Figure 8: Ablation demonstration for time-steps T ′ (a) a beige sliced tomato and a spotted bowl; (b) a blue clock tower is against the gray sky; (c) a rabbit and a gray chair. Timestep T ′ We test T ′ = 0, T ′ = 25 (ours), and T ′ = 50 as depicted in [PITH_FULL_IMAGE:figures/f…
Figure 10
Figure 10. Figure 10: Additional qualitative comparison on the AnE dataset. We compared our model using the same prompt and random seed as SD [34], SG [32], AnE [6], and EMAMA [44], with each column sharing the same random seed. 4 [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Additional qualitative comparison on the DVMP dataset. We compared our model using the same prompt and random seed as SD [34], SG [32], AnE [6], and EMAMA [44], with each column sharing the same random seed. 5 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Understanding Galaxy Morphology Evolution Through Cosmic Time via Redshift Conditioned Diffusion Models

    astro-ph.GA 2024-11 conditional novelty 5.0 of 10

    A redshift-conditioned diffusion model generates galaxy images whose ellipticity, size, Sersic index, and isophotal area track the morphology-redshift trends observed in the Hyper Suprime-Cam survey.

Reference graph

Works this paper leans on

46 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Karanam, Kshitijh Joseph, Ak- shara Saxena, Karan Goswami, and Balaji Vasan Srinivasan

    Ankit Agarwal, Shreyank N. Karanam, Kshitijh Joseph, Ak- shara Saxena, Karan Goswami, and Balaji Vasan Srinivasan. A-star: Test-time attention segregation and retention for text- to-image synthesis. In ICCV, 2023. 3

    work page 2023

  2. [2]

    Blended diffusion for text-driven editing of nat- ural images

    Omer Avrahami, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Blended diffusion for text-driven editing of nat- ural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208– 18218, 2022. 2, 3

    work page 2022

  3. [3]

    Large scale GAN training for high fidelity natural image synthesis

    Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019. 2

    work page 2019

  4. [4]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 2

    work page 2023

  5. [5]

    Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation

    Yingshan Chang, Yasi Zhang, Zhiyuan Fang, Yingnian Wu, Yonatan Bisk, and Feng Gao. Skews in the phe- nomenon space hinder generalization in text-to-image gen- eration. arXiv preprint arXiv:2403.16394, 2024. 2

    work page Pith review arXiv 2024

  6. [6]

    Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models

    Hila Chefer, Yuval Alaluf, Yaniv Nitzan, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models. ACM TOG, 2023. 2, 3, 5, 6, 7, 8, 4

    work page 2023

  7. [7]

    Wide stochastic networks: Gaussian limit and pac-bayesian training

    Eugenio Clerico, George Deligiannidis, and Arnaud Doucet. Wide stochastic networks: Gaussian limit and pac-bayesian training. In International Conference on Algorithmic Learn- ing Theory, pages 447–470. PMLR, 2023. 4

    work page 2023

  8. [8]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. InNeurIPS, pages 8780–8794,

Show all 46 references
  1. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  2. [10]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bj ¨orn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873–12883, 2021. 2, 3

    2021

  3. [11]

    Training- free structured diffusion guidance for compositional text-to- image synthesis

    Wenhu Feng, Xuehai He, Tsung-Jui Fu, Varun Jampani, Arun Raj Kumar Akula, Murali Narayana, Subhajit Basu, Xiaowei Evan Wang, and William Yang Wang. Training- free structured diffusion guidance for compositional text-to- image synthesis. In ICLR, 2023. 2, 3, 7

    2023

  4. [12]

    An image is worth one word: Personalizing text-to- image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patash- nik, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to- image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022. 2

    2022 arXiv

  5. [13]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, pages 2672–2680, 2014. 2

    2014

  6. [14]

    Prompt-to-prompt im- age editing with cross-attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yotam Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross-attention control. In ICLR, 2023. 2

    2023

  7. [15]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 2, 3

    2022 arXiv

  8. [16]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. In NeurIPS, pages 6840–6851,

  9. [17]

    spaCy 2: Natural lan- guage understanding with Bloom embeddings, convolutional neural networks and incremental parsing

    Matthew Honnibal and Ines Montani. spaCy 2: Natural lan- guage understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017. 4, 2

    2017

  10. [18]

    Denoising diffusion restoration models.arXiv preprint arXiv:2201.11793, 2022

    Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models.arXiv preprint arXiv:2201.11793, 2022. 3

    2022 arXiv

  11. [19]

    Diffusionclip: Text-guided im- age manipulation using diffusion models

    Seonghyeon Kim, Minjae Seo, Seunghyun Kim, Minsu Cho, and Suha Kwak. Diffusionclip: Text-guided im- age manipulation using diffusion models. arXiv preprint arXiv:2110.02711, 2022. 2

    2022 arXiv

  12. [20]

    Kingma, Tim Salimans, Ben Poole, and Jonathan Ho

    Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In NeurIPS, pages 21696– 21707, 2021. 2

    2021

  13. [21]

    Dichotomize and generalize: Pac-bayesian bi- nary activated deep neural networks

    Ga ¨el Letarte, Pascal Germain, Benjamin Guedj, and Franc ¸ois Laviolette. Dichotomize and generalize: Pac-bayesian bi- nary activated deep neural networks. Advances in Neural Information Processing Systems, 32, 2019. 4

    2019

  14. [22]

    Divide and bind: Improving long-term compositionality in text-to-image synthesis

    Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide and bind: Improving long-term compositionality in text-to-image synthesis. In 34th British Machine Vision Con- ference (BMVC), 2023. 2, 7

    2023

  15. [23]

    GLIGEN: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: Open-set grounded text-to-image generation. arXiv preprint arXiv:2301.07093, 2023. 2, 3

    2023 arXiv

  16. [24]

    Tenenbaum

    Nan Liu, Shuang Li, Yu Du, Antonio Torralba, and Joshua B. Tenenbaum. Composable diffusion: Learning energy-based models for compositional visual generation. In ECCV, 2022. 2, 3, 7

    2022

  17. [25]

    Some PAC-Bayesian theorems

    David McAllester. Some PAC-Bayesian theorems. In Con- ference on Learning Theory (COLT), 1999. 2, 3, 4, 5, 7, 1

    1999

  18. [26]

    SDEdit: Guided image synthe- sis and editing with stochastic differential equations

    Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthe- sis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021. 3

    2021 arXiv

  19. [28]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09672, 2021. 2

    2021 arXiv

  20. [29]

    Energy-based cross atten- tion for bayesian context update in text-to-image diffusion models

    Gyeong Yong Park, Jaehoon Kim, Bumsub Kim, Se- ung Wook Lee, and Jong Chul Ye. Energy-based cross atten- tion for bayesian context update in text-to-image diffusion models. arXiv preprint arXiv:2306.09869, 2023. 3, 7 9

    2023 arXiv

  21. [30]

    Im- age transformer

    Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Im- age transformer. arXiv preprint arXiv:1802.05751, 2018. 2

    2018 arXiv

  22. [31]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and et al. Learning transferable visual models from natural language supervision. In Int. Conf. Machine Learning , pages 8748–

  23. [32]

    Linguistic bind- ing in diffusion models: Enhancing attribute correspondence through attention map alignment

    Rotem Rassin, Elad Hirsch, Dror Glickman, Shauli Rav- fogel, Yoav Goldberg, and Gal Chechik. Linguistic bind- ing in diffusion models: Enhancing attribute correspondence through attention map alignment. In CVPR, 2023. 2, 3, 5, 6, 7, 8, 4

    2023

  24. [33]

    Generative ad- versarial text to image synthesis

    Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Lo- geswaran, Bernt Schiele, and Honglak Lee. Generative ad- versarial text to image synthesis. InICML, pages 1060–1069. PMLR, 2016. 2

    2016

  25. [34]

    Geometry-free view synthesis: Transformers and no 3d pri- ors

    Robin Rombach, Patrick Esser, and Bj ¨orn Ommer. Geometry-free view synthesis: Transformers and no 3d pri- ors. ICCV, 2021. 2, 5, 6, 4

    2021

  26. [35]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In NeurIPS, pages 1–13,

  27. [36]

    Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion model...

    2022 arXiv

  28. [37]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric A Weiss, Niru Mah- eswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arXiv:1503.03585, 2015. 2

    2015 arXiv

  29. [38]

    Score-based generative mod- eling through stochastic differential equations

    Yang Song and Stefano Ermon. Score-based generative mod- eling through stochastic differential equations. In ICLR,

  30. [39]

    Nor- malized flat minima: Exploring scale invariant definition of flat minima for neural networks using pac-bayesian analy- sis

    Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Nor- malized flat minima: Exploring scale invariant definition of flat minima for neural networks using pac-bayesian analy- sis. In International Conference on Machine Learning, pages 9636–9647. PMLR, 2020. 4

    2020

  31. [40]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, pages 5998– 6008, 2017. 2

    2017

  32. [41]

    Latent energy-based odyssey: Black-box optimization via expanded exploration in the energy-based latent space

    Peiyu Yu, Dinghuai Zhang, Hengzhi He, Xiaojian Ma, Ruiyao Miao, Yifan Lu, Yasi Zhang, Deqian Kong, Ruiqi Gao, Jianwen Xie, et al. Latent energy-based odyssey: Black-box optimization via expanded exploration in the energy-based latent space. arXiv preprint arXiv:2405.16730,

    arXiv

  33. [42]

    Object- conditioned energy-based model for attention map alignment in text-to-image diffusion models

    Yasi Zhang, Peiyu Yu, and Ying Nian Wu. Object- conditioned energy-based model for attention map alignment in text-to-image diffusion models. In Synthetic Data for Computer Vision Workshop @ CVPR 2024, 2024. 2

    2024

  34. [43]

    Flow priors for linear inverse problems via iterative corrupted trajectory matching

    Yasi Zhang, Peiyu Yu, Yaxuan Zhu, Yingshan Chang, Feng Gao, Ying Nian Wu, and Oscar Leong. Flow priors for linear inverse problems via iterative corrupted trajectory matching. arXiv preprint arXiv:2405.18816, 2024. 2

    2024 arXiv

  35. [44]

    Object- conditioned energy-based attention map alignment in text-to- image diffusion models

    Yasi Zhang, Peiyu Yu, and Ying Nian Wu. Object- conditioned energy-based attention map alignment in text-to- image diffusion models. In Computer Vision – ECCV 2024, pages 55–71, Cham, 2025. Springer Nature Switzerland. 2, 3, 5, 6, 7, 8, 4

    2024

  36. [45]

    Statistical guaran- tees for lifelong reinforcement learning using pac-bayesian theory

    Zhi Zhang, Chris Chow, Yasi Zhang, Yanchao Sun, Haochen Zhang, Eric Hanchen Jiang, Han Liu, Furong Huang, Yuchen Cui, and Oscar Hernan Madrid Padilla. Statistical guaran- tees for lifelong reinforcement learning using pac-bayesian theory. arXiv preprint arXiv:2411.00401, 2024. 3

    2024 arXiv

  37. [46]

    Non-vacuous generalization bounds at the imagenet scale: a pac-bayesian compression approach

    Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P Adams, and Peter Orbanz. Non-vacuous generalization bounds at the imagenet scale: a pac-bayesian compression approach. arXiv preprint arXiv:1804.05862, 2018. 4

    2018 arXiv

  38. [47]

    Think twice before you act: Improving inverse problem solving with mcmc

    Yaxuan Zhu, Zehao Dou, Haoxin Zheng, Yasi Zhang, Ying Nian Wu, and Ruiqi Gao. Think twice before you act: Improving inverse problem solving with mcmc. arXiv preprint arXiv:2409.08551, 2024. 2 10 Unlocking the Potential of Text-to-Image Diffusion with PAC-Bayesian Theory Supple...

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.