Pith. sign in

REVIEW 3 major objections 5 minor 80 references

Without retraining, AnchorSteer improves text-to-image faithfulness by anchoring prompt semantics in the initial noise and adding a mid-generation 'Think-Erase-Retouch' correction loop, lifting GenEval SDXL from 55.9 to 63.4.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

AnchorSteer improves text-to-image faithfulness by anchoring initial noise with CLIP/DAS-derived semantics (LP-SDS) and correcting errors during denoising with a VLM-driven Think-Erase-Retouch loop.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection Practical training-free alignment recipe with real benchmark gains, but the anchoring mechanism is oversold and the experimental controls are not yet tight enough. the 3 major comments →

arxiv 2607.26647 v1 pith:PF3ZK6IV submitted 2026-07-29 cs.CV

Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time

classification cs.CV
keywords text-to-image alignmentcompositional faithfulnessinference-time optimizationnoise initializationsemantic anchoringscore distillation samplingdenoising trajectory steeringdiffusion models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that faithful text-to-image generation under complex compositional prompts can be substantially improved at inference time, without retraining, by controlling both the starting noise and the denoising path. Semantic Anchoring uses a CLIP model to synthesize a text-aligned prior image, refines it through a latent-space score distillation step that pulls it onto the diffusion manifold, and then forward-noises it into the initial latent so the sampler starts from a prompt-relevant state. Reflective Steering periodically decodes the intermediate image, lets a vision-language model diagnose missing and inconsistent content, and runs a targeted erase-retouch loop that suppresses bad semantics and reinforces missing ones. On GenEval the method raises SDXL's overall score from 55.9 to 63.4 and improves most T2I-CompBench++ categories, with similar gains on a transformer-based DiT backbone, suggesting the effect is architecture-independent.

Core claim

The central claim is that the two sources of compositional failure—an uninformative initial noise and an uncorrected denoising trajectory—can be addressed jointly at inference time by injecting semantics into the initial latent and by making the trajectory self-corrective. The paper advances a specific mechanism: distill CLIP's semantic prior into the diffusion latent space via Latent-Prior Score Distillation Sampling, then diffuse that refined latent forward to produce an initial noise that is both text-aligned and distribution-consistent. Along the trajectory, Think-Erase-Retouch uses a vision-language model to turn misalignment into two actionable signals—a negative prompt for content to

What carries the argument

The load-bearing mechanism is LP-SDS (Latent-Prior Score Distillation Sampling), an optimization objective that distills CLIP visual priors into the diffusion model's latent distribution by following the frozen diffusion model's noise-prediction residual; the paper shows this is equivalent to minimizing a time-weighted KL divergence between the perturbed latent distribution and the model's conditional distribution, pulling the prior onto the diffusion manifold. The second mechanism is the Think-Erase-Retouch steering loop: a VLM decodes the Tweedie-predicted clean image, separates errors into missing (D_pos) and inconsistent (D_neg) prompts, erases via negative-guided DDIM inversion, and ret

Load-bearing premise

The core bet is that after DDPM forward noising to step T=50, the terminal latent still carries enough of the refined semantic prior to steer generation; if diffusion erases that signal, Semantic Anchoring contributes little and the reported gains rest almost entirely on Reflective Steering.

What would settle it

Run AnchorSteer on GenEval with Semantic Anchoring disabled: sample z_T purely as N(0,I) (keeping the same seed and Reflective Steering unchanged). If the overall score does not drop materially below 63.39 (or if it matches the +Steer-only 59.87), the claim that semantic information survives forward diffusion is falsified. A complementary check: measure the distance between the empirical distribution of z_T and N(0,I) or test whether shuffling the latent's spatial layout (as in Table S5) removes the gain.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Semantic Anchoring alone raises structural and attribute metrics (Counting up by about 14 points on SDXL GenEval, Attribute Binding by 2.75), indicating that prompt-aligned initial noise reduces early-stage semantic drift.
  • Reflective Steering alone boosts relational reasoning (Two Objects up by 10.6 points on SDXL), showing that mid-generation diagnosis and correction prevent error propagation.
  • The two components are complementary: full AnchorSteer beats either component alone on both SDXL and HunyuanDiT, so initialization and trajectory control address different failure modes.
  • Gains transfer across architectures—U-Net and DiT backbones both improve—suggesting the method is not tied to one model family.
  • The method costs about 68.5 seconds per image on SDXL versus 3.7 seconds for standard sampling, but it also improves a distilled four-step model, pointing to a viable speed-quality trade-off.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the semantic-leakage premise holds more broadly, LP-SDS could become a general prior-injection primitive for other conditioning signals (depth, color palettes, style) that suffer the same CLIP-to-diffusion domain gap.
  • The Think-Erase-Retouch loop suggests a modular test for controllability: the same VLM diagnosis could drive different correction policies (e.g., attention-based erasure instead of guided inversion), allowing component-wise isolation of what matters—diagnosis vs. correction.
  • Table S5 hints that the anchored initialization's benefit is not a mere distribution-shift artifact: a latent from a mismatched prompt performs no better than Gaussian, while the prompt-matched latent does. A stronger test would vary how much semantic signal survives by starting the noising process from earlier steps.
  • Because the VLM runs on decoded intermediate images, the method's ceiling is partly bounded by the VLM's diagnostic accuracy; performance differences across VLMs may track diagnosis quality more than diffusion backbone choice—a testable prediction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes AnchorSteer, a training-free, inference-time framework for improving text-image alignment in text-to-image diffusion models. It consists of two components: Semantic Anchoring, which builds a text-aligned initial noise by extracting a CLIP-based semantic prior via Direct Ascent Synthesis, refining it in latent space with LP-SDS, and then applying DDPM forward diffusion; and Reflective Steering, a VLM-driven Think-Erase-Retouch loop that diagnoses semantic deviations and corrects them through negative-guided inversion and positive-guided re-denoising. Experiments on GenEval and T2I-CompBench++ with SDXL and HunyuanDiT report consistent gains over standard sampling and several baselines, with ablations claiming complementary benefits of the two components. The paper also includes quality/diversity metrics, a user study, runtime analysis, and an extension to distilled models.

Significance. If the reported results are reproducible, AnchorSteer would be a practically useful contribution: it is training-free, architecture-agnostic, and improves compositional faithfulness on both U-Net and DiT backbones. The empirical scope is a strength: the paper evaluates on two benchmarks, compares against multiple training-free and training-based baselines, and includes ablations, a user study, and runtime analysis. I do not see a circular-evaluation problem, since the VLM critic is pretrained and the final outcomes are scored by external benchmarks. However, the mechanism attributed to Semantic Anchoring is not established by the evidence as presented, and the lack of uncertainty quantification and the tuning-on-test details make the headline margins difficult to assess.

major comments (3)
  1. [Sec. 4.1 (DDPM Forward Diffusion for Noise Initialization)] The claim “z_T still follows N(0,I)” is mathematically incorrect for a fixed z*: the conditional distribution is N(sqrt(alpha_bar_T) z*, (1 - alpha_bar_T) I). This is load-bearing, because the proposed mechanism requires z_T to preserve prompt-specific semantic information while remaining distributionally consistent. With SDXL’s 50-step DDIM schedule, alpha_bar_T is very small, so the mean shift is tiny; the first denoising step amplifies signal and noise by the same factor, so a simple SNR argument does not rescue the claim. The only direct empirical support, Table S5 in Sec. F, is a single run with no error bars and uses z_shu_T as a null, which is not matched: a shuffled latent has different norm/correlation and does not separate prompt-specific leakage from generic structured-initialization effects. Please provide repeated-seed experiments with matched nulls (e.g., z* from another pr
  2. [Sec. 5.1 and Table S2 (hyperparameter selection)] The hyperparameters N_lp and tau are selected on GenEval (Table S2), and the main GenEval results in Tables 1, 2, and 5 are reported on the same benchmark. This creates a risk of selection-on-test. Several reported differences are small (e.g., T2I-CompBench++ Non-Spatial 0.3139 vs 0.3171), and no standard errors, confidence intervals, or seed-level results are provided anywhere. The claim of “consistent” improvement needs error bars over multiple seeds and a stated validation protocol. This is especially important for the single-run Table S5 and for the +Anchor vs +Steer decomposition in Table 5, which is used to support the complementarity claim.
  3. [Sec. 5.3 / Table 4 (ImageReward, quality and diversity)] The ImageReward score jumps from 0.4891 to 0.8781 while HPSv2, PickScore, and Aesthetic improve modestly. Such a large gain on one preference model, reported without error bars or qualitative analysis, is surprising and should be verified. In addition, the diversity results in Table S1 are based on 20 prompts per task and 10 images per prompt; this sampling protocol should be stated in the main text, and confidence intervals would help substantiate the diversity claims.
minor comments (5)
  1. [Eq. (5) and surrounding text] The KL interpretation should be presented as an analogy or heuristic; as written it implies an exact identity that is not established by the SDS derivation, and the paper’s own derivation omits the Jacobian term.
  2. [Abstract / Sec. 1] There are capitalization and grammar errors: “we proposeAnchorSteer” in the abstract and “First, atnoise initialization” in Sec. 1. Please proofread.
  3. [Sec. F / Table S5] The semantic-preservation experiment is load-bearing but appears only in the supplementary material. It should either be moved into the main text or at least explicitly referenced from Sec. 4.1 with a one-sentence summary of its limitations.
  4. [Algorithm S1 / Eqs. (6)-(9)] Both the Erase and Retouch steps use D_neg in the negative-prompt position; please clarify the behavior when D_neg is empty and how the algorithm avoids double-counting erasure during retouch.
  5. [References] Several references are dated 2026; please verify that they are publicly available or update them. Reproducibility would also be improved by releasing code and exact evaluation scripts.

Circularity Check

0 steps flagged

No significant circularity: AnchorSteer's improvements are measured on external benchmarks; no component's output is definitionally equal to its input.

full rationale

AnchorSteer's pipeline has three main stages: (1) DAS extracts a CLIP prior from the prompt; (2) LP-SDS refines that prior in VAE latent space with the frozen diffusion model's score, using the standard SDS gradient (Eq. 4) with the U-Net Jacobian omitted; (3) DDPM forward noising produces z_T, and Reflective Steering later inserts a VLM-based Think-Erase-Retouch loop during denoising. None of these stages fits a parameter to GenEval or T2I-CompBench++ labels; the reported gains in Tables 1-3 are computed by external benchmark protocols, and the VLM critic (Qwen-VL) is a pretrained model not fine-tuned on those benchmarks. The use of the same diffusion model in LP-SDS and in the final sampling is a standard self-distillation/initialization technique, not a reduction of the evaluation to the method's own outputs. The assertion in Sec. 4.1 that z_T = sqrt(alpha_T) z* + sqrt(1-alpha_T) eps 'still follows N(0,I)' is not exact for fixed nonzero z*, but this is a correctness/technical concern about signal strength, not a circularity; the semantic-leakage claim is tested empirically in Table S5 rather than assumed in the metric. Self-citations in the reference list (e.g., [24-26,72]) appear only in related-work or motivational contexts and are not load-bearing for the derivation. Therefore the central claim has independent content and no circular step can be exhibited.

Axiom & Free-Parameter Ledger

7 free parameters · 8 axioms · 0 invented entities

The central claim rests on several frozen pretrained models (diffusion backbone, CLIP, VLM) and on hand-chosen hyperparameters (N_das, N_lp=400, tau=1, gamma_e=1, gamma_r=5, LP-SDS schedule/templates). No new physical entities are introduced. The strongest unproven premise is semantic leakage through full DDPM noising to the terminal timestep.

free parameters (7)
  • N_lp (LP-SDS optimization steps) = 400
    Chosen by ablation on GenEval (Table S2); performance peaks at 400.
  • inspection stride tau = 1
    Ablated on GenEval (Table S2); smaller stride improves score, tau=1 selected.
  • erase guidance scale gamma_e = 1.0
    Set in implementation details; no ablation reported, likely tuned.
  • retouch guidance scale gamma_r = 5.0
    Set in implementation details; no ablation reported, likely tuned.
  • DAS optimization hyperparameters (steps, learning rate, initialization) = not specified
    N_das and optimizer are not given in main text or supplement; required to reproduce the semantic prior.
  • LP-SDS weighting w(t) and noise schedule = not specified
    Scalar factors are absorbed into w(t); the concrete schedule is not stated and affects the gradient.
  • VLM instruction templates = hand-designed
    The constraint-extraction, image-inspection, and relation-check prompts are hand-crafted and determine the D_pos/D_neg mapping.
axioms (8)
  • domain assumption DDPM forward diffusion preserves enough semantic information at the terminal timestep to affect generation
    Eq. 4.1 and Sec. F rely on prior work [11] and Table S5 rather than a quantitative bound; if alpha_bar_T is near zero, the injected signal may vanish.
  • domain assumption CLIP embeddings and DAS produce a semantic prior meaningfully aligned with compositional prompts
    Sec. 4.1; no guarantee that DAS from OpenCLIP ViT-B/32 captures multi-object spatial relations, and optimization details are unspecified.
  • domain assumption VLM can reliably diagnose missing and inconsistent content from intermediate decoded latents
    Sec. 4.2 and Sec. G; intermediate x0 predictions are blurry and VLM may hallucinate, and correctness of D_pos/D_neg directly controls correction.
  • standard math DDIM inversion is locally reversible enough for erase-retouch edits
    Eqs. 7-8 use an approximate inversion rule; inversion errors can accumulate over repeated correction loops.
  • standard math SDS Jacobian omission is a valid approximation for latent optimization
    Eqs. S5-S6 follow [46]; this is a known approximation, not an exact gradient.
  • domain assumption The pretrained diffusion model can render the reinforced positive prompt c+ at the current timestep without retraining
    The Retouch phase assumes that adding missing terms to the prompt and using CFG is sufficient to recover missing content, with no guarantee for unseen combinations.
  • domain assumption Negative-prompt guidance suppresses erroneous semantics without harming other content
    Eq. 6 treats erasure as a directed operation; in practice negative prompts can remove unrelated content.
  • domain assumption GenEval and T2I-CompBench++ metrics measure the claimed text-image faithfulness
    The benchmarks are treated as ground truth; their detectors/LLM evaluators have their own biases and noise.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time." pith.science (2026). https://pith.science/paper/PF3ZK6IV

@misc{pith2026260726647,
  author       = {Pith},
  title        = {Pith review of: Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PF3ZK6IV}},
  note         = {Machine review of arXiv:2607.26647}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

While text-to-image diffusion models achieve impressive visual quality, they frequently struggle to maintain precise alignment with complex compositional prompts. An effective strategy is to improve the inference process of diffusion models, thereby better leveraging their pretrained priors to address misalignment. Existing training-free methods can be divided into two categories. The first category focuses on improving the randomly sampled initial noise, either performing costly search over noise pools or manipulating sampled noise without ensuring reliable semantic injection. The second category focuses on improving the denoising trajectory, lacking explicit mechanisms to timely diagnose and correct semantic errors. we propose \textbf{AnchorSteer}, a training-free framework that exerts fine-grained control over \textbf{both initialization} and \textbf{the denoising trajectory}. AnchorSteer consists of two synergistic components: \textbf{Semantic Anchoring} replaces uninformative Gaussian noise with text-aligned initializations via CLIP-based prior extraction and a novel Latent-Prior Score Distillation Sampling (LP-SDS) objective. Specifically, LP-SDS distills CLIP visual priors into the knowledge distribution of diffusion models, mitigating the domain gap between CLIP-based priors and diffusion-based priors. \textbf{Reflective Steering} transforms passive denoising with an active Think--Erase--Retouch loop that enables mid-generation self-correction. It leverages VLM-based diagnosis to detect semantic deviations and performs targeted latent refinement to suppress erroneous content and recover missing attributes. Extensive experiments on GenEval and T2I-CompBench++ demonstrate that AnchorSteer consistently outperforms existing baselines in text--image alignment while preserving high visual quality.

Figures

Figures reproduced from arXiv: 2607.26647 by Feiming Wei, Pengcheng Luan, Tao Zhang, Wenxian Yu, Xinyi Wang, Yalin Su, Yuyang Huang.

Figure 1
Figure 1. Figure 1: Visual comparison of text-to-image results. Our AnchorSteer demonstrates better text–image alignment across various aspects. Abstract. While text-to-image diffusion models achieve impressive vi￾sual quality, they frequently struggle to maintain precise alignment with complex compositional prompts. An effective strategy is to improve the inference process of diffusion models, thereby better leveraging their… view at source ↗
Figure 2
Figure 2. Figure 2: Existing methods for noise initialization and their limitations. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of AnchorSteer. (Top) Semantic Anchoring for Initial Noise [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: CLIP-based semantic priors and their refinement via LP-SDS. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of LP-SDS. The semantic prior Iprior produced by DAS is en￾coded into the latent space as z0 = EVAE(Iprior) and iteratively refined via LP-SDS gradients from a frozen diffusion U-Net. The optimized latent z ∗ is projected onto the diffusion manifold, bridging the domain gap between CLIP-based synthesis and diffusion-based generation while preserving text-aligned semantics. where q(zt | z) = N … view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison on challenging compositional prompts. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of semantic correction during denoising. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 9 linked inside Pith

  1. [1]

    In: The Fourteenth International Conference on Learning Representations (2026), https://openreview.net/forum?id=xEWooSOgaz

    Ahn, D., Kang, J., Lee, S., Min, J., Kim, M., Jang, W., Cho, H., Paul, S., Kim, S., Cha, E., Jin, K.H., Kim, S.: A noise is worth diffusion guidance. In: The Fourteenth International Conference on Learning Representations (2026), https://openreview.net/forum?id=xEWooSOgaz

  2. [2]

    arXiv preprint arXiv:2309.16609 (2023) 16 X

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023) 16 X. Wang et al

  3. [3]

    In: The Thirteenth International Conference on Learning Representations (2024),https: //openreview.net/forum?id=MKvQH1ekeY

    Bai, L., Shao, S., Zhou, Z., Qi, Z., Xu, Z., Xiong, H., Xie, Z.: Zigzag dif- fusion sampling: Diffusion models can self-improve via self-reflection. In: The Thirteenth International Conference on Learning Representations (2024),https: //openreview.net/forum?id=MKvQH1ekeY

  4. [4]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 18392–18402 (2023)

  5. [5]

    In: Proceedings of the International Conference on Computer Vision (ICCV) (2021)

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the International Conference on Computer Vision (ICCV) (2021)

  6. [6]

    In: Findings of the Association for Computational Linguistics: EMNLP 2024

    Chang, Z., Li, M., Wang, J., Liu, Y., Wang, Q., Liu, Y.: Repairing catastrophic- neglect in text-to-image diffusion models via attention-guided feature enhance- ment. In: Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 11379–11390 (2024)

  7. [7]

    Chatterjee, A., Stan, G.B.M., Aflalo, E., Paul, S., Ghosh, D., Gokhale, T., Schmidt, L., Hajishirzi, H., Lal, V., Baral, C., et al.: Getting it right: Improving spatial consistencyintext-to-imagemodels.In:EuropeanConferenceonComputerVision. pp. 204–222. Springer (2024)

  8. [8]

    ACM trans- actions on Graphics (TOG)42(4), 1–10 (2023)

    Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM trans- actions on Graphics (TOG)42(4), 1–10 (2023)

  9. [9]

    In: The Thirteenth International Con- ference on Learning Representations (2025),https://openreview.net/forum?id= E77uvbOTtp

    Chung, H., Kim, J., Park, G.Y., Nam, H., Ye, J.C.: CFG++: Manifold-constrained classifier free guidance for diffusion models. In: The Thirteenth International Con- ference on Learning Representations (2025),https://openreview.net/forum?id= E77uvbOTtp

  10. [10]

    In: Forty-first international conference on machine learning (2024)

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al.: Scaling rectified flow transformers for high-resolution image synthesis. In: Forty-first international conference on machine learning (2024)

  11. [11]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

    Everaert, M.N., Fitsios, A., Bocchio, M., Arpa, S., Süsstrunk, S., Achanta, R.: Exploiting the Signal-Leak Bias in Diffusion Models. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 4025–4034 (January 2024)

  12. [12]

    Feng, H., Ding, Z., Xia, Z., Niklaus, S., Abrevaya, V., Black, M.J., Zhang, X.: Explorativeinbetweeningoftimeandspace.In:EuropeanConferenceonComputer Vision. pp. 378–395. Springer (2024)

  13. [13]

    Feng, W., He, X., Fu, T.J., Jampani, V., Akula, A.R., Narayana, P., Basu, S., Wang, X.E., Wang, W.Y.: Training-free structured diffusion guidance for composi- tionaltext-to-imagesynthesis.In:TheEleventhInternationalConferenceonLearn- ing Representations (2023),https://openreview.net/forum?id=PUIqjT4rzq7

  14. [14]

    Fort, S., Whitaker, J.: Direct ascent synthesis: Revealing hidden generative capa- bilities in discriminative models (2025),https://arxiv.org/abs/2502.07753

  15. [15]

    Advances in Neural Information Processing Systems36, 52132–52152 (2023)

    Ghosh, D., Hajishirzi, H., Schmidt, L.: Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems36, 52132–52152 (2023)

  16. [16]

    Grimal, P., Borgne, H.L., Ferret, O.: Text-to-image alignment in denoising-based models through step selection (2025),https://arxiv.org/abs/2504.17525

  17. [17]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Guo, X., Liu, J., Cui, M., Li, J., Yang, H., Huang, D.: Initno: Boosting text- to-image diffusion models via initial noise optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9380– 9389 (2024) Anchoring and Steering Diffusion 17

  18. [18]

    Harrington, A., Koepke, A., Karthik, S., Darrell, T., Efros, A.A.: It’s never too late: Noiseoptimizationforcollapserecoveryintraineddiffusionmodels.In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 43124–43134 (2026)

  19. [19]

    In: The Eleventh In- ternational Conference on Learning Representations (2023),https://openreview

    Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross-attention control. In: The Eleventh In- ternational Conference on Learning Representations (2023),https://openreview. net/forum?id=_CDixzkzeyb

  20. [20]

    Advances in neural information processing systems33, 6840–6851 (2020)

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems33, 6840–6851 (2020)

  21. [21]

    In: NeurIPS 2021 Work- shop on Deep Generative Models and Downstream Applications (2021),https: //openreview.net/forum?id=qw8AKxfYbI

    Ho, J., Salimans, T.: Classifier-free diffusion guidance. In: NeurIPS 2021 Work- shop on Deep Generative Models and Downstream Applications (2021),https: //openreview.net/forum?id=qw8AKxfYbI

  22. [22]

    Huang, K., Duan, C., Sun, K., Xie, E., Li, Z., Liu, X.: T2I-CompBench++: An EnhancedandComprehensiveBenchmarkforCompositionalText-to-ImageGener- ation.IEEETransactionsonPatternAnalysisMachineIntelligence(01),1–17(Jan 5555),https://doi.ieeecomputersociety.org/10.1109/TPAMI.2025.3531907

  23. [23]

    Advances in Neural Information Processing Systems36, 78723–78747 (2023)

    Huang, K., Sun, K., Xie, E., Li, Z., Liu, X.: T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems36, 78723–78747 (2023)

  24. [24]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Huang, Y., Chen, Y., Ding, L., Zhang, X., Dai, W., Zou, J., Xiong, H., Tian, Q.: Im-zero: Instance-level motion controllable video generation in a zero-shot manner. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 7265–7275 (2025)

  25. [25]

    In: European Conference on Computer Vision

    Huang, Y., Chen, Y., Liu, Y., Zhang, X., Dai, W., Xiong, H., Tian, Q.: Domain- fusion: Generalizing to unseen domains with latent diffusion models. In: European Conference on Computer Vision. pp. 480–498. Springer (2024)

  26. [26]

    Advances in Neural Information Processing Systems38, 108999–109019 (2026)

    Huang, Y., Chen, Y., Zhou, J., Dai, W., Zhang, X., Zou, J., Xiong, H., Tian, Q.: Diffusion-driven progressive target manipulation for source-free domain adap- tation. Advances in Neural Information Processing Systems38, 108999–109019 (2026)

  27. [27]

    Zenodo (2021)

    Ilharco, G., Wortsman, M., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., et al.: Openclip. Zenodo (2021)

  28. [28]

    Advances in Neural Information Processing Systems37, 76177–76209 (2024)

    Jiang, D., Song, G., Wu, X., Zhang, R., Shen, D., Zong, Z., Liu, Y., Li, H.: Co- mat: Aligning text-to-image diffusion model with image-to-text concept matching. Advances in Neural Information Processing Systems37, 76177–76209 (2024)

  29. [29]

    Advances in neural information processing systems35, 26565–26577 (2022)

    Karras,T.,Aittala,M.,Aila,T.,Laine,S.:Elucidatingthedesignspaceofdiffusion- based generative models. Advances in neural information processing systems35, 26565–26577 (2022)

  30. [30]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition

    Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R.C., Schindler, K.: Re- purposing diffusion-based image generators for monocular depth estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition. pp. 9492–9502 (2024)

  31. [31]

    In: The Fourteenth International Con- ference on Learning Representations (2026),https://openreview.net/forum?id= 11dzFZ2UM1

    Kim, K., Kim, S.: Model already knows the best noise: Bayesian active noise selec- tion via attention in video diffusion model. In: The Fourteenth International Con- ference on Learning Representations (2026),https://openreview.net/forum?id= 11dzFZ2UM1

  32. [32]

    Advances in neural information processing systems36, 36652–36663 (2023) 18 X

    Kirstain,Y.,Polyak,A.,Singer,U.,Matiana,S.,Penna,J.,Levy,O.:Pick-a-pic:An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems36, 36652–36663 (2023) 18 X. Wang et al

  33. [33]

    In: European Conference on Computer Vision (ECCV)

    Koo, G., Yoon, S., Hong, J.W., Yoo, C.D.: Flexiedit: Frequency-aware latent re- finement for enhanced non-rigid editing. In: European Conference on Computer Vision (ECCV). pp. 363–379. Lecture Notes in Computer Science, Springer (2024)

  34. [34]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Kumari, N., Zhang, B., Zhang, R., Shechtman, E., Zhu, J.Y.: Multi-concept cus- tomization of text-to-image diffusion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1931–1941 (2023)

  35. [35]

    Labs, B.F.: Flux.https://github.com/black-forest-labs/flux(2024)

  36. [36]

    Li, Z., Zhang, J., Lin, Q., Xiong, J., Long, Y., Deng, X., Zhang, Y., Liu, X., Huang, M., Xiao, Z., Chen, D., He, J., Li, J., Li, W., Zhang, C., Quan, R., Lu, J., Huang, J., Yuan, X., Zheng, X., Li, Y., Zhang, J., Zhang, C., Chen, M., Liu, J., Fang, Z., Wang, W., Xue, J., Tao, Y., Zhu, J., Liu, K., Lin, S., Sun, Y., Li, Y., Wang, D., Chen, M., Hu, Z., Xia...

  37. [37]

    Lian, L., Li, B., Yala, A., Darrell, T.: LLM-grounded diffusion: Enhancing prompt understandingoftext-to-imagediffusionmodelswithlargelanguagemodels.Trans- actions on Machine Learning Research (2024),https://openreview.net/forum? id=hFALpTb4fR, featured Certification

  38. [38]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liang, Z., Yuan, Y., Gu, S., Chen, B., Hang, T., Cheng, M., Li, J., Zheng, L.: Aesthetic post-training diffusion models from generic preferences with step-by- step preference optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13199–13208 (2025)

  39. [39]

    In: European conference on computer vision

    Liu, N., Li, S., Du, Y., Torralba, A., Tenenbaum, J.B.: Compositional visual gen- eration with composable diffusion models. In: European conference on computer vision. pp. 423–439. Springer (2022)

  40. [40]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu,Y.,Zhang,Y.,Jaakkola,T.,Chang,S.:Correctingdiffusiongenerationthrough resampling. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8713–8723 (2024)

  41. [41]

    In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Liu, Y., Zhang, Y., Jaakkola, T., Chang, S.: Correcting diffusion generation through resampling. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 8713–8723 (2024).https://doi.org/10.1109/ CVPR52733.2024.00832

  42. [42]

    arXiv preprint arXiv:2501.09732 (2025)

    Ma, N., Tong, S., Jia, H., Hu, H., Su, Y.C., Zhang, M., Yang, X., Li, Y., Jaakkola, T., Jia, X., et al.: Inference-time scaling for diffusion models beyond scaling de- noising steps. arXiv preprint arXiv:2501.09732 (2025)

  43. [43]

    Transactions on Ma- chineLearningResearch(2024),https://openreview.net/forum?id=a68SUt6zFt, featured Certification

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: DINOv2: Learning robust visual feat...

  44. [44]

    Peebles,W.,Xie,S.:Scalablediffusionmodelswithtransformers.In:Proceedingsof the IEEE/CVF international conference on computer vision. pp. 4195–4205 (2023)

  45. [45]

    In: The Twelfth International Conference on Learning Representa- tions (2024),https://openreview.net/forum?id=di52zR8xgf

    Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: SDXL: Improving latent diffusion models for high-resolution im- age synthesis. In: The Twelfth International Conference on Learning Representa- tions (2024),https://openreview.net/forum?id=di52zR8xgf

  46. [46]

    In: The Eleventh International Conference on Learning Representations (2023),https://openreview.net/forum?id=FjNys5c7VyY Anchoring and Steering Diffusion 19

    Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. In: The Eleventh International Conference on Learning Representations (2023),https://openreview.net/forum?id=FjNys5c7VyY Anchoring and Steering Diffusion 19

  47. [47]

    Qi, Z., Bai, L., Xiong, H., Xie, Z.: Not all noises are created equally:diffusion noise selection and optimization (2024),https://arxiv.org/abs/2407.14041

  48. [48]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

  49. [49]

    Advances in Neural Information Processing Systems36, 3536–3559 (2023)

    Rassin, R., Hirsch, E., Glickman, D., Ravfogel, S., Goldberg, Y., Chechik, G.: Lin- guistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. Advances in Neural Information Processing Systems36, 3536–3559 (2023)

  50. [50]

    In: Proceedings of the Fifth BlackboxNLP workshop on analyzing and interpreting neural networks for NLP

    Rassin, R., Ravfogel, S., Goldberg, Y.: Dalle-2 is seeing double: Flaws in word-to- concept mapping in text2image models. In: Proceedings of the Fifth BlackboxNLP workshop on analyzing and interpreting neural networks for NLP. pp. 335–345 (2022)

  51. [51]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)

  52. [52]

    Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: Dream- booth:Finetuningtext-to-imagediffusionmodelsforsubject-drivengeneration.In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 22500–22510 (2023)

  53. [53]

    In: The Thirteenth International Confer- ence on Learning Representations (2024)

    Sadat, S., Hilliges, O., Weber, R.M.: Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In: The Thirteenth International Confer- ence on Learning Representations (2024)

  54. [54]

    Advances in neural information processing systems35, 36479–36494 (2022)

    Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text- to-image diffusion models with deep language understanding. Advances in neural information processing systems35, 36479–36494 (2022)

  55. [55]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Samuel, D., Ben-Ari, R., Raviv, S., Darshan, N., Chechik, G.: Generating images of rare concepts using pre-trained diffusion models. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 4695–4703 (2024)

  56. [56]

    In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2022),https://openreview.net/forum?id=M3Y74vmsMcY

    Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C.W., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kun- durthy, S.R., Crowson, K., Schmidt, L., Kaczmarczyk, R., Jitsev, J.: LAION-5b: An open large-scale dataset for training next generation image-text models. In: Thirty-sixth Conference on Neural Information ...

  57. [57]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Shen, D., Song, G., Xue, Z., Wang, F.Y., Liu, Y.: Rethinking the spatial incon- sistency in classifier-free diffusion guidance. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9370–9379 (2024)

  58. [58]

    Song,J.,Meng,C.,Ermon,S.:Denoisingdiffusionimplicitmodels.In:International Conference on Learning Representations (2021),https://openreview.net/forum? id=St1giarCHLP

  59. [59]

    In: Interna- tional Conference on Learning Representations (2021),https://openreview.net/ forum?id=PxTIG12RRHS

    Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score- based generative modeling through stochastic differential equations. In: Interna- tional Conference on Learning Representations (2021),https://openreview.net/ forum?id=PxTIG12RRHS

  60. [60]

    In: Forty-second In- ternational Conference on Machine Learning (2025),https://openreview.net/ forum?id=JpbqiD7n9r 20 X

    Tang, Z., Peng, J., Tang, J., Hong, M., Wang, F., Chang, T.H.: Inference-time alignment of diffusion models with direct noise optimization. In: Forty-second In- ternational Conference on Machine Learning (2025),https://openreview.net/ forum?id=JpbqiD7n9r 20 X. Wang et al

  61. [61]

    Team, Q.: Qwen2.5-vl (January 2025),https://qwenlm.github.io/blog/qwen2. 5-vl/

  62. [62]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion

    Tian, J., Aggarwal, L., Colaco, A., Kira, Z., Gonzalez-Franco, M.: Diffuse attend and segment: Unsupervised zero-shot segmentation using stable diffusion. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. pp. 3554–3563 (2024)

  63. [63]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., Naik, N.: Diffusion model alignment using direct preference optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8228–8238 (2024)

  64. [64]

    arXiv preprint arXiv:2409.12191 (2024)

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., Lin, J.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

  65. [65]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Wang, R., Chen, Z., Chen, C., Ma, J., Lu, H., Lin, X.: Compositional text-to-image synthesis with attention map control of diffusion models. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 5544–5552 (2024)

  66. [66]

    Wu, C., Li, J., Zhou, J., Lin, J., Gao, K., Yan, K., ming Yin, S., Bai, S., Xu, X., Chen, Y., Chen, Y., Tang, Z., Zhang, Z., Wang, Z., Yang, A., Yu, B., Cheng, C., Liu, D., Li, D., Zhang, H., Meng, H., Wei, H., Ni, J., Chen, K., Cao, K., Peng, L., Qu, L., Wu, M., Wang, P., Yu, S., Wen, T., Feng, W., Xu, X., Wang, Y., Zhang, Y., Zhu, Y., Wu, Y., Cai, Y., L...

  67. [67]

    In: NeurIPS 2023 Workshop on Regu- latable ML (2023),https://openreview.net/forum?id=8WH2t9F0Ip

    Wu, S., Tang, S., Aydore, S., Kearns, M., Roth, A.: Membership inference attack on diffusion models via quantile regression. In: NeurIPS 2023 Workshop on Regu- latable ML (2023),https://openreview.net/forum?id=8WH2t9F0Ip

  68. [68]

    In: European Conference on Computer Vision (ECCV) (2024)

    Wu,T.,Si,C.,Jiang,Y.,Huang,Z.,Liu,Z.:Freeinit:Bridgingtheinitializationgap in video diffusion models. In: European Conference on Computer Vision (ECCV) (2024)

  69. [69]

    arXiv preprint arXiv:2306.09341 (2023)

    Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., Li, H.: Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341 (2023)

  70. [70]

    In: Proceedingsofthe37thInternationalConferenceonNeuralInformationProcessing Systems

    Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., Dong, Y.: Imagere- ward: learning and evaluating human preferences for text-to-image generation. In: Proceedingsofthe37thInternationalConferenceonNeuralInformationProcessing Systems. pp. 15903–15935 (2023)

  71. [71]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Xu, R., Xi, W., Wang, X., Mao, Y., Cheng, Z.: Stylessp: Sampling startpoint en- hancement for training-free diffusion-based method for style transfer. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

  72. [72]

    arXiv preprint arXiv:2511.07756 (2025)

    Yan, S., Li, M., Xinliang, B., Yang, J., Zhang, Y., Xiong, G., Lan, Y., Zhang, T., Zhai, W., Zha, Z.J.: Beyond randomness: Understand the order of the noise in diffusion. arXiv preprint arXiv:2511.07756 (2025)

  73. [73]

    Yu, J., Wang, Y., Zhao, C., Ghanem, B., Zhang, J.: Freedom: Training-free energy- guidedconditionaldiffusionmodel.In:ProceedingsoftheIEEE/CVFInternational Conference on Computer Vision. pp. 23174–23184 (2023)

  74. [74]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Zeng, Q., Song, J., Zheng, H., Jiang, H., Song, M.: D^2-dpm: Dual denoising for quantized diffusion probabilistic models. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 9806–9814 (2025).https://doi.org/10. 1609/aaai.v39i9.33063 Anchoring and Steering Diffusion 21

  75. [75]

    Zhang,G.,Fu,B.,Fan,Q.,Zhang,Q.,Liu,R.,Gu,H.,Zhang,H.,Liu,X.:Compass: Enhancingspatialunderstandingintext-to-imagediffusionmodels.In:Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15253–15265 (2025)

  76. [76]

    In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreason- able effectiveness of deep features as a perceptual metric. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 586–595 (2018). https://doi.org/10.1109/CVPR.2018.00068

  77. [77]

    In: International Conference on Computer Vision (2025)

    Zhou, Z., Shao, S., Bai, L., Zhang, S., Xu, Z., Han, B., Xie, Z.: Golden noise for diffusion models: A learning framework. In: International Conference on Computer Vision (2025)

  78. [78]

    2w(t) ∂zt ∂z ⊤ ∂ˆϵϕ(zt;c, t) ∂zt ⊤ (ˆϵϕ(zt;c, t)−ϵ) # . (S3) Sincez t =α tz+σ tϵ, we have ∂zt ∂z =α tI. (S4) Substituting yields ∇zLDiff =E t,ϵ

    Zhuang, C., Hu, Y., Gao, P.: Magnet: We never know how text-to-image diffusion models work, until we learn how vision-language models function. Advances in Neural Information Processing Systems37, 57115–57149 (2024) 22 X. Wang et al. Supplementary Material This supplementary material provides additional details and experimental results to complement the m...

  79. [79]

    As shown in Table S2 (top), the CLIP-only prior (Nlp=0) improves the standard baseline, while apply- ing LP-SDS optimization (Nlp >0) further boosts performance

    Effect ofN lp.We varyN lp ∈ {0,200,400,600}. As shown in Table S2 (top), the CLIP-only prior (Nlp=0) improves the standard baseline, while apply- ing LP-SDS optimization (Nlp >0) further boosts performance. This confirms both the effectiveness of the semantic prior and the need to bridge the domain gapbetweenCLIPembeddingsandthe diffusionlatent space.Perf...

  80. [80]

    and", "or

    Effect of Inspection Strideτ.We testτ∈ {1,2,5}. As shown in Ta- ble S2 (bottom), smaller stride (more frequent inspection) consistently improves alignment, particularly on compositional categories such asTwo-object,Count- ing, andPosition, where semantic deviations are most prone to accumulate. Notably, densely applying the correction loop does not compro...

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.