Pith. sign in

REVIEW 4 major objections 5 minor 45 references

Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedback

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that training a ControlNet with lightweight probes that check its own intermediate features at every denoising step improves both spatial control and image quality.

desk verdict A plausible new training objective for ControlNet that deserves a referee, but the mixed numbers and an unaddressed probe-calibration issue mean the SOTA claim is premature. read the letter →

arxiv 2507.02321 v1 pith:WPUBCVUO submitted 2025-07-03 cs.CV

Nina Konovalova , Maxim Nikolaev , Andrey Kuznetsov , Aibek Alanov This is my paper
classification cs.CV
keywords diffusionmodelscontrollableimagegenerationControlNetintermediatefeaturesupervisionalignmentlossdepthedgetext-to-image
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reward-based methods like ControlNet++ only enforce alignment with the input control on the final denoising steps, because their single-step image predictions are too blurry to read at early, noisy stages. This paper argues that the intermediate features of the diffusion UNet still contain spatial structure at those early steps, and that a small timestep-conditioned convolutional probe can recover the control signal directly from those features. InnerControl trains such probes to produce pseudo ground-truth controls at every denoising step, then adds an alignment loss that penalizes mismatches between the predicted and target condition throughout the whole trajectory. The authors report that this improves control fidelity and generation quality, and that combining it with ControlNet++ gives the best depth-alignment results among the compared methods.

What carries the argument

The load-bearing component is the lightweight, timestep-conditioned convolutional probe $H(\cdot,t)$, trained to map intermediate UNet decoder features to the spatial control signal (depth, HED edges, or LineArt). For depth, the probe reads self-attention outputs from the decoder rather than convolutional features, following the observation that self-attention encodes structure. The probe's predictions at every timestep provide pseudo ground-truth controls, allowing the alignment loss to push the ControlNet under training toward features that are consistent with the input condition at each step. This makes the probe a differentiable inner voice that reports whether the generation is still obeying the control, without needing a full image reconstruction.

What would settle it

Measure the probe's depth or edge predictions against known input controls at high-noise timesteps (for example, t greater than 800) for a trained ControlNet: if the probe's RMSE is no better than one-step DPT predictions at those steps, the early-stage signal is not there and the alignment loss cannot be doing the claimed work. Alternatively, train with the full alignment loss and check whether thin-edge fidelity on a dense edge benchmark drops below ControlNet++ while RMSE improves, which would confirm that the fine-detail limitation is load-bearing.

Watch

Extended reading notes

Core claim

The paper's central claim is that intermediate diffusion features carry enough spatial structure, even at very noisy early steps, for a lightweight convolutional probe $H(\cdot,t)$ to reconstruct the conditioning signal, and that enforcing consistency between this reconstruction and the input condition at every timestep improves both controllability and image fidelity. The authors support this by comparing probe-based depth estimates with DPT estimates from one-step predictions: probe estimates stay aligned with ground truth at high noise levels, while DPT degrades. The method adds the alignment loss $\mathcal{L}_{\text{align}} = \mathcal{L}(\mathbf{c}_{\text{spatial}}, H[\text{ControlNet}(\mathbf{c}_{\text{spatial}}, \mathbf{c}_{\text{text}}, x_T, t)], t)$ to the standard diffusion and reward losses, applies it over 920 of 1000 denoising steps, and reports reduced depth RMSE with competitive FID and CLIP scores on MultiGen-20M for HED, LineArt, and depth conditioning.

Load-bearing premise

Everything rests on the small neural network that reads the control signal from the model's internal features being accurate at the noisiest steps; the paper itself concedes that small probes can miss fine details such as thin edges.

Editorial extensions

If this is right

  • The reward loss of ControlNet++ can be extended to early denoising steps when paired with probe-based alignment, without the FID degradation seen when reward loss alone is applied early.
  • Depth control fidelity improves: depth RMSE drops by about 7.9% over ControlNet++ and 10.2% over Ctrl-U at guidance 7.5, with slightly better FID than Ctrl-U.
  • The probe gives dense per-step supervision, so the ControlNet is aligned along the whole trajectory rather than only at the end.
  • The method is stated to be model-agnostic: any per-timestep signal extractor could replace the lightweight probe, making the approach a training strategy rather than a fixed architecture.
  • The alignment loss can be combined with uncertainty-aware reward modeling pipelines such as Ctrl-U, a direction the paper explicitly leaves for future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the paper leaves implicit is that the probe could serve as a cheap stepwise diagnostic for other conditional diffusion models: if a model's intermediate features stop encoding the condition, the probe flags the divergence before the final image is formed.
  • The acknowledged thin-edge limitation suggests a direct scaling test: replacing the small conv probe with a stronger per-step estimator should improve fine-detail alignment, and the paper's own framing predicts that better probes yield better results.
  • The depth results favor self-attention features over convolutional features, implying the choice of feature layer should be matched to the control type; normals, segmentation, or other geometry controls may likewise benefit from attention-based probes.
  • Applying the alignment loss at every one of 920 steps is computationally heavy, so an unstated trade-off is whether sampling a subset of early steps at higher frequency could capture most of the benefit at lower cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces InnerControl, a training strategy for ControlNet that adds an alignment loss computed by lightweight, timestep-conditioned probe networks H(·,t), which predict the conditioning signal (depth, HED edges, or LineArt edges) from intermediate UNet decoder features. The alignment loss is applied over 920 of 1000 denoising steps and is combined with the ControlNet++ reward loss. Experiments on MultiGen-20M report depth RMSE, edge SSIM, FID, and CLIP-score, and the authors conclude that the method improves both control alignment and generation fidelity and achieves state-of-the-art performance on edge and depth control. The code is released.

Significance. The core idea is well motivated: early diffusion steps determine global structure, and one-step denoised images at high noise levels are poor inputs for off-the-shelf estimators such as DPT. Training probes directly on intermediate diffusion features is a plausible and potentially reusable mechanism, and the paper provides a useful comparison of probe-based versus DPT-based depth estimation in Figs. 2 and 3. Strengths include the public code, the use of a standard benchmark, and ablations that vary the number of steps for the alignment and reward losses. The main weaknesses are the lack of uncertainty quantification, the absence of a check on probe validity under training distribution shift, and several claims in the abstract and conclusion that are not supported by the reported numbers.

major comments (4)
  1. [Sec. 4.2, Eq. (7)] The alignment loss L_alignment = L(c_spatial, H[ControlNet(...), t]) is computed with a probe H(·,t) that is trained on features from a frozen pretrained SD1.5/ControlNet, but is then used to supervise the ControlNet being optimized. The paper never checks that H remains an accurate estimator on the training-time feature distribution. If the trained ControlNet's decoder features drift (for example, toward features that H maps to the desired control), the alignment loss can be reduced without genuinely improving control extraction. This is not merely hypothetical: the paper's own ablation in Table 2 shows that alignment-only training substantially changes the generated-image statistics (FID 22.43 vs 18.61 for the ControlNet baseline), indicating feature drift. Please report the probe's own error on fixed held-out controls before and after InnerControl training, and monitor the correlation between H's estimates and a reference estimator (e.g., DPT on final images) throughout training; without such a check, the central training signal in Eq. (7) is not established.
  2. [Sec. 5.2, Table 1] The abstract and conclusion claim state-of-the-art performance and improvements in both control alignment and generation fidelity, but Table 1 does not support this across the reported settings. At guidance 7.5, depth FID is 18.29 for InnerControl versus 16.66 for ControlNet++; at guidance 3.0, depth FID is 14.67 versus 14.56. HED SSIM is lower than Ctrl-U at both guidance scales (0.8207 vs 0.8401 and 0.8305 vs 0.8522), and LineArt SSIM is lower than ControlNet++ at both scales (0.8258 vs 0.8399 and 0.8395 vs 0.8515). Only depth RMSE improves consistently. Please either report the numbers honestly and qualify the claims, or add controlled comparisons and statistical tests that support the stronger statement.
  3. [Sec. 5.1 and Sec. 5.3, Table 2] The paper reports that metrics are averaged over four random seeds but gives no standard deviations or significance tests. This is load-bearing because several central comparisons involve small margins (e.g., LineArt FID 12.08 vs 12.04, depth FID 14.67 vs 14.56, HED CLIP 31.28 vs 31.32). Without error bars or per-seed results, the reader cannot determine whether the claimed improvements are meaningful. In addition, the ablation in Table 2 shows that the alignment loss alone severely degrades FID (22.43 vs 18.61 for the baseline) while improving RMSE, and the advantage of adding it on top of the reward loss is modest and varies with the step threshold (RMSE 28.93 at 200/200, 28.56 at 400/200, 28.41 at 600/200, 27.50 at 920/200). The statement that alignment loss 'does not increase FID' is confusing because the comparison mixes different step ranges; please clarify and provide uncertainty estimates.
  4. [Sec. 4.2 and Sec. 7.1] The training procedure for the probe H(·,t) is under-specified. The paper states that H is a lightweight convolutional network conditioned on timestep embeddings, but does not specify the supervision data, the exact set of UNet features used (decoder convolutions versus self-attention outputs), the timestep sampling distribution for training H, or whether H is trained before or after the initial ControlNet finetuning. Since H is the source of pseudo-ground-truth controls in Eq. (7), these details are essential for reproducibility and for assessing the probe-calibration concern raised in the first major comment. The limitation in Sec. 7.3 that small probes may miss fine details such as thin edges is directly relevant to the edge-control experiments and should be discussed in the main text.
minor comments (5)
  1. [Sec. 4.2, Eq. (7)] The notation in Eq. (7) is imprecise: ControlNet is called with x_T (pure noise) rather than a general timestep x_t, and H is applied to intermediate UNet features, not to the return value of ControlNet as the expression 'H[ControlNet(...), t]' suggests. Please rewrite the equation to show the actual data flow.
  2. [Sec. 5.1, Baselines paragraph] There is a typo in the baselines list: 'Uni-ControlNet [20] and and UniControl [21]' contains a duplicated 'and'.
  3. [Fig. 5 caption] The caption contains a typo: 'LeneArt' should be 'LineArt'.
  4. [Table 1] The ControlNet-SDXL row reports only depth RMSE and has no FID or CLIP-score entries; please explain why these are omitted or fill in the missing values.
  5. [Abstract and Sec. 4.2] The phrasing 'We prove the efficiency' is too strong for an empirical demonstration; 'we demonstrate' would be more appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the alignment loss compares an externally supplied control to a separately trained probe's estimate, so the central claim does not reduce to its inputs.

full rationale

InnerControl's training signal is L_alignment = L(c_spatial, H[ControlNet(...), t]) (Eq. 7), where H is a lightweight network trained to map intermediate UNet features to control signals and c_spatial is the user-provided condition. The target is therefore external to the model being trained, not defined in terms of that model's own output. The probe H is trained on real feature-control pairs following the architecture of Readout Guidance, an external reference, and the paper compares H against DPT on fixed pretrained features to motivate its use; this is an empirical validation, not a circular definition. The paper's own limitation (Sec. 7.3) concedes that small probes may miss fine details, which is an effectiveness limitation and a distribution-shift risk once ControlNet weights move during training, but that risk is a correctness and robustness concern, not a circularity of the derivation. No load-bearing self-citation chain or imported uniqueness theorem appears; citations to [22] and [34] are external prior work. Hence the central claim, that enforcing probe-based alignment across denoising steps improves control fidelity, is self-contained against external benchmarks and does not reduce to its inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the probe is a standard neural module. The free parameters are limited to loss weights and step thresholds, none of which are fitted to the evaluation data in a way that would make the central claim circular.

free parameters (4)
  • beta (alignment loss weight) = 1.0
    Chosen uniformly for all experiments without ablation study.
  • alpha (reward loss weight) = 0.5 (depth), 1.0 (HED/LineArt)
    Inherited from ControlNet++.
  • alignment steps threshold = 920
    Set to all denoising steps; motivated by motivation analysis.
  • reward steps threshold = 200 (edge), 400 (depth)
    Taken from ControlNet++.
assumptions (4)
  • standard math Diffusion forward and reverse processes as in Eq. (1)-(3).
    Background for the training objective.
  • domain assumption Intermediate UNet features encode spatial structure even at early denoising steps.
    Cited from [22]; underlies the feasibility of probe-based alignment.
  • domain assumption Lightweight probes can accurately reconstruct control signals from these features at all steps.
    Validated by Figure 2 but only for depth; no quantitative evidence for edges.
  • ad hoc to paper Adding L_alignment across all steps does not degrade final image quality.
    The ablation shows alignment alone degrades FID; only the combination with reward loss yields acceptable quality. The paper assumes the combined loss is stable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedback." pith.science (2026). https://pith.science/paper/WPUBCVUO

@misc{pith2026250702321,
  author       = {Pith},
  title        = {Pith review of: Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WPUBCVUO}},
  note         = {Machine review of arXiv:2507.02321}
}
read the original abstract

Despite significant progress in text-to-image diffusion models, achieving precise spatial control over generated outputs remains challenging. ControlNet addresses this by introducing an auxiliary conditioning module, while ControlNet++ further refines alignment through a cycle consistency loss applied only to the final denoising steps. However, this approach neglects intermediate generation stages, limiting its effectiveness. We propose InnerControl, a training strategy that enforces spatial consistency across all diffusion steps. Our method trains lightweight convolutional probes to reconstruct input control signals (e.g., edges, depth) from intermediate UNet features at every denoising step. These probes efficiently extract signals even from highly noisy latents, enabling pseudo ground truth controls for training. By minimizing the discrepancy between predicted and target conditions throughout the entire diffusion process, our alignment loss improves both control fidelity and generation quality. Combined with established techniques like ControlNet++, InnerControl achieves state-of-the-art performance across diverse conditioning methods (e.g., edges, depth).

Figures

Figures reproduced from arXiv: 2507.02321 by the authors.

Figure 1
Figure 1. Visualizing the trade-off between control consistency (RMSE) and image fidelity (FID) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Left: RMSE between depth estimated from a final image and DPT depth prediction for single-step predicted images (green line) and estimated depth from intermediate features (blue line) for SD1.5 generation. Right: RMSE between control depth and DPT depth prediction for single￾step predicted images (green line) and estimated depth from intermediate features (blue line) for ControlNet. Input 1 step 200 steps 400 steps … view at source ↗
Figure 3
Figure 3. Results of one-step prediction (up) at varying noise levels (from low to high), correspond￾ing depth prediction generated using the DPT estimator [42] (middle) and depth, estimated from intermediate UNet features (bottom). 4.2 Alignment on early steps While standard discriminative models such as DPT [42] struggle to extract precise control signals from blurry images, prior works demonstrate that intermediate diffusi… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Pipeline overview. We schematically illustrate the main idea of our InnerControl framework, highlighting the integration of the alignment loss. The main difference from ControlNet++ is Alignment module, which processes intermediate features extracted from the UNet deco…
Figure 5
Figure 5. Figure 5: Qualitative Comparison with Baselines: Side-by-side results for HED (top), depth (middle) and LeneArt control (bottom) using identical prompts and two guidance scales (3.0 and 7.5). Our method produces more accurate and aligned to input control results compared to comp…
Figure 6
Figure 6. Figure 6: Visualization of difference between extracted signal from intermediate features and input [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Quality comparison for attention-based and convolution-based predictions for depth maps [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Visualization of one-step prediction, corresponding DPT depth estimation, and depth [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Visualization of one-step prediction, corresponding DPT depth estimation and depth [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: More visualizations for InnerControl (ours) method (depth maps) [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: More visualizations for InnerControl (ours) method (LineArt) [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: More visualizations for InnerControl (ours) method (HED) [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 23 canonical work pages

  1. [1]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI conference on artificial intelligence , volume 38, pages 4296–4304, 2024

    work page 2024

  2. [2]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023

    2023

  3. [3]

    Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721, 2023

    arXiv 2023

  4. [4]

    ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback: Project Page: liming-ai

    Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, Chen Chen, and others. ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback: Project Page: liming-ai. github. io/ControlNet_Plus_Plus. In European Conference on Computer Vision, pages 129–147. Springer, 2024

    work page 2024

  5. [5]

    Ctrl-U: Robust Conditional Image Generation via Uncertainty-aware Reward Modeling

    Guiyu Zhang, Huan-ang Gao, Zijian Jiang, Hao Zhao, and Zhedong Zheng. Ctrl-u: Robust condi- tional image generation via uncertainty-aware reward modeling.arXiv preprint arXiv:2410.11236, 2024

    work page Pith review arXiv 2024

  6. [6]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  7. [7]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    arXiv 2010

  8. [8]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021

    2021

Show all 45 references
  1. [9]

    Deep unsuper- vised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR, 2015

    2015

  2. [10]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021

    2021 arXiv

  3. [11]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821–8831. PMLR, 2021

    2021

  4. [12]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  5. [13]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, and others. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural inform...

    2022

  6. [14]

    ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers

    Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, and others. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022

    2022 arXiv

  7. [15]

    Hierarchical text-conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  8. [16]

    Cocktail: Mixing multi-modality control for text-conditional image generation

    Minghui Hu, Jianbin Zheng, Daqing Liu, Chuanxia Zheng, Chaoyue Wang, Dacheng Tao, and Tat-Jen Cham. Cocktail: Mixing multi-modality control for text-conditional image generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 10

    2023

  9. [17]

    Com- poser: Creative and controllable image synthesis with composable conditions

    Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Com- poser: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778, 2023

    2023 arXiv

  10. [18]

    ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systems

    Denis Zavadski, Johann-Friedrich Feiden, and Carsten Rother. ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systems. In European Conference on Computer Vision, pages 343–362. Springer, 2024

    2024

  11. [19]

    Relactrl: Relevance-guided efficient control for diffusion transformers

    Ke Cao, Jing Wang, Ao Ma, Jiasong Feng, Zhanjie Zhang, Xuanhua He, Shanyuan Liu, Bo Cheng, Dawei Leng, Yuhui Yin, and others. Relactrl: Relevance-guided efficient control for diffusion transformers. arXiv preprint arXiv:2502.14377, 2025

    2025

  12. [20]

    Uni-controlnet: All-in-one control to text-to-image diffusion models

    Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36:11127–11150, 2023

    2023

  13. [21]

    Unicontrol: A unified diffusion model for controllable visual generation in the wild

    Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, and others. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint arXiv:2305.11147, 2023

    2023 arXiv

  14. [22]

    Beyond surface statistics: Scene repre- sentations in a latent diffusion model, 2023

    Yida Chen, Fernanda Viégas, and Martin Wattenberg. Beyond surface statistics: Scene repre- sentations in a latent diffusion model, 2023. URL https://arxiv. org/abs/2306.05720

    2023 arXiv

  15. [23]

    Label-efficient semantic segmentation with diffusion models

    Dmitry Baranchuk, Ivan Rubachev, Andrey V oynov, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126, 2021

    2021 arXiv

  16. [24]

    Unsupervised semantic correspondence using stable diffusion.Advances in Neural Information Processing Systems, 36:8266–8279, 2023

    Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasac- chi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion.Advances in Neural Information Processing Systems, 36:8266–8279, 2023

    2023

  17. [25]

    Text-to-image diffusion models are zero shot classifiers

    Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero shot classifiers. Ad- vances in Neural Information Processing Systems, 36:58921–58937, 2023

    2023

  18. [26]

    Diffusion model as representation learner

    Xingyi Yang and Xinchao Wang. Diffusion model as representation learner. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18938–18949, 2023

    2023

  19. [27]

    Denoising diffusion autoencoders are unified self-supervised learners

    Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15802–15812, 2023

    2023

  20. [28]

    Emer- gent correspondence from image diffusion

    Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36:1363–1389, 2023

    2023

  21. [29]

    Clean- DIFT: Diffusion Features without Noise

    Nick Stracke, Stefan Andreas Baumann, Kolja Bauer, Frank Fundel, and Björn Ommer. Clean- DIFT: Diffusion Features without Noise. arXiv preprint arXiv:2412.03439, 2024

    2024

  22. [30]

    Emerdiff: Emerg- ing pixel-level semantic knowledge in diffusion models

    Koichi Namekata, Amirmojtaba Sabour, Sanja Fidler, and Seung Wook Kim. Emerdiff: Emerg- ing pixel-level semantic knowledge in diffusion models. arXiv preprint arXiv:2401.11739 , 2024

    2024 arXiv

  23. [31]

    What the daam: Interpreting stable diffusion using cross attention

    Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture. What the daam: Interpreting stable diffusion using cross attention. arXiv preprint arXiv:2210.04885, 2022

    arXiv

  24. [32]

    Your diffusion model is secretly a zero-shot classifier

    Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2206–2217, 2023

    2023

  25. [33]

    Diffusiondet: Diffusion model for object detection

    Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 19830–19843, 2023. 11

    2023

  26. [34]

    Readout guidance: Learning control from diffusion features

    Grace Luo, Trevor Darrell, Oliver Wang, Dan B Goldman, and Aleksander Holynski. Readout guidance: Learning control from diffusion features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8217–8227, 2024

    2024

  27. [35]

    Diffusion hyperfeatures: Searching through time and space for semantic correspondence

    Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems, 36:47500–47510, 2023

    2023

  28. [36]

    Distillation of diffusion features for semantic correspondence

    Frank Fundel, Johannes Schusterbauer, Vincent Tao Hu, and Björn Ommer. Distillation of diffusion features for semantic correspondence. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6762–6774. IEEE, 2025

    2025

  29. [37]

    CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation

    Yifeng Xu, Zhenliang He, Shiguang Shan, and Xilin Chen. CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation. arXiv preprint arXiv:2410.09400, 2024

    2024 arXiv

  30. [38]

    X-adapter: Adding universal compatibility of plugins for upgraded diffusion model

    Lingmin Ran, Xiaodong Cun, Jia-Wei Liu, Rui Zhao, Song Zijie, Xintao Wang, Jussi Keppo, and Mike Zheng Shou. X-adapter: Adding universal compatibility of plugins for upgraded diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,...

    2024

  31. [39]

    Ctrl-adapter: An efficient and versatile framework for adapting diverse controls to any diffusion model.arXiv preprint arXiv:2404.09967, 2024

    Han Lin, Jaemin Cho, Abhay Zala, and Mohit Bansal. Ctrl-adapter: An efficient and versatile framework for adapting diverse controls to any diffusion model.arXiv preprint arXiv:2404.09967, 2024

    2024 arXiv

  32. [40]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22511–22521, 2023

    2023

  33. [41]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  34. [42]

    Vision transformers for dense prediction

    René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision , pages 12179– 12188, 2021

    2021

  35. [43]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017

    2017

  36. [44]

    Diffusers: State-of-the-art diffusion models

    Patrick V on Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. 2022

    2022

  37. [45]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 12 7 Appendix 7.1 Implementation Details Training details. We utilize the...

    2016

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.