Pith. sign in

REVIEW 4 major objections 7 minor 62 references

FlexControl: Computation-Aware ControlNet with Differentiable Router for Text-to-Image Generation

T0 review · 4 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FlexControl replaces manual ControlNet block choice with a trainable router that switches control blocks on per timestep, reporting better FID at matched or reduced FLOPs on UNet and DiT backbones.

desk verdict A credible, well-motivated router for per-block ControlNet gating with useful empirical gains, but the paper never tests its central claim that learned routing beats a fixed schedule. read the letter →

arxiv 2502.10451 v2 pith:5D7AH2ZJ submitted 2025-02-11 cs.LG cs.GR

Zheng Fang , Lichuan Xiang , Xu Cai , Kaicheng Zhou , Hongkai Wen This is my paper · ORCID
classification cs.LGcs.GR
keywords controllabletext-to-imagegenerationControlNetdynamicroutingdifferentiablegatingcomputation-awarelossGumbel-SigmoidUNetDiT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the usual practice of deciding once which ControlNet blocks should inject conditioning, and when, is a heuristic that wastes compute and can reduce fidelity. FlexControl instead copies every diffusion block, places a lightweight router in front of each, and lets the router decide per input latent and per denoising step whether the block runs, with Gumbel-Sigmoid relaxation for end-to-end training and a computation-aware loss that targets a chosen fraction of the full branch's FLOPs. On depth, canny-edge, and segmentation tasks with SD1.5, the trained model reports FID 11.65, 11.37, and 14.80, against ControlNet's 17.76, 15.23, and 21.33, at 280 G FLOPs versus 233 G for ControlNet and 561 G for ControlNet-Large. The same recipe transfers to the DiT-based SD3.0 backbone, where it again outperforms ControlNet at lower or comparable cost. If these results hold, learned per-block routing is a better use of the control budget than fixed architectures or brute-force duplication of the diffusion model.

What carries the argument

The load-bearing object is the router unit, a gating head that accounts for under 1% of total parameters, placed before each copied control block. For UNet it compresses the block's spatial feature by global average pooling and maps the pooled vector through an MLP to a scalar; for DiT it averages over tokens and over channels separately and fuses global and local features. A Sigmoid followed by a threshold at 0.5 converts the scalar to the mask $M_l\in\{0,1\}$, so the forward chain either computes $h_l=F_l(h_{l-1},c,t)$ and injects $y^l_c$, or bypasses that block through $\operatorname{skip}_l(\cdot)$, skipping both the control block and its zero module. During training the mask is relaxed with Gumbel-Sigmoid noise so gradients reach the router, and the cost term $L_C$ drives the realized FLOP fraction toward the target sparsity $\gamma$.

What would settle it

Reproduce the SD1.5 depth-map evaluation at $\gamma=0.5$, and compare against ControlNet run with its control branch active only for the late half of the 20 DDIM steps, matching FLOPs by keeping the same active block count per step. If the static schedule matches FlexControl's FID (11.65) and depth RMSE within sampling noise, the per-sample, per-block router has not earned its advantage. A second check: feed the same timestep different latents and see whether the masks vary; if the activation pattern is nearly input-independent, FlexControl has learned a timestep schedule, not a data-dependent policy.

Watch

Extended reading notes

Core claim

FlexControl's central claim is that conditional control in diffusion models should be routed dynamically. A binary mask $M_l\in\{0,1\}$ computed from the block's input latent $h_{l-1}$ decides whether the $l$-th copied control block runs and whether its zero-module output is injected into the frozen backbone; since $h_{l-1}$ carries the timestep embedding, the same router yields a per-timestep schedule and, through the latent, a per-sample schedule. Training uses a Gumbel-Sigmoid approximation of the mask in the backward pass while the forward pass uses the hard mask, and the total loss is $L_\theta = L_{SD} + \lambda_C L_C$, where $L_C$ penalizes the batch-averaged ratio of FlexControl FLOPs to ControlNet-Large FLOPs when it deviates from a target sparsity $\gamma$. With $\gamma=0.5$ on SD1.5, FlexControl reports an average FID of 12.61 against 18.11 for ControlNet and 14.05 for ControlNet-Large, higher CLIP scores, and controllability metrics (RMSE, SSIM, mIoU) that match or beat ControlNet-Large. The router's learned pattern is sparse in early denoising and dense in late denoising, which the paper interprets as the late stage carrying most of the conditional-fidelity burden.

Load-bearing premise

The claim stands or falls on the learned router beating a fixed schedule (such as turning control on only in late denoising steps) at the same FLOPs budget, which Section A3 admits would already improve efficiency, and the paper never runs that comparison, so if a static schedule matches FlexControl's numbers the data-driven contribution reduces to relearning that schedule.

Editorial extensions

If this is right

  • Manual block-selection heuristics in ControlNet pipelines become unnecessary: the router learns the schedule from data, so different tasks get different configurations without extra architecture search or retraining.
  • The sparsity knob $\gamma$ gives a continuous quality-efficiency dial; at $\gamma=0.5$, FlexControl matches or beats the all-blocks ControlNet-Large while using roughly half its FLOPs.
  • The same gating recipe works on UNet and DiT backbones, so dynamic block routing is not tied to a particular diffusion architecture or sampler.
  • The learned early-sparse, late-dense activation pattern implies that always-on early control spends most of its FLOPs where it helps least, and later denoising steps deserve more of the control budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct experiment the paper does not run: compare FlexControl at $\gamma=0.5$ with a static schedule that activates ControlNet blocks only in the late half of denoising, matched for FLOPs. Section A3 concedes such a schedule can already improve efficiency, so if it matches FlexControl's FID and mIoU, the data-driven router reduces to learning a known schedule.
  • The router's per-sample decisions could be repurposed as a controllability signal: the number, position, and timing of activated blocks may indicate how much conditioning a given sample needs, which could drive test-time compute allocation in other conditional generation systems.
  • Because the cost loss only penalizes a FLOP ratio, the same gating-plus-sparsity recipe can be applied to any dual-stream conditional model, including adapters and multi-expert pipelines, not just ControlNet-style architectures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes FlexControl, a ControlNet variant that copies all diffusion blocks into a large trainable branch (ControlNet-Large) and adds a lightweight router unit that generates binary masks to activate or skip each control block at every denoising step. The router is trained end-to-end with a Gumbel-Sigmoid relaxation, and a computation-aware loss L_C penalizes deviation from a target sparsity gamma. Experiments are reported on SD1.5 and SD3.0 backbones for depth, canny, and segmentation conditions, claiming improved FID, CLIP score, and controllability metrics over ControlNet, ControlNet++, T2I-Adapter, and ControlNet-Large, with reduced FLOPs at lower sparsity levels. The paper includes ablations over gamma, speed measurements, and an analysis of learned activation patterns in the supplementary material.

Significance. The problem addressed is genuine: ControlNet deployment currently relies on ad-hoc choices of which blocks to activate and when, and a data-driven router that optimizes a quality-compute trade-off would be practically valuable. The proposed design is architecture-agnostic, validated on both UNet and DiT backbones, and introduces a concrete computation-aware loss. The paper also provides quantitative comparisons, ablations, and speed measurements, which is commendable. However, the central empirical claim that learned routing outperforms simple fixed schedules is not directly tested, and the main efficiency comparison is undermined by a mismatch between the stated and actual FLOPs at gamma=0.5. If the authors add the missing static-schedule baseline and matched-compute comparisons, the contribution would be solid.

major comments (4)
  1. [Sec. 4.1, Table 3] The statement in Sec. 4.1 that setting gamma=0.5 aligns FlexControl's FLOPs with ControlNet's is contradicted by Table 3, which reports 280 G FLOPs for FlexControl gamma=0.5 versus 233 G for ControlNet on SD1.5. The headline FID gains in Table 1 (e.g., 11.65 vs 17.76 on depth) are therefore obtained at roughly 20% higher FLOPs, not at matched compute. This is load-bearing because the paper claims 'similar computational cost to the original ControlNet' and 'preserves or enhances image fidelity while also reducing computational overhead.' The authors should either correct the text, use gamma=0.3 as the matched-compute comparison, or present an explicit FLOP-matched evaluation.
  2. [Sec. A3] The supplementary states that 'activating ControlNet only in the late stage... can simply improve the inference efficiency, and no retraining is involved,' and Figs. 1(c), A1-A3 show that the learned activation pattern is largely timestep-determined: sparse early, dense late, with only modest variation across conditions. The paper never compares FlexControl against a fixed schedule that activates the same block set in the late stage with the same FLOPs budget. Without this control, the core claim that data-driven, sample-adaptive routing outperforms a simple heuristic is untested. This is load-bearing because the novelty claim in the Introduction is precisely the learned, computation-aware router.
  3. [Tables 1, 2, 4, 5] All quality and controllability metrics are reported as single numbers without error bars, confidence intervals, or multiple seeds. Some headline differences are small (e.g., FlexControl gamma=0.5 FID 11.65 vs ControlNet-Large 12.45 in Table 1), and FID estimates on finite test sets carry sampling noise. The paper should report variance across seeds or repeated evaluations, or at least provide the number of generated images used for FID and the statistical significance of the differences, to support the claims of consistent improvement.
  4. [Sec. A2, Sec. 4.1] The training protocol for the baselines is not documented. Section A2 provides training settings for FlexControl (optimizer, learning rate, batch size, iterations, warm-up) but does not state whether ControlNet, ControlNet++, T2I-Adapter, and ControlNet-Large were trained under the same data, resolution, number of steps, and compute budget. If the baselines are taken from public checkpoints trained on different data or for different durations, the comparisons in Tables 1 and 2 conflate training-set differences with architectural and routing differences. The authors should specify the provenance of all baselines or retrain them under identical conditions.
minor comments (7)
  1. [Abstract] There are grammatical errors, e.g., 'With introducing a computation-aware loss' should be 'By introducing a computation-aware loss,' and 'it benefit' should be 'it benefits.'
  2. [Sec. 3.3, Eq. (15)] The temperature hyperparameter is written as 'TP' without a subscript; it should be T_P to distinguish it from the threshold T in Eq. (11), and the notation should be introduced consistently in the text.
  3. [Fig. 1 caption] The caption contains a typo: '50% spasity' should be '50% sparsity.'
  4. [Sec. 3.1] 'Formulaly' should be 'Formally' in the first paragraph of the preliminaries.
  5. [Sec. 4.1] The sentence 'We exam generation controllability' should read 'We examine generation controllability.'
  6. [Sec. 4.3, Table 4 discussion] The phrase 'while increasing the blocks budgets observed performance increasing' is unclear; it should be revised to 'increasing the block budget leads to better performance.'
  7. [Algorithm 1] In the pseudocode, 'though Eq. (8)' should be 'through Eq. (8).'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: FlexControl's claims are empirical benchmark results; the missing static-schedule baseline is an evidence gap, not a circular reduction.

full rationale

FlexControl's central claims are empirical benchmark results, not a derivation that assumes its own conclusion. The router is trained end-to-end with the standard diffusion loss (Eq. 18) and a computation-aware cost loss (Eq. 19), where the target sparsity gamma is a user-set hyperparameter rather than a fitted quantity later relabeled as a prediction. The reported FID, CLIP score, RMSE, SSIM, and mIoU values (Tables 1, 2, 4, 5) are measured on held-out test sets and compared against published baselines and an internal ControlNet-Large baseline. No equation in the paper defines the router's quality outcome in terms of the measured metrics, and no load-bearing result is justified by a self-citation. The supplementary statement that 'activating ControlNet only in the late stage... can simply improve the inference efficiency' (Sec. A3) identifies a plausible static-schedule alternative, but the absence of a quantitative comparison against such a schedule is a missing baseline or correctness concern, not a circularity. The FLOPs reduction at gamma < 1 is a direct consequence of the cost objective, not a circular prediction of quality. No self-definitional, fitted-input-renamed-as-prediction, or self-citation-chain pattern is present, so the circularity score is 0.

Assumptions & free parameters 7 free parameters · 5 assumptions · 1 invented entities

The central claim rests on several hand-set hyperparameters (gamma, lambda_C, threshold, temperature, alpha1/alpha2, warm-up schedule) and on standard assumptions about diffusion training, Gumbel-Sigmoid approximation, and dataset representativeness. The only new entity is the router unit, which is the method itself and is supported by internal experiments.

free parameters (7)
  • gamma (target sparsity) = 0.3, 0.5, 0.7
    User-selected budget in the cost loss (Eq. 19); the paper reports results at three levels and chooses gamma=0.5 for headline comparisons.
  • lambda_C = 0.5
    Weight of cost loss in Eq. 20, set by hand; no sensitivity analysis is provided.
  • threshold T = 0.5
    Binarization threshold in Eq. 11, set by default to 0.5.
  • Gumbel-Sigmoid temperature TP = 5
    Temperature hyperparameter in Eq. 15, set by default to 5.
  • alpha1, alpha2 = 0.5, 0.5
    Blending weights for global and local features in DiT router, Eq. 14, fixed to 0.5.
  • O (router feature dimension) = C/64
    Reduced feature dimension for DiT router, default setting.
  • Warm-up steps = 10K (SD1.5), 5K (SD3.0)
    Mask fixed to 1 during warm-up; schedule chosen by hand in Sec A2.
assumptions (5)
  • standard math DDPM and flow-matching training losses (Eqs. 3 and 5) hold for the pretrained backbones.
    Used in Sec 3.1 as background for L_SD and the noise-prediction objective.
  • domain assumption Frozen pretrained SD1.5 and SD3.0 models provide a high-quality prior that controlled generation should not disrupt.
    Core to the ControlNet philosophy; assumed throughout Sec 3.2 and relied on by the frozen-backbone design.
  • domain assumption Gumbel-Sigmoid with straight-through sampling provides a valid differentiable approximation to the discrete mask.
    Eqs. 15 to 17; a standard approximation, but the paper gives no analysis of the bias introduced by the hard/soft mask mismatch.
  • domain assumption FLOPs ratio F_Flex/F_Large is a reliable proxy for actual computational cost and is compatible with gradient training.
    Used in cost loss Eq. 19; the paper computes FLOPs from block activations but does not discuss deviations from measured latency or training-time overhead.
  • domain assumption The three evaluation datasets (MultiGen-20M, LLAVA-558K, ADE20K) are representative for their corresponding control tasks.
    Sec A2; no analysis of domain shift or selection effects, and baseline checkpoints may have been trained on different data.
invented entities (1)
  • Router unit with Gumbel-Sigmoid binary mask independent evidence
    purpose: Dynamically selects which control blocks to activate at each denoising step, enabling sparsity and reducing FLOPs.
    The router's behavior is empirically characterized in Sec 4 and A3 (activation patterns vs timestep and sparsity), and its decisions produce measurable quality/compute trade-offs, giving it a falsifiable handle within the paper, though no external validation yet exists.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FlexControl: Computation-Aware ControlNet with Differentiable Router for Text-to-Image Generation." pith.science (2026). https://pith.science/paper/5D7AH2ZJ

@misc{pith2026250210451,
  author       = {Pith},
  title        = {Pith review of: FlexControl: Computation-Aware ControlNet with Differentiable Router for Text-to-Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5D7AH2ZJ}},
  note         = {Machine review of arXiv:2502.10451}
}
read the original abstract

ControlNet offers a powerful way to guide diffusion-based generative models, yet most implementations rely on ad-hoc heuristics to choose which network blocks to control-an approach that varies unpredictably with different tasks. To address this gap, we propose FlexControl, a novel framework that copies all diffusion blocks during training and employs a trainable gating mechanism to dynamically select which blocks to activate at each denoising step. With introducing a computation-aware loss, we can encourage control blocks only to activate when it benefit the generation quality. By eliminating manual block selection, FlexControl enhances adaptability across diverse tasks and streamlines the design pipeline, with computation-aware training loss in an end-to-end training manner. Through comprehensive experiments on both UNet (e.g., SD1.5) and DiT (e.g., SD3.0), we show that our method outperforms existing ControlNet variants in certain key aspects of interest. As evidenced by both quantitative and qualitative evaluations, FlexControl preserves or enhances image fidelity while also reducing computational overhead by selectively activating the most relevant blocks. These results underscore the potential of a flexible, data-driven approach for controlled diffusion and open new avenues for efficient generative model design. The code will soon be available at https://github.com/Anonymousuuser/FlexControl.

Figures

Figures reproduced from arXiv: 2502.10451 by the authors.

Figure 1
Figure 1. Dynamically inject conditional controls for image generation based on timestep and specific sample. (a) The architecture of the ControlNet. (b) The architecture of the proposed FlexControl. (c) Statistics on the number of activated control blocks of the FlexControl at each denoising step. Here, “50% spasity” indicates that the number of floating-point operations (FLOPs) of activated blocks is limited to 50% of the t… view at source ↗
Figure 2
Figure 2. Overview of dynamic routing guided by the router unit. (a) In the training stage, Gumbel noise is added to the discrete mask to assist the gradient backpropagation. (b) In the inference stage, the router unit controls whether to activate the control block and whether to inject conditional control into the frozen block of the backbone according to the input latent variable. Once output the instruction of inactive, th… view at source ↗
Figure 3
Figure 3. Qualitative comparison of controllable generation methods. FlexControl achieves higher fidelity and structure preservation across Depth Map, Canny Edge, and Segmentation Mask conditions, reducing distortions (boxes) seen in other methods. It better aligns with input conditions while maintaining visual quality. 4. Experiment We evaluate FlexControl against state-of-the-art methods across different image conditions: d… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Comparison of FlexControl and existing methods on SD3.0 for edge preservation. FlexControl maintains better spa￾tial consistency and object integrity across different sparsity levels, while ControlNet-based methods introduce distortions and incon￾sistencies (highlighte…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 32 canonical work pages

  1. [1]

    All are worth words: A vit backbone for diffusion models

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 22669–22679, 2023. 2

    work page 2023

  2. [2]

    One transformer fits all distributions in multi-modal diffu- sion at scale

    Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffu- sion at scale. InInternational Conference on Machine Learn- ing, pages 1692–1717. PMLR, 2023. 2

    work page 2023

  3. [3]

    A computational approach to edge detection

    John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelli- gence, pages 679–698, 1986. 12

    work page 1986

  4. [4]

    Training-free layout control with cross-attention guidance

    Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5343–5353, 2024. 2

    2024

  5. [5]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 12

    work page 2024

  6. [6]

    RepControlNet: ControlNet Reparameterization

    Zhaoli Deng, Kaibin Zhou, Fanyi Wang, and Zhenpeng Mi. Repcontrolnet: Controlnet reparameterization. arXiv preprint arXiv:2408.09240, 2024. 3

    work page Pith review arXiv 2024

  7. [7]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 2

    2021

  8. [8]

    Repvgg: Making vgg-style convnets great again

    Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 13733–13742, 2021. 3

    work page 2021

Show all 62 references
  1. [9]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 8

    2021

  2. [10]

    Scaling recti- fied flow transformers for high-resolution image synthesis, march 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis, march 2024. URL http://arxiv. org/abs/2403.0320...

    2024 arXiv

  3. [11]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  4. [12]

    Structural pruning for diffusion models

    Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. Advances in Neural Informa- tion Processing Systems, 2023. 3

    2023

  5. [13]

    Not all prompts are made equal: Prompt-based pruning of text-to-image diffusion models

    Alireza Ganjdanesh, Reza Shirkavand, Shangqian Gao, and Heng Huang. Not all prompts are made equal: Prompt-based pruning of text-to-image diffusion models. arXiv preprint arXiv:2406.12042, 2024. 3

    2024 arXiv

  6. [14]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 13

    2017

  7. [15]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  8. [16]

    Plug-and- play diffusion distillation

    Yi-Ting Hsiao, Siavash Khodadadeh, Kevin Duarte, Wei-An Lin, Hui Qu, Mingi Kwon, and Ratheesh Kalarot. Plug-and- play diffusion distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13743–13752, 2024. 3

    2024

  9. [17]

    Cock- tail: Mixing multi-modality control for text-conditional im- age generation

    Minghui Hu, Jianbin Zheng, Daqing Liu, Chuanxia Zheng, Chaoyue Wang, Dacheng Tao, and Tat-Jen Cham. Cock- tail: Mixing multi-modality control for text-conditional im- age generation. In Thirty-seventh Conference on Neural In- formation Processing Systems, 2023. 3

    2023

  10. [18]

    Composer: Creative and controllable im- age synthesis with composable conditions

    Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable im- age synthesis with composable conditions. arXiv preprint arXiv:2302.09778, 2023. 2

    2023 arXiv

  11. [19]

    Reversion: Diffusion-based relation inversion from images

    Ziqi Huang, Tianxing Wu, Yuming Jiang, Kelvin CK Chan, and Ziwei Liu. Reversion: Diffusion-based relation inversion from images. arXiv preprint arXiv:2303.13495, 2023

    2023 arXiv

  12. [20]

    Avatar- craft: Transforming text into neural human avatars with pa- rameterized shape and pose control

    Ruixiang Jiang, Can Wang, Jingbo Zhang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Avatar- craft: Transforming text into neural human avatars with pa- rameterized shape and pose control. In Proceedings of the IEEE/CVF International Conference on Computer Vision ,...

    2023

  13. [21]

    Humansd: A native skeleton-guided diffusion model for human image generation

    Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu. Humansd: A native skeleton-guided diffusion model for human image generation. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 15988–15998, 2023. 2

    2023

  14. [22]

    Brushnet: A plug-and-play image inpaint- ing model with decomposed dual-branch diffusion

    Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: A plug-and-play image inpaint- ing model with decomposed dual-branch diffusion. arXiv preprint arXiv:2403.06976, 2024. 2, 4

    2024 arXiv

  15. [23]

    On architectural compression of text-to- image diffusion models

    Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. On architectural compression of text-to- image diffusion models. arXiv preprint arXiv:2305.15798,

    arXiv

  16. [24]

    Variational diffusion models

    Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural infor- mation processing systems, 34:21696–21707, 2021. 2

    2021

  17. [25]

    Multi-architecture multi-expert diffusion models

    Yunsung Lee, JinYoung Kim, Hyojun Go, Myeongho Jeong, Shinhyeok Oh, and Seungtaek Choi. Multi-architecture multi-expert diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 13427–13436,

  18. [26]

    Controlnet ++: Improving conditional controls with efficient consistency feedback

    Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaon- ing Wang, Xuefeng Xiao, and Chen Chen. Controlnet ++: Improving conditional controls with efficient consistency feedback. In European Conference on Computer Vision , pages 129–147. Springer, 2025. 3, 7

    2025

  19. [27]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521, 2023. 2, 7

    2023

  20. [28]

    Flow matching for generative mod- eling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling. In International Conference on Learning Representa- tions, 2023. 3

    2023

  21. [29]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 6, 12

    2024

  22. [30]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. 3

    2022 arXiv

  23. [31]

    Instaflow: One step is enough for high-quality diffusion- based text-to-image generation

    Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion- based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023. 3

    2023

  24. [32]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296–4304, 20...

    2024

  25. [33]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In Interna- tional conference on machine learning, 2022. 2

    2022

  26. [34]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  27. [35]

    Controlnext: Powerful and effi- cient control for image and video generation

    Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming- Chang Yang, and Jiaya Jia. Controlnext: Powerful and effi- cient control for image and video generation. arXiv preprint arXiv:2408.06070, 2024. 3

    2024 arXiv

  28. [36]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 2, 7

    2023 arXiv

  29. [37]

    Unicontrol: A unified diffusion model for controllable visual generation in the wild

    Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint arXiv:2305.11147, 2023. 3

    2023 arXiv

  30. [38]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  31. [39]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 2

    2020

  32. [40]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Confer- ence for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. 13

    2020

  33. [41]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937–13949,

  34. [42]

    Dynamic spatial sparsification for efficient vi- sion transformers and convolutional neural networks

    Yongming Rao, Zuyan Liu, Wenliang Zhao, Jie Zhou, and Jiwen Lu. Dynamic spatial sparsification for efficient vi- sion transformers and convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10883–10897, 2023. 5

    2023

  35. [43]

    Image deblurring with domain generalizable diffusion models

    Mengwei Ren, Mauricio Delbracio, Hossein Talebi, Guido Gerig, and Peyman Milanfar. Image deblurring with domain generalizable diffusion models. arXiv preprint arXiv:2212.01789, 1, 2022. 2

    2022 arXiv

  36. [44]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 8

    2022

  37. [45]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  38. [46]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  39. [47]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  40. [48]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 2

    2015

  41. [49]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In International Conference on Learning Representations, 2021. 13

    2021

  42. [50]

    Stable diffusion v1.5 model card

    Stability. Stable diffusion v1.5 model card. https://huggingface.co/runwayml/stable-diffusion-v1-5,

  43. [51]

    Maxvit: Multi-axis vision transformer

    Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In European conference on computer vision, pages 459–479. Springer, 2022. 2

    2022

  44. [52]

    Pretraining is all you need for image-to-image translation

    Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. arXiv preprint arXiv:2205.12952, 2022. 8

    2022 arXiv

  45. [53]

    Instancediffusion: Instance- level control for image generation

    Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Ro- hit Girdhar, and Ishan Misra. Instancediffusion: Instance- level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6232–6242, 2024. 2

    2024

  46. [54]

    Reco: Region-controlled text-to-image genera- tion

    Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image genera- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 14246–14255,

  47. [55]

    Improving ef- ficiency of diffusion models via multi-stage framework and tailored multi-decoder architectures

    Huijie Zhang, Yifu Lu, Ismail Alkhouri, Saiprasad Rav- ishankar, Dogyoon Song, and Qing Qu. Improving ef- ficiency of diffusion models via multi-stage framework and tailored multi-decoder architectures. arXiv preprint arXiv:2312.09181, 2023. 3

    2023

  48. [56]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 1, 2, 4, 7, 13

    2023

  49. [57]

    Controllable text-to-image generation with gpt-

    Tianjun Zhang, Yi Zhang, Vibhav Vineet, Neel Joshi, and Xin Wang. Controllable text-to-image generation with gpt-

  50. [58]

    arXiv preprint arXiv:2305.18583, 2023. 2

    2023 arXiv

  51. [59]

    Uni-controlnet: All-in-one control to text-to-image diffusion models

    Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Sys- tems, 36, 2024. 3, 6, 12

    2024

  52. [60]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 633–641,

  53. [61]

    Please use a brief sentence with as few words as pos- sible to summarize the picture

    Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, and Yi Yang. Migc: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6818– 6828, 2024. 2 FlexControl: Computation-Aware Cont...

    2024

  54. [62]

    We generate five groups of images, and the average results are reported

    and RFlow [10] sampler, implementing 20 denoising steps to generate images without incorporating any nega- tive prompts. We generate five groups of images, and the average results are reported. A3. Dynamic Route Exploration In order to improve the parameter utilization of Cont...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.