Pith. sign in

REVIEW 3 major objections 4 minor 57 references

Scale Your Instructions: Enhance the Instruction-Following Fidelity of Unified Image Generation Model by Self-Adaptive Attention Scaling

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that instruction neglect in unified image generation models such as OmniGen comes from the input image's cross-attention activations overwhelming those of individual sub-instructions, and that a training-free…

desk verdict Solid training-free fix for multi-instruction neglect in OmniGen, but the cross-attention consistency assumption needs quantitative support before the mechanism is fully convincing. read the letter →

arxiv 2507.16240 v1 pith:3TMKSGLJ submitted 2025-07-22 cs.CV

Chao Zhou , Tianyi Wei , Nenghai Yu This is my paper
classification cs.CV
keywords instruction-followingfidelityunifiedimagegenerationcross-attentionmapseditingattentionscalingOmniGentraining-freeinferencevisualconditional
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Unified image generation models such as OmniGen accept free-form text and image inputs in one sequence, but when a prompt contains several sub-instructions the model tends to drop some of them. This paper claims the drop-out is caused by a conflict in the cross-attention maps at key denoising steps: the input image activates strongly in the region where a sub-instruction should appear, suppressing that instruction's influence. To fix it, the paper proposes Self-Adaptive Attention Scaling (SaaS), an inference-time method that computes a spatial mask and a per-instruction scaling factor from the cross-attention map at step t, then applies the scale to the masked region at step t−1. SaaS needs no training and no test-time optimization, and the paper reports that it makes OmniGen follow every sub-instruction in multi-part editing and in depth- and segmentation-conditional generation, at roughly one percent extra latency.

What carries the argument

The load-bearing object is the cross-attention map of the unified transformer, defined in the paper as the attention from noise-latent tokens to the concatenation of input-image tokens and text-instruction tokens. SaaS operates on maps averaged over layers and heads, smoothed with a Gaussian filter, and aggregated per sub-instruction by summing the maps of its tokens. A mask is obtained by min-max normalizing the resulting map and thresholding it (τ = 0.4 for editing, τ = 0.2 for conditional generation), and the scaling factor is the ratio of the input image's summed activation to the sub-instruction's summed activation within the mask. The mechanism that makes this cheap is temporal consistency: the same mask and factor are carried one step backward, so the method performs one forward pass with no extra optimization.

What would settle it

Measure the overlap, e.g., the mean Dice or IoU score, between the binarized sub-instruction mask extracted at step t and the mask extracted at step t−1 across all 50 denoising steps on a large prompt set. If the overlap drops sharply in the early steps where SaaS is active, or falls below a usable level for global edits, then the temporal-consistency premise is falsified and SaaS should not work as described.

Watch

Extended reading notes

Core claim

The central claim is that instruction neglect in OmniGen originates in the cross-attention maps of the early denoising steps: for a neglected sub-instruction, the region where that sub-instruction should exert influence is dominated by high activations from the input image tokens, so the instruction's signal is overwhelmed. The paper supports this with perturbation experiments that identify the vital steps and layers, and with visualizations showing the activation overlap. On top of that diagnosis, SaaS rebalances the competition: it builds a binary mask of each sub-instruction's region from smoothed, averaged cross-attention maps, computes the ratio of the input image's summed activation to the sub-instruction's summed activation inside the mask, and uses that ratio to scale the sub-instruction's attention up at the neighboring timestep. Because cross-attention maps are reported to be consistent between adjacent timesteps, the mask and factor computed at step t can be applied to step t−1 without recomputation.

Load-bearing premise

The entire method rests on the assumption that cross-attention maps stay nearly identical from one denoising step to the next, so a mask and scaling factor computed at step t remain valid when applied at step t−1; if the maps shift, SaaS will amplify the wrong image regions or the wrong instructions.

Editorial extensions

If this is right

  • On the EMU-Edit and PIE-Bench evaluations, SaaS lifts CLIP-T and PickScore well above the baseline models, with the largest gains on prompts containing two to four sub-instructions.
  • The same procedure transfers to visual conditional generation: images generated from depth maps and segmentation maps score higher than OmniGen on all four reported metrics (CLIP-I, DINO-v2, CLIP-T, PickScore).
  • Fixed scaling factors cannot replace the adaptive scheme: a factor that is right for one sub-instruction over-edits another, while SaaS computes a different factor for each.
  • SaaS is a drop-in inference module, adding no training and negligible cost (about 0.3 seconds and 2 MB of VRAM over the base model).
  • The appendix shows SaaS also improves instruction following on a different unified editing model (MIGE), indicating the mechanism is not tied to OmniGen alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The conflict diagnosis is correlational: the paper shows overlap between image activations and neglected-instruction regions, but does not prove the overlap causes the neglect; a controlled experiment that artificially suppresses image activations in those regions would separate cause from symptom.
  • The method's reliance on cross-attention stability across adjacent timesteps is only shown qualitatively; because the appendix reports that the optimal mask threshold differs between local and global edits, one can expect the mask-transfer step to be most fragile precisely when an edit changes global layout.
  • The same mask-and-ratio logic could be applied to other attention-based generative models, including autoregressive multimodal generators, whenever cross-attention maps are exposed; testing on video or multi-object scenes, where conflicts are sharper, would stress-test the conflict hypothesis.
  • Scaling factors are computed from raw activation sums, so a sub-instruction with intrinsically low text attention yet correct behavior could be needlessly amplified; an alternative normalizing by instruction length or by the instruction's own baseline activation might reduce over-editing cases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper investigates text-instruction neglect in the unified image generation model OmniGen. Through perturbation analysis it identifies the vital steps and layers of the denoising process, and by inspecting cross-attention maps it observes that neglected sub-instructions have activation regions that conflict with the input image activations. It proposes Self-Adaptive Attention Scaling (SaaS), which at denoising step t extracts a binary mask and a scaling factor for each sub-instruction and applies them at step t-1 to rescale the cross-attention values in the masked regions. Experiments on instruction-based image editing and visual conditional image generation report consistent improvements in CLIP-T and PickScore over OmniGen and several baselines, with no additional training and negligible overhead.

Significance. If the quantitative results are reliable, SaaS is a practically useful, training-free inference-time intervention: it improves instruction-following in a unified generation model while adding only about 1% latency and negligible memory. The paper contributes a clean perturbation-based diagnostic, a non-circular derivation of the scaling factors from observed attention activations, and a generalizability check on a different architecture (MIGE). The main limitation is that the central mechanism rests on an assumption of cross-attention consistency across adjacent timesteps that is only supported qualitatively.

major comments (3)
  1. [Sec. 3.3.3, Eq. (5)] The binary mask M_t[T_i] computed at step t is applied to the cross-attention at step t-1, and the paper justifies this by the claim that cross-attention maps are 'highly consistent across adjacent timesteps' (Fig. 2c). This is a load-bearing assumption: because the mask is binary, any position outside M_t[T_i] receives zero attention for that sub-instruction at step t-1, so a misaligned mask could suppress the very instruction the method aims to amplify. I ask for a quantitative consistency measure over the steps where SaaS is active: e.g., per-step IoU of the masks M_t[T_i] versus M_{t-1}[T_i], or cosine similarity of the smoothed maps A_t[T_i] and A_{t-1}[T_i], reported separately for local and global edits. Appendix D's observation that the threshold behavior differs between local and global edits suggests that mask stability is not guaranteed, and the current qualitative evidence (a single example in Fig. 2c) is insufficient to establish the validity of the one-step lag.
  2. [Sec. 3.3.1, Appendix D] The threshold tau is not self-adaptive: the implementation uses tau=0.4 for instruction-based editing and tau=0.2 for visual conditional generation, and Appendix D states that 'lower thresholds work better for global editing, while higher thresholds suit local editing.' This means the user must know the edit type a priori and choose tau accordingly. Table 5 reports very small PickScore variation across thresholds on one example, but Fig. 11 shows clear visual failures when tau is mismatched to the edit type. The paper should provide a per-task sensitivity analysis with multiple examples and either justify a fixed value for each task category or give a principled automatic selection rule that does not require knowing whether the edit is local or global. As written, the method's robustness claim ('the choice of threshold is not critical') is contradicted by the local/global caveat immediately below.
  3. [Sec. 4.2, Tables 1, 3, 6] All quantitative results are single-run point estimates without error bars, confidence intervals, or significance tests. This is particularly important for the central claim of 'superior instruction-following fidelity,' since the PickScore gaps are large (e.g., 0.462 vs. 0.233 in Table 1, single-instruction) and might still be stable, but the CLIP-I/DINO-v2 differences relative to OmniGen are small and the text in Sec. 4.2 attributes them to instruction neglect rather than to method noise. I recommend reporting results over multiple random seeds (or, at minimum, bootstrapped confidence intervals over the evaluation set) and a paired significance test for the user study in Table 2. Without these, the reader cannot assess whether the reported improvements are statistically reliable or driven by a few favorable examples.
minor comments (4)
  1. [Sec. 3.3.2, Eq. (4)] Equation (4) appears to have a typographical error: the formula shows 'P(At[I] · Mt[Ti])P(At[Ti] · Mt[Ti])' without indicating the division operator. It should read 'P(At[I] · Mt[Ti]) / P(At[Ti] · Mt[Ti])'. Please fix.
  2. [Sec. 3.3.1] The cross-attention map is described as 32x32 'assuming the generated image resolution is 512x512.' Clarify how the map resolution scales with output resolution and whether the method requires a fixed output size.
  3. [Sec. 4.1] The implementation details mention image guidance SI=1.6 and text guidance ST=2.5; these abbreviations are not defined in the main text. Please define SI and ST.
  4. [Sec. 4.2, Table 2] The user study description reports 32 participants and 50 image pairs per participant, but does not state the total number of editing examples per condition (single vs. multiple sub-instruction) or whether the same images were used across participants. Please specify the protocol more precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SaaS scaling factors are computed from observed cross-attention maps at inference time and validated on external benchmarks.

full rationale

The paper's derivation is self-contained and does not reduce to its inputs by construction. The central quantities, mask M_t[T_i] and scaling factor alpha_t[T_i], are computed from observed cross-attention activations at timestep t (Eqs. 2-4) and applied at timestep t-1 (Eq. 5); they are not fitted to the evaluation metrics. The threshold tau is a simple hand-set hyperparameter, and Appendix D (Table 5) shows the result is stable across a range of thresholds, so it is not a fitted parameter renamed as a prediction. The adjacent-timestep consistency assumption is an empirical premise illustrated in Fig. 2c; even if it were quantitatively weak, that would be a correctness risk rather than circularity. Evaluation is performed on held-out standard benchmarks (EMU-Edit, PIE-Bench, MultiGen-20M, ADE20K) with external metrics (CLIP-T, PickScore, user study), so the claimed improvements are not equivalent to the method's definitions. Self-citations (refs. 43, 44, 50) appear only in related-work context and are not load-bearing for the mechanism. Therefore no specific circular step can be quoted, and the paper warrants a score of 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The main free choices are the mask threshold tau (task-specific), the number of initial steps where SaaS is applied, the set of attention layers used for mask extraction, and the timestep-dependent coefficient xi (set to 1). The core axioms are the empirical consistency of cross-attention across adjacent timesteps and the causal hypothesis that input-image activations suppress instruction activations.

free parameters (4)
  • Mask threshold tau = 0.4 for editing, 0.2 for conditional generation
    Hand-set per task; appendix shows it is not fully insensitive (e.g., too low for local editing causes unrealistic appearance, too high for global editing causes failure).
  • SaaS window (initial steps) = 20 of 50 steps
    Empirically chosen; SaaS is most effective in early denoising stages.
  • Vital layers used for cross-attention averaging = unspecified in main text (deep layers)
    The paper says shallow layers are negligible but does not specify exact layer indices; appendix shows performance differences with layer subsets.
  • Timestep-related hyperparameter xi_t = 1
    Set to 1 for simplicity; not optimized.
assumptions (3)
  • domain assumption Cross-attention maps are highly consistent across adjacent timesteps
    Asserted from Fig. 2c and used to transfer masks and scaling factors from step t to step t-1.
  • domain assumption Instruction neglect is caused by conflict between input image and instruction activations in cross-attention maps
    Central hypothesis; no causal proof, only correlational observation.
  • domain assumption Thresholded mask extraction identifies regions of influence of each sub-instruction
    Assumes min-max normalization and thresholding produce faithful spatial masks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scale Your Instructions: Enhance the Instruction-Following Fidelity of Unified Image Generation Model by Self-Adaptive Attention Scaling." pith.science (2026). https://pith.science/paper/3TMKSGLJ

@misc{pith2026250716240,
  author       = {Pith},
  title        = {Pith review of: Scale Your Instructions: Enhance the Instruction-Following Fidelity of Unified Image Generation Model by Self-Adaptive Attention Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3TMKSGLJ}},
  note         = {Machine review of arXiv:2507.16240}
}
read the original abstract

Recent advancements in unified image generation models, such as OmniGen, have enabled the handling of diverse image generation and editing tasks within a single framework, accepting multimodal, interleaved texts and images in free form. This unified architecture eliminates the need for text encoders, greatly reducing model complexity and standardizing various image generation and editing tasks, making it more user-friendly. However, we found that it suffers from text instruction neglect, especially when the text instruction contains multiple sub-instructions. To explore this issue, we performed a perturbation analysis on the input to identify critical steps and layers. By examining the cross-attention maps of these key steps, we observed significant conflicts between neglected sub-instructions and the activations of the input image. In response, we propose Self-Adaptive Attention Scaling (SaaS), a method that leverages the consistency of cross-attention between adjacent timesteps to dynamically scale the attention activation for each sub-instruction. Our SaaS enhances instruction-following fidelity without requiring additional training or test-time optimization. Experimental results on instruction-based image editing and visual conditional image generation validate the effectiveness of our SaaS, showing superior instruction-following fidelity over existing methods. The code is available https://github.com/zhouchao-ops/SaaS.

Figures

Figures reproduced from arXiv: 2507.16240 by the authors.

Figure 1
Figure 1. Results of SaaS. OmniGen [46] has a tendency to overlook some sub-instructions, while our SaaS can effectively mitigate this issue. Notably, SaaS does not require any extra training or test-time optimization. Zoom in for better visualization. Abstract Recent advancements in unified image generation mod￾els, such as OmniGen, have enabled the handling of diverse image generation and editing tasks within a single frame… view at source ↗
Figure 2
Figure 2. Cross-attention maps for the input image and different sub-instructions. We can get three key observations: (a) we can pre-identify the regions where each sub-instruction will appear according to the corresponding cross-attention map; (b) the regions of activation for the neglected sub-instruction are highly conflicting with those for the input image, where the input image dominates (red box); (c) the cross-attentio… view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: DINO-v2 similarity [23] by perturbed layers. Com￾paring the similarity from top to bottom (blue curve) and from bottom to top (yellow curve), the perturbations in shallow layers have minimal effect on the image generation process. 3.3. Self-Adaptive Attention Scaling G…
Figure 5
Figure 5. Figure 5: Framework of SaaS. SaaS is designed to enhance the instruction-following fidelity of unified image generation models. At the denoising step t, a unique mask of each sub-instruction is extracted according to the cross-attention map averaged by the vital layers. Subseque…
Figure 6
Figure 6. Figure 6: Qualitative comparisons of instruction-based image editing. We present edited images from the baselines using the same input for each column. From top to bottom: input image, our method, OmniGen [46], IP2P [2], and MagicBrush [52]. The text above each column represents…
Figure 7
Figure 7. Figure 7: Qualitative comparison of visual conditional image generation. The images on the left are generated from the depth map, while those on the right are generated from the segmentation map. Our SaaS method outperforms OmniGen in both instruction-following fidelity and imag…
Figure 8
Figure 8. Figure 8: Qualitative comparisons between SaaS and scale with a fixed factor. We compared SaaS with fixed scaling factors of 2 and 5. The results show that while direct scaling has some effect, our SaaS method outperforms this fixed scaling approach. Zoom in for better visualiza…
Figure 10
Figure 10. Figure 10: Demonstration of SaaS on challenging editing tasks. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 9
Figure 9. Figure 9: Cases of SaaS on MIGE. Zoom in for better visualiza [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 11
Figure 11. Figure 11: Visual comparison of editing results under different [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Visual comparisons between various steps and layers. [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Additional qualitative comparison of instruction-based image editing. We compare our SaaS with these state-of-the-art image editing methods. Zoom in for better visualization [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Additional qualitative comparison of visual conditional image generation. We compare our SaaS method with OmniGen in the generation tasks from the depth map and the segmentation map. The text below each image represents the corresponding instruction. For better visual…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 39 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jy- oti Aneja, and Ahmed Awadallah. Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abs/2404.14219, 2024. 3

    arXiv 2024

  2. [2]

    Tim Brooks, Aleksander Holynski, and Alexei A. Efros. In- structpix2pix: Learning to follow image editing instructions. In 2023 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 18392–18402, 2023. 3, 6, 7

    work page 2023

  3. [3]

    Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xi- aohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV) , pages 22560–22570,

  4. [4]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. 2022 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 11305–11315,

    work page 2022

  5. [5]

    Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models. ACM Trans. Graph., 42(4), 2023. 4

    work page 2023

  6. [6]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101,

  7. [7]

    Intern vl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, and Sen Xing. Intern vl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In 2024 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 24185–24198, 2024. 3

    work page 2024

  8. [8]

    Tam- ing Transformers for High-Resolution Image Synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Tam- ing Transformers for High-Resolution Image Synthesis . In 2021 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 12868–12878, Los Alami- tos, CA, USA, 2021. IEEE Computer Society. 3

    work page 2021

Show all 57 references
  1. [9]

    Scaling rec- tified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rec- tified flow transformers for high-resolution image syn...

    2024

  2. [10]

    Focus on your instruction: Fine-grained and multi-instruction image editing by atten- tion modulation

    Qin Guo and Tianwei Lin. Focus on your instruction: Fine-grained and multi-instruction image editing by atten- tion modulation. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6986–6996,

    2024

  3. [11]

    Prompt-to-prompt image editing with cross-attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with cross-attention control. In The Eleventh Inter- national Conference on Learning Representations, 2023. 2

    2023

  4. [12]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 13

    2022 arXiv

  5. [13]

    An edit friendly ddpm noise space: Inversion and manipulations

    Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In 2024 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 12469– 12478, 2024. 3

    2024

  6. [14]

    Pnp inversion: Boosting diffusion-based editing with 3 lines of code

    Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Pnp inversion: Boosting diffusion-based editing with 3 lines of code. InThe Twelfth International Conference on Learning Representations, 2024. 5

    2024

  7. [15]

    What’s in the im- age? a deep-dive into the vision of vision language models

    Omri Kaduri, Shai Bagon, and Tali Dekel. What’s in the im- age? a deep-dive into the vision of vision language models. arXiv preprint arXiv:2411.17491, 2024. 3

    2024 arXiv

  8. [16]

    Diffusion models for open-vocabulary segmenta- tion

    Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for open-vocabulary segmenta- tion. In Computer Vision – ECCV 2024: 18th European Con- ference, Milan, Italy, September 29–October 4, 2024, Pro- ceedings, Part V , page 299–317, Berlin, Hei...

    2024

  9. [17]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Ma- tiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In Thirty-seventh Conference on Neural Information Process- ing Systems, 2023. 5

    2023

  10. [18]

    Open-vocabulary object segmenta- tion with diffusion models

    Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Open-vocabulary object segmenta- tion with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 4

    2023

  11. [19]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maxim- ilian Nickel, and Matthew Le. Flow matching for genera- tive modeling. In The Eleventh International Conference on Learning Representations, 2023. 3

    2023

  12. [20]

    Ace++: Instruction- based image creation and editing via context-aware content filling

    Chaojie Mao, Jingfeng Zhang, Yulin Pan, Zeyinzi Jiang, Zhen Han, Yu Liu, and Jingren Zhou. Ace++: Instruction- based image creation and editing via context-aware content filling. arXiv preprint arXiv:2501.02487, 2025. 13

    2025 arXiv

  13. [21]

    Null-text inversion for editing real images using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In CVPR, pages 6038–6047,

  14. [22]

    T2i-adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Th...

    2024

  15. [23]

    Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael ...

    2024

  16. [24]

    DiffuseV AE: Efficient, controllable and high- fidelity generation from low-dimensional latents

    Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Ab- hishek Kumar. DiffuseV AE: Efficient, controllable and high- fidelity generation from low-dimensional latents. Transac- tions on Machine Learning Research, 2022. 2

    2022

  17. [25]

    Peebles and Saining Xie

    William S. Peebles and Saining Xie. Scalable diffusion mod- els with transformers. 2023 IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 4172–4182, 2022. 3

    2023

  18. [26]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth Interna- tional Conference on Learning Representations, 2024. 1

    2024

  19. [27]

    Unicontrol: A unified diffusion model for controllable visual generation in the wild

    Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, Stefano Ermon, Yun Fu, and Ran Xu. Unicontrol: A unified diffusion model for controllable visual generation in the wild. In Thirty-seventh Conferen...

    2023

  20. [28]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Con...

    2021

  21. [29]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learn- ing Research, 21(140):1–67, 2020. 2

    2020

  22. [30]

    High-Resolution Image Synthesis with Latent Diffusion Models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models . In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674–10685, Los Alamitos, CA, USA,

  23. [31]

    Semantic image inversion and editing using rec- tified stochastic differential equations

    L Rout, Y Chen, N Ruiz, C Caramanis, S Shakkottai, and W Chu. Semantic image inversion and editing using rec- tified stochastic differential equations. In arXiv preprint arxiv:2410.10792, 2024. 3

    2024 arXiv

  24. [32]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. 2023 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 22500–22510,

    2023

  25. [33]

    Emu edit: Precise image editing via recognition and gener- ation tasks

    Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and gener- ation tasks. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8871–8879,

    2024

  26. [34]

    Generative Multimodal Mod- els are In-Context Learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative Multimodal Mod- els are In-Context Learners . In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1...

    2024

  27. [35]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. ArXiv, abs/2405.09818, 2024. 3

    2024 arXiv

  28. [36]

    Qwen2.5-vl, 2025

    Qwen Team. Qwen2.5-vl, 2025. 3, 5

    2025

  29. [37]

    Mige: A unified framework for multimodal instruction-based image generation and editing

    Xueyun Tian, Wei Li, Bingbing Xu, Yige Yuan, Yuanzhuo Wang, and Huawei Shen. Mige: A unified framework for multimodal instruction-based image generation and editing. arXiv preprint arXiv:2502.21291, 2025. 12

    2025 arXiv

  30. [38]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1921–1930, 2023. 2

    1921

  31. [39]

    Neural discrete representation learning

    Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems , page 6309–6318, Red Hook, NY , USA, 2017. Curran Associates Inc. 3

    2017

  32. [40]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Ke-Yang Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  33. [41]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, and Liangdong Wang. Emu3: Next-token prediction is all you need. CoRR, abs/2...

    2024 arXiv

  34. [42]

    Hairclipv2: Unifying hair editing via proxy feature blending

    Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Weiming Zhang, Gang Hua, and Nenghai Yu. Hairclipv2: Unifying hair editing via proxy feature blending. InProceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 23589–23599, 2023. 3

    2023

  35. [43]

    Enhancing mmdit-based text-to-image models for similar subject generation

    Tianyi Wei, Dongdong Chen, Yifan Zhou, and Xingang Pan. Enhancing mmdit-based text-to-image models for similar subject generation. arXiv preprint arXiv:2411.18301, 2024. 3

    2024 arXiv

  36. [44]

    Freeflux: Understanding and exploiting layer-specific roles in rope-based mmdit for versatile image editing

    Tianyi Wei, Yifan Zhou, Dongdong Chen, and Xingang Pan. Freeflux: Understanding and exploiting layer-specific roles in rope-based mmdit for versatile image editing. arXiv preprint arXiv:2503.16153, 2025. 3

    2025 arXiv

  37. [45]

    Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using dif- fusion models

    Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using dif- fusion models. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1206–1217, 2023. 4

    2023

  38. [46]

    Omnigen: Unified image generation

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xin- grun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. CoRR, abs/2409.11340, 2024. 1, 2, 3, 6, 7, 8

    2024 arXiv

  39. [47]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. In The Thirteenth International Conference on ...

    2025

  40. [48]

    Characteristic analysis of otsu threshold and its ap- plications

    Xiangyang Xu, Shengzhou Xu, Lianghai Jin, and Enmin Song. Characteristic analysis of otsu threshold and its ap- plications. Pattern recognition letters, 32(7):956–961, 2011. 12

    2011

  41. [49]

    Enhancing instruction-following capability of visual-language models by reducing image redundancy

    Te Yang, Jian Jia, Xiangyu Zhu, Weisong Zhao, Bo Wang, Yanhua Cheng, Yan Li, Shengyuan Liu, Quan Chen, Peng Jiang, et al. Enhancing instruction-following capability of visual-language models by reducing image redundancy. arXiv preprint arXiv:2411.15453, 2024. 3

    2024 arXiv

  42. [50]

    RoCC: Robust Covert Communication Based on Cross-Modal Information Retrieval.Journal of Im- age and Graphics, 29(2):369–381, 2024

    Zhang Yanming, Chen Kejiang, Ding Jinyang, Zhang Weim- ing, and Yu Nenghai. RoCC: Robust Covert Communication Based on Cross-Modal Information Retrieval.Journal of Im- age and Graphics, 29(2):369–381, 2024. 3

    2024

  43. [51]

    Peeling back the layers: Interpreting the story- telling of vit

    Jingjie Zeng, Zhihao Yang, Qi Yang, Liang Yang, and Hongfei Lin. Peeling back the layers: Interpreting the story- telling of vit. In Proceedings of the 32nd ACM International Conference on Multimedia, page 7298–7306, New York, NY , USA, 2024. Association for Computing Machinery. 3

    2024

  44. [52]

    Magicbrush: A manually annotated dataset for instruction- guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction- guided image editing. In Thirty-seventh Conference on Neu- ral Information Processing Systems Datasets and Bench- marks Track, 2023. 6, 7

    2023

  45. [53]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In IEEE International Conference on Computer Vision (ICCV),

  46. [54]

    Ultraedit: Instruction-based fine-grained image editing at scale

    Haozhe Zhao, Xiaojian Shawn Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-based fine-grained image editing at scale. Advances in Neural Information Pro- cessing Systems, 37:3058–3093, 2024. 13

    2024

  47. [55]

    Semantic under- standing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. Int. J. Com- put. Vision, 127(3):302–321, 2019. 5

    2019

  48. [56]

    graffiti

    Chunting Zhou, LILI YU, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi- modal model. In The Thirteenth International Conference on Le...

    2025

  49. [2022]

    IEEE Computer Society. 1

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.