Pith. sign in

REVIEW 4 major objections 5 minor 63 references

R-Genie: Reasoning-Guided Generative Image Editing

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces R-Genie, a 1.3B reasoning-guided image editor that couples a multimodal LLM's implicit-intent reasoning with discrete diffusion, and claims superior editing accuracy over nine baselines on a new benchmark, REditBench.

desk verdict R-Genie's REditBench numbers are self-referential: the model trains on the same SDXL/CLIP-generated targets it is evaluated against, so the claimed superiority is likely benchmark imitation, not reasoning. read the letter →

arxiv 2505.17768 v2 pith:RT3YGQU4 submitted 2025-05-23 cs.CV

Dong Zhang , Lingfeng He , Rui Yan , Fei Shen , Jinhui Tang This is my paper
classification cs.CV
keywords reasoning-guidededitingimagemultimodallargelanguagemodeldiscretediffusionreasoning-attentionbridgeintentioninferenceREditBenchworldknowledge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a new image-editing paradigm it calls reasoning-guided generative editing: the model receives a high-level query such as "Whose clothes are more likely to make him invisible in the wild? Switch him or her with a female," and must infer which object to change, what change to make, and how to keep the rest of the scene intact. To support this task, the authors build REditBench, over 1,000 image-instruction-edit triples with reasoning contexts, and propose R-Genie, which couples a multimodal LLM with a discrete diffusion editor. The central claim is that a joint reasoning-attention bridge lets the model ground linguistic reasoning in spatially precise visual features, and that on REditBench R-Genie attains the best CLIP similarity, lowest background L2 loss, and highest RISEBench score among nine baselines, with only 1.3B parameters.

What carries the argument

The reasoning-attention bridge (Eq. 4) is the central mechanism; it takes the MLLM's <EDIT> token projection as a query and the visual tokens as key/value pairs to compute spatially-aware editing weights, so linguistic reasoning is grounded in specific image regions. This sits inside a hierarchical reasoning module (Eq. 3) that iteratively refines reasoning states against pooled visual context, and an editing-conditioned discrete diffusion model that reconstructs masked image tokens conditioned on the edit signal. Hybrid alignment (Eq. 6) freezes the text encoder and trains a visual encoder with time-dependent contrastive and reconstruction losses.

What would settle it

Take a set of reasoning instructions where the correct edit is intentionally not what a strong inpainting model would produce (e.g., replace the sunburn-liable person with a green lizard), generate ground truths by human artists instead of SDXL, and re-run the comparison; if R-Genie no longer outperforms the baselines, the reported superiority is an artifact of the benchmark construction.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a single 1.3B model can perform complex, multi-step editing that requires world knowledge and intention inference by letting a frozen language model generate edit tokens and a trainable diffusion module synthesize the pixels. The key evidence is the quantitative comparison in Table 1: R-Genie scores 62.14% CLIP similarity, 2.01% L2 background loss, and 64.0 RISEBench, edging out larger unified and editing-specific models. The paper also reports ablations showing that its hierarchical reasoning module and reasoning-attention bridge stabilize cross-modal alignment, which it says diverged in 67% of trials without them.

Load-bearing premise

The load-bearing premise is that the REditBench ground-truth edited images—produced by SDXL inpainting and kept via CLIP-based semantic consistency—faithfully represent the intended reasoning-based edits; if they do not, R-Genie is trained and evaluated to imitate the inpainter rather than to reason.

Editorial extensions

If this is right

  • If correct, image editing can move from explicit instructions to abstract, intention-based requests, reducing the prompt-engineering burden on users.
  • The REditBench dataset provides a reusable testbed for measuring reasoning in editing, which could standardize evaluation for this task.
  • The 1.3B parameter efficiency suggests that reasoning-guided editing does not require 10B+ models, pointing toward more deployable systems.
  • The reasoning-attention bridge is a reusable component that could be applied to other text-to-image grounding tasks beyond editing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ground truth in REditBench is generated by SDXL inpainting and filtered by CLIP-based semantic consistency; if those targets are biased toward SDXL's preferences, the comparative results may partly reflect that bias rather than pure reasoning ability.
  • A natural testable extension would be to regenerate the benchmark targets with a different inpainting model and verify whether R-Genie's margin over baselines persists.
  • The paper's reasoning claim could be sharpened by ablating the <REASON> token and measuring the performance drop, clarifying how much of the gain comes from reasoning as opposed to visual grounding alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes R-Genie, a 1.3B-parameter model that combines the Phi-1.5 language model (via Show-o) with a discrete diffusion model for 'reasoning-guided image editing,' where a user gives an implicit, knowledge-dependent instruction (e.g., 'who is most likely to get sunburned?') and the model must identify the target and produce the edit. To support this, the authors construct REditBench, a dataset of 1,070 image-instruction-edit triples built from RefCOCO/RefCOCO+ with target images generated by SDXL inpainting and filtered by CLIP. R-Genie uses a reasoning-attention bridge and hierarchical reasoning module to condition the diffusion process on MLLM-derived <EDIT> tokens. On REditBench, R-Genie reports the best CLIP similarity (62.14), lowest L2 background loss (2.01), and highest RISEBench score (64.0) among ten methods, and a user study favoring R-Genie. The central claim is that this demonstrates superior reasoning-guided editing accuracy.

Significance. The task formulation and the architectural combination of an MLLM with discrete diffusion are original and potentially useful, and the construction of a dedicated benchmark addresses a real gap. The paper also includes ablations and a user study. If the headline numbers were obtained under a fair, non-circular protocol, R-Genie would be a meaningful advance. However, as reported, the evaluation cannot separate the method's contribution from in-distribution training on machine-generated targets; the paper's significance for the reasoning-editing claim is therefore not yet established. The strengths are the conceptual framing, the benchmark construction effort, and the breadth of the comparison; the weakness is the validity of the evidence.

major comments (4)
  1. [Table 1; Sections 3.2, 5.1] The claim of superior editing accuracy rests on a circular evaluation. R-Genie is trained for 100 epochs on the 850-sample REditBench training split, whereas all nine baselines are evaluated without any adaptation to REditBench. Because the REditBench target images are generated by SDXL-1.0 inpainting and then kept by CLIP-based semantic consistency filtering (Section 3.2), and because the headline metrics include CLIP Similarity and RISEBench Score, R-Genie's high scores may reflect in-distribution training against the same machine-generated target distribution that defines the evaluation signal. The paper provides no evaluation on an independent benchmark (e.g., the RISEBench dataset of [60]) with human-verified targets, nor any fine-tuned baseline, so the superiority claim is not supported. Please add a zero-shot evaluation on an external benchmark and, ideally, fine-tune the baselines on the same training split to isolate the architectural contribution.
  2. [Section 5.1; Table 1] The definition of RISEBench Score is missing. The text says 'Drawing upon [60], this composite is used to measure evaluates multi-dimensional aspects, including instruction adherence, appearance consistency, and generation plausibility,' but no formula, weighting, or reference to a specific evaluation protocol is given. Since this metric is one of the four headline numbers in Table 1 and is used in the superiority claim, the reader cannot interpret or reproduce it. Please specify the exact computation, the source of the target text, and whether the official code of [60] was used.
  3. [Section 5.4; Table 2] The ablation does not support the conclusion that the proposed modules contribute positively. The intermediate row (+ HRM and RAB) drops CLIP similarity from 53.29 to 42.99 and raises L2 background loss from 4.18 to 5.34, yet the full model (All) reaches 62.14 and 2.01. The text attributes the intermediate failure to 'naive visual-textual alignment through direct loss minimization' becoming unstable in 67% of trials, but no instability statistics or variance are reported, and the contribution of the 'Hybrid Opt' component alone is never isolated. In addition, this section refers to 'procedurally-generated synthetic training data,' which is not described in Section 3.2 and appears to contradict the dataset construction described there (RefCOCO-derived annotations with SDXL-inpainted targets). Please clarify the data generation process and provide a complete ablation that isolates each component.
  4. [Section 5.5; Table 3] The user study does not substantiate the claim of 'statistically significant preference rates.' The study compares R-Genie against only InstructPix2Pix and MGIE, not against the other seven baselines; no significance test, confidence interval, or inter-annotator agreement is reported; and the multi-select protocol makes the 'Chosen Frequency' aggregation ambiguous (e.g., frequencies sum to 236 across 220 items). Moreover, the study is performed on the same REditBench data on which R-Genie was trained, so it inherits the circularity of the main evaluation. Please report the full statistical analysis (e.g., bootstrap or paired test) and, ideally, include a human evaluation on an external set.
minor comments (5)
  1. [Sections 4.3 and 5.1] There is an inconsistency in the diffusion-noise coefficient: Eq. (6) says λ1 = 1 − αt and λ2 = αt with αt following the diffusion noise schedule, but Section 5.1 states 'αt is set to 0.5 as in [23,52].' Please reconcile these statements.
  2. [Section 5.2] The reported improvements of '1.8% and 10.7%' over the second-best methods (VILA-U and Janus) are numerically inconsistent with Table 1: CLIP similarity improves by 1.07 absolute (1.75% relative), and L2 BG loss by 0.18 absolute (8.2% relative). Please clarify how these percentages are computed.
  3. [Section 4.2] The phrase 'context preservation gates as in [52]' is not defined; please provide the relevant equation or a precise citation to the appendix where this mechanism is described.
  4. [Section 5.1] The description of AP as 'a data-driven measure of image naturalness' is inaccurate; the LAION Aesthetic Predictor scores aesthetic quality, not naturalness. Please rephrase.
  5. [Appendix] There are several typographical errors: 'Appendex' in the appendix heading, 'resymbles' in Figure A7, and 'lnstruction-followed' in reference [46]. Please correct them.

Circularity Check

1 steps flagged · score 6.0 of 10

R-Genie's headline accuracy numbers are measured on the same REditBench it was fine-tuned on, whose SDXL+CLIP-generated targets also define the CLIP-based evaluation signal.

  1. fitted input called prediction [Sec. 3.2 (Benchmark Dataset), Sec. 5.1 (Experimental Setting), Sec. 5.2 (Comparative Analysis), Table 1]
    "High-fidelity edited images are generated using state-of-the-art inpainting models, particularly Stable-Diffusion-XL-1.0 [34] ... Rigorous quality assurance is conducted through CLIP-based semantic consistency verification and human review ... For a fair result comparison, the training procedure consists of 100 epochs ... All experiments are conducted on our proposed REditBench under identical conditions."

    R-Genie is trained for 100 epochs on REditBench's 850-sample training split (Sec. 5.1), then its claimed superiority in Table 1 is computed on the same benchmark's 220-sample val split (Sec. 5.2). The benchmark's ground-truth edits are SDXL-1.0 inpainting outputs admitted by CLIP semantic-consistency filtering (Sec. 3.2), and the primary accuracy metric is also CLIP cosine similarity between edited regions and text. The 'reasoning-guided editing accuracy' being predicted therefore reduces, by construction, to how well the model reproduces the SDXL+CLIP target-generation pipeline that produced its training labels; the numbers do not separate correct reasoning from imitation of the automatic inpainter's output distribution.

full rationale

The central empirical claim (Table 1: best CLIP similarity 62.14%, lowest L2 BG loss 2.01%, highest RISEBench 64.0) is a benchmark-bound result, not a derived theorem. The circular component is the evaluation loop: the benchmark defines the task, the training targets, and the scoring signal through the same SDXL+CLIP pipeline, and R-Genie is the only method fine-tuned on that benchmark (100 epochs on the 850-sample training split), while the nine baselines are evaluated without comparable training. This makes the reported superiority partially self-referential: a model optimized to imitate SDXL-inpainted, CLIP-filtered targets will tend to score well on a CLIP-similarity metric of the same type, so the result does not independently establish world-knowledge reasoning. There is no equation-level tautology and no load-bearing self-citation chain; the confound is a benchmark-construction/evaluation loop, which is why the score is 6 rather than higher.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a new dataset (REditBench) built from RefCOCO and SDXL inpainting, and on learned components (HRM and RAB) whose individual contributions are not isolated in the ablation. The main free choices are the loss weighting and undisclosed curation thresholds. No genuinely new physical entities are introduced; the <REASON>/<EDIT> tokens are implementation-level constructs with no independent evidence.

free parameters (3)
  • Loss balance coefficients λ1 and λ2 = λ1 = 1 - α_t, λ2 = α_t, with α_t = 0.5
    Section 4.3 sets these weights to balance InfoNCE and reconstruction losses; no sensitivity analysis is reported, and the value follows prior work ([23,52]).
  • CLIP curation threshold for REditBench = Not reported
    Section 3.2 filters synthetic edits by CLIP-based semantic consistency and human review, but the threshold is undisclosed; it determines which triples enter the benchmark and shapes all results.
  • Training schedule = 100 epochs, learning rate 3e-4, weight decay 0
    Section 5.1 fixes these choices without ablation; they are part of the empirical setup and affect convergence and final scores.
assumptions (4)
  • domain assumption REditBench synthetic ground truth generated by SDXL inpainting is a valid representation of the intended reasoning edit.
    Section 3.2: edited images are produced by SDXL inpainting and CLIP-filtered. If these targets are wrong or biased, both training and evaluation inherit the error.
  • domain assumption Pretrained Show-o and Phi-1.5 provide sufficient base understanding and generation ability for the new task.
    Section 4.1 and 5.1 adopt these backbones without validating that their visual and textual capabilities are adequate; all measured gains are relative to this base.
  • domain assumption CLIP similarity, L2 background loss, LAION aesthetic predictor, and RISEBench score measure edit correctness.
    Section 5.1 defines these as principal metrics; no evidence is given that they track human judgment of reasoning accuracy beyond the small user study.
  • domain assumption The token prediction objective in Eq. 5 is sufficient for faithful editing conditioned on the <EDIT> token.
    Section 4.2 states the reconstruction loss but does not prove the <EDIT> token carries enough information to specify the edit; this is assumed by the architecture.

how reviews work

0 comments
Cite this review

Pith. "Pith review of R-Genie: Reasoning-Guided Generative Image Editing." pith.science (2026). https://pith.science/paper/RT3YGQU4

@misc{pith2026250517768,
  author       = {Pith},
  title        = {Pith review of: R-Genie: Reasoning-Guided Generative Image Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RT3YGQU4}},
  note         = {Machine review of arXiv:2505.17768}
}
read the original abstract

While recent advances in image editing have enabled impressive visual synthesis capabilities, current methods remain constrained by explicit textual instructions and limited editing operations, lacking deep comprehension of implicit user intentions and contextual reasoning. In this work, we introduce a new image editing paradigm: reasoning-guided generative editing, which synthesizes images based on complex, multi-faceted textual queries accepting world knowledge and intention inference. To facilitate this task, we first construct a comprehensive dataset featuring over 1,000 image-instruction-edit triples that incorporate rich reasoning contexts and real-world knowledge. We then propose R-Genie: a reasoning-guided generative image editor, which synergizes the generation power of diffusion models with advanced reasoning capabilities of multimodal large language models. R-Genie incorporates a reasoning-attention mechanism to bridge linguistic understanding with visual synthesis, enabling it to handle intricate editing requests involving abstract user intentions and contextual reasoning relations. Extensive experimental results validate that R-Genie can equip diffusion models with advanced reasoning-based editing capabilities, unlocking new potentials for intelligent image synthesis.

Figures

Figures reproduced from arXiv: 2505.17768 by the authors.

Figure 1
Figure 1. By integrating multimodal large language models, we endow generative image editing [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Two examples of the annotated pixel-accurate image-instruction-edit triples. Left: atomic [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The pipeline of R-Genie. R-Genie employs a MLLM to process an introduced <REASON> token alongside textual and visual input tokens. The MLLM-generated <EDIT> token is subsequently routed through a reasoning-attention bridge and a hierarchical reasoning module, which perform bidirectional reasoning by integrating visual features through cross-modal interactions. Finally, a discrete diffusion model reconstructs the tar… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative result comparisons with other instruction-based image editing methods. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 18 canonical work pages

  1. [7]

    Got: Unleashing reasoning capability of multimodal large language model for visual generation and editing.arXiv preprint arXiv:2503.10639, 2025

    Rongyao Fang, Chengqi Duan, Kun Wang, Linjiang Huang, Hao Li, Shilin Yan, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, et al. Got: Unleashing reasoning capability of multimodal large language model for visual generation and editing.arXiv preprint arXiv:2503.10639, 2025

    arXiv 2025

  2. [60]

    Envisioning beyond the pixels: Benchmarking reasoning-informed visual editing.arXiv preprint arXiv:2504.02826, 2025

    Xiangyu Zhao, Peiyuan Zhang, Kexian Tang, Hao Li, Zicheng Zhang, Guangtao Zhai, Junchi Yan, Hua Yang, Xue Yang, and Haodong Duan. Envisioning beyond the pixels: Benchmarking reasoning-informed visual editing.arXiv preprint arXiv:2504.02826, 2025. 13

    arXiv 2025

  3. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [2]

    Instructpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18392–18402, 2023

    2023

  5. [3]

    Personalizing multimodal large language models for image captioning: an experimental analysis

    Davide Bucciarelli, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Personalizing multimodal large language models for image captioning: an experimental analysis. arXiv preprint arXiv:2412.03665, 2024

    arXiv 2024

  6. [4]

    The revolution of multimodal large language models: a survey.arXiv preprint arXiv:2402.12451, 2024

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The revolution of multimodal large language models: a survey.arXiv preprint arXiv:2402.12451, 2024

    arXiv 2024

  7. [5]

    A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt.arXiv preprint arXiv:2303.04226, 2023

    Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S Yu, and Lichao Sun. A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt.arXiv preprint arXiv:2303.04226, 2023

    arXiv 2023

  8. [6]

    Diffusion models in vision: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10850–10869, 2023

    Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10850–10869, 2023

    2023

Show all 63 references
  1. [8]

    Guid- ing instruction-based image editing via multimodal large language models.arXiv preprint arXiv:2309.17102, 2023

    Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guid- ing instruction-based image editing via multimodal large language models.arXiv preprint arXiv:2309.17102, 2023

    2023 arXiv

  2. [9]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. InEuropean Conference on Computer Vision, pages 148–166. Springer, 2024

    2024

  3. [10]

    Exploiting clip self-consistency to automate image augmentation for safety critical scenarios

    Sujan Sai Gannamaneni, Frederic Klein, Michael Mock, and Maram Akila. Exploiting clip self-consistency to automate image augmentation for safety critical scenarios. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3594–3604, 2024

    2024

  4. [11]

    Image style transfer using convolutional neural networks

    Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016

    2016

  5. [12]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024. 10

    2024 arXiv

  6. [13]

    Instructdiffusion: A generalist modeling interface for vision tasks

    Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Houqiang Li, Han Hu, et al. Instructdiffusion: A generalist modeling interface for vision tasks. InProceedings of the IEEE/CVF Conference on computer vision and pattern recogniti...

    2024

  7. [14]

    Artificial general intelligence: concept, state of the art, and future prospects

    Ben Goertzel. Artificial general intelligence: concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5(1):1, 2014

    2014

  8. [15]

    Diffusion models in low-level vision: A survey

    Chunming He, Yuqi Shen, Chengyu Fang, Fengyang Xiao, Longxiang Tang, Yulun Zhang, Wangmeng Zuo, Zhenhua Guo, and Xiu Li. Diffusion models in low-level vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  9. [16]

    Prompt-to-prompt image editing with cross attention control.arXiv preprint arXiv:2208.01626, 2022

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control.arXiv preprint arXiv:2208.01626, 2022

    2022 arXiv

  10. [17]

    Smartedit: Exploring complex instruction- based image editing with multimodal large language models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction- based image editing with multimodal large language models. InProceedings of the IEEE/CVF Conference...

    2024

  11. [18]

    Image-to-image translation with conditional adversarial networks

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017

    2017

  12. [19]

    Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding.arXiv preprint arXiv:2504.04423, 2025

    Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding.arXiv preprint arXiv:2504.04423, 2025

    2025 arXiv

  13. [20]

    A style-based generator architecture for generative adversarial networks, 2019

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019

    2019

  14. [21]

    Imagic: Text-based real image editing with diffusion models

    Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6007–6017, 2023

    2023

  15. [22]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014

    2014

  16. [23]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024

    2024

  17. [24]

    Visual question answering instruction: Unlocking multimodal large language model to domain-specific visual multitasks

    Jusung Lee, Sungguk Cha, Younghyun Lee, and Cheoljong Yang. Visual question answering instruction: Unlocking multimodal large language model to domain-specific visual multitasks. arXiv preprint arXiv:2402.08360, 2024

    2024 arXiv

  18. [25]

    Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814, 2024

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814, 2024

    2024 arXiv

  19. [26]

    Textbooks are all you need ii: phi-1.5 technical report.arXiv preprint arXiv:2309.05463, 2023

    Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report.arXiv preprint arXiv:2309.05463, 2023

    2023 arXiv

  20. [27]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  21. [28]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean Conference on Computer Vision, pages 38–55. Springer,...

    2024

  22. [29]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  23. [30]

    Adapedit: Spatio-temporal guided adaptive edit- ing algorithm for text-based continuity-sensitive image editing

    Zhiyuan Ma, Guoli Jia, and Bowen Zhou. Adapedit: Spatio-temporal guided adaptive edit- ing algorithm for text-based continuity-sensitive image editing. InProceedings of the AAAI Conference on Artificial Intelligence, pages 4154–4161, 2024

    2024

  24. [31]

    Hd-painter: High-resolution and prompt-faithful text-guided image in- painting with diffusion models

    Hayk Manukyan, Andranik Sargsyan, Barsegh Atanyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Hd-painter: High-resolution and prompt-faithful text-guided image in- painting with diffusion models. InThe Thirteenth International Conference on Learning Representations, 2023

    2023

  25. [32]

    Toward verifiable and reproducible human evaluation for text- to-image generation

    Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin’ichi Satoh. Toward verifiable and reproducible human evaluation for text- to-image generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2023

  26. [33]

    State of the art on diffusion models for visual computing

    Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T Barron, Amit Bermano, Eric Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al. State of the art on diffusion models for visual computing. InComputer Graphics Forum, page e15063. Wiley Online Library, 2024

    2024

  27. [34]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  28. [35]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  29. [36]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  30. [37]

    Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information processing systems, 35:36479–36494, 2022

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information ...

    2022

  31. [38]

    Laion- 5b: An open large-scale dataset for training next generation image-text models.Advances in neural information processing systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models.Advances in neural informa...

    2022

  32. [39]

    Imagdressing-v1: Customizable virtual dressing

    Fei Shen, Xin Jiang, Xin He, Hu Ye, Cong Wang, Xiaoyu Du, Zechao Li, and Jinhui Tang. Imagdressing-v1: Customizable virtual dressing. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 6795–6804, 2025

    2025

  33. [40]

    Imagpose: A unified conditional framework for pose-guided person generation.Advances in neural information processing systems, 37:6246–6266, 2024

    Fei Shen and Jinhui Tang. Imagpose: A unified conditional framework for pose-guided person generation.Advances in neural information processing systems, 37:6246–6266, 2024

    2024

  34. [41]

    Imaggarment-1: Fine- grained garment generation for controllable fashion design.arXiv preprint arXiv:2504.13176, 2025

    Fei Shen, Jian Yu, Cong Wang, Xin Jiang, Xiaoyu Du, and Jinhui Tang. Imaggarment-1: Fine- grained garment generation for controllable fashion design.arXiv preprint arXiv:2504.13176, 2025

    2025 arXiv

  35. [42]

    Learning by planning: Language-guided global image editing

    Jing Shi, Ning Xu, Yihang Xu, Trung Bui, Franck Dernoncourt, and Chenliang Xu. Learning by planning: Language-guided global image editing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13590–13599, 2021

    2021

  36. [43]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024. 12

    2024 arXiv

  37. [44]

    Metamorph: Multimodal under- standing and generation via instruction tuning.arXiv preprint arXiv:2412.14164, 2024

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal under- standing and generation via instruction tuning.arXiv preprint arXiv:2412.14164, 2024

    2024 arXiv

  38. [45]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  39. [46]

    Gpt4video: A unified multimodal large language model for lnstruction-followed understanding and safety-aware generation

    Zhanyu Wang, Longyue Wang, Zhen Zhao, Minghao Wu, Chenyang Lyu, Huayang Li, Deng Cai, Luping Zhou, Shuming Shi, and Zhaopeng Tu. Gpt4video: A unified multimodal large language model for lnstruction-followed understanding and safety-aware generation. InProceedings of the 32nd A...

    2024

  40. [47]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    arXiv

  41. [48]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. InForty-first International Conference on Machine Learning, 2024

    2024

  42. [49]

    Multimodal large language models make text-to-image generative models align better.Advances in Neural Information Processing Systems, 37:81287–81323, 2024

    Xun Wu, Shaohan Huang, Guolong Wang, Jing Xiong, and Furu Wei. Multimodal large language models make text-to-image generative models align better.Advances in Neural Information Processing Systems, 37:81287–81323, 2024

    2024

  43. [50]

    Vila-u: a unified foundation model integrating visual understanding and generation.arXiv preprint arXiv:2409.04429, 2024

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation.arXiv preprint arXiv:2409.04429, 2024

    2024 arXiv

  44. [51]

    Omnigen: Unified image generation, 2024

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation, 2024

    2024

  45. [52]

    Show-o: One single trans- former to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  46. [53]

    Smartbrush: Text and shape guided object inpainting with diffusion model

    Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22428–22437, 2023

    2023

  47. [54]

    A survey on video diffusion models.ACM Computing Surveys, 57(2):1–42, 2024

    Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video diffusion models.ACM Computing Surveys, 57(2):1–42, 2024

    2024

  48. [55]

    Progressive instance-aware feature learning for compositional action recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):10317–10330, 2023

    Rui Yan, Lingxi Xie, Xiangbo Shu, Liyan Zhang, and Jinhui Tang. Progressive instance-aware feature learning for compositional action recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):10317–10330, 2023

    2023

  49. [56]

    Higcin: Hierarchical graph-based cross inference network for group activity recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):6955–6968, 2020

    Rui Yan, Lingxi Xie, Jinhui Tang, Xiangbo Shu, and Qi Tian. Higcin: Hierarchical graph-based cross inference network for group activity recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):6955–6968, 2020

    2020

  50. [57]

    Mmginpainting: Multi-modality guided image inpainting based on diffusion models.IEEE Transactions on Multimedia, 2024

    Cong Zhang, Wenxia Yang, Xin Li, and Huan Han. Mmginpainting: Multi-modality guided image inpainting based on diffusion models.IEEE Transactions on Multimedia, 2024

    2024

  51. [58]

    Mm- llms: Recent advances in multimodal large language models.arXiv preprint arXiv:2401.13601, 2024

    Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm- llms: Recent advances in multimodal large language models.arXiv preprint arXiv:2401.13601, 2024

    2024 arXiv

  52. [59]

    Magicbrush: A manually annotated dataset for instruction-guided image editing.Advances in Neural Information Processing Systems, 36:31428–31449, 2023

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing.Advances in Neural Information Processing Systems, 36:31428–31449, 2023

    2023

  53. [61]

    Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    2024 arXiv

  54. [62]

    Unpaired image-to-image translation using cycle-consistent adversarial networks

    Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. InProceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017. 14 Appendex To enhance the reproducibili...

    2017

  55. [63]

    produces outputs with marginal modifications relative to source images, revealing constrained multimodal reasoning and adaptive generation capacities. In contrast, while OmniGen [51] demon- strates partial success in certain cases, its performance remains inconsistent - notabl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.