Pith. sign in

REVIEW 5 major objections 8 minor 44 references

InsightEdit: Towards Better Instruction Following for Image Editing

T0 review · 5 major / 8 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that instruction-based image editing improves when the diffusion model conditions on MLLM-reasoned visual features, not just text, and that a new 2.5-million-pair dataset trains this ability.

desk verdict Real engineering, a sizable new dataset, and an external-benchmark win, but the SOTA claim leans on a VIEScore judge that the authors also used to filter training data. read the letter →

arxiv 2411.17323 v1 pith:WRQC4ZH2 submitted 2024-11-26 cs.CV

Yingjing Xu , Jie Kong , Jiazhi Wang , Xiao Pan , Bo Lin , Qiang Liu This is my paper · ORCID
classification cs.CV
keywords instruction-basedimageeditingdiffusionmodelsmultimodallargelanguagevisualconditioningdatasetbackgroundconsistencyVIEScoreinstructionfollowing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that instruction-based image editing fails today for two fixable reasons: the training datasets are too simple and low-quality, and the models condition on the text instruction while ignoring the visual content of the image. To fix the data side, it builds AdvancedEdit, an automated pipeline producing 2.5 million editing pairs at high resolution by masking objects, generating targets with mask-based editors, recaptioning instructions into simple and complex forms, and filtering with a multimodal-LLM quality score. To fix the conditioning side, it adds a visual stream that carries image features reasoned by a multimodal large language model into the diffusion denoiser, alongside the usual text stream. The resulting model, InsightEdit, is reported to outperform InstructPix2Pix, MagicBrush, InstructDiffusion, MGIE, and SmartEdit on the paper's AdvancedEdit-Eval and on Reason-Edit, with the largest margins on complex instructions and background preservation.

What carries the argument

The load-bearing object is the two-stream bridging mechanism. The text branch takes the hidden states of the [MM] tokens that the MLLM is trained to predict, refines them with a Q-Former and a bidirectional information module (BIM), and feeds the result as the text condition. The image branch's Image Alignment Adapter (IAA) maps those same hidden states through an MLP mapper into a CLIP embedding, trained against the CLIP features of the target image, so the condition carries what the edited result should look like. A decoupled cross-attention layer, borrowed from image-prompt adapters, lets the UNet attend to the image stream with an adjustable weight $\lambda$. The supporting data pipeline uses object extraction, mask generation, mask-based editing, instruction recaptioning, and VIEScore filtering to produce the AdvancedEdit pairs.

What would settle it

Run a forced-choice human study on the 300 AdvancedEdit-Eval image pairs: if people do not prefer InsightEdit's edits over SmartEdit's at roughly the rate the VIEScore gap (0.831 vs 0.682) predicts, the claimed state-of-the-art edge is an artifact of the shared metric. A second check would rebuild the dataset with a different human-aligned filter and see whether the advantage survives.

Watch

Extended reading notes

Core claim

The central claim is that a mask-free editing model can follow complex instructions and keep unedited regions intact only when the diffusion process receives both the text instruction and the visual semantics extracted from the source image, not text alone. InsightEdit implements this with a two-stream bridging module: the text stream processes the hidden states of the [MM] tokens produced by a multimodal large language model through a Q-Former and a bidirectional information module, while the image stream's Image Alignment Adapter (IAA) maps the same hidden states into a CLIP embedding that is supervised by the target image's CLIP features. A decoupled cross-attention layer in the UNet fuses the two streams at a tunable weight. Trained on the AdvancedEdit dataset, the paper reports a VIEScore of 0.831 on AdvancedEdit-Eval and 0.947 on Reason-Edit reasoning scenarios, against SmartEdit's 0.682 and 0.835, with better PSNR, SSIM, LPIPS, and CLIPScore.

Load-bearing premise

The evaluation assumes the automated VIEScore metric matches human judgment closely enough that selecting training pairs with it and then measuring success with it is a fair test.

Editorial extensions

If this is right

  • Because the image condition enters through a decoupled cross-attention layer with a tunable weight $\lambda$, the same bridging design can be grafted onto other diffusion backbones without retraining the whole UNet.
  • The AdvancedEdit pipeline's remove/add/replace pairs give other mask-free editing models a ready-made training resource for complex instructions and background consistency.
  • Training on advanced recaptioned instructions raises both instruction-following scores and background-consistency metrics, indicating that data complexity and visual conditioning reinforce each other.
  • The IAA loss ties a hidden-state embedding directly to the target image's CLIP embedding, which is why the edited object matches the intended concept while the unedited scene stays close to the source.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the text branch unablated: zeroing or masking the text-condition stream at inference would isolate how much of the gain comes from the visual stream alone, which is a direct test the current ablations do not run.
  • Because VIEScore both filters the training data and reports the headline gains, the numerical advantage could partly reflect the model matching the metric's own preferences; a human preference study on the same 300 pairs would separate the two effects.
  • The data pipeline's steps (caption, extract objects, generate masks, edit, recaption, filter) are task-agnostic, so the same recipe could be carried to video editing or object-level video manipulation if a mask-based video editor is available; the paper does not explore this.
  • The paper itself lists its remaining limits as the instruction understanding of the MLLM and the power of the diffusion backbone, so the ceiling of the approach is tied to progress in those two components.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper proposes InsightEdit, an instruction-based image editing method that introduces a two-stream bridging mechanism to inject both textual and visual features from an MLLM into a diffusion model, along with a new Image Alignment Adapter (IAA) for target-image supervision. The authors also present an automated data construction pipeline that generates the AdvancedEdit dataset (2.5M editing pairs) and a 300-pair AdvancedEdit-Eval benchmark. Experiments report state-of-the-art VIEScore results on AdvancedEdit-Eval and Reason-Edit, with ablations showing gains from IAA and from the AdvancedEdit data.

Significance. If validated, the AdvancedEdit dataset could be a valuable large-scale resource for instruction-based image editing, and the IAA mechanism is a plausible way to inject target-image information explicitly. The two-stream bridging architecture is incremental but reasonable. The main significance, however, is undermined by evaluation issues: the primary metric (VIEScore) is also used as a data filter, the self-created benchmark is from the same pipeline, and the headline comparisons confound the method contribution with additional training data. The paper would be substantially strengthened by a human study, controlled comparisons, and significance testing.

major comments (5)
  1. [Section 3.1 Step 5, Section 5.2] The paper uses VIEScore both to filter the AdvancedEdit training pairs (Section 3.1 Step 5) and as the primary semantic metric on the self-created AdvancedEdit-Eval (Section 5.2). This creates a selection bias: the model is trained to imitate targets that were selected for high VIEScore, and the same judge is then used for evaluation. Since the baselines were not trained on data filtered this way, the reported VIEScore gap (e.g., 0.831 vs. 0.682 for SmartEdit in Table 2) may reflect this bias rather than a genuine editing-quality advantage. The concern extends to the Reason-Edit results, where VIEScore is also the judge. Please add a human preference study or an independent judge (e.g., a different MLLM) and report the correlation with VIEScore.
  2. [Section 5.2, Tables 2 and 3] The 'InsightEdit with AdvancedEdit' rows have access to 202,822 additional training pairs from a dataset constructed with the same pipeline as the evaluation set, while the baselines do not. This confounds the method contribution with the data contribution. For example, in Table 3, 'InsightEdit with AdvancedEdit' outperforms SmartEdit by a large margin, but it also uses substantially more training data. To support the architectural claim, the authors should present an architecture-only comparison where InsightEdit and SmartEdit are trained on identical data (the 'InsightEdit' row in Table 3 is one such comparison, but the paper still highlights the AdvancedEdit row as SOTA). The data contribution should be framed separately, not as evidence for the architecture.
  3. [Section 5.2, Tables 2-5] All quantitative results are reported as single runs without error bars or significance tests. VIEScore is an MLLM-based, likely stochastic judge; differences such as 0.831 vs. 0.738 (Table 2) or 7.545 vs. 7.307 (Table 4) could be within run-to-run variation. The authors should provide multiple runs or bootstrap confidence intervals for the primary comparisons, and ideally a paired significance test for the head-to-head comparisons.
  4. [Tables 2, 4, and 5] The VIEScore scale is inconsistent across tables: Table 2 reports values around 0.3–0.8, while Tables 4 and 5 report values around 7.0–7.5 for the same configuration (e.g., InsightEdit with AdvancedEdit has identical PSNR/SSIM/LPIPS/CLIPScore in Table 2 and Table 4, but VIEScore of 0.831 in Table 2 and 7.545 in Table 4). This inconsistency is unexplained and makes it impossible to compare results across tables. The authors should clarify the normalization procedure and ensure a uniform scale throughout the paper.
  5. [Section 5.1, Section 5.2] The paper states that VIEScore 'might better align with human preferences' but provides no human evaluation. Given that the main metric is an MLLM and is also used as a data filter, human validation is essential to support the central claim. A human preference study on a subset of AdvancedEdit-Eval and Reason-Edit (e.g., 100 examples each, comparing InsightEdit vs. SmartEdit) should be added.
minor comments (8)
  1. [Section 5.2] Typo: 'qualititive' should be 'qualitative'.
  2. [Figure 4] The label 'InstructPixPix' should be 'InstructPix2Pix'.
  3. [Figure 2] Some extracted text in the figure has spacing artifacts (e.g., 'v ersion', 'ob j ect'); please check the original figures for clean rendering.
  4. [Table 1] The check and cross symbols are not defined in the caption; please clarify their meaning.
  5. [Section 3.1 Step 4] The prompt used for instruction recaptioning is not provided; including the exact prompt would improve reproducibility.
  6. [Section 4.2 and 4.3] The hyperparameters r (number of MM tokens) and lambda in Eq. (6) are not specified; please report their values in the implementation details.
  7. [Figure 3] The acronyms IAA, BIM, Q-Former, and MM are used in the figure but the caption does not expand them; please add a legend or describe them in the caption.
  8. [General] The paper references an appendix multiple times (e.g., 'detailed in the Appendix'), but the version under review contains no appendix; please ensure the appendix is included in the final submission.

Circularity Check

1 steps flagged · score 4.0 of 10

SOTA claim rests on VIEScore, which is both the filter used to build AdvancedEdit training pairs and the primary judge on AdvancedEdit-Eval; the reported advantage is partly an artifact of evaluating with the same scorer used for data selection.

  1. fitted input called prediction [Section 3.1 Step 5; Section 5.1 Metrics; Section 5.2 Comparison with State-of-the-art]
    "Step 5: Quality Evaluation. To construct a high-quality dataset, we assess and filter the image-editing pairs using VIEScore [20], which aligns evaluations with human preferences via MLLM. ... To more accurately evaluate editing effects and align with human preferences, we employ VIEScore [20], which utilizes MLLM to evaluate editing performance from two perspectives: instruction-following and background consistency."

    The same MLLM-based scoring function VIEScore is used twice: in Step 5 to filter which training pairs are kept in AdvancedEdit, and in Section 5.1-5.2 as the primary semantic metric on the paper's own AdvancedEdit-Eval benchmark. The model is trained on 202,822 AdvancedEdit pairs that survived VIEScore filtering, so its outputs are optimized, by imitation, to match targets that VIEScore rated highly. The AdvancedEdit-Eval set is introduced in the same dataset section as 300 curated removal/addition/replacement pairs, with no described human annotation, external source, or explicit disjoint split from training.

full rationale

The main claimed result—state-of-the-art VIEScore on AdvancedEdit-Eval—is partially circular because the same MLLM judge is used both to filter the training data (Section 3.1 Step 5) and to score the primary evaluation metric (Section 5.1-5.2). The AdvancedEdit-Eval set is presented as 300 curated pairs from the same removal/addition/replacement pipeline, with no external or human-validated construction described, so the model is effectively trained to imitate VIEScore-selected targets and then evaluated by VIEScore on a self-created benchmark. This selection bias can inflate the reported advantage over baselines. However, the circularity is only partial: Reason-Edit is an external benchmark, the low-level metrics are not the filter, and the architecture contributions (IAA, two-stream bridging) are assessed independently of the metric-selection loop. No load-bearing self-citation chain appears, and the IAA loss is supervised from actual target image features rather than from the evaluation metric. Score 4 reflects a real but partial circularity in the central claim, not a fully forced derivation.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The free parameters are hand-set hyperparameters and unspecified thresholds in the data pipeline that affect the central results. The axioms are domain assumptions about the reliability of external tools and the validity of the evaluation metric.

free parameters (5)
  • Image condition weight lambda in decoupled cross-attention (Eq. 6)
    Controls the influence of image features versus text features during inference. The paper says it can be adjusted but does not report the value used for the results, so it is a hand-set hyperparameter that affects outputs.
  • Number of IAA output tokens N = 4
    Set to 4 by default per Section 4.2; chosen by hand and affects the granularity of image conditioning.
  • Number of MM tokens r
    Added to the LLaVA vocabulary; the paper does not specify the count, which affects how much multi-modal information is carried.
  • GroundedSAM mask confidence threshold
    Used in Step 2 of the pipeline to filter masks; 'predefined threshold' is not specified, affecting which objects are edited.
  • VIEScore filtering thresholds
    Step 5 filters editing pairs based on VIEScore ratings; the exact acceptance criteria are not given, affecting dataset quality and composition.
assumptions (4)
  • domain assumption VIEScore accurately reflects human preferences for instruction following and background consistency.
    The paper relies on VIEScore for both dataset filtering and evaluation (Sections 3.1 and 5.2), citing [20]. If this alignment is weak, the central SOTA claim is undermined.
  • domain assumption Mask-based editing models (BrushNet, Power-Paint) generate high-quality target images without artifacts that mislead training.
    The dataset pipeline uses these models to create target images (Step 3). If they introduce failures invisible to VIEScore, the training pairs may be noisy.
  • domain assumption CLIP target image embeddings provide a suitable supervision signal for aligning MLLM hidden states with editing goals.
    The IAA loss minimizes distance between Mapper(h) and CLIP(Itar) (Eq. 5). This presumes CLIP embeddings capture edit-relevant visual semantics.
  • domain assumption GPT-4o and GroundedSAM produce accurate captions, object lists, and masks at scale.
    The pipeline depends on these tools for object extraction and mask generation (Steps 1-2). Errors would propagate to instructions and edit targets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InsightEdit: Towards Better Instruction Following for Image Editing." pith.science (2026). https://pith.science/paper/WRQC4ZH2

@misc{pith2026241117323,
  author       = {Pith},
  title        = {Pith review of: InsightEdit: Towards Better Instruction Following for Image Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WRQC4ZH2}},
  note         = {Machine review of arXiv:2411.17323}
}
read the original abstract

In this paper, we focus on the task of instruction-based image editing. Previous works like InstructPix2Pix, InstructDiffusion, and SmartEdit have explored end-to-end editing. However, two limitations still remain: First, existing datasets suffer from low resolution, poor background consistency, and overly simplistic instructions. Second, current approaches mainly condition on the text while the rich image information is underexplored, therefore inferior in complex instruction following and maintaining background consistency. Targeting these issues, we first curated the AdvancedEdit dataset using a novel data construction pipeline, formulating a large-scale dataset with high visual quality, complex instructions, and good background consistency. Then, to further inject the rich image information, we introduce a two-stream bridging mechanism utilizing both the textual and visual features reasoned by the powerful Multimodal Large Language Models (MLLM) to guide the image editing process more precisely. Extensive results demonstrate that our approach, InsightEdit, achieves state-of-the-art performance, excelling in complex instruction following and maintaining high background consistency with the original image.

Figures

Figures reproduced from arXiv: 2411.17323 by the authors.

Figure 1
Figure 1. We propose InsightEdit, an end-to-end instruction-based image editing model, trained on high-quality data and designed to fully harness the capabilities of Multimodal Large Language Models (MLLM), achieving high-quality edits with strong instruction-following and background consistency. Dataset Editing Pairs Resolution Good Background Consistency Complex Instructions High Visual Quality Automated Pipeline InstructPi… view at source ↗
Figure 2
Figure 2. The overall data construction pipeline. (1) Captioning & Object Extraction: Utilizing VLM to generate a global caption from the source image, and further get an object JSON list contains both simple caption and detailed caption. (2) Mask Generation: Utilizing GroundedSAM to obtain the corresponding mask of each object. (3) Editing Pair Construction: Utilizing mask-based image editing model to construct target image … view at source ↗
Figure 3
Figure 3. The overall architecture of InsightEdit. It mainly consists of three parts: (1) Comprehension Module: A comprehension module that leverages MLLM to perceive and comprehend the image editing task; (2) Bridging Module: A bridging module that better interacts and extracts both the textual and image features; (3) Generation Module: A generation module that receives editing guidance via diffusion model to generate the ta… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison on AdvancedEdit. InsightEdit shows superior instruction following and background consistency capability. humans over other methods, whether regarding background preservation or the ability to follow complex instructions [PITH_FULL_IMAGE:figures/…
Figure 6
Figure 6. Figure 6: Demonstration of the effectiveness of AdvancedEdit. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 9 canonical work pages

  1. [1]

    Blended latent diffusion

    Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM transactions on graphics (TOG) , 42 (4):1–11, 2023. 2

    work page 2023

  2. [2]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 1, 2, 3, 6

    work page 2023

  3. [3]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 1209–1218, 2018. 6

    2018

  4. [4]

    Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xi- aohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 22560–22570, 2023. 1

    2023

  5. [5]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 5

    2020

  6. [6]

    Learning to follow object-centric image editing instructions faithfully

    Tuhin Chakrabarty, Kanishk Singh, Arkadiy Saakyan, and Smaranda Muresan. Learning to follow object-centric image editing instructions faithfully. arXiv preprint arXiv:2310.19145, 2023. 2, 3

    arXiv 2023

  7. [7]

    Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021. 6

    2021

  8. [8]

    Guiding instruction-based im- age editing via multimodal large language models

    Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guiding instruction-based im- age editing via multimodal large language models. arXiv preprint arXiv:2309.17102, 2023. 1

    arXiv 2023

Show all 44 references
  1. [9]

    Instructdiffusion: A generalist modeling inter- face for vision tasks

    Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Houqiang Li, Han Hu, et al. Instructdiffusion: A generalist modeling inter- face for vision tasks. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Reco...

    2024

  2. [10]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 1, 3

    2022 arXiv

  3. [11]

    Image quality metrics: Psnr vs

    Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366–2369. IEEE, 2010. 6

    2010

  4. [12]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 5

    2021 arXiv

  5. [13]

    Diffusion model-based image editing: A survey

    Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Shifeng Chen, and Liangliang Cao. Diffusion model-based image editing: A survey. arXiv preprint arXiv:2402.17525, 2024. 1

    2024 arXiv

  6. [14]

    Smartedit: Exploring complex instruction-based image editing with multimodal large lan- guage models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction-based image editing with multimodal large lan- guage models. In Proceedings of the IEEE/CVF Conferen...

    2024

  7. [15]

    Hq-edit: A high-quality dataset for instruction-based image editing

    Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and Cihang Xie. Hq-edit: A high-quality dataset for instruction-based image editing. arXiv preprint arXiv:2404.09990, 2024. 2

    2024 arXiv

  8. [16]

    Brushnet: A plug-and-play image inpaint- ing model with decomposed dual-branch diffusion

    Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: A plug-and-play image inpaint- ing model with decomposed dual-branch diffusion. arXiv preprint arXiv:2403.06976, 2024. 1, 3

    2024 arXiv

  9. [17]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In Proceedings of the 2014 con- ference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014. 6

    2014

  10. [18]

    Adam: A method for stochastic opti- mization

    Diederik P Kingma. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980, 2014. 6

    2014 arXiv

  11. [19]

    Gen- erating images with multimodal language models

    Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Gen- erating images with multimodal language models. Advances in Neural Information Processing Systems, 36, 2024. 3, 5

    2024

  12. [20]

    Viescore: Towards explainable metrics for conditional image synthesis evaluation

    Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. arXiv preprint arXiv:2312.14867, 2023. 4, 6

    2023 arXiv

  13. [21]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.00692,

    arXiv

  14. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 5

    2023

  15. [23]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  16. [24]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 5

    2024

  17. [25]

    Language models are few-shot learners

    Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 1, 2020. 3 9

    2005 arXiv

  18. [26]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  19. [27]

    Grounded sam: Assembling open-world models for diverse visual tasks,

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kun- chang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks,

  20. [28]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2

    2022

  21. [29]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  22. [30]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 5

    2023 arXiv

  23. [31]

    Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting

    Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont- Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting. In Proceedings of the IEEE/CVF c...

    2023

  24. [32]

    Smartbrush: Text and shape guided object inpainting with diffusion model

    Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22428–22437, 2023

    2023

  25. [33]

    Dreaminpainter: Text-guided subject-driven image inpaint- ing with diffusion models

    Shaoan Xie, Yang Zhao, Zhisheng Xiao, Kelvin CK Chan, Yandong Li, Yanwu Xu, Kun Zhang, and Tingbo Hou. Dreaminpainter: Text-guided subject-driven image inpaint- ing with diffusion models. arXiv preprint arXiv:2312.03771,

    arXiv

  26. [34]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 6

    2024 arXiv

  27. [35]

    Editworld: Simulat- ing world dynamics for instruction-following image editing

    Ling Yang, Bohan Zeng, Jiaming Liu, Hong Li, Minghao Xu, Wentao Zhang, and Shuicheng Yan. Editworld: Simulat- ing world dynamics for instruction-following image editing. arXiv preprint arXiv:2405.14785, 2024. 2, 3

    2024 arXiv

  28. [36]

    An improved baseline for reasoning segmentation with large language model

    Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.17240, 2023. 6

    2023 arXiv

  29. [37]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  30. [38]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016. 6

    2016

  31. [39]

    Magicbrush: A manually annotated dataset for instruction- guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction- guided image editing. Advances in Neural Information Pro- cessing Systems, 36, 2024. 1, 2, 3, 6

    2024

  32. [40]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 3

    2023

  33. [41]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 6

    2018

  34. [42]

    Hive: Harnessing human feedback for instructional visual editing

    Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  35. [43]

    Ultraedit: Instruction-based fine-grained im- age editing at scale

    Haozhe Zhao, Xiaojian Ma, Liang Chen, Shuzheng Si, Ru- jie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-based fine-grained im- age editing at scale. arXiv preprint arXiv:2407.05282, 2024. 2, 3

    2024 arXiv

  36. [44]

    A task is worth one word: Learning with task prompts for high-quality versatile image inpainting

    Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. arXiv preprint arXiv:2312.03594, 2023. 3 10

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.