MultiRef: Controllable Image Generation with Multiple Visual References

Dongping Chen; Gaoyang Jiang; Petr Sushko; Ranjay Krishna; Ruoxi Chen; Shiyun Lang; Sinan Wang; Siyuan Wu; Yao Wan

REVIEW 4 major objections 7 minor 1 cited by

MultiRef: Controllable Image Generation with Multiple Visual References

T0 review · 4 major / 7 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current image generators fail to combine multiple visual references, a new 1,990-sample benchmark shows.

desk verdict Real multi-reference benchmark and dataset, but the judge choice and headline numbers contradict the paper's own tables. read the letter →

arxiv 2508.06905 v3 pith:3ZVIK2U6 submitted 2025-08-09 cs.CV

Ruoxi Chen , Dongping Chen , Siyuan Wu , Sinan Wang , Shiyun Lang , Petr Sushko , Gaoyang Jiang , Yao Wan

show 1 more author

Ranjay Krishna

This is my paper

classification cs.CV

keywords controllableimagegenerationmulti-referenceconditioningbenchmarkdatasetRefBlendeditingMLLM-as-a-judgeunifiedmodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that today's image generation systems cannot reliably follow multiple visual references at once—even though real creative work often blends several source images. It introduces MultiRef-bench, a benchmark of 990 synthetic and 1,000 real-world tasks, built with a new synthetic data engine called RefBlend and a 38k-image dataset of the same name. Across three unified models and six agentic frameworks, the best tested model, OmniGen, reaches only 66.6% on synthetic samples and 79.0% on real-world cases relative to the golden answer. The wider point is that multi-reference conditioning is a distinct, measurable capability that current single-source or loosely chained systems do not possess.

What carries the argument

The argument is carried by a two-part measurement apparatus. RefBlend is the synthetic data engine: it extracts ten reference types (depth, Canny edges, masks, bounding boxes, poses, sketches, semantic maps, captions, subjects, styles) from source images and combines them under a reference compatibility graph that forbids contradictory pairings, yielding 33 valid reference combinations and a filtered dataset of 38,076 samples. MultiRef-bench then pairs this synthetic test split with 1,000 human-filtered Reddit editing tasks, and evaluates outputs through rule-based metrics (IoU, MSE, mAP, CLIP scores) plus a fine-tuned MLLM-as-a-judge that the paper validates against human scores for instruc

What would settle it

Re-run human scoring on the benchmark's spatial-condition subsets—pose and bounding box—where the fine-tuned judge's Pearson correlation is only around 0.26 and 0.40; if human rankings of the tested models on those subsets differ from the GPT-4o-mini-based rankings, the claimed performance ordering would not hold on the conditions the benchmark most emphasizes.

Watch

Extended reading notes

Core claim

The central claim is that no currently available generator reliably integrates content from multiple visual references, and that this failure can be observed and quantified. On the synthetic half of MultiRef-bench, the best unified model, ACE, averages 0.553 reference fidelity against a ground-truth average of 0.709, while OmniGen averages 0.464; on the real-world half, OmniGen scores 0.496 instruction following and 0.421 source fidelity, far below the golden answer's 0.767 and 0.706. Compositional pipelines such as Claude + SD3.5 produce competitive image quality but degrade instruction adherence and source fidelity, suggesting that separating a perceiver from a generator fundamentally hurt

Load-bearing premise

The reported model rankings rest on trusting the MLLM judge's correlation with human judgment for instruction following and source fidelity, and on treating each Reddit output image as the single correct answer to its editing request.

Editorial extensions

If this is right

MultiRef-bench gives any claimed general-purpose generator a concrete test: 1,990 multi-reference tasks with per-condition metrics for spatial, semantic, and style control.
Unified end-to-end architectures are the more promising direction for multi-reference generation, since separated perceiver-plus-generator frameworks consistently lose instruction following and source fidelity.
Captions are load-bearing for current models: removing a caption degrades depth fidelity and aesthetics across models, while sometimes improving sketch and semantic-map fidelity.
Input reference order is not commutative: changing the order of reference images shifts condition-specific fidelity more than global image quality, so prompt or input design has real effects on multi-reference success.
Spatial constraints are the weakest reference types across nearly all models, making bounding-box and pose fidelity a natural first target for future multi-reference training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the judge-based ordering is sound, the 38k synthetic RefBlend samples become a ready training resource: a model fine-tuned on multi-reference pairings could close a large share of the reported gap, and MultiRef-bench's synthetic/real split would directly measure that gain.
The paper's own judge validation shows low human correlation on pose (Pearson 0.257) and bounding boxes (0.404) in its fine-tuned model; model rankings on those spatial conditions may be noisier than rankings on style or subject, and a human re-scoring of those subsets would be a cheap robustness test.
Because the real-world ground truth is a single selected Reddit output, tasks with multiple valid edits may underestimate model ability; a version of the benchmark that scores against several accepted outputs would be a natural extension.
The task taxonomy—replacement, addition, style, spatial manipulation, attribute transfer—could transfer to video or 3D editing, where multi-source conditioning is similarly untested.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know first. The empirical and data contributions are real: MultiRef-bench has 990 synthetic samples with 33 reference combinations and 1,000 real-world Reddit editing tasks, plus a 38k-image dataset built by the RefBlend compatibility-graph engine. The rule-based metrics for depth, mask, bbox, sketch, and semantic map are sensible, and the qualitative finding that current generators struggle to combine several visual references is likely correct. The paper earns credit for constructing a first-of-its-kind evaluation resource.

The soft spots are concentrated in the evaluation layer. Section 2.3 says GPT-4o-mini was chosen as the judge 'for its superior alignment with human judgment shown in Table 3,' and Appendix C.3 repeats that. But Table 3 and Table 10 show GPT-4o outperforming GPT-4o-mini on the two headline metrics, Instruction Following and Source Fidelity, on both realistic and synthetic splits (0.624 vs 0.530 and 0.668 vs 0.632 for IF). Since the central rankings in Tables 4 and 11 ride on IF/SF, this internal contradiction is not cosmetic. The judge-human correlations are also moderate (0.53–0.67) relative to human-human agreement (0.66–0.72), so small score gaps between models are not a solid basis for ordering. The abstract's 66.6% and 79.0% figures are not derived anywhere in the stated aggregation; they lack an audit trail.

The real-world part also assumes a single golden output per Reddit task, though the same request can have several valid results. That matters for absolute scores, though less for relative model comparisons on the same ground truth. One concern from the stress-test is overstated: the fine-tuned Qwen model is used to filter the synthetic dataset, not to evaluate models, so the claimed self-referential loop is not present. Dataset filtering could still introduce bias, but that is a different issue.

This is a paper a serious referee should see, with the explicit request to fix the judge justification and re-derive the headline numbers. The benchmark and dataset themselves are useful; I'd cite the resource while not quoting the percentages until the evaluation layer is cleaned up.

Referee Report

4 major / 7 minor

Summary. The paper introduces MultiRef, a benchmark and dataset for multi-reference image generation. It combines 990 synthetically generated samples produced by the RefBlend engine (10 reference types, 33 reference combinations) with 1,000 real-world tasks sampled from Reddit's r/PhotoshopRequest community. The authors evaluate three open-source unified models (OmniGen, ACE, Show-o) and six agentic frameworks (e.g., ChatDiT, LLM+SD) using rule-based metrics for spatial conditions and an MLLM-based judge for semantic-level assessments. They report that even the best existing system, OmniGen, performs far below the golden answer, with, e.g., 0.496 instruction-following on real-world tasks, and they release a 38k-image synthetic dataset.

Significance. The contribution is potentially valuable. A reproducible benchmark targeting multi-reference conditioning is missing, and the paper assembles a large dataset, a data engine, detailed reference definitions, and publicly available materials. The rule-based spatial metrics (IoU, MSE, mAP) are well matched to the conditions they measure. However, the quantitative headline results are not currently trustworthy: the choice of primary judge contradicts the paper's own validation tables, and the abstract's aggregate percentages are not derivable from the reported tables. The real-world part also assumes a single golden output for edits that often admit multiple valid solutions. If these issues are addressed, the benchmark could become a useful community resource.

major comments (4)

[§2.3, Table 3, Appendix C.3] The stated reason for choosing GPT-4o-mini as the primary judge is contradicted by the paper's own validation. Section 2.3 says GPT-4o-mini is selected for 'superior alignment with human judgment shown in Table 3', but Table 3 reports GPT-4o as having higher Pearson correlation than GPT-4o-mini on IF and SF (Realistic: 0.624 vs 0.530 for IF, 0.613 vs 0.514 for SF; Synthetic: 0.668 vs 0.632, 0.659 vs 0.616). Appendix C.3 repeats the claim, contrary to Table 10. Since the IF/SF scores in Tables 4 and 11 are produced by GPT-4o-mini, the reported model ordering and performance gaps are not numerically supported by the criterion the paper itself states. Re-evaluation with GPT-4o, or an explicit justification based on the dimensions where GPT-4o-mini actually wins, is needed before the central conclusions can be relied upon.
[Abstract and Section 2.3/Tables 4 and 11] The headline numbers 66.6% and 79.0% have no stated aggregation. OmniGen's scores in Table 4 (Synthetic: IQ 0.730, IF 0.532, SF 0.438) and Table 11 (Real-world: IQ 0.555, IF 0.496, SF 0.421) do not directly produce 0.666 or 0.790 under any formula described in Section 2.3. If these are ratios to the golden answer, the conversion is not specified. Please provide the exact computation or remove the percentages from the abstract and Section 1.
[§2.1, §C.1, Table 11] The real-world benchmark treats one selected Reddit output as the unique golden answer. Section 2.1 and C.1 say that when multiple outputs exist, annotators select only one, and Table 11 computes IF/SF against this single ground truth. But many editing requests admit several valid output images; a model producing a different but correct result is scored as failing. This can inflate reported failure rates and may reorder models on the real-world subset. The paper should include multiple accepted outputs per task, or report reference-free / rule-based metrics alongside the comparison to the single gold.
[§2.2 Step 4, §2.3, Appendix B.4/B.5] The evaluation pipeline has a self-referential component and the human-alignment correlations are moderate. The fine-tuned judge (Qwen-2.5-VL-7B-FT) is trained on the authors' own human annotations (B.5), used to filter MultiRef in Step 4, and used in evaluation, while GPT-4o-mini is used for overall IF/SF. Table 3 shows human-human agreement is 0.665 (Realistic IF) and 0.721 (Synthetic IF), versus 0.530 and 0.632 for GPT-4o-mini. The score differences between evaluated systems are comparable to the judge-human disagreement (Table 11: OmniGen IF 0.496 vs Claude+SD3.5 0.589). The reported ordering could therefore change with a different validated judge. Please report confidence intervals, a human-evaluated subset for the main comparisons, or a robustness analysis across judges.

minor comments (7)

[§2.2 vs §C.2] Section 2.2 says RefBlend produces samples 'across 34 reference combinations', while the abstract and C.2 say 33 combinations. Please correct the discrepancy.
[§2.1] Section 2.1 reports collecting 2,300 queries and keeping 45% (1,000 examples), but 45% of 2,300 is approximately 1,035. Clarify the quality-control criterion or the exact counts.
[Table 1] The legend lists 'Semantic Content Specification' twice (Green and Pink); one of these categories is presumably different. Also define the symbols (✓, ➞, and the blank/other marker) in the caption.
[Table 4] Several metrics are labeled '↓' (lower better), but the AVG column seems to combine them with '↑' metrics. State the normalization formula (e.g., 1 - normalized error) so that AVG is interpretable.
[Appendix C.3] The appendix says the judge validation used 'a selected test set of 300 samples for either Realistic and Synthetic dataset'. Report the exact test-set sizes and whether Table 3 and Table 10 are based on the same sets, ideally with confidence intervals.
[References] Reference [2] is titled 'Claude 3.5: A Sonnet', but the text mentions 'Claude-3.7-Sonnet' in the experimental setup. Please align the reference and the model version.
[Abstract/Introduction] The phrase 'state-of-the-art systems' is broader than the evaluated set (three open-source unified models and six agentic frameworks). Recommend scoping the claim to the evaluated models, since Emu2-Gen was omitted and proprietary generators are not exhaustive.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: the benchmark is an empirical evaluation, not a derivation; minor self-citations are not load-bearing.

full rationale

The central claim is an empirical benchmark result: nine systems are run on 1,990 held-out examples and compared to golden outputs via rule-based metrics (IoU, MSE, CLIP) and judge-based scores (IQ/IF/SF). No reported score is obtained by fitting a parameter to the target outcome; the golden outputs (original images for the synthetic split, selected Reddit edits for the real-world split) are external to the evaluated systems. The fine-tuned Qwen-2.5-VL judge is used for data filtering and for supplementary qualitative reference-fidelity comments, while the headline IQ/IF/SF numbers in Tables 4 and 11 come from GPT-4o-mini; both judge models are validated against human annotations, so the filter/evaluator overlap is a methodological self-reference rather than a definitional reduction. The self-citations to [5] (MLLM-as-a-Judge) and [51] (RealEdit) are to methods rather than to a uniqueness theorem, and the paper reports independent human validation, so they are not load-bearing. I therefore find no step where a prediction is equivalent to its input by construction. Two non-circular correctness risks are worth flagging in the verdict: Section 2.3 says GPT-4o-mini was chosen for superior alignment with human judgment, but Table 3/10 shows GPT-4o has higher IF/SF Pearson correlations on both realistic and synthetic data; and the abstract's 66.6% and 79.0% figures are not derivable from Tables 4/11 in the text. These affect evidentiary weight, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 3 invented entities

The central claim depends on a judge model that was trained on the same authors' human annotations and used both to filter the data and to evaluate the models. The judge shows low agreement on two of the weakest reported conditions (pose 0.257, bbox 0.404). The benchmark also assumes unique ground truth for real-world edits, while the synthetic portion depends on hand-written compatibility rules and automated extractors. No code or judge weights are shipped, only the dataset.

free parameters (4)

IoU filtering threshold 0.8 = 0.8
Chosen in Section 2.2, Step 4 for filtering semantic-map references; motivates the claim that the dataset is high-quality.
Judge score filtering threshold 3 = 3
In Appendix B.4, references scoring under 3 are filtered. This threshold shapes the 38k dataset composition.
30 samples per reference combination = 30
Appendix C.2 states 30 samples per combination for the 990 synthetic benchmark; this is a design choice that sets the benchmark size.
Judge training set sizes 16,590 and 1,750 = 16,590 and 1,750
The judge was fine-tuned on these human-scored samples; its behavior is an artifact of this choice.

assumptions (4)

domain assumption The fine-tuned MLLM judge's scores approximate human judgment on all 10 reference types.
Enters in Section 2.3 and Appendix B.4; the reported Pearson correlations are as low as 0.257 (pose) and 0.404 (bbox), so this assumption is only partially validated.
domain assumption A unique golden output exists for each real-world Reddit request.
Section 2.1 selects one output per query as ground truth; multiple valid edits exist for most real requests.
domain assumption Rule-based references (depth, edges, masks) are generated accurately by existing extractors.
Section 2.2 Step 1 relies on SAM2, Grounded SAM2, Depth Anything V2, and threshold-based Canny; errors in extraction propagate directly to the synthetic benchmark.
ad hoc to paper The reference compatibility rules correctly define all valid multi-reference combinations.
Section 2.2 Step 2 introduces mutual exclusivity and dependency rules by hand; violations or missed valid combinations would bias the 33 set and the conclusions.

invented entities (3)

RefBlend independent evidence
purpose: Synthetic data engine that extracts references and combines them by compatibility rules to produce benchmarking and training samples.
It is a software pipeline whose outputs (the 990 and 38k datasets) are publicly released, so it can be inspected and used independently.
Fine-tuned judge model (Qwen-2-VL-7B-FT)
purpose: Scores alignment and quality for filtering synthetic data and evaluating reference fidelity.
No weights, code, or API are released; only aggregate Pearson/MAE numbers are given, so external falsification is not possible.
MultiRef-bench independent evidence
purpose: Benchmark consisting of 990 synthetic and 1,000 real-world multi-reference tasks.
The dataset is publicly released and can be rerun by independent groups.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MultiRef: Controllable Image Generation with Multiple Visual References." pith.science (2026). https://pith.science/paper/3ZVIK2U6

@misc{pith2026250806905,
  author       = {Pith},
  title        = {Pith review of: MultiRef: Controllable Image Generation with Multiple Visual References},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3ZVIK2U6}},
  note         = {Machine review of arXiv:2508.06905}
}

read the original abstract

Visual designers naturally draw inspiration from multiple visual references, combining diverse elements and aesthetic principles to create artwork. However, current image generative frameworks predominantly rely on single-source inputs -- either text prompts or individual reference images. In this paper, we focus on the task of controllable image generation using multiple visual references. We introduce MultiRef-bench, a rigorous evaluation framework comprising 990 synthetic and 1,000 real-world samples that require incorporating visual content from multiple reference images. The synthetic samples are synthetically generated through our data engine RefBlend, with 10 reference types and 33 reference combinations. Based on RefBlend, we further construct a dataset MultiRef containing 38k high-quality images to facilitate further research. Our experiments across three interleaved image-text models (i.e., OmniGen, ACE, and Show-o) and six agentic frameworks (e.g., ChatDiT and LLM + SD) reveal that even state-of-the-art systems struggle with multi-reference conditioning, with the best model OmniGen achieving only 66.6% in synthetic samples and 79.0% in real-world cases on average compared to the golden answer. These findings provide valuable directions for developing more flexible and human-like creative tools that can effectively integrate multiple sources of visual inspiration. The dataset is publicly available at: https://multiref.github.io/.

Figures

Figures reproduced from arXiv: 2508.06905 by the authors.

**Figure 2.** An overview of RefBlend. It consists of reference generation (in yellow) and instruction prompt generation (in blue). First, various references (canny, depth, etc) are extracted from an original image. Then, a basic instruction prompt is formed from selected compatible references. Finally, the enhanced prompt is integrated with references to construct a sample [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Left, Middle: Distribution analysis of textual content length and image count for synthetic and real-world parts. Right: [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (12 more)

**Figure 4.** Figure 4: Real-world image generation conditioned on multi [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Images generated under multiple references. More are provided in the Supplementary Materials. GT: Ground Truth. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Ablation study on the impact of different reference [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: Reference Distribution in MultiRef [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: Distribution of human preference scores (1-5) for image-text alignment and image quality across various conditioning [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

**Figure 9.** Figure 9: Human Annotation Platform C Details of MultiRef-bench MultiRef-bench consists of 1,990 examples. The 1,000 examples from the real-world part represent real-world tasks sampled from the Reddit community r/PhotoshopRequest. The synthetic 990 examples are a test set split…

**Figure 10.** Figure 10: Samples in MultiRef-bench. The first row present real-world examples while others are from the synthetic part [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

**Figure 11.** Figure 11: Image generation conditioned on a combination of two references. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]

**Figure 12.** Figure 12: Image generation conditioned on a combination of three references. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

**Figure 13.** Figure 13: Ablation study on depth map and mask conditioning formats. Left: Depth-conditioned generation results comparing [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

**Figure 14.** Figure 14: Ablation study on the impact of captions. This figure illustrates the differences in image generation when a text [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]

**Figure 15.** Figure 15: Ablation studies: (a) bounding box conditioning formats; (b) impact of input reference order. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Emu3.5: Native Multimodal Models are World Learners
cs.CV 2025-10 unverdicted novelty 6.0 of 10

Emu3.5 is a native multimodal world model pre-trained on over 10 trillion vision-language tokens with next-token prediction, post-trained via reinforcement learning, and accelerated by Discrete Diffusion Adaptation fo...

Reference graph

Works this paper leans on

84 extracted references · 49 canonical work pages · cited by 1 Pith paper

[1]

Namhyuk Ahn, Junsoo Lee, Chunggi Lee, Kunhee Kim, Daesik Kim, Seung-Hun Nam, and Kibeom Hong. 2024. Dreamstyler: Paint by style inversion with text- to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 674–681

2024
[2]

Anthropic. 2024. Claude 3.5: A Sonnet. https://www.anthropic.com/news/claude- 3-5-sonnet. Accessed: 2024-09-04

2024
[3]

Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 843–852

2023
[4]

Caroline Chan, Frédo Durand, and Phillip Isola. 2022. Learning to generate line drawings that convey geometry and semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 7915–7925

2022
[5]

Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024. Mllm-as-a- judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning

2024
[6]

Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen. 2024. Subject-driven text-to-image generation via apprenticeship learning. Advances in Neural Information Processing Systems 36 (2024)

work page 2024
[7]

Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao
[8]

Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang, Nanxuan Zhao, Yilin Wang, et al. 2024. UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics. arXiv preprint arXiv:2412.07774 (2024)

arXiv 2024

Show all 84 references

[9]

Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, et al. 2024. MJ- Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? arXiv preprint arXiv:2407.04842 (2024)

2024 arXiv
[10]

Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. 2021. VITON- HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization. In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR)

2021
[11]

Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski
[12]

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. 2024. Scaling rectified flow transformers for high-resolution image synthesis. In Forty- first International Conference on M...

2024
[13]

Flux. 2024. Black Forest Labs. https://blackforestlabs.ai/

2024
[14]

Ziqi Gao, Weikai Huang, Jieyu Zhang, Aniruddha Kembhavi, and Ranjay Krishna
[15]

Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094 (2024)

2024 arXiv
[16]

GeminiTeam. 2023. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805 [cs.CL]

2023 arXiv
[17]

arXiv preprint arXiv:2412.08221 (2024)

Generate Any Scene: Evaluating and Improving Text-to-Vision Generation with Scene Graph Programming. arXiv preprint arXiv:2412.08221 (2024)

2024
[18]

Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. 2025. Can We Generate Images with CoT? Let’s Verify and Reinforce Image Generation Step by Step. arXiv preprint arXiv:2501.13926 (2025)

2025 arXiv
[19]

Zhen Han, Zeyinzi Jiang, Yulin Pan, Jingfeng Zhang, Chaojie Mao, Chenwei Xie, Yu Liu, and Jingren Zhou. 2024. ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer. arXiv preprint arXiv:2410.00086 (2024)

2024 arXiv
[20]

Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. 2023. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems 36 (2023), 52132–52152

2023
[21]

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718 (2021)

2021 arXiv
[22]

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017)

2017
[23]

Zhen Han, Chaojie Mao, Zeyinzi Jiang, Yulin Pan, and Jingfeng Zhang. 2024. StyleBooth: Image Style Editing with Multimodal Instruction. arXiv preprint arXiv:2404.12154 (2024)

2024 arXiv
[24]

Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Kr- ishna, and Noah A Smith. 2023. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision...

2023
[25]

Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2023. T2i- compbench: A comprehensive benchmark for open-world compositional text-to- image generation. Advances in Neural Information Processing Systems 36 (2023), 78723–78747

2023
[26]

Hexiang Hu, Kelvin CK Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, et al. 2024. Instruct- Imagen: Image generation with multi-modal instruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

2024
[27]

Chengyou Jia, Changliang Xia, Zhuohang Dang, Weijia Wu, Hangwei Qian, and Minnan Luo. 2024. ChatGen: Automatic Text-to-Image Generation From FreeStyle Chatting. arXiv preprint arXiv:2411.17176 (2024)

2024 arXiv
[28]

Xuan Ju, Ailing Zeng, Jianan Wang, Qiang Xu, and Lei Zhang. 2023. Human- art: A versatile human-centric dataset bridging natural and artificial scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 618–629

2023
[29]

Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Chen Liang, Tong Shen, Han Zhang, Huanzhang Dou, Yu Liu, and Jingren Zhou. 2024. ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers. arXiv preprint arXiv:2412.12571 (2024)

2024 arXiv
[30]

Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. 2025. ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback. In European Conference on Computer Vision . Springer, 129–147

2025
[31]

Tianle Li, Max Ku, Cong Wei, and Wenhu Chen. 2023. Dreamedit: Subject-driven image editing. arXiv preprint arXiv:2306.12624 (2023)

2023 arXiv
[32]

Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu
[33]

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 1931–1941

1931
[34]

Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. 2024. Evaluating text-to-visual gen- eration with image-to-text generation. In European Conference on Computer Vision. Springer, 366–384

2024
[35]

Xiaoyu Liu, Yuxiang Wei, Ming Liu, Xianhui Lin, Peiran Ren, Xuansong Xie, and Wangmeng Zuo. 2024. Smartcontrol: Enhancing controlnet for handling rough visual conditions. In European Conference on Computer Vision . Springer, 1–17

2024
[36]

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. 2023. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22511–22521

2023
[37]

Chen Liang, Lianghua Huang, Jingwu Fang, Huanzhang Dou, Wei Wang, Zhi- Fan Wu, Yupeng Shi, Junge Zhang, Xin Zhao, and Yu Liu. 2024. IDEA-Bench: How Far are Generative Models from Professional Designing? arXiv preprint arXiv:2412.11767 (2024)

2024 arXiv
[38]

Nithin Gopalakrishnan Nair, Jeya Maria Jose Valanarasu, and Vishal M Patel
[39]

OpenAI. 2024. GPT-4O Mini: Advancing Cost-Efficient Intelligence. https:// openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/. Accessed: 2024-09-04

2024
[40]

Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. 2024. Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

2024
[41]

Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. 2024. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 4296–4304

2024
[42]

Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. 2024. UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild.Advances in Neural Information Processing Systems...

2024
[43]

In European Conference on Computer Vision

Maxfusion: Plug&play multi-modal generation in text-to-image diffusion models. In European Conference on Computer Vision . Springer, 93–110
[44]

Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. 2024. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 (2024)

2024 arXiv
[45]

OpenAI. 2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/ Accessed: 2024-06-06

2024
[46]

Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei
[47]

arXiv preprint arXiv:2310.02992 (2023)

Kosmos-g: Generating images in context with multimodal large language models. arXiv preprint arXiv:2310.02992 (2023)

2023 arXiv
[48]

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural i...

2022
[49]

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenh...

2024 arXiv
[50]

Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2024. Genera- tive multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

2024
[51]

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In MM ’25, October 27–31, 2025, Dublin, Ireland Chen et al. Proceedings of the IEEE/CVF conference on computer vision and patte...

2022
[52]

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 22500–22510

2023
[53]

Babak Saleh and Ahmed Elgammal. 2015. Large-scale classification of fine- art paintings: Learning the right metric on the right feature. arXiv preprint arXiv:1505.00855 (2015)

2015 arXiv
[54]

Zhizhong Wang, Lei Zhao, and Wei Xing. 2023. Stylediffusion: Controllable disentangled style transfer via diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 7677–7689

2023
[55]

Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. 2024. Emu edit: Precise image editing via recognition and generation tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

2024
[56]

Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. 2024. Omnigen: Unified image generation. arXiv preprint arXiv:2409.11340 (2024)

2024 arXiv
[57]

Peter Sushko, Ayana Bharadwaj, Zhi Yang Lim, Vasily Ilin, Ben Caffee, Dong- ping Chen, Mohammadreza Salehi, Cheng-Yu Hsieh, and Ranjay Krishna. 2025. REALEDIT: Reddit Edits As a Large-scale Empirical Dataset for Image Transfor- mations. arXiv:2502.03629 [cs.CV] https://arxiv.o...

2025 arXiv
[58]

Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang
[59]

arXiv preprint arXiv:2411.15098 (2024)

OminiControl: Minimal and Universal Control for Diffusion Transformer. arXiv preprint arXiv:2411.15098 (2024)

2024 arXiv
[60]

Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. 2024. Genartist: Multimodal llm as an agent for unified image generation and editing. Advances in Neural Information Processing Systems 37 (2024), 128374–128395

2024
[61]

Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. 2024. Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems 36 (2024)

2024
[62]

Cong Wei, Zheyang Xiong, Weiming Ren, Xinrun Du, Ge Zhang, and Wenhu Chen
[63]

arXiv:2411.07199 [cs.CV] https://arxiv.org/abs/2411.07199

OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision. arXiv:2411.07199 [cs.CV] https://arxiv.org/abs/2411.07199

arXiv
[64]

Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. 2023. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22490–22499. 5 Acknowledgments...

2023
[65]

Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. 2024. Show-o: One Single Transformer to Unify Multimodal Understanding and Generation. arXiv preprint arXiv:2408.12528 (2024)

2024 arXiv
[66]

Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi
[67]

In Proceedings of the IEEE/CVF International Conference on Computer Vision

Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 7754– 7765
[68]

Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. 2024. Depth Anything V2. arXiv:2406.09414 (2024)

2024 arXiv
[69]

Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. 2023. Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 14246–14255

2023
[71]

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding conditional con- trol to text-to-image diffusion models. InProceedings of the IEEE/CVF International Conference on Computer Vision . 3836–3847

2023
[72]

Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. 2024. Uni-controlnet: All-in-one control to text-to- image diffusion models. Advances in Neural Information Processing Systems 36 (2024)

2024
[74]

Canny edge

to generate the depth map with default parameters. Canny edge. A Canny edge map is a binary image that shows the boundaries and edges detected in an original image using the Canny edge detection algorithm. It identifies edges by looking for areas of rapid intensity change in t...

2025
[75]

Element Replacement - Face swaps - Object substitutions - Background replacements - Text replacements - Component swaps (wheels, screens, etc.)
[76]

Element Addition - Adding people to scenes - Adding objects to environments - Adding details or elements to objects - Adding text or graphics - Adding visual effects
[77]

Style and Appearance Modifications - Color adjustments - Lighting modifications - Artistic style transfers - Texture changes - Visual quality enhancements
[78]

Spatial/Environment Manipulations - Repositioning ele- ments - Combining multiple images into layouts - Changing scale or proportion - Adjusting orientation or alignment - Creating composite images
[79]

category

Attribute Transfers - Transferring expressions between faces - Applying visual characteristics across images - Main- taining specific features while changing others - Matching visual properties (lighting, color) - Transferring specific de- tails while preserving context MultiR...

2025
[80]

First, the INPUT IMAGE(S) that the user wants to edit
[81]

Then, the user’s ORIGINAL INSTRUCTION (which might be noisy or unclear)
[82]

Based on comparing these, please:

Finally, the OUTPUT IMAGE after editing. Based on comparing these, please:
[83]

Infer what specific edit was performed
[84]

Edit image <image1> by [specific editing instruction using clear terminology]

Write a clear, precise prompt that would help an AI model achieve this exact edit Your prompt should follow this format: "Edit image <image1> by [specific editing instruction using clear terminology]" Here are some examples of good output prompts: - "Edit image <image1> by tak...

2025
[2023]

Advances in Neural Information Processing Systems 36 (2023), 16222–16239

Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing Systems 36 (2023), 16222–16239

2023
[2024]

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Anydoor: Zero-shot object-level image customization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 6593–6602

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Namhyuk Ahn, Junsoo Lee, Chunggi Lee, Kunhee Kim, Daesik Kim, Seung-Hun Nam, and Kibeom Hong. 2024. Dreamstyler: Paint by style inversion with text- to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 674–681

2024

[2] [2]

Anthropic. 2024. Claude 3.5: A Sonnet. https://www.anthropic.com/news/claude- 3-5-sonnet. Accessed: 2024-09-04

2024

[3] [3]

Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 843–852

2023

[4] [4]

Caroline Chan, Frédo Durand, and Phillip Isola. 2022. Learning to generate line drawings that convey geometry and semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 7915–7925

2022

[5] [5]

Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024. Mllm-as-a- judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning

2024

[6] [6]

Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen. 2024. Subject-driven text-to-image generation via apprenticeship learning. Advances in Neural Information Processing Systems 36 (2024)

work page 2024

[7] [7]

Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao

[8] [8]

Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang, Nanxuan Zhao, Yilin Wang, et al. 2024. UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics. arXiv preprint arXiv:2412.07774 (2024)

arXiv 2024