Pith. sign in

REVIEW 5 major objections 6 minor 66 references

Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Combining a controlled dataset of 50K minimally edited image pairs with a feature-level consistency objective lets open 7B models match or beat GPT-4o at detecting fine-grained visual differences, and reduces hallucination on standard…

desk verdict The dataset and benchmark are genuine resources, but the advertised feature-level consistency loss never appears in the methods, and internal inconsistencies in the reported numbers deepen the concern. read the letter →

arxiv 2506.07227 v1 pith:GXIOO42R submitted 2025-06-08 cs.CV cs.CL

Tianyi Bai , Yuxuan Fan , Jiantao Qiu , Fupeng Sun , Jiayi Song , Junlin Han , Zichen Liu , Conghui He
show 2 more authors
Wentao Zhang Binhang Yuan
This is my paper
classification cs.CVcs.CL
keywords multimodallargelanguagemodelshallucinationfine-grainedvisualunderstandingcontrolledimageeditingdifferencedetectionfeatureconsistencylosssupervisedfine-tuningMLLMevaluationbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that multimodal LLMs hallucinate on fine-grained visual differences because they lack two things: training data containing tightly controlled, minimally edited image pairs, and a learning objective that keeps visual representations stable under small edits. To supply both, the authors build a pipeline that filters source images, applies minimal semantic edits with an image generator, and writes aligned captions plus difference descriptions, producing the Micro Edit Dataset (MED) with over 50K pairs across 11 edit categories. They fine-tune open models with an SFT loss that feeds the feature difference between the paired images into the language head and trains it to emit the difference description. On the MED benchmark, the best fine-tuned 7B model reaches 51.61% accuracy, slightly ahead of GPT-4o, and the method posts consistent gains on POPE, MMStar, BLINK, and MME, which the paper reads as reduced hallucination and stronger grounding.

What carries the argument

The machinery is a two-stage loop. First, a controlled data pipeline: Qwen2.5-VL-72B filters images from DOCCI and Visual Genome and writes edit instructions targeting eleven hallucination-prone change types (attribute, count, position, object presence, and others); Gemini Flash 2.0 executes minimal edits; CLIP cosine similarity above 0.7 keeps changes subtle; and caption completion, edited-image captioning, and difference-description generation are done stepwise by Qwen2.5-VL-72B and Qwen3-32B, with 1,000 pairs human-verified. Second, the SFT objective of Equation 7, written as a captioning loss applied to the feature difference of the paired images, $l_{\mathrm{cap}}(Z_\theta[I_\theta(x_i) - I_\theta(\hat{x}_i)], S_\phi(t_i,\hat{t}_i))$: the model must produce the difference sentence from the vector difference of the two image embeddings, which is the mechanism that stabilizes embeddings under small edits and connects visual change to language change.

What would settle it

Take a held-out set of image pairs from the same editing pipeline and write the difference descriptions entirely by human inspection, then ask the fine-tuned model the same questions; if accuracy on the human-written set falls back toward the base-model level while remaining high on pipeline-written questions, the reported gains come from fitting synthetic phrasing rather than from sharper visual perception. A cheaper version of the same test is to have humans verify a random sample of the synthetic difference descriptions against the actual images and measure the error rate of the supervision itself.

Watch

Extended reading notes

Core claim

The paper's central claim is that fine-grained visual hallucination in multimodal LLMs is mostly a data and objective problem rather than an architecture problem: once models are trained on minimally edited image pairs whose captions and difference sentences are aligned to the actual visual change, their ability to detect and describe subtle differences rises sharply. On the Micro Edit Detection benchmark, Qwen2.5-VL-7B improves from 39.74% to 51.61% after fine-tuning, overtaking GPT-4o (51.32%) and Doubao-1.5-vision-pro (51.81%) while remaining well below the human accuracy of 95.21%. Similar gains appear on the real-world MED-Real set and on standard hallucination and perception benchmarks (POPE, MMStar, BLINK, MME), and the paper's ablation attributes part of the gain to fine-tuning the vision encoder alone and the rest to joint ViT-plus-LLM fine-tuning. The load-bearing comparison is that the method outperforms GPT-4o on difference detection accuracy while reducing hallucinations, which is the claim the paper asks readers to accept.

Load-bearing premise

The training targets and the benchmark answers are both produced by the same automated captioning and difference-writing pipeline, with only 1,000 of the 50,000+ pairs checked by humans, so the whole method presupposes that those synthetic difference descriptions actually match the true visual changes in the images.

Editorial extensions

If this is right

  • Open 7B models fine-tuned on MED reach parity with frontier closed models on fine-grained difference detection: Qwen2.5-VL-7B (Ours) scores 51.61% on the MED benchmark versus 51.32% for GPT-4o.
  • Training on minimal edits transfers to standard benchmarks, with consistent gains on POPE, MMStar Coarse Perception, BLINK Visual Correspondence and Counting, and MME across all four tested backbones.
  • The 11-category taxonomy exposes where models fail most (Action, Comparison, Negation are the hardest categories for closed models), giving a diagnostic grid for future hallucination work.
  • Fine-tuning the vision encoder alone delivers solid gains, and jointly fine-tuning ViT and LLM adds cross-modal alignment benefits, so the visual representation is a large part of the story.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the MED benchmark is built by the same pipeline that generated the training data, the headline benchmark numbers likely overstate the true perceptual gain; the MED-Real results, where three models improve but Qwen2-VL-7B is flat at 82.86%, are the more conservative estimate of real-world transfer.
  • The feature-difference-to-language supervision is a general recipe that extends naturally beyond image pairs, for example to video frame pairs, multi-step edit chains, or temporal change descriptions, where the same mechanism could ground change language in representation differences.
  • A direct testable extension would be scaling the human-verified subset: if the synthetic supervision is the bottleneck the paper's own framing implies, MED accuracy should rise monotonically with the fraction of human-verified pairs in the training mix.
  • The method inversely suggests a cheap diagnostic for any MLLM: probe it with pairs from the 11 categories to locate which change types its representations fail to encode, then target training data at exactly those categories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes a controlled image-editing pipeline that generates minimally edited image pairs with aligned captions, and uses it to build the Micro Edit Dataset (MED, over 50K pairs across 11 fine-grained edit categories) and a Micro Edit Detection benchmark (165 synthetic + 35 real-world pairs). The authors further propose a supervised fine-tuning framework that they claim includes a feature-level consistency loss promoting stable visual embeddings under small edits, and they evaluate the resulting models on MED, MED-Real, POPE, MMStar, BLINK, MMVP, and MME. The paper reports gains over base models and commercial systems such as GPT-4o.

Significance. The dataset and benchmark are potentially useful resources: the controlled editing pipeline is clearly described, the authors release code and data, and the inclusion of a 35-pair MED-Real set shows awareness of distribution-leakage concerns. If the claimed learning objective existed and the results were statistically supported, the paper would make a meaningful contribution to fine-grained multimodal reasoning. However, the central methodological novelty advertised throughout the paper—a feature-level consistency loss—is not defined or implemented anywhere in the method or experiments, and several key numbers in the text contradict Table 1. The only loss actually trained is a standard difference-captioning SFT objective, so all reported gains could be attributed to supervised fine-tuning on MED data alone. The benchmark is small (165 main questions, ~15 per category), no significance tests are provided, and the headline comparison with GPT-4o rests on a 0.29-point difference. These issues substantially reduce the significance of the empirical claims as they stand.

major comments (5)
  1. [Abstract; Section 4.3, Eq. (7); Table 3] The 'feature-level consistency loss' advertised in the Abstract, Section 1, and Conclusions is never defined. Section 4.3 derives only the SFT loss in Eq. (7), which is an autoregressive captioning loss applied to the difference feature Iθ(xi) − Iθ(ˆxi); subtracting image features as decoder input is not a regularizer or a consistency objective. No auxiliary term on the image encoder appears in the loss derivation, the training details in Appendix C, or the ablation in Table 3, which compares ViT-only versus joint fine-tuning and does not isolate a consistency-loss component. The central claim that the proposed loss 'promotes stable visual embeddings under small edits' is therefore unsupported, and the reported gains could all be produced by SFT on MED data alone.
  2. [Section 5.2 vs Table 1] The text and Table 1 contradict each other on baseline accuracies. Section 5.2 reports Qwen2.5-VL-7B improving 'from 40.24% to 51.61%', but Table 1 lists the baseline as 39.74%. For LLaVA-V1.6-7B, the text states an improvement 'from 40.44% to 44.04% (+3.60)', whereas Table 1 shows 31.04% for the base model and 40.44% for the fine-tuned model. These discrepancies affect the claimed gains and must be reconciled before the results can be assessed.
  3. [Section 5.2; Table 1; Abstract] No error bars, confidence intervals, or significance tests are reported anywhere. With only 165 main benchmark questions and roughly 15 per category, the differences the paper highlights—such as Qwen2.5-VL-7B (Ours) at 51.61% versus GPT-4o at 51.32%, a 0.29-point gap—are within sampling noise. The Abstract's claim that the method 'improves difference detection accuracy and reduces hallucinations compared to strong baselines, including GPT-4o' is therefore not supported as stated. The authors should provide paired tests, bootstrap intervals, or per-category confidence intervals, and temper the comparative claims accordingly.
  4. [Sections 3.2–3.3; Appendix B] The MED benchmark is generated from the same controlled editing pipeline and the same difference-description QA format used to construct the SFT training data, with only the benchmark/training pair overlap excluded. Consequently, gains on the main MED benchmark may partly reflect format matching rather than improved fine-grained perception. The MED-Real set is a welcome partial mitigation because it uses real-world image pairs from MMVP, but it contains only 35 pairs and still uses the same automated question-answer generation scheme. The paper should report per-pair or per-category results on MED-Real, and explicitly discuss the residual risk of format-level leakage.
  5. [Section 3.2; Step 4] The training targets—difference descriptions and benchmark answers—are generated by Qwen2.5-VL-72B, Qwen3-32B, GPT-4o, and related models, with only 1,000 of 50,000+ pairs human-verified. If these synthetic descriptions do not accurately reflect the actual visual edits, the fine-tuned models would fit generated text rather than true perceptual distinctions, and a benchmark built from the same pipeline would hide that failure. The paper should either substantially expand the human verification sample or provide an independent cross-check (for example, evaluation by a different, held-out VLM or a human study on a random subset) to establish the validity of the supervision signal.
minor comments (6)
  1. [Eqs. (1)–(2)] The notation lclip(Iθ(x) − Tθ(t)) writes a subtraction between image and text embeddings, but a CLIP-style contrastive loss typically operates on paired embeddings rather than on their vector difference; please clarify the intended operation.
  2. [Section 2] The paragraph heading 'Evaluating Multimodal LLMsThe evaluation' is missing a space or paragraph break; please fix the formatting.
  3. [Table 2; Section 5.3] Table 2 shows that LLaVA-V1.6-7B (Ours) has a decreased MME score (1441.89 to 1420.57) despite the text claiming 'consistent and generalizable performance gains'; the paper should acknowledge and explain such red cells rather than only stating overall improvements.
  4. [Appendix C; Section 5.4] The LLaVA-V1.6-7B training configuration has 'Unfreeze Vision Tower False', which conflicts with the paper's emphasis on vision-encoder fine-tuning as a mechanism; please clarify how the proposed feature-level objective is realized when the vision tower remains frozen.
  5. [Reference [63]] Reference [63] is cited as the source of the MMVP benchmark, but the listed title describes a 'Multimodal Mocap Dataset with Vision and Pressure Sensors', which appears to be the wrong reference; please verify and correct the citation.
  6. [Section 3.1; Table 1] The CLIP similarity thresholds (0.7 for filtering and 0.95 for benchmark selection) are free parameters that affect dataset difficulty and composition; the paper should include a sensitivity analysis or at least justify the chosen thresholds.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the MED benchmark is a held-out test split, and independent benchmarks (MED-Real, POPE, MMStar, BLINK, MME) break any training/evaluation loop.

full rationale

The paper's claimed derivation is self-contained and not circular. The training objective in Eq. 7 is a supervised fine-tuning loss on the MED instruction data; the MED benchmark is a held-out split whose samples are 'excluded from the fine-tuning dataset' (Section 3.3), and whose question-answer sets are 'manually verified.' The authors explicitly anticipate the same-pipeline concern: 'To address potential concerns about overfitting to synthetic data, as both the training set and benchmark stem from the same controlled editing pipeline, we augment the evaluation with real-world image pairs' (Section 3.3). They then report gains on MED-Real, POPE, MMStar, BLINK, and MME (Section 5.3), which are external to the MED training distribution. No parameter is fitted to benchmark answers, and no equation reduces a prediction to its own input. The abstract's 'feature-level consistency loss' is not actually defined in Eq. 7 (only a difference-captioning loss appears), but that is an evidence/correctness gap, not a circularity: a missing term cannot make the derivation circular because it is never instantiated. Score 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the trustworthiness of automatically generated supervision and on the assumption that standard MLLMs can consume difference inputs. The two CLIP thresholds are hand-set and shape both training data and benchmark difficulty.

free parameters (2)
  • CLIP similarity threshold (training filter) = 0.7
    Pairs below 0.7 cosine similarity are discarded in Section 3.1; this threshold controls the 'minimal edit' definition and affects all training data.
  • CLIP similarity threshold (benchmark selection) = >0.95
    Benchmark pairs are selected with CLIP similarity above 0.95 in Section 3.3; this controls benchmark difficulty and is not derived from theory.
assumptions (4)
  • domain assumption The edit instructions produced by Qwen-2.5-VL-72B correspond to actual minimal visual edits made by Gemini Flash 2.0.
    The entire dataset rests on the semantic alignment of the generated edit prompt with the executed edit; only a 1,000-sample manual check is reported (Section 3.2).
  • domain assumption The difference descriptions generated by Qwen3-32B and Qwen2.5-VL-72B are accurate ground-truth for the visual difference.
    These descriptions serve as SFT targets and benchmark answers without human verification at scale (Section 3.2).
  • domain assumption The MLLM can represent the pair of images as a feature difference I(x1) - I(x2) and decode it into a difference caption; no architectural change is made to enable this.
    The loss in Eq. 7 assumes the model can consume the difference of visual features, but the paper fine-tunes standard MLLMs without adding a difference operation (Section 4.3).
  • domain assumption The theoretical noise model (Eq. 5) with probability η describes real MLLM training data.
    This is asserted without empirical evidence and is not used to derive the proposed method (Section 4.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning." pith.science (2026). https://pith.science/paper/GXIOO42R

@misc{pith2026250607227,
  author       = {Pith},
  title        = {Pith review of: Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GXIOO42R}},
  note         = {Machine review of arXiv:2506.07227}
}
read the original abstract

Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks but still struggle with fine-grained visual differences, leading to hallucinations or missed semantic shifts. We attribute this to limitations in both training data and learning objectives. To address these issues, we propose a controlled data generation pipeline that produces minimally edited image pairs with semantically aligned captions. Using this pipeline, we construct the Micro Edit Dataset (MED), containing over 50K image-text pairs spanning 11 fine-grained edit categories, including attribute, count, position, and object presence changes. Building on MED, we introduce a supervised fine-tuning (SFT) framework with a feature-level consistency loss that promotes stable visual embeddings under small edits. We evaluate our approach on the Micro Edit Detection benchmark, which includes carefully balanced evaluation pairs designed to test sensitivity to subtle visual variations across the same edit categories. Our method improves difference detection accuracy and reduces hallucinations compared to strong baselines, including GPT-4o. Moreover, it yields consistent gains on standard vision-language tasks such as image captioning and visual question answering. These results demonstrate the effectiveness of combining targeted data and alignment objectives for enhancing fine-grained visual reasoning in MLLMs.

Figures

Figures reproduced from arXiv: 2506.07227 by the authors.

Figure 1
Figure 1. GPT-4o exhibits hallucination errors on fine-grained visual question answering. Instances are sys￾tematically selected to highlight failure cases in GPT-4o (Date accessed: May 08, 2025) visual question answering (VQA) performance across five hallucination-prone categories [15, 30]: Object, Universality, Counting, Action, and Spatial. Each row shows two examples where GPT-4o provides fluent yet inaccurate answers due… view at source ↗
Figure 2
Figure 2. Overview of the Micro Edit Dataset (MED) construction pipeline. We begin with MLLM-guided fil￾tering and controlled visual editing based on hallucination-prone change types. Caption alignment is performed via step-wise prompting to ensure consistency between original, edited images, and textual descriptions. The resulting data supports both supervised fine-tuning and benchmark evaluation for fine-grained visual reas… view at source ↗
Figure 3
Figure 3. Examples from the MED Benchmark covering all 11 composition types. These categories reflect common sources of hallucination in MLLMs [15, 30] and include both basic changes (Object, Attribute, Scene, Spatial, Action, Part) and more complex transformations (Counting, Comparison, Differentiation, Negation, Universality). Each example shows an image pair, a difference question, and multiple-choice answers including the… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Example samples from the Micro Edit Dataset (MED) used for supervised fine-tuning. Each sample consists of a minimally edited image pair, a question prompting for the most notable difference, and a visually grounded answer. Answers are refined to highlight concrete sem…
Figure 5
Figure 5. Figure 5: More examples in the MED benchmark 18 [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: More examples in the MED benchmark 19 [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Prompt for Qwen-2.5-VL-72B to filter editable images [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Prompt for Qwen-2.5-VL-72B to generate editing prompt [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Prompt for Qwen-2.5-VL-72B to generate complete original description for original images [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Prompt for Qwen-2.5-VL-72B to generate description for edited images [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Prompt for Qwen-2.5-VL-72B to generate difference description [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Prompt for Qwen-2.5-VL-72B to get judge 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Prompt for Qwen-2.5-VL-72B to get second judge [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Prompt for Qwen3-32B to generate SFT data [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Question Template of Benchmark [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: Prompt for GPT-4o to generate right answer [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: Prompt for GPT-4o to generate wrong answer [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

66 extracted references · 33 canonical work pages

  1. [1]

    URL https://api.semanticscholar.org/CorpusID:276612236

    Claude 3.7 sonnet system card. URL https://api.semanticscholar.org/CorpusID:276612236

    work page

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 10

    arXiv 2025

  4. [4]

    Hallucination of multimodal large language models: A survey .CoRR, 2024

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey .CoRR, 2024

    work page 2024

  5. [5]

    Flux.1 fill [dev]

    Black Forest Labs. Flux.1 fill [dev]. https://huggingface.co/black-forest-labs/FLUX.1-Fill-dev , 2024. Accessed: 2025-04-19

    work page 2024

  6. [6]

    Ledits++: Limitless image editing using text-to-image models

    Manuel Brack, Felix Friedrich, Katharia Kornmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolin´ario Passos. Ledits++: Limitless image editing using text-to-image models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8861–8870, 2024

    work page 2024

  7. [7]

    Instructpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18392–18402, 2023

    work page 2023

  8. [8]

    The revolution of multimodal large language models: a survey .arXiv preprint arXiv:2402.12451, 2024

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The revolution of multimodal large language models: a survey .arXiv preprint arXiv:2402.12451, 2024

    arXiv 2024

Show all 66 references
  1. [9]

    Vlmimic: Vision language models are visual imitation learner for fine-grained actions.Advances in Neural Information Processing Systems, 37:77860–77887, 2024

    Guangyan Chen, Meiling Wang, Te Cui, Yao Mu, Haoyang Lu, Tianxing Zhou, Zicai Peng, Mengxiao Hu, Haizhou Li, Li Yuan, et al. Vlmimic: Vision language models are visual imitation learner for fine-grained actions.Advances in Neural Information Processing Systems, 37:77860–77887, 2024

    2024

  2. [10]

    Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    2024 arXiv

  3. [11]

    Anydoor: Zero-shot object-level image customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 6593–6602, 2024

    2024

  4. [12]

    Unireal: Universal image generation and editing via learning real-world dynamics.arXiv preprint arXiv:2412.07774, 2024

    Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang, Nanxuan Zhao, Yilin Wang, et al. Unireal: Universal image generation and editing via learning real-world dynamics.arXiv preprint arXiv:2412.07774, 2024

    2024 arXiv

  5. [13]

    Diffusion self-guidance for controllable image generation

    Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing Systems, 36:16222–16239, 2023

    2023

  6. [14]

    Mme-survey: A comprehensive survey on evaluation of multimodal llms.arXiv preprint arXiv:2411.15296, 2024

    Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. Mme-survey: A comprehensive survey on evaluation of multimodal llms.arXiv preprint arXiv:2411.15296, 2024

    2024 arXiv

  7. [15]

    Tldr: Token-level detective reward model for large vision language models.arXiv preprint arXiv:2410.04734, 2024

    Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, and Lawrence Chen. Tldr: Token-level detective reward model for large vision language models.arXiv preprint arXiv:2410.04734, 2024

    2024 arXiv

  8. [16]

    Towards a benchmark of multimodal large language models for industrial engineering

    Markus Michael Geipel. Towards a benchmark of multimodal large language models for industrial engineering. In 2024 IEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA), pp. 1–4. IEEE, 2024

    2024

  9. [17]

    Generative adversarial networks.Communications of the ACM, 63(11):139–144, 2020

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley , Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks.Communications of the ACM, 63(11):139–144, 2020

    2020

  10. [18]

    Gemini 2.5 flash

    Google DeepMind. Gemini 2.5 flash. https://deepmind.google/technologies/gemini/flash/, 2025. Accessed: 2025-04-26

    2025

  11. [19]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay , Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904–6913, 2017

    2017

  12. [20]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  13. [21]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 11

    2020

  14. [22]

    Diffusion model-based image editing: A survey .arXiv preprint arXiv:2402.17525, 2024

    Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Liangliang Cao, and Shifeng Chen. Diffusion model-based image editing: A survey .arXiv preprint arXiv:2402.17525, 2024

    2024 arXiv

  15. [23]

    Smartedit: Exploring complex instruction-based image editing with multimodal large language models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction-based image editing with multimodal large language models. In Proceedings of the IEEE/CVF Conference...

    2024

  16. [24]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 6700–6709, 2019

    2019

  17. [25]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  18. [26]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401–4410, 2019

    2019

  19. [27]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer ...

    2017

  20. [28]

    Evaluating and improving compositional text-to-visual generation

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Evaluating and improving compositional text-to-visual generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  21. [29]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  22. [30]

    Evaluating text-to-visual generation with image-to-text generation

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. InEuropean Conference on Computer Vision, pp. 366–384. Springer, 2024

    2024

  23. [31]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  24. [32]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  25. [33]

    Step1x-edit: A practical framework for general image editing.arXiv preprint arXiv:2504.17761, 2025

    Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al. Step1x-edit: A practical framework for general image editing.arXiv preprint arXiv:2504.17761, 2025

    2025 arXiv

  26. [34]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022

    2022 arXiv

  27. [35]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pp

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pp. 216–233. Springer, 2024

    2024

  28. [36]

    Customizable image synthesis with multiple subjects.Advances in neural information processing systems, 36:57500–57519, 2023

    Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Customizable image synthesis with multiple subjects.Advances in neural information processing systems, 36:57500–57519, 2023

    2023

  29. [37]

    Magicquill: An intelligent interactive image editing system.arXiv preprint arXiv:2411.09703, 2024

    Zichen Liu, Yue Yu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Wen Wang, Zhiheng Liu, Qifeng Chen, and Yujun Shen. Magicquill: An intelligent interactive image editing system.arXiv preprint arXiv:2411.09703, 2024

    2024 arXiv

  30. [38]

    Fine-grained image editing by pixel-wise guidance using diffusion models.arXiv preprint arXiv:2212.02024, 2022

    Naoki Matsunaga, Masato Ishii, Akio Hayakawa, Kenji Suzuki, and Takuya Narihira. Fine-grained image editing by pixel-wise guidance using diffusion models.arXiv preprint arXiv:2212.02024, 2022

    2022 arXiv

  31. [39]

    Mm1: methods, analysis and insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Anton Belyi, et al. Mm1: methods, analysis and insights from multimodal llm pre-training. In European Conference on Computer Vision, pp. 304–323. ...

    2024

  32. [40]

    Sdedit: Guided image synthesis and editing with stochastic differential equations.arXiv preprint arXiv:2108.01073, 2021

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations.arXiv preprint arXiv:2108.01073, 2021

    2021 arXiv

  33. [41]

    Dragondiffusion: Enabling drag-style manipulation on diffusion models

    Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.02421, 2023

    2023 arXiv

  34. [42]

    Docci: Descriptions of connected and contrasting images

    Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et al. Docci: Descriptions of connected and contrasting images. InEuropean Conference on Computer Vision, pp. 291–309. Springer, 2024

    2024

  35. [43]

    Drag your gan: Interactive point-based manipulation on the generative image manifold

    Xingang Pan, Ayush Tewari, Thomas Leimk¨uhler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image manifold. InACM SIGGRAPH 2023 conference proceedings, pp. 1–11, 2023

    2023

  36. [44]

    Evaluating llm–generated multimodal diagnosis from medical images and symptom analysis.arXiv preprint arXiv:2402.01730, 2024

    Dimitrios P Panagoulias, Maria Virvou, and George A Tsihrintzis. Evaluating llm–generated multimodal diagnosis from medical images and symptom analysis.arXiv preprint arXiv:2402.01730, 2024

    2024 arXiv

  37. [45]

    Synthesize diagnose and optimize: Towards fine-grained vision-language understanding

    Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. Synthesize diagnose and optimize: Towards fine-grained vision-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13279–13288, 2024

    2024

  38. [46]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy , Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry , Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, ...

    2021

  39. [47]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684–10695, 2022

    2022

  40. [48]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317–8326, 2019

    2019

  41. [49]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  42. [50]

    Emu: Generative pretraining in multimodality .arXiv preprint arXiv:2307.05222, 2023

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality .arXiv preprint arXiv:2307.05222, 2023

    2023 arXiv

  43. [51]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14...

    2024

  44. [52]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  45. [53]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568–9578, 2024

    2024

  46. [54]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  47. [55]

    Image inpainting with external-internal learning and monochromic bottleneck

    Tengfei Wang, Hao Ouyang, and Qifeng Chen. Image inpainting with external-internal learning and monochromic bottleneck. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5120–5129, 2021

    2021

  48. [56]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  49. [57]

    Uni-paint: A unified framework for multimodal image inpainting with pre- trained diffusion model

    Shiyuan Yang, Xiaodong Chen, and Jing Liao. Uni-paint: A unified framework for multimodal image inpainting with pre- trained diffusion model. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 3190–3199, 2023. 13

    2023

  50. [58]

    Imagebrush: Learning visual in-context instructions for exemplar-based image manipulation

    Yifan Yang, Houwen Peng, Yifei Shen, Yuqing Yang, Han Hu, Lili Qiu, Hideki Koike, et al. Imagebrush: Learning visual in-context instructions for exemplar-based image manipulation. Advances in Neural Information Processing Systems, 36:48723–48743, 2023

    2023

  51. [59]

    Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    2022 arXiv

  52. [60]

    Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  53. [61]

    Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Lingfeng Ren, Linjie Li, Jianfeng Wang, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Lijuan Wang, and Xinchao Wang. Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities. arXiv preprint arXiv:2408.00765, 2024

    2024 arXiv

  54. [62]

    Walkvlm: Aid visually impaired people walking by vision language model.arXiv preprint arXiv:2412.20903, 2024

    Zhiqiang Yuan, Ting Zhang, Ying Deng, Jiapei Zhang, Yeshuang Zhu, Zexi Jia, Jie Zhou, and Jinchao Zhang. Walkvlm: Aid visually impaired people walking by vision language model.arXiv preprint arXiv:2412.20903, 2024

    2024 arXiv

  55. [63]

    Mmvp: A multimodal mocap dataset with vision and pressure sensors

    He Zhang, Shenghao Ren, Haolei Yuan, Jianhui Zhao, Fan Li, Shuangpeng Sun, Zhenghao Liang, Tao Yu, Qiu Shen, and Xun Cao. Mmvp: A multimodal mocap dataset with vision and pressure sensors. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp....

    2024

  56. [64]

    Mllms know where to look: Training-free perception of small visual details with multimodal llms.arXiv preprint arXiv:2502.17422, 2025

    Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Mllms know where to look: Training-free perception of small visual details with multimodal llms.arXiv preprint arXiv:2502.17422, 2025

    2025 arXiv

  57. [65]

    Magicbrush: A manually annotated dataset for instruction-guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems, 36:31428–31449, 2023

    2023

  58. [66]

    123- ABCD,\

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System D...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.