Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing

Yang Liu; Yuxin Peng; Zhuoying Li; Zhu Xu

REVIEW 3 major objections 5 minor 3 cited by

Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces BPM, a metric for instruction-based image editing that explicitly separates editing-relevant from editing-irrelevant regions, and claims it achieves the highest agreement with human evaluation among existing…

desk verdict A sensible two-tier metric for instruction-based editing, but the headline number is undercut by an undefined Smodify for add/remove instructions and alpha tuning on the evaluation set. read the letter →

arxiv 2506.13827 v1 pith:XFCZWCAD submitted 2025-06-15 cs.GR cs.AI

Zhuoying Li , Zhu Xu , Yuxin Peng , Yang Liu This is my paper

classification cs.GRcs.AI

keywords instruction-basedimageeditingevaluationmetricregion-awarescoringsemantic-awareCLIPdirectionalsimilaritypreservationandmodificationLLMinstructionparsingmask-guided

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces BPM (Balancing Preservation and Modification), an automated metric for instruction-based image editing that separates an image into regions that should be edited and regions that should stay untouched. It claims that by evaluating these two parts separately, BPM aligns with human judgments more closely than existing automatic metrics, including vision-language-model-based evaluators like GPT-4o. The metric also provides interpretable sub-scores for position, size, semantic compliance, and preservation, and includes a training-free way to use the detected editing regions as guidance to improve editing models themselves.

What carries the argument

The key machinery is the explicit disentanglement of the image into editing-relevant and editing-irrelevant regions, driven by a three-step pipeline: LLM instruction parsing, detector/segmentor localization, and two rule-based plus two embedding-based judges. The load-bearing identity is the directional CLIP similarity computed on cropped bounding-box regions, cos_sim(CLIP_I(A_edit)-CLIP_I(A_origin), CLIP_T(o2)-CLIP_T(o1)), which measures whether the change in image features moves in the same direction as the change in text features. This identity lets the metric assess modification against the instruction while implicitly checking that unrelated attributes are retained, and the preservation score uses an L2 distance on the complement of the union of the original and edited masks to enforce that untouched regions are unchanged.

What would settle it

Take an edit where the detector misses the edited object (for instance, an instruction that adds a small object that the detector does not recognize, or an edit that changes an object beyond recognizability), and check BPM's score against human ratings; a single systematic failure of localization that flips the relative ranking of two clearly good and clearly bad edits would refute the metric's reliability without a fallback.

Watch

Extended reading notes

Core claim

The paper claims that the central failure of existing editing metrics is that they evaluate the whole image as one unit, even though a good edit requires opposite treatments for different regions: the editing-relevant region must change according to the instruction, while everything else must stay intact. BPM addresses this by first parsing the instruction with an LLM to identify the source and target objects and any required size or position changes, then using a detector and segmentor to locate those objects in both the original and edited images. A Region-Aware Judge scores whether the edited region's position and size match the instruction, and a Semantic-Aware Judge uses directional CLIP similarity on the cropped edited regions plus an L2 distance on the preserved regions to score content compliance and preservation. The final BPM score is a weighted combination (0.7*S_semantic + 0.3*S_region), and experiments report that this score has the highest alignment with human preference rankings among all compared metrics on both local edits and global edits.

Load-bearing premise

The metric assumes that the LLM parsing and the detector and segmentor reliably locate every region that the editing process touches in both the original and edited images, so that the right pixels are scored.

Editorial extensions

If this is right

If BPM's human-alignment results hold, it provides a cheap and interpretable alternative to human evaluation for benchmarking instruction-based editing models at scale, with speed around 1.5 seconds per image versus about 11.4 seconds for GPT-4o according to the paper's comparison.
The region-localization step can be reused as training-free guidance: applying classifier-free guidance only inside the detected editing regions improved preservation and modification scores for IP2P, FTIP2P, and MGIE in the paper's experiments, and 91% of user-preference samples rated the guided output as better or equal.
Because the metric returns separate scores for position, size, modification, and preservation, it can diagnose which aspect of an editing model fails, rather than giving a single opaque number.
The metric could be applied to other instruction-following visual tasks where a subset of pixels must change and the rest must stay fixed, such as object removal, inpainting, or attribute-sensitive generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper does not pursue is using the BPM sub-scores as training rewards or loss components for editing models, since the component structure gives dense, differentiable-style feedback on exactly which criterion failed.
The metric's reliance on open-set detection and segmentation means its practical ceiling is set by those tools; as better open-vocabulary segmentors arrive, the same scoring scheme should improve without any change to the BPM formulas.
The paper's 97% localization accuracy on 100 samples suggests the main risk is not typical parsing errors but instructions that refer to objects indirectly (e.g., 'it' when the referent is not named), which the paper itself notes as the failure mode in its object-identification errors.
One implicit claim worth testing is whether the directional CLIP similarity stays reliable when the edited object changes class (e.g., clock to street sign) versus when it only changes appearance; the paper evaluates both types but does not break down performance by edit type.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper proposes BPM, an automatic metric that separates editing-relevant from editing-irrelevant regions and scores them differently. That decomposition is the right instinct, and the specific combination—LLM parsing, Grounding DINO/SAM localization, masked CLIP directional similarity, and masked L2 preservation—is genuinely new as a package. The human-alignment results, based on 960 responses and several model pairs, directionally beat the baselines, including GPT-4o with a standard prompt. The ground-truth test (87% favoring the well-edited image) is a clean sanity check, and the component ablations (Tables 5–7) support the design choices. That is real work worth taking seriously.

But there is a definitional gap that directly threatens the headline. For add and remove instructions, Sec. 3.2 sets o1 or o2 to "None". Alg. 2 line 10 then computes CLIPT(o2) - CLIPT(o1), which is undefined. The appendix only handles corner cases for fsize and fposition, not for Smodify. MagicBrush, the local-editing test set, explicitly includes adding and removing objects, so a sizable share of the 100 instructions likely falls into these categories. Either the released code implements an undocumented special case, or the reported 0.799 average was computed on a subset or with a different formula. Either way, the specification does not support the central claim as written.

Second, alpha = 0.7 was selected from Table 9 on a 100-entry subset of the same evaluation data used for the main alignment test. That is fitting a parameter to the evaluation target, so some of the reported advantage over baselines is expected even without a real difference. This is addressable—fix alpha from a separate validation split or report a range—but it does weaken the current numbers.

Third, normalization and thresholds are underspecified: the IOU thresholds in fposition and the Norn operation in Alg. 2 are never defined, so exact reproduction is not possible from the paper. That is a minor but annoying omission.

Also minor: the mask-guidance application is evaluated partly with BPM itself, which risks circularity, and the 91% "better or equal" user study is small. These are fixable with a more independent evaluation.

The paper does include honest checks: 97–99% parsing accuracy and a 4.01/5 human rating for mask quality on 100 samples, which are good signs that the localization step works in practice.

Bottom line: the idea is solid and the core direction is right, but the undefined Smodify case is a load-bearing flaw in the current specification, and the alpha tuning reduces confidence in the reported advantage. I would send this to peer review—it deserves referee time—but with the expectation of significant revision to close the definitional gap and clean up the evaluation protocol.

Referee Report

3 major / 5 minor

Summary. This paper proposes BPM, an automatic metric for instruction-based image editing. The metric first uses an LLM to parse the instruction into source/target objects and required size/position changes, then uses a detector and segmentor to localize those objects in the original and edited images. A region-aware judge checks size and position alignment, while a semantic-aware judge computes a directional CLIP similarity on the edited region and an L2-based preservation score on the complementary region. The two scores are combined with weight alpha. The authors report human-alignment experiments on local edits from MagicBrush and global edits from PIE-Bench, a ground-truth test, component ablations, and a mask-guidance application for improving editing models. The central claim is that BPM achieves the highest alignment with human evaluation among the compared metrics.

Significance. If the central claim holds, BPM would be a useful automated evaluator for instruction-based image editing, because it explicitly separates modification and preservation regions and provides interpretable component scores. The paper has several strengths: a concrete algorithmic pipeline, a human study with 960 responses, validation of parsing and mask quality, component-level ablations, and a released project page. The main caveats are that the S_modify definition is incomplete for add/remove instructions and that the balance weight alpha is tuned on a subset of the evaluation data. These issues are fixable but must be addressed before the headline alignment result can be taken at face value.

major comments (3)

[§3.2 and Alg. 2 (line 10)] For add/remove instructions, §3.2 sets o1 or o2 to 'None' and initializes masks and boxes to the counterpart. Algorithm 2 line 10 then computes S_modify as the cosine similarity between CLIP image-difference vectors and CLIPT(o2) - CLIPT(o1), but CLIPT('None') is undefined. No fallback is documented: Appendix A.4 covers f_size and f_position only, not S_modify. Because the MagicBrush sample explicitly covers 'adding objects' and 'removing objects', the reported Alignment Score of 0.799 in Table 1 must either have been computed on a subset that excludes those instructions or using an undocumented special case. Please specify the exact fallback, for example by defining S_modify for the single-object case, or exclude such samples and report a per-instruction-type breakdown.
[Appendix B.1 / Table 9 / §4.1] The balance weight alpha is set to 0.7 because it achieves the highest human-alignment score on a 100-entry subset of the same evaluation data (Table 9). Since BPM is defined by the weighted sum in Alg. 1 line 12, and since the headline comparison in Table 1 uses this tuned value, the comparison is partially circular: the metric itself has effectively been fit to the evaluation target. Please either fix alpha as a predetermined constant, tune it on a disjoint validation set, or report the sensitivity of all pairwise alignment scores over the full range of alpha.
[§3.3 and §3.2 (thresholds and localization failure)] The region-aware judge depends on an IOU threshold and an area-ratio threshold that are never specified in the main text or appendix; without these values, f_size and f_position are not reproducible, and their binary 0/1 outputs are sensitive to the chosen thresholds. Additionally, the pipeline has no specified behavior when the detector or segmentor returns no detection, multiple detections, or an incorrect segmentation for o1 or o2. The only evidence of localization quality is the 100-sample validation in §4.3 and Table 13, and Appendix B.4 notes the LLM can misparse instructions containing 'it' because it cannot see the image. Since S_region and S_semantic are computed exactly on the localized masks, such failures can change the score for affected samples and therefore the reported alignment. Please report the thresholds, define the failure fallback, and quantify the alignment after excluding or flagging failure cases.

minor comments (5)

[Alg. 2 line 12 and §3.4] The min-max normalization Norm is not defined over its support; S_modify and S_preserve are scalars per sample, so please specify whether normalization is performed per sample, per batch, or over the whole evaluation set.
[§4.1] The alignment formula uses strict inequalities 1_{H1>H2} and 1_{M1>M2}, but ties in either human or metric scores are not addressed; please state how ties are counted.
[Table 3] The reported inference speed lacks hardware and batch-size information; please add the experimental setting so the speed comparison is meaningful.
[§3.1] There is a typo in the phrase 'fpsosition(·)', which should read 'f_position(·)'.
[§4.4 / Table 8] Evaluating the mask-guidance enhancement with the BPM component scores themselves may favor the method by construction; please also report improvement using the independent user-study percentages from Table 15 and, if possible, an external metric.

Circularity Check

2 steps flagged · score 4.0 of 10

BPM has independent human-alignment grounding, but alpha is tuned on the evaluation target and the mask-guided enhancement is partly self-scored with its own mask-defined components.

fitted input called prediction [Appendix B.1 and Sec. 4.1 (Implementation Details), with reported results in Tables 1-2]
"We conduct experiments with different weight factor α between Ssemantic and Sregion, i.e., BP M= α ∗ Ssemantic + (1 − α) ∗ Sregion, the results are shown in Table. 9. Among all combinations, α = 0.7 reaches the best human evaluation alignment. ... For scaling weight α for Ssemantic, we set it to 0.7 (i.e. BPM = 0.7 ∗ Ssemantic + 0.3 ∗ Sregion)."

The scalar α is selected by maximizing human-alignment on a 100-entry subset, and the appendix does not state that this subset is held out from the human-alignment evaluation reported in Tables 1-2. Because the reported local-edit alignment is the same target used to select the weighting parameter, that portion of the headline result is an in-sample optimum rather than an independent prediction. The global-edit results and separate user studies provide independent support, so this is partial rather than complete circularity.
self definitional [Sec. 3.4 Eqs. (2)-(3), Sec. 3.5 Eq. (4), and Table 8]
"Ap_origin = [1− Morigin ∪ Medit] ⊙ Iorigin; Ap_edit = [1− Morigin ∪ Medit] ⊙ Iedit ... sT ∗ (ϵθ(zt, t, Iorigin, Tedit) − ϵθ(zt, t, Iorigin, ∅)) ⊙ Mall ... Mall = Morigin ∪ Medit."

The enhancement uses Mall = Morigin ∪ Medit to restrict instruction guidance to exactly the pixels that the preservation score excludes from its L2 comparison. Thus any method that suppresses changes outside Mall will mechanically raise Spreserve, making the 'Preservation' gains in Table 8 partly self-confirming rather than independent evidence of quality improvement. The paper mitigates this with a separate user study, but the tabulated component-score comparison is constructed from the same mask-defined regions that the guidance is designed to protect.

full rationale

The core metric is benchmarked externally against human ratings, so the central claim is not simply defined into existence. The CLIP directional similarity and masked L2 components are computed from stated image/text inputs and masks, with no load-bearing self-citation or imported uniqueness theorem. The main circularity-related concerns are (1) the balance factor α is tuned on the human-alignment target and then reported as part of the headline alignment, and (2) the mask-guided enhancement is partly evaluated with BPM's own mask-defined preservation score, so Table 8's improvements are partially enforced by construction. The global-edit comparison and independent user studies keep the central claim substantially supported, so the score is moderate rather than severe. The reviewer-flagged undefined Smodify for add/remove instructions is a specification gap and reproducibility risk, but it is not itself a circularity: no equation in the paper reduces that result to its own input by definition.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard ML assumptions about CLIP semantics, detection quality, LLM parsing reliability, and human ground truth, plus one fitted scalar alpha. No new physical or ontological entities are introduced; the only new construct is the BPM score, which is a composition of existing models and distance functions.

free parameters (3)

balance weight alpha = 0.7
Chosen as the value with best human alignment on a 100-entry subset (Appendix B.1, Table 9), then used in main experiments. This is a free parameter fitted to the evaluation target.
position saliency IOU threshold = not specified
fposition requires bounding-box IOU below or above a threshold (Sec. 3.3), but the threshold value is never given.
size saliency area ratio threshold = not specified
fsize requires area ratio greater or less than a threshold for larger and smaller, and near 1 for unchanged (Sec. 3.3), but the threshold values are not reported.

assumptions (6)

domain assumption CLIP encoders provide a semantically consistent space for measuring image and text edit directions.
Smodify uses cosine similarity between image-edit and text-edit differences in CLIP space (Eq. 1), inherited from Kim et al. 2022.
domain assumption The union of Morigin and Medit masks covers all pixels affected by the edit.
Preservation score computes L2 on the complement of Morigin union Medit (Eqs. 2-3). If an edit changes pixels outside detected masks such as shadows, reflections, or style shifts, preservation is misjudged.
domain assumption Grounding DINO and Grounded SAM localize source and target objects accurately in both images.
Region localization in Alg. 1 lines 5-6 feeds every downstream score. The paper validates on 100 samples but does not provide failure analysis for detection.
domain assumption LLM parsing reliably extracts object names and size and position states from instructions.
Line 4 of Alg. 1 and Sec. 3.2; the authors validate 97-99 percent accuracy on 100 samples, but the metric has no handling for parse failures.
domain assumption Human ratings are an appropriate ground truth for editing quality.
Human alignment tests treat pairwise human preferences as the target. This is a standard assumption for perceptual metrics.
domain assumption Pixel L2 distance is a valid measure of content preservation in irrelevant regions.
Spreserve equals 1 minus L2 between preserved crops (Alg. 2 line 11). Pixel L2 is sensitive to shifts, compression, and benign appearance changes.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing." pith.science (2026). https://pith.science/paper/XFCZWCAD

@misc{pith2026250613827,
  author       = {Pith},
  title        = {Pith review of: Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XFCZWCAD}},
  note         = {Machine review of arXiv:2506.13827}
}

read the original abstract

Instruction-based image editing, which aims to modify the image faithfully according to the instruction while preserving irrelevant content unchanged, has made significant progress. However, there still lacks a comprehensive metric for assessing the editing quality. Existing metrics either require high human evaluation costs, which hinder large-scale evaluation, or are adapted from other tasks and lose task-specific concerns, failing to comprehensively evaluate both instruction-based modification and preservation of irrelevant regions, resulting in biased evaluation. To tackle this, we introduce a new metric called Balancing Preservation and Modification (BPM), tailored for instruction-based image editing by explicitly disentangling the image into editing-relevant and irrelevant regions for specific consideration. We first identify and locate editing-relevant regions, followed by a two-tier process to assess editing quality: Region-Aware Judge evaluates whether the position and size of the edited region align with the instruction, and Semantic-Aware Judge further assesses the instruction content compliance within editing-relevant regions as well as content preservation within irrelevant regions, yielding comprehensive and interpretable quality assessment. Moreover, the editing-relevant region localization in BPM can be integrated into image editing approaches to improve editing quality, demonstrating its broad applicability. We verify the effectiveness of the BPM metric on comprehensive instruction-editing data, and the results show the highest alignment with human evaluation compared to existing metrics, indicating its efficacy. Code is available at: https://joyli-x.github.io/BPM/

Figures

Figures reproduced from arXiv: 2506.13827 by the authors.

**Figure 2.** Overall Pipeline of BPM . Firstly, LLM is utilized to parse and analyze editing instruction, generates responses to identify the source and target object, as well as the object size and position changing state requirement during editing. Then we conduct Region-Aware Judge to verify editing follows the instruction for region size and position, yielding region-aware score Sregion. For Semantic-Aware Judge, we utilize … view at source ↗

**Figure 3.** Evaluation functions for region-aware judge. Size Alignment: The pipeline is shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (11 more)

**Figure 4.** Figure 4: Visualized example of evaluation metrics comparison. 4.3. Ablation Study Several ablation studies are conducted to separately verify the effectiveness of (1) each component score in BPM . (2) LLM instruction parsing. (3) our directional similarity. Effectiveness of Com…

**Figure 5.** Figure 5: Visualized example of image editing mask guidance. Effectiveness of Directional Similarity: we compared our directional similarity with original CLIPScore, i.e., the image-text alignment of edited object. From the results in Table. 7 we can see that our directional sim…

**Figure 6.** Figure 6: Prompts for parsing instruction [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: The instructions for user study [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 8.** Figure 8: Screen-shot of our user study page. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

**Figure 9.** Figure 9: The instructions for user study of preference test [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

**Figure 10.** Figure 10: Screen-shot of our user study page for preference test. Users were asked to evaluate whether the left edited image is better than the right one from five dimensions. B.6. Additional Component Score Verification Due to limited space, the full component score verificati…

**Figure 11.** Figure 11: Visualized of human score collection procedure. By offering references for identical input (Generated image 1-4 in each line), the annotators can judge the editing quality more reasonably. On the contrary, cross-sampled score collection lacks absolute comparison. For …

**Figure 12.** Figure 12: Visualization of mask comparison. We compare our editing region mask with two approaches, GPT-4o(OpenAI, 2024) and DiffEdit(Couairon et al., 2022). For GPT-4o, we prompt it to output the bounding box coordinates of editing region; for DiffEdit, we visualize the attent…

**Figure 13.** Figure 13: Visualized Evaluation comparison among different metrics. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

**Figure 14.** Figure 14: More visualization of mask-guided editing. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Tunable Polariton Canalization in Natural van der Waals Oxide
physics.optics 2026-04 unverdicted novelty 7.0 of 10

Untreated alpha-V2O5 exhibits frequency-tunable in-plane polariton canalization with unidirectional Poynting-vector flow, mapped by infrared nano-imaging and a permittivity phase diagram.
Exposing Blindspots: Cultural Bias Evaluation in Generative Image Models
cs.CV 2025-10 conditional novelty 6.0 of 10

When countries are not named, image models default to US-like modern styles, and iterative image editing erodes cultural fidelity that CLIPScore misses but human raters and a culture-aware VQA metric catch.
Identity-Preserving Text-to-Video Generation via Training-Free Prompt, Image, and Guidance Enhancement
cs.CV 2025-09 conditional novelty 5.0 of 10

A training-free prompt, image, and guidance enhancement framework improves face consistency and video quality for identity-preserving text-to-video generation, winning the ACM Multimedia 2025 IPVG challenge.

Reference graph

Works this paper leans on

22 extracted references · 5 canonical work pages · cited by 3 Pith papers

[1]

M., Mas- siceti, D., Sanjabi, M., Hu, S

Basu, S., Saberi, M., Bhardwaj, S., Chegini, A. M., Mas- siceti, D., Sanjabi, M., Hu, S. X., and Feizi, S. Editval: Benchmarking diffusion based text-guided image editing methods. arXiv preprint arXiv:2310.02426,

arXiv
[4]

Y ., Yang, Y ., and Gan, Z

Fu, T.-J., Hu, W., Du, X., Wang, W. Y ., Yang, Y ., and Gan, Z. Guiding instruction-based image editing via multimodal large language models. arXiv preprint arXiv:2309.17102,

arXiv
[5]

Stylegan-nada: Clip-guided domain adaptation of image generators

Gal, R., Patashnik, O., Maron, H., Chechik, G., and Cohen- Or, D. Stylegan-nada: Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108.00946,

arXiv
[6]

Ace: All-round creator and editor following instructions via diffusion transformer

Han, Z., Jiang, Z., Pan, Y ., Zhang, J., Mao, C., Xie, C., Liu, Y ., and Zhou, J. Ace: All-round creator and editor following instructions via diffusion transformer. arXiv preprint arXiv:2410.00086,

arXiv
[8]

L., and Choi, Y

Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y . Clipscore: A reference-free evaluation metric for im- age captioning. arXiv preprint arXiv:2104.08718,

arXiv
[9]

Hui, M., Yang, S., Zhao, B., Shi, Y ., Wang, H., Wang, P., Zhou, Y ., and Xie, C

URL https://arxiv.org/abs/ 2207.12598. Hui, M., Yang, S., Zhao, B., Shi, Y ., Wang, H., Wang, P., Zhou, Y ., and Xie, C. Hq-edit: A high-quality dataset for instruction-based image editing,

arXiv
[10]

Johnson, J., Alahi, A., and Fei-Fei, L

URL https: //arxiv.org/abs/2404.09990. Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In ECCV,

arXiv
[11]

Could it be eggs?

From the result we can conclude that for the task of parsing instructions, current advanced LLM show similar alignment performance between metric and human evaluation. Specifically, Gemma-9B can achieve performance on par with GPT-4o. 13 Balancing Preservation and Modification: A Region and Semantic-Aware Metric for Instruction-Based Image Editing Figure ...

work page 2022

Show all 22 references

[12]

Imagic: Text-based real image editing with diffusion models

Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276,

arXiv
[13]

and Ye, J

Kim, G. and Ye, J. C. Diffusionclip: Text-guided image manipulation using diffusion models. arXiv preprint arXiv:2110.02711,

arXiv
[14]

Li, J., Li, H., Erfani, S., Feng, L., Bailey, J., and Liu, F

URL https://arxiv.org/ abs/2312.14867. Li, J., Li, H., Erfani, S., Feng, L., Bailey, J., and Liu, F. Visual-text cross alignment: Refining the similar- ity score in vision-language models. arXiv preprint arXiv:2406.02915,

arXiv
[15]

Grounding dino: Marrying dino with grounded pre-training for open-set object detection

Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499,

arXiv
[16]

https://cdn.openai.com/papers/GPTV_ System_Card.pdf/ [Accessed: 22-09-2024]. OpenAI. Introducing gpt-4o: our fastest and most afford- able flagship model,

2024
[17]

Radford, A., Kim, J

https://openai.com/ index/hello-gpt-4o/ [Accessed: 22-09-2024]. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. arXiv preprint ...

2024 arXiv
[19]

Our prompts include three parts, separately targets for object identify, size state and position state judgement. A.2. User Study for Human Alignment Test In this part, we provide details about the user study regarding the degree to which the metrics align with human ratings. ...

2024
[20]

A.3. User Study for Mask Guided Enhancement Quality To analyze whether the editing performance have improved after adding mask guidance, we conducted preference tests with three human evaluators comparing images produced by the original models and those generated by models wit...

2022
[22]

Ours” denotes edited image under our mask guidance is preferred by user, “Equal

and DiffEdit(Couairon et al., 2022). For GPT-4o, we prompt it to output the bounding box coordinates of editing region; for DiffEdit, we visualize the attention map that serves as their mask guidance during editing process. 18 Balancing Preservation and Modification: A Region ...

2022
[2016]

Direct inversion: Boosting diffusion-based editing with 3 lines of code

Ju, X., Zeng, A., Bian, Y ., Liu, S., and Xu, Q. Direct inversion: Boosting diffusion-based editing with 3 lines of code. arXiv preprint arXiv:2310.01506,

arXiv
[2021]

Hierarchical text-conditional image generation with clip latents

Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125,

arXiv
[2022]

Diffedit: Diffusion-based semantic image editing with mask guidance

Couairon, G., Verbeek, J., Schwenk, H., and Cord, M. Diffedit: Diffusion-based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11427,

arXiv
[2023]

Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800,

arXiv
[2024]

Prompt-to-prompt im- age editing with cross attention control

Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y ., and Cohen-Or, D. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626,

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

M., Mas- siceti, D., Sanjabi, M., Hu, S

Basu, S., Saberi, M., Bhardwaj, S., Chegini, A. M., Mas- siceti, D., Sanjabi, M., Hu, S. X., and Feizi, S. Editval: Benchmarking diffusion based text-guided image editing methods. arXiv preprint arXiv:2310.02426,

arXiv

[2] [4]

Y ., Yang, Y ., and Gan, Z

Fu, T.-J., Hu, W., Du, X., Wang, W. Y ., Yang, Y ., and Gan, Z. Guiding instruction-based image editing via multimodal large language models. arXiv preprint arXiv:2309.17102,

arXiv

[3] [5]

Stylegan-nada: Clip-guided domain adaptation of image generators

Gal, R., Patashnik, O., Maron, H., Chechik, G., and Cohen- Or, D. Stylegan-nada: Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108.00946,

arXiv

[4] [6]

Ace: All-round creator and editor following instructions via diffusion transformer

Han, Z., Jiang, Z., Pan, Y ., Zhang, J., Mao, C., Xie, C., Liu, Y ., and Zhou, J. Ace: All-round creator and editor following instructions via diffusion transformer. arXiv preprint arXiv:2410.00086,

arXiv

[5] [8]

L., and Choi, Y

Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y . Clipscore: A reference-free evaluation metric for im- age captioning. arXiv preprint arXiv:2104.08718,

arXiv

[6] [9]

Hui, M., Yang, S., Zhao, B., Shi, Y ., Wang, H., Wang, P., Zhou, Y ., and Xie, C

URL https://arxiv.org/abs/ 2207.12598. Hui, M., Yang, S., Zhao, B., Shi, Y ., Wang, H., Wang, P., Zhou, Y ., and Xie, C. Hq-edit: A high-quality dataset for instruction-based image editing,

arXiv

[7] [10]

Johnson, J., Alahi, A., and Fei-Fei, L

URL https: //arxiv.org/abs/2404.09990. Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In ECCV,

arXiv

[8] [11]

Could it be eggs?

From the result we can conclude that for the task of parsing instructions, current advanced LLM show similar alignment performance between metric and human evaluation. Specifically, Gemma-9B can achieve performance on par with GPT-4o. 13 Balancing Preservation and Modification: A Region and Semantic-Aware Metric for Instruction-Based Image Editing Figure ...

work page 2022