Evaluation-Verification Reward for Consistent Multi-Reference Image Editing

Chao Shen; Chenhao Lin; Lei Sun; Meng Yu; Pengfei Zhang; Xiangxiang Chu; Xiaochen Lv; Yingmao Miao

REVIEW 3 major objections 5 minor 28 references

A two-stage evaluation-verification reward filters hallucinated multimodal-judge claims and turns them into a stable reinforcement-learning signal for multi-reference image editing.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 15:05 UTC pith:4MDKTTFD

load-bearing objection A real engineering idea—verifier-filtered MLLM rewards—but the verifier's reliability is asserted, not measured, and the metrics come from the same model family as the training reward. the 3 major comments →

arxiv 2607.29025 v1 pith:4MDKTTFD submitted 2026-07-31 cs.CV

Evaluation-Verification Reward for Consistent Multi-Reference Image Editing

Yingmao Miao , Pengfei Zhang , Xiaochen Lv , Meng Yu , Lei Sun , Xiangxiang Chu , Chao Shen , Chenhao Lin This is my paper

classification cs.CV

keywords multi-reference image editingreinforcement learningmultimodal LLM evaluatorvisual grounding verificationreward modeldiffusion model fine-tuninghallucination suppressionhuman preference alignment

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the bottleneck preventing reinforcement learning from improving multi-reference image editing is the reward signal, not the editor: existing rewards cannot judge consistency and harmony across several images, and multimodal LLMs used as zero-shot judges either hallucinate during long reasoning or cannot perform the deduction needed for short judgments. The proposed EVR reward splits evaluation into five criteria and, for each criterion, asks an MLLM evaluator to produce several short rationale-plus-score hypotheses. A verifier then checks each claim against the actual pixels and keeps only the claims it can ground, producing a per-criterion weighted score and a final geometric-mean reward. With this signal, an off-the-shelf editor can be fine-tuned with RL without architectural changes, improving reference consistency, context disambiguation, instruction consistency, and harmony, and reaching a 67% human-preference win rate over the base model while matching or beating far larger closed systems. The core insight a sympathetic reader should take away is that verifying a specific factual assertion is simpler and more visual than open-ended evaluation, so hallucination-prone reasoning can be turned into a trustworthy training signal.

Core claim

EVR's central claim is that a reward can be made both detailed and trustworthy by separating generation of judgments from verification of those judgments. For each of five dimensions—reference consistency, scene consistency, visual harmony, instruction consistency, and visual quality—an MLLM evaluator writes five short rationales with scores, sampling multiple hypotheses so that valid observations survive even when some rationales hallucinate. A verifier aggregates every factual claim, checks it against the reference images, the edited image, and the instruction, and assigns a binary sign to each rationale indicating whether its claims are visually grounded. The dimension reward is the avera

What carries the argument

The load-bearing mechanism is the evaluator-verifier pair inside EVR. The evaluator is an MLLM prompted to judge one dimension at a time with short chains of thought, generating multiple independent hypotheses so that even when some are hallucinated, others capture true visual facts. The verifier—drawing on the same MLLM family—extracts each factual claim and decides whether it is supported by the pixels, enabling claim-level grounding that filters text-induced hallucination. The reward is then the geometric mean of five dimension rewards, each a credibility-weighted average of evaluator scores, with weights equal to the verifier's binary accept/reject signs. A data pipeline that assembles s

Load-bearing premise

The method assumes the Verifier can reliably tell, from the pixels, whether a textual claim made by the Evaluator is true—even though both are drawn from the same MLLM family, so the Verifier may inherit the same text-driven hallucination it is supposed to filter.

What would settle it

Take a held-out set of edited images with deliberately planted errors (a rotated watch face, a floating object, background bleed), generate evaluator rationales, and have the Verifier label each claim true or false; compute verifier accuracy separately for each of the five dimensions. If accuracy is near chance on harmony or instruction-consistency claims, or if the verifier accepts plausible-sounding false claims as readily as it rejects them, then the verified reward carries no signal and the reported RL gains should disappear when it is used alone.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

An existing open editor can be preference-aligned through RL with only a low-rank adapter, improving exactly the dimensions (reference consistency, harmony, instruction following) that generic rewards miss.
A small 8-billion-parameter evaluator with the verifier produces editing quality comparable to larger unverified evaluators, so verification can substitute for raw model scale.
The geometric-mean aggregation makes reward hacking harder: copying the reference scene inflates one dimension but collapses others, so the total reward penalizes the shortcut.
Training on synthetic two-reference tuples transfers to real-world images and to settings with more than two references, up to the base editor's generative limits.
The EVR reward aligns with human preference better than five alternative MLLM reward formulations, giving a fine-grained signal suitable for online RL rather than just a metric.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same evaluate-then-verify recipe could be carried to other generation domains where MLLM judges are known to hallucinate—text-to-video, 3D, or audio—where a claim-level grounding step may filter the same failure mode.
The paper's non-i.i.d. finding (the evaluator repeats similar hallucinations rather than varying independently) implies that averaging scores is structurally the wrong fix and that verifier accuracy, not evaluator diversity, is the binding constraint on reward quality—a claim the paper does not fully test.
Because the method cannot rescue the base model when five or more references trigger denoising collapse, coupling EVR with stronger base generators is a natural next test; the paper stops at the current editor's prior.
Replacing the same-family MLLM verifier with an independently trained or human-supervised grounding model would directly test whether the gains come from verification itself or from the verifier sharing the evaluator's biases.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A real engineering idea—verifier-filtered MLLM rewards—but the verifier's reliability is asserted, not measured, and the metrics come from the same model family as the training reward.

read the letter

The paper has a genuinely useful idea: instead of trusting an MLLM's open-ended evaluation, have it produce multiple rationales and then use a second pass to check each claim against the actual pixels. Applied to multi-reference editing, that's a clean way to make reward signals more discriminative. The synthetic data pipeline for N=2 editing is also practical, and the ablations are honest—removing the verifier or CoT does hurt, and the paper openly discusses failure cases at N≥5. Credit where it's due: the qualitative improvements in consistency and harmony are visible, and the 67% user-study win rate over the base model is a real result, even if the study is small (12 experts, no significance testing).

But the central mechanism is under-validated. The paper claims that verification is 'fundamentally simpler and more visually grounded' than open-ended evaluation, yet it never directly measures verifier precision, recall, or error patterns. The human-alignment number (0.707) is for the full reward, not for the verifier's accept/reject decisions. The top-n availability rate (89% at K=5) tells you that at least one claim usually passes, not that the passed claims are correct. And since the verifier is the same MLLM family as the evaluator, it could inherit the same text-induced hallucinations it's meant to filter. This isn't a fatal flaw—the ablations suggest EVR helps even with a small verifier—but the load-bearing assumption needs direct evidence.

There's also a circularity problem. The policy is optimized against Qwen-based EVR and then judged by Qwen3.5-plus on the same five dimensions. Some of the reported gain may just be alignment between the reward and the judge's biases. A human study partially breaks this, but the main tables are MLLM scores. And Table 1 and Table 3 report different numbers for the same base model (e.g., Scene Consistency 4.62 vs 4.37, Harmony 2.22 vs 3.34) without clarification—that's the kind of inconsistency that makes reviewers lose trust. The fallback to unverified mean scores when no claim passes (11% of samples at K=5) also reintroduces the exact noise the method is designed to remove.

Should it go to review? Yes. The mechanism is novel and the experiments, despite the gaps, support the qualitative conclusions. But the authors need to add a direct verifier error analysis, use an external reward model for the main metrics, fix the table inconsistencies, and release code/data. Without those, the strongest claims about 'reliable and fine-grained reward signals' remain unproven. If you're working in this area, reading it is worthwhile; citing it, I'd wait for a revised version with cleaner evidence.

Referee Report

3 major / 5 minor

Summary. The paper proposes EVR, a multi-dimensional MLLM-based reward for RL fine-tuning of multi-reference image editing. An Evaluator generates K rationale-and-score pairs per dimension across five criteria; a Verifier accepts or rejects each rationale based on visual grounding; verified scores are aggregated by a geometric mean. Using DiffusionNFT and a synthetic data pipeline, the authors fine-tune Qwen-Image-Edit and report improvements over the base model, with qualitative examples and a user study showing preference over baselines and NanoBanana. The paper includes prompt templates, implementation details, and a discussion of failure cases at large N.

Significance. EVR targets a genuine bottleneck: the lack of reliable, multi-dimensional reward signals for multi-reference image editing. If the verifier assumption holds, the two-stage Evaluator-Verifier design is a practical way to suppress hallucination in MLLM-based rewards, and the synthetic data pipeline is scalable and generally useful. Strengths include a human preference study, transparent prompt templates, and explicit acknowledgment of failure cases for N≥5. However, the central mechanism is not directly validated: the Verifier's decisions are never audited, and several quantitative reporting inconsistencies undermine confidence in the main claims. The contribution is plausible and potentially significant but not yet fully established.

major comments (3)

[§4.4, Tables 1 and 3] The same configuration (Qwen-Image-Edit-2509 base; +EVR-RL(Qwen3-VL-8B)) is reported with strikingly different numbers: Table 1 gives RC 4.36, H 2.63, Score 0.68, whereas Table 3 gives RC 4.15, H 4.32, Score 0.79. Section 4.1 says that only 'metrics in main results' use the qwen3.5-plus API; the caption of Table 3 does not state the metric source. If Table 3 uses the local Qwen3-VL-8B, then the ablation results are confounded with the model used for the training reward and cannot be compared to Table 1. The metric source for every table must be stated and the discrepancy explained; this is load-bearing because the ablations are intended to support the mechanism.
[§3.3.2 and Appendix A.2] The central claim that the Verifier grounds evaluator claims in pixels and produces reliable rewards is not directly tested. The Evaluator and Verifier are the same MLLM family (Qwen3-VL-8B), and Appendix A.2 documents that evaluator hypotheses are non-i.i.d. and systematically biased. If the Verifier inherits the same text priors, it can accept hallucinated rationales. The paper's evidence is indirect (Table 2 human alignment, Table 3 w/o Verifier, Table 5 availability). A manual error audit of Verifier accept/reject decisions—precision, recall, error patterns—is needed, plus an analysis showing that verification changes scores in the intended direction. Without this, the abstract's claim of 'reliable and fine-grained reward signals' is not established.
[§3.3.3 Eq. (9) and Appendix A.1] Eq. (9) defines r_d = sum(v_i S_i)/(sum v_i + ε), which gives r_d ≈ 0 when no claim passes verification. Appendix A.1 states that the implementation falls back to the mean candidate score when verification fails. The paper does not give the exact fallback formula or its effect on the gradient, and Table 5 shows this fallback applies to ~11% of samples at K=5. Since the fallback reintroduces unverified scores, the method as implemented differs from the stated reward. The full reward specification must be given, and the impact of the fallback on the training signal analyzed.

minor comments (5)

[Abstract and §1] The title and text repeatedly write 'Evaluation–erification Reward' instead of 'Evaluation–Verification Reward' (missing 'V').
[Appendix B.5] The text references 'our Limitations section,' but no Limitations section appears in the manuscript. Add one or fix the cross-reference.
[§3.2] The dataset is described as 10K triplets with 1K used as a test set; the number of triplets used for RL training is not stated explicitly. Please clarify.
[§4.3, Table 2] The human-alignment comparison lacks details: dataset size, sample distribution, annotation protocol, and whether the raters overlap with those in §4.5. Without these, the 0.707 alignment is difficult to interpret.
[§4.5 and Appendix B.2] User-study win rates are reported as point estimates without confidence intervals or significance tests; the abstract's '67%' should be reconciled with the '65.0%' and '67.0%' figures in B.2.

Circularity Check

0 steps flagged

No significant circularity: EVR reward and RL pipeline are self-contained; limitations are unvalidated assumptions, not circular reductions.

full rationale

EVR's reward (Eq. 9) is computed from Evaluator rationales/scores and Verifier accept/reject signs; the policy is optimized with DiffusionNFT (Eq. 4) using that reward. No parameter is fit to the reported main-result quantities, and no target result is assumed into the reward by construction. The Evaluator and Verifier are zero-shot MLLMs, not trained on the test set. Validation includes independent components: human alignment (Table 2), a 12-expert user study (Sec. 4.5, B.2), and main metrics computed with a different judge (qwen3.5-plus API) than the training-time judge (Qwen3-VL-8B). The paper's own self-citations (Miao et al. 2025; Yu et al. 2026) appear only as related-work support and are not load-bearing for the central derivation. The main weakness—that Verifier correctness is assumed and the Verifier shares the Evaluator's MLLM family—is an empirical correctness/robustness concern, not a circular reduction: no equation equates the reported improvement to the reward inputs. Therefore no specific circular step can be substantiated.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The main contributions are design choices and domain assumptions rather than fitted constants. The reward includes hand-set parameters (K, criteria set, epsilon, normalization) and several assumptions about MLLM reliability and synthetic-data coherence; no new physical entities are introduced.

free parameters (5)

K (number of evaluator hypotheses per dimension) = 5
Chosen via an availability ablation on 500 samples (Appendix A.2, Table 5); higher K raises verifier CoT length and cost, so 5 is a hand-tuned trade-off that shapes the reward.
Evaluation dimension set = RC, SC, H, IC, Q
The five criteria are hand-selected by the authors (Section 3.3.1); the reward and the main evaluation metrics both depend on this choice.
Stability constant epsilon = small constant (unspecified)
Used in the denominator of Eq. (9) for numerical stability; value not reported.
Reward normalization = R_final = (R-1)/4
Maps the 1-5 dimension scores to [0,1] for DiffusionNFT (Appendix A.1); arbitrary linear scaling.
DiffusionNFT guidance strength (1/beta) = 1.0
Hyperparameter from DiffusionNFT (Appendix A.1); affects policy update magnitude though not the EVR criterion itself.

axioms (6)

domain assumption An MLLM Verifier can reliably decide whether an Evaluator's factual claim is supported by the pixels, and this verification is easier than open-ended evaluation.
Stated in Section 3.3.2 and used to construct Eq. (8); the whole EVR mechanism depends on it, but no systematic error analysis is provided.
domain assumption Qwen3-VL-8B's visual judgments, after verification, align with human preferences for multi-reference editing.
Empirical claim supported only by the human-alignment table (Table 2) and user study (B.2); no confidence intervals.
domain assumption The geometric mean of five dimension rewards prevents reward hacking by requiring all dimensions to be high.
Proposed in Section 3.3.3 following Luo et al.; the w/o Dim ablation supports sensitivity but no formal guarantee.
domain assumption Synthetic object/scene tuples generated by GPT + Z-Image + inpainting are semantically coherent enough for RL.
Section 3.2 relies on this to create 10K training triples; arbitrary mismatches would derail policy optimization.
standard math Flow Matching/Rectified Flow and DiffusionNFT objectives from cited works are correct.
Equations (1)-(6) are imported from prior literature and assumed to hold.
ad hoc to paper Verification failure falls back to the unverified mean candidate score.
Appendix A.1: 'Upon verifier failure, we adopt a fallback strategy of using the mean candidate score'; this reintroduces the unverified signal exactly when the verifier finds no grounded claim.

pith-pipeline@v1.3.0-daily-deepseek · 17292 in / 17446 out tokens · 156281 ms · 2026-08-03T15:05:46.379799+00:00 · methodology

0 comments

read the original abstract

While recent image editing models have made rapid progress, multi-reference editing remains challenging, particularly in maintaining visual consistency across references and ensuring overall visual harmony. Reinforcement learning has proven highly effective for text-to-image generation and single-image editing, but its extension to multi-reference editing is hindered by the absence of suitable reward models that capture multi-image relational constraints. Moreover, naively using multimodal large language models(MLLMs) as zero-shot evaluators faces a key tension between hallucination-prone long-form reasoning and the limited deductive power of short-form judgments. We address these issues with a Multi-dimensional Evaluation-Verification Reward(EVR). EVR decomposes evaluation into distinct visual criteria; for each criterion, an MLLM Evaluator generates multiple candidate hypotheses, and a Verifier grounds each claim in concrete visual evidence to accept or reject it, producing reliable and fine-grained reward signals. Together with a scalable data pipeline, our method enables RL fine-tuning of off-the-shelf editors without architectural changes. Extensive experiments show substantial gains over the base Qwen-Image-Edit, improving consistency and harmony to match or surpass NanoBanana.

discussion (0)

Reference graph

Works this paper leans on

28 extracted references

[1]

EVR for Multi-Reference Image Editing 20

No Match: The object from Reference Image is completely absent in Edited Image, or a different object (different category or identity) appears in its place. EVR for Multi-Reference Image Editing 20
[2]

a chair” instead of “the red office chair

Weak Match: An object of the same general category is present (e.g., “a chair” instead of “the red office chair”), but key identifying features (color, shape, style, brand, or distinctive markings) are missing or altered beyond recognition
[3]

Excessive structure or ghosting appears

Partial Match: The correct object is likely intended, but critical attributes are inconsistent (e.g., wrong color, material changed, major design elements missing), making it visually distinct from the reference. Excessive structure or ghosting appears
[4]

Match: The object in Edited Image closely matches the one in Reference Image in type, form, color, texture, and distinctive details; minor differences due to lighting, viewpoint, partial occlusion, or editing artifacts are acceptable
[5]

Response Format (Please specify the reason for the deduction within the <Reason> tag

Exact Match: The object in Edited is visually faithful to the one in Reference Image across all defining characteristics—same model, condition, color, pose, and unique identifiers—with only natural variations expected from image editing (e.g., perspective shift, brightness adjustment). Response Format (Please specify the reason for the deduction within th...
[6]

Inconsistent: The non-edited parts of the scene are fundamentally altered—e.g., indoor/outdoor switch, distorted room geometry, warped background, or human anatomy broken; clear contamination from the source object’s context
[7]

Mostly Inconsistent: Scene type is recognizable, but major artifacts in unchanged areas: severe pose distortion, mismatched global lighting, perspective shift, or visible background bleed affecting realism
[8]

Partially Consistent: Core structure is preserved, but subtle issues in non-edited regions—e.g., slight pose drift, inconsistent ambient occlusion, minor texture warping, or soft lighting mismatches—reduce plausibility
[9]

Only negligible imperfections (e.g., tiny shadow edge discrepancy) that don’t break immersion

Mostly Consistent: Unchanged content (scene + subject) remains visually stable; lighting, perspective, and materials align well. Only negligible imperfections (e.g., tiny shadow edge discrepancy) that don’t break immersion
[10]

No trace of external contamination or structural alteration

Fully Consistent: The entire non-edited portion is indistinguishable from the original—perfect preservation of pose, environment, lighting, and style. No trace of external contamination or structural alteration. Response Format (Please specify the reason for the deduction within the <Reason> tag. Respond to the score directly within the <Score> tag.): <Re...
[11]

collage effect

Poor Harmony: The object appears obviously pasted in—sharp cut-out edges, physically impossible positions or poses, implausible scale/proportions for the setting, or semantically absurd placements. collage effect
[12]

These issues significantly detract from realism despite attempts at blending

Fair Harmony: While the object could belong to the plausible category for the scene, there are noticeable issues related to physical rules or common sense, such as objects intersecting unnaturally (clipping issues), or unrealistic interactions between objects. These issues significantly detract from realism despite attempts at blending
[13]

Ordinary Harmony: Objects are placed generally conform to the physical logic of the scene. However, it may violate the patterns of daily use, such as slight clipping artifacts or unrealistic object interactions (such as EVR for Multi-Reference Image Editing 21 inappropriate wearing patterns, unreasonable orientations, etc., positions that do not conform t...
[14]

Aspects like lighting direction, shadow softness, and color grading are also well-matched

Good Harmony: The placement and interaction of the object closely follow physical laws and common usage patterns, with only minute flaws that might escape notice by most viewers. Aspects like lighting direction, shadow softness, and color grading are also well-matched. Only expert scrutiny might reveal editing traces
[15]

There are no visual cues suggesting compositing—it reads as a single, authentic photograph

Excellent Harmony: The inserted object is indistinguishable from native scene content, perfectly aligned with physical rules, daily use patterns, and visual consistency. There are no visual cues suggesting compositing—it reads as a single, authentic photograph. Response Format (Please specify the reason for the deduction within the <Reason> tag. Respond t...
[16]

place a coffee cup on the table

Bad Execution: The editing instruction is largely ignored or misinterpreted. The wrong object is inserted, placed in an irrelevant location, or depicted in a way that contradicts the directive (e.g., instructed to “place a coffee cup on the table” but a teapot appears on the floor). Severe violations of physics or context further undermine fidelity
[17]

Poor Execution: The general intent of the instruction is recognizable, but key details are inaccurate or incomplete—wrong orientation, incorrect scale, missing required interactions (e.g., cup floating above table instead of resting on it), or partial fulfillment (only part of a multi-step instruction is followed)
[18]

Minor deviations exist—such as slight misalignment, weak contact with supporting surfaces, or subtle inconsistencies in usage context—but the core request is fulfilled

Fair Execution: The instruction is mostly followed correctly: the right object appears in the specified location with appropriate orientation and basic interaction. Minor deviations exist—such as slight misalignment, weak contact with supporting surfaces, or subtle inconsistencies in usage context—but the core request is fulfilled
[19]

Object identity, position, pose, and interaction align closely with the directive

Good Execution: The instruction is executed with high precision. Object identity, position, pose, and interaction align closely with the directive. Physical contact, support, and spatial relationships appear natural. Visual blending (lighting, shadows, color) supports realism, though tiny imperfections may remain upon close inspection
[20]

Every aspect—object type, placement, orientation, physical interaction (e.g., proper support, no clipping), and semantic context—matches the directive exactly

Excellent Execution: The instruction is implemented flawlessly. Every aspect—object type, placement, orientation, physical interaction (e.g., proper support, no clipping), and semantic context—matches the directive exactly. The result looks like a native part of the original photograph, with no detectable errors in logic, physics, or execution. Response F...
[21]

Bad: Extremely blurry, underexposed with significant noise, indiscernible subjects, collapsed subjects, and chaotic composition
[22]

Poor: Noticeable blur, poor lighting, washed-out colors, and awkward composition with cut-off subjects, incomplete object
[23]

Fair: In focus with adequate lighting, dull colors, decent composition but lacks creativity
[24]

Good: Sharp, good exposure, vibrant colors, thoughtful composition with a clear focal point
[25]

Response Format (Please specify the reason for the deduction within the <Reason> tag

Excellent: Exceptional clarity, perfect exposure, rich colors, masterful composition with emotional impact. Response Format (Please specify the reason for the deduction within the <Reason> tag. Respond to the score directly within the <Score> tag.): <Reason>...</Reason><Score>1-5</Score> EVR for Multi-Reference Image Editing 22 C.2 Verifier Prompt The fol...
[26]

**Aggregate Claims**: Extract every factual claim, observation, or alleged flaw mentioned across all <Reason> sections
[27]

The deduction statements regarding wearing patterns and physical laws in each evaluation may not be groundless

**Verify Against Evidence**: According to the edited image, determine which of these claims are actually true, false, or unsupported. The deduction statements regarding wearing patterns and physical laws in each evaluation may not be groundless. Please treat these evaluations with caution and carefully examine them
[28]

Only if all conditions are met should the evaluation be marked as valid (1)

**Evaluate Each Assessment**: For each original evaluation, check: - Whether its reasoning relies only on verified facts, - Whether it correctly applies the rubric definitions, - Whether its final score logically follows from accurate observations, - Whether it fails to mention a critical flaw that is both present and relevant to the rubric. Only if all c...

[1] [1]

EVR for Multi-Reference Image Editing 20

No Match: The object from Reference Image is completely absent in Edited Image, or a different object (different category or identity) appears in its place. EVR for Multi-Reference Image Editing 20

[2] [2]

a chair” instead of “the red office chair

Weak Match: An object of the same general category is present (e.g., “a chair” instead of “the red office chair”), but key identifying features (color, shape, style, brand, or distinctive markings) are missing or altered beyond recognition

[3] [3]

Excessive structure or ghosting appears

Partial Match: The correct object is likely intended, but critical attributes are inconsistent (e.g., wrong color, material changed, major design elements missing), making it visually distinct from the reference. Excessive structure or ghosting appears

[4] [4]

Match: The object in Edited Image closely matches the one in Reference Image in type, form, color, texture, and distinctive details; minor differences due to lighting, viewpoint, partial occlusion, or editing artifacts are acceptable

[5] [5]

Response Format (Please specify the reason for the deduction within the <Reason> tag

Exact Match: The object in Edited is visually faithful to the one in Reference Image across all defining characteristics—same model, condition, color, pose, and unique identifiers—with only natural variations expected from image editing (e.g., perspective shift, brightness adjustment). Response Format (Please specify the reason for the deduction within th...

[6] [6]

Inconsistent: The non-edited parts of the scene are fundamentally altered—e.g., indoor/outdoor switch, distorted room geometry, warped background, or human anatomy broken; clear contamination from the source object’s context

[7] [7]

Mostly Inconsistent: Scene type is recognizable, but major artifacts in unchanged areas: severe pose distortion, mismatched global lighting, perspective shift, or visible background bleed affecting realism

[8] [8]

Partially Consistent: Core structure is preserved, but subtle issues in non-edited regions—e.g., slight pose drift, inconsistent ambient occlusion, minor texture warping, or soft lighting mismatches—reduce plausibility

[9] [9]

Only negligible imperfections (e.g., tiny shadow edge discrepancy) that don’t break immersion

Mostly Consistent: Unchanged content (scene + subject) remains visually stable; lighting, perspective, and materials align well. Only negligible imperfections (e.g., tiny shadow edge discrepancy) that don’t break immersion

[10] [10]

No trace of external contamination or structural alteration

Fully Consistent: The entire non-edited portion is indistinguishable from the original—perfect preservation of pose, environment, lighting, and style. No trace of external contamination or structural alteration. Response Format (Please specify the reason for the deduction within the <Reason> tag. Respond to the score directly within the <Score> tag.): <Re...

[11] [11]

collage effect

Poor Harmony: The object appears obviously pasted in—sharp cut-out edges, physically impossible positions or poses, implausible scale/proportions for the setting, or semantically absurd placements. collage effect

[12] [12]

These issues significantly detract from realism despite attempts at blending

Fair Harmony: While the object could belong to the plausible category for the scene, there are noticeable issues related to physical rules or common sense, such as objects intersecting unnaturally (clipping issues), or unrealistic interactions between objects. These issues significantly detract from realism despite attempts at blending

[13] [13]

Ordinary Harmony: Objects are placed generally conform to the physical logic of the scene. However, it may violate the patterns of daily use, such as slight clipping artifacts or unrealistic object interactions (such as EVR for Multi-Reference Image Editing 21 inappropriate wearing patterns, unreasonable orientations, etc., positions that do not conform t...

[14] [14]

Aspects like lighting direction, shadow softness, and color grading are also well-matched

Good Harmony: The placement and interaction of the object closely follow physical laws and common usage patterns, with only minute flaws that might escape notice by most viewers. Aspects like lighting direction, shadow softness, and color grading are also well-matched. Only expert scrutiny might reveal editing traces

[15] [15]

There are no visual cues suggesting compositing—it reads as a single, authentic photograph

Excellent Harmony: The inserted object is indistinguishable from native scene content, perfectly aligned with physical rules, daily use patterns, and visual consistency. There are no visual cues suggesting compositing—it reads as a single, authentic photograph. Response Format (Please specify the reason for the deduction within the <Reason> tag. Respond t...

[16] [16]

place a coffee cup on the table

Bad Execution: The editing instruction is largely ignored or misinterpreted. The wrong object is inserted, placed in an irrelevant location, or depicted in a way that contradicts the directive (e.g., instructed to “place a coffee cup on the table” but a teapot appears on the floor). Severe violations of physics or context further undermine fidelity

[17] [17]

Poor Execution: The general intent of the instruction is recognizable, but key details are inaccurate or incomplete—wrong orientation, incorrect scale, missing required interactions (e.g., cup floating above table instead of resting on it), or partial fulfillment (only part of a multi-step instruction is followed)

[18] [18]

Minor deviations exist—such as slight misalignment, weak contact with supporting surfaces, or subtle inconsistencies in usage context—but the core request is fulfilled

Fair Execution: The instruction is mostly followed correctly: the right object appears in the specified location with appropriate orientation and basic interaction. Minor deviations exist—such as slight misalignment, weak contact with supporting surfaces, or subtle inconsistencies in usage context—but the core request is fulfilled

[19] [19]

Object identity, position, pose, and interaction align closely with the directive

Good Execution: The instruction is executed with high precision. Object identity, position, pose, and interaction align closely with the directive. Physical contact, support, and spatial relationships appear natural. Visual blending (lighting, shadows, color) supports realism, though tiny imperfections may remain upon close inspection

[20] [20]

Every aspect—object type, placement, orientation, physical interaction (e.g., proper support, no clipping), and semantic context—matches the directive exactly

Excellent Execution: The instruction is implemented flawlessly. Every aspect—object type, placement, orientation, physical interaction (e.g., proper support, no clipping), and semantic context—matches the directive exactly. The result looks like a native part of the original photograph, with no detectable errors in logic, physics, or execution. Response F...

[21] [21]

Bad: Extremely blurry, underexposed with significant noise, indiscernible subjects, collapsed subjects, and chaotic composition

[22] [22]

Poor: Noticeable blur, poor lighting, washed-out colors, and awkward composition with cut-off subjects, incomplete object

[23] [23]

Fair: In focus with adequate lighting, dull colors, decent composition but lacks creativity

[24] [24]

Good: Sharp, good exposure, vibrant colors, thoughtful composition with a clear focal point

[25] [25]

Response Format (Please specify the reason for the deduction within the <Reason> tag

Excellent: Exceptional clarity, perfect exposure, rich colors, masterful composition with emotional impact. Response Format (Please specify the reason for the deduction within the <Reason> tag. Respond to the score directly within the <Score> tag.): <Reason>...</Reason><Score>1-5</Score> EVR for Multi-Reference Image Editing 22 C.2 Verifier Prompt The fol...

[26] [26]

**Aggregate Claims**: Extract every factual claim, observation, or alleged flaw mentioned across all <Reason> sections

[27] [27]

The deduction statements regarding wearing patterns and physical laws in each evaluation may not be groundless

**Verify Against Evidence**: According to the edited image, determine which of these claims are actually true, false, or unsupported. The deduction statements regarding wearing patterns and physical laws in each evaluation may not be groundless. Please treat these evaluations with caution and carefully examine them

[28] [28]

Only if all conditions are met should the evaluation be marked as valid (1)

**Evaluate Each Assessment**: For each original evaluation, check: - Whether its reasoning relies only on verified facts, - Whether it correctly applies the rubric definitions, - Whether its final score logically follows from accurate observations, - Whether it fails to mention a critical flaw that is both present and relevant to the rubric. Only if all c...