{"id":"bb6cd969-832c-4136-9f6f-6d8aef8aa9eb","arxiv_id":"2507.07317","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An automatically generated training dataset and a fine-tuned LLaVA-NeXT model produce an image editing evaluation scorer that aligns with human preference and serves as a reward model for improving editing models.","lead":"The paper builds an automatic pipeline that generates over 100,000 training examples for judging how well an image edit follows an instruction, then fine-tunes an 8-billion-parameter vision-language model to score edits. The resulting scorer beats open-source VLMs and Gemini-Pro 1.5 on several benchmarks, and can be used as a reward model to improve an editing model's outputs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AURORA-Bench evaluation likely overlaps ADIEE training data; the headline gains may reflect memorization rather than learned evaluation.","rationale":"The reader identified heuristic label fidelity (Sec. 3.1 and Eq. 4) as the weakest assumption and recommended conditional acceptance pending label ablations. I partially agree, but the most load-bearing vulnerability is leakage: the paper's own Appendix C names MagicBrush and Emu-Edit as AURORA-Bench sources, and Section 4.1 uses MagicBrush and Emu-Edit test images to build training data. No split or deduplication is reported, so the benchmark superiority and reward-model gains may be inflated by memorization rather than by a genuinely learned evaluation rule. This concern is distinct from the heuristic-label issue because it attacks the validity of the evaluation itself. The concrete overlap check would settle it. Since the reader already chose CONDITIONAL and the concern is serious but not yet proven, the verdict remains CONDITIONAL pending the leakage test; if nontrivial overlap is found, the verdict should move to REJECT.","tokens_in":21201,"tokens_out":5868,"duration_ms":66834,"concrete_test":"Compute the overlap between the ADIEE training set and AURORA-Bench point-wise/pair-wise evaluation samples: for each AURORA-Bench sample, match input images by perceptual hash (or exact file hash when available) and match normalized edit instructions by exact/approximate string match against the 9,935 MagicBrush samples, 1,100 Emu-Edit instructions, and 21,382 SEED sequences. If any overlap exists, rerun Spearman correlation and pairwise accuracy on only the disjoint AURORA-Bench subset. If the disjoint-subset correlation drops below the reported 0.4734 toward GPT-4o levels, or if the pairwise accuracy falls below 55.56, the headline gains are not evidence of generalization. Report the overlap count and the disjoint-subset metrics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that automatically generated, heuristically labeled data trains a scorer that generalizes beyond its training sources. That claim requires the evaluation benchmarks to be disjoint from the training set. Section 4.1 builds ADIEE training data from 9,935 MagicBrush samples, 1,100 instructions on Emu-Edit test images, and 21,382 SEED-Data-Edit-Part3 multi-turn sequences. Appendix C states that AURORA-Bench point-wise/pair-wise samples are collected from MagicBrush, Action-Genome, Something-Something, Epic-Kitchen, Kubric, CLEVR, WhatsUp, and Emu-Edit. Thus the benchmark explicitly contains samples from two of the exact sources used to construct the training set. The paper reports no deduplication, no train/eval split by source, and no overlap analysis. If AURORA-Bench reuses MagicBrush or Emu-Edit images/instructions that appear in ADIEE training, the scorer can memorize the edit and its expected score, inflating the reported 0.4734 Spearman correlation, the 4.75% pairwise gain, and the reward-model improvement. This is more load-bearing than the heuristic-label concern: even a perfectly human-aligned labeler cannot rescue the central claim if the test set is contaminated. The appendix also shows GPT-4o with 1-shot reaches 0.4779 on AURORA point-wise, slightly above ADIEE's 0.4734, so the state-of-the-art comparison is already protocol-sensitive and needs scrutiny.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ADIEE, a pipeline that automatically constructs a large training dataset for instruction-guided image editing evaluation by (i) generating edited outputs with nine editing methods on MagicBrush and Emu-Edit images and assigning heuristic scores, and (ii) extracting multi-turn edit sequences from SEED-Data-Edit-Part3. The authors fine-tune LLaVA-NeXT-8B with a special score token and an MLP decoder to predict a 0–10 quality score, and evaluate the resulting scorer on ImagenHub, GenAI-Bench, and AURORA-Bench. They also use the scorer as a reward model to fine-tune MagicBrush, reporting an improvement in the scorer's own evaluation on ImagenHub. The paper claims state-of-the-art alignment with human ratings across all benchmarks.","tokens_in":21567,"tokens_out":9938,"duration_ms":95310,"significance":"If the claims hold, ADIEE offers a scalable, transparent alternative to proprietary VLM judges, and a practical reward model for improving editing models. The paper releases code and models and includes a human preference study, which are strengths. However, the central generalization claim is currently threatened by (a) likely overlap between ADIEE's training sources and AURORA-Bench, (b) inconsistent numbers across the main text and appendix, and (c) a protocol-sensitivity issue where a 1-shot GPT-4o baseline exceeds the proposed scorer on AURORA-Bench point-wise. These issues are load-bearing for the headline claims and require substantial revision.","major_comments":[{"comment":"The training set is built from 9,935 MagicBrush samples, 1,100 instructions on Emu-Edit test images, and 21,382 SEED-Data-Edit-Part3 sequences (Section 4.1). Appendix C states that AURORA-Bench point-wise and pair-wise samples are collected from MagicBrush and Emu-Edit, among other sources. The paper reports no deduplication or overlap analysis between ADIEE training data and the AURORA-Bench evaluation sets. If the same images or instructions appear in both, the reported Spearman correlation (0.4734) and pairwise accuracy gains could reflect memorization rather than generalized evaluation ability. The authors should quantify the overlap and re-evaluate on the subset of AURORA-Bench that does not stem from training sources (e.g., video-derived samples), and report the correlation on that subset separately.","section":"§4.1 and Appendix C"},{"comment":"There are major numerical inconsistencies for the same ADIEE scorer across the manuscript. Table 2 reports GenAI-Bench accuracy 59.96 and AURORA-Bench pair-wise 55.56; Table 5 reports 59.41 and 52.38 for the same conditions; Appendix Table 7 reports 59.41 and 52.88. The abstract's +4.75% gain on AURORA pair-wise is only consistent with the main-text Table 2 value of 55.56, not with the appendix value. Similarly, Section 4.4 reports that reward-conditioned MagicBrush improves the ImagenHub score from 5.90 to 6.43, while Section 4.5 mentions a score of 6.67 for 'our method' and 4.27 for the without-reward-conditioning variant. These discrepancies must be reconciled and the source of variation (e.g., different evaluation subsets, random seeds, or post-processing) must be stated explicitly.","section":"Tables 2, 5, and Appendix Table 7"},{"comment":"Appendix Table 6 shows that GPT-4o with a 1-shot in-context example reaches 0.4779 Spearman correlation on AURORA-Bench point-wise, which is higher than the proposed ADIEE scorer's 0.4734. The main text, however, compares against zero-shot baselines and claims a +17.24% improvement over state-of-the-art. This makes the SOTA claim protocol-sensitive. The authors should either include the 1-shot baselines in the main tables, or justify why zero-shot is the appropriate comparison and temper the abstract and conclusion accordingly.","section":"Appendix Table 6 and Abstract"},{"comment":"The reward-model improvement on ImagenHub (+8.98%, from 5.90 to 6.43) is measured by the ADIEE scorer itself, which is the very model being proposed. The human pairwise study in Table 4 provides some external validation, but it is not directly linked to the ImagenHub score improvement and lacks any significance test. In addition, the count column in Table 4 (914 + 752 + 1190 = 2856) does not match the stated protocol of 56 participants x 50 comparisons (2800). The authors should report human-rated quality scores for the baseline and fine-tuned models, or at least provide a statistical test on the preference counts, and correct the count mismatch.","section":"§3.4 and §4.4, Tables 3–4"},{"comment":"The heuristic score assignments are strong and untested assumptions. Section 3.1 assigns score 0 to all outputs from DiffEdit, Pix2Pix-Zero, SDEdit, and Text2LIVE, and uses a CLIP-D threshold of tau = 0.2 to identify partial edits. Equation (4) assigns intermediate multi-turn edits a score proportional to the fraction of completed instructions, and assigns 0.5 to over-edited outputs. These rules are not validated against human perception. Since ImagenHub and AURORA-Bench include outputs from some of these same editing methods, the scorer could partially align with human ratings by learning method-specific biases. A small human-annotation study on a sample of the generated training data, or an ablation showing that the results are stable to changes in these thresholds, is needed to support the claim that the labels track human judgment.","section":"§3.1 and Eq. (4)"}],"minor_comments":[{"comment":"The introduction states that 'no public training datasets exist to fine-tune open-source VLMs' for image-editing evaluation, but Related Work Section 2.3 cites OmniEdit and RewardEdit20K, which are public training datasets built with proprietary models. The claim should be qualified to refer to human-annotated or open-label datasets.","section":"Abstract and §2.3"},{"comment":"The sentence 'we apply a CLIP-D threshold of tau_CLIP-D = 0.2, which is used in IP2P to keep successful edits, and assign this score to samples below the threshold' is ambiguous: in IP2P, a higher CLIP-D typically indicates a more successful edit, so it is unclear whether score 0.5 is assigned to samples below or above the threshold. Please clarify the intended direction.","section":"§3.1"},{"comment":"The total number of human preference comparisons (2856) does not match the stated 56 participants each comparing 50 pairs (2800). Please verify the numbers or the procedure.","section":"Table 4"},{"comment":"The section title 'Alation Study' should be 'Ablation Study'.","section":"§4.5"},{"comment":"The caption reads 'face are blocked'; this should be 'faces are blocked'.","section":"Figure 8 caption"},{"comment":"Several entries report 0.0 for Qwen2.5-VL (1-shot) on GenAI-Bench and for Fuyu, CogVLM, and OpenFlamingo on AURORA-Bench pair-wise; these likely reflect parsing or formatting failures rather than true accuracies and should be marked as unavailable instead of 0.0.","section":"Appendix Table 7"},{"comment":"The reward-feedback-learning result (score 6.30) is only mentioned in the text; presenting it in a table alongside the reward-conditioning result would improve reproducibility.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the likely overlap between ADIEE's training sources and AURORA-Bench, which directly threatens the central generalization claim. The numerical inconsistencies between main tables and appendix are also concerning because the abstract's headline numbers cannot be reproduced from the appendix. The protocol sensitivity to 1-shot prompting adds another layer of risk. These issues are fixable with additional analysis and revised reporting, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core idea is genuinely useful: treat ground-truth edits as positive examples and input images as negatives, and use multi-turn edit sequences to synthesize partial credit. That is a cheap way to build a large training set for an editing judge, and the paper shows real gains over open-source VLMs. Second, the headline numbers are not yet trustworthy because the evaluation set overlaps the training set. AURORA-Bench draws samples from MagicBrush and Emu-Edit, both of which are training sources. The paper reports no deduplication or overlap analysis. If the test images and instructions are seen in training, the scorer can memorize rather than generalize. This is the first thing to check before believing the +17.24% correlation gain.\n\nWhat is actually new: the observation that editing datasets implicitly provide evaluation data, the multi-turn scoring function in Eq. 4, and the use of CLIP/DINO only to find easy negatives rather than as metrics. The ablations show the score token and MLP matter. The reward-conditioned fine-tuning of MagicBrush is a reasonable application, and the human preference study, while small, is independent of the scorer.\n\nSoft spots, in proportion. The overlap issue is serious and load-bearing. The paper must report source-level splits, deduplication, and ideally a clean-room evaluation on truly unseen data. The second concern is the labeling heuristics: assigning 0 to all DiffEdit/Pix2Pix-Zero/SDEdit/Text2LIVE outputs and 0.5 to over-edits is crude; it may teach score boundaries rather than human preference. The paper should validate the labels on a small human-annotated subset. Minor: no error bars anywhere, and the reward-model gain is partly circular because the scorer both labels the fine-tuning data and measures the improvement. Also note that GPT-4o with 1-shot reaches 0.4779 on AURORA point-wise, above ADIEE's 0.4734; the paper's claim is carefully worded ('open-source and Gemini-Pro 1.5') but the practical edge over strong proprietary baselines is thin.\n\nWho this is for: anyone building automatic evaluators for image editing, especially practitioners who want a training recipe. It deserves a serious referee, but the referee should demand an overlap analysis and a human-labeled validation of the heuristics. I would not cite it for the absolute numbers until those are resolved.","headline":"Useful training recipe for an editing judge, but the test set overlaps the training data and the headline gains need a clean-room check.","tokens_in":22019,"tokens_out":2351,"would_cite":false,"duration_ms":24419,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a VLM judge for instruction-guided image editing, trained entirely on a 100K-sample dataset labeled by automatic heuristics rather than human annotation, outperforms all open-source VLMs and Gemini-Pro 1.5 on…","keywords":["instruction-guided image editing","automatic evaluation","vision-language model judge","reward model","synthetic training data","human alignment","multi-turn editing","CLIP directional similarity"],"falsifier":"Take a random sample of the constructed 100K training examples, have several human raters assign scores, and compute the agreement between the heuristic label and the human average. Alternatively, feed the scorer pairs where the heuristic labels are demonstrably wrong—such as a high-quality DiffEdit output or a multi-turn intermediate image that already satisfies the full instruction—and check whether the scorer still follows the heuristic rather than the image content.","tokens_in":21017,"feed_emoji":"🖼️","tokens_out":5302,"duration_ms":52098,"temperature":0.7,"pith_summary":"This paper tries to show that a reliable judge for instruction-guided image editing can be built without any human preference labels. It creates over 100,000 training examples automatically, by treating existing editing datasets and models as sources of successful, partially successful, and failed edits, and labels them with simple heuristics. A vision-language model fine-tuned on those labels scores edits more consistently with human raters than any open-source VLM and than Gemini-Pro 1.5 on three benchmarks. The same scorer also serves as a reward model that raises the average evaluation score of the MagicBrush editing model from 5.90 to 6.43. A sympathetic reader would care because cheap, transparent automatic evaluation would replace costly human annotation and opaque proprietary judges.","feed_headline":"Auto-labeled images train an editing judge that beats open VLMs","feed_subtitle":"A fine-tuned 8B scorer agrees with humans better than Gemini-Pro 1.5 and lifts MagicBrush's average score by 9 percent.","key_machinery":"The central mechanism is a fully automatic labeling pipeline. For single-turn edits, it scores each generated output by comparing it to the input and ground-truth images: CLIP directional similarity below 0.2 marks partial success, CLIP-I and DINO-I below the dataset's fifth percentile mark failure, and all outputs from DiffEdit, Pix2Pix-Zero, SDEdit, and Text2LIVE are treated as failures because those models produce inconsistent edits. For multi-turn sequences, the score function f in Eq. 4 assigns 0 to earlier images, the fraction $(k-j_1)/(j_2-j_1)$ to intermediate turns, 1 to the ground truth, and 0.5 to over-edited later images. These examples fine-tune a LLaVA-NeXT-8B whose vocabulary is extended with a special [SCORE] token, and a small MLP decoder turns that token's embedding into the numeric rating; LoRA keeps the fine-tune efficient.","core_discovery":"On the paper's own terms, the discovery is that instruction-guided editing data implicitly contains evaluation signal: the original input image is a failed edit, the ground-truth output is a perfect edit, and intermediate images in multi-turn sequences are partial edits whose score is the fraction of instructions completed. The paper augments this with outputs from nine editing models, labels low-quality-model outputs as failures and uses CLIP directional similarity thresholds to flag partial successes, then trains a LLaVA-NeXT-8B with a special [SCORE] token whose embedding is decoded by an MLP into a number. Trained this way, the scorer achieves a 0.4734 Spearman correlation with human ratings on AURORA-Bench point-wise and 59.96 percent pair-wise accuracy on GenAI-Bench, surpassing all open-source VLMs and Gemini-Pro 1.5; on ImagenHub it beats all open-source VLMs and Gemini-Pro 1.5 while trailing only GPT-4o. As a reward model, it lifts the editing model's average ImagenHub score from 5.90 to 6.43, a gain the paper reports as +8.98 percent.","pith_inferences":["This reader infers that the method's ceiling is set by the heuristics: if CLIP and DINO miss fine-grained failures that humans notice, the scorer will inherit that blind spot.","This reader infers the reward-model gain may partly reflect the scorer rewarding the same coarse semantics it was trained to detect, so improvements on human preference could be smaller than the +8.98 percent metric suggests.","This reader infers the same recipe could be applied to other instruction-following generative domains, such as video or 3D editing, wherever a ground-truth anchor and a multi-turn sequence exist.","A testable extension would be to train the scorer with a small human-annotated correction set on top of the heuristics and measure whether alignment improves further."],"forward_implications":["A VLM judge trained entirely on automatically generated labels can beat strong open-source and proprietary evaluators, so human-annotated evaluation datasets are not a prerequisite for good alignment.","The scorer can pick the best edit among candidates, enabling automated model selection and leaderboard ranking without manual inspection.","Using the scorer as a reward signal improves the editing model's output quality by a measurable margin (+8.98 percent average score), so automatic judges can substitute for human feedback in fine-tuning.","Because the dataset creation recipe relies only on editing datasets and generic CLIP/DINO thresholds, it should transfer to new editing methods and benchmarks without new annotation effort."],"supporting_citations":[{"why":"Provides the base MagicBrush dataset, the editing model later improved, and the instruction-guided samples used as positive training examples.","marker":"[83]"},{"why":"Supplies InstructPix2Pix and the CLIP-D threshold of 0.2 used to label partial edit successes.","marker":"[9]"},{"why":"Supplies the SEED-Data-Edit multi-turn edit sequences used to generate partial-score training samples.","marker":"[22]"},{"why":"Defines the 0-10 scoring protocol and question templates used to evaluate and prompt the scorer.","marker":"[37]"},{"why":"Supplies the special-token-plus-MLP decoding design that turns a [SCORE] token embedding into a numeric rating.","marker":"[39]"},{"why":"Is the LLaVA-NeXT-8B backbone that is fine-tuned into the ADIEE scorer.","marker":"[46]"},{"why":"Provides the CLIP model used for CLIP directional similarity and CLIP-I thresholds in negative-sample detection.","marker":"[57]"},{"why":"Provides DINOv2 features used for the fifth-percentile similarity thresholds that flag visually different failed edits.","marker":"[54]"},{"why":"Provides the ImagenHub benchmark with human ratings used for correlation evaluation and the human-to-human upper bound.","marker":"[38]"},{"why":"Provides AURORA-Bench point-wise and pair-wise benchmarks and the AURORA editing model used for positive samples.","marker":"[36]"}],"fun_headline_variants":["Auto-labeled edits train a judge that beats Gemini-Pro","8B scorer from generated data outranks open VLMs","Self-made dataset teaches an editing judge","Train an image-editing judge with auto-labeled data","Auto-generated data yields a top editing scorer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the heuristic labels—DiffEdit, Pix2Pix-Zero, SDEdit, and Text2LIVE outputs always fail, CLIP-D below 0.2 means partial success, and an intermediate turn's score is the completed fraction of instructions—match what human raters would say; if any of these rules is wrong in the bulk of samples, the scorer's human alignment is an artifact of the labeling scheme.","fun_headline_variants_meta":{"raw":{"variants":["Auto-labeled edits train a judge that beats Gemini-Pro","8B scorer from generated data outranks open VLMs","Self-made dataset teaches an editing judge","Train an image-editing judge with auto-labeled data","Auto-generated data yields a top editing scorer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1433,"prompt_tokens":1079,"completion_tokens":354,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":281}},"tokens_in":695,"tokens_out":354,"duration_ms":3864,"temperature":1.0,"reasoning_tokens":281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:43:21.851841+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the constructed 100K training examples, have several human raters assign scores, and compute the agreement between the heuristic label and the human average. Alternatively, feed the scorer pairs where the heuristic labels are demonstrably wrong—such as a high-quality DiffEdit output or a multi-turn intermediate image that already satisfies the full instruction—and check whether the scorer still follows the heuristic rather than the image content.","supporting_citations":[{"cited_title":"Magicbrush: A manually annotated dataset for instruction- guided image editing","cited_arxiv_id":null,"evidence_quote":"Provides the base MagicBrush dataset, the editing model later improved, and the instruction-guided samples used as positive training examples."},{"cited_title":"Llava-next: Im- proved reasoning, ocr, and world knowledge, January 2024","cited_arxiv_id":null,"evidence_quote":"Is the LLaVA-NeXT-8B backbone that is fine-tuned into the ADIEE scorer."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP model used for CLIP directional similarity and CLIP-I thresholds in negative-sample detection."},{"cited_title":"Imagenhub: Standardizing the evaluation of conditional image generation models","cited_arxiv_id":null,"evidence_quote":"Provides the ImagenHub benchmark with human ratings used for correlation evaluation and the human-to-human upper bound."},{"cited_title":"Learning Action and Reasoning-Centric Image Editing from Videos and Simulations","cited_arxiv_id":null,"evidence_quote":"Provides AURORA-Bench point-wise and pair-wise benchmarks and the AURORA editing model used for positive samples."}],"review_version":1}