{"id":"4a10feb1-ef65-42d6-adf3-9bee29c07ca7","arxiv_id":"2507.16193","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A large human-annotated benchmark of AI-edited images (EBench-18K) plus a fine-tuned LMM metric (LMM4Edit) that predicts human preference scores across three dimensions and answers editing-specific questions.","lead":"This paper introduces EBench-18K, a benchmark of over 18,000 AI-edited images with human preference scores, and LMM4Edit, a multimodal AI model that automatically scores edited images on quality, alignment, and preservation. The resource could help developers compare image editing models without running costly human studies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Train/test split may leak source-image identity; if the 4:1 split is by edited image rather than by source, LMM4Edit's reported SOTA SRCC could be inflated.","rationale":"The reader's weakest_assumption focused on the 512x512 presentation and Z-score rescaling of human MOS. That is a legitimate external-validity caveat, but it does not threaten the paper's internal claim: the correlations in Tables 3-4 are defined relative to exactly those collected MOSs. The split ambiguity is a sharper, internal-validity threat. If the split is per edited image, the same source image appears in both training and test, and because LMM4Edit's input includes the source image, the model can short-circuit by recognizing the source and predicting its associated score distribution. This would inflate LMM4Edit's absolute SRCCs and, because the model is much larger than traditional IQA baselines, could also inflate its relative advantage over fine-tuned LMM baselines. The concern is concrete and falsifiable: either the released split code shows a content-disjoint split, or rerunning with a source-disjoint split will reveal the magnitude of the effect. I do not treat this as proof of fraud; it is an unstated design choice that could change the conclusion. The reader's conditional verdict is appropriate: the paper should be accepted only with the split clarified or the analysis redone on a source-disjoint split. Hence the verdict is unchanged, but the condition is sharpened. I also credit the paper's strengths: a large human-annotated benchmark, three-dimensional MOSs plus QA pairs, and an open repository, which are real contributions regardless of this validity question.","tokens_in":28697,"tokens_out":10315,"duration_ms":104909,"concrete_test":"Inspect the released repository's data-split code. If the split is per edited image or unspecified, rerun Table 3 with a content-disjoint split: assign all 17 edited images of each source image to the same fold (e.g., 5-fold cross-validation over the 1,080 source images) and recompute SRCC/PLCC for LMM4Edit and the fine-tuned baselines. If LMM4Edit's margin over InternVL2.5 shrinks materially (e.g., >0.03 SRCC) or its ranking changes, the headline claim is inflated by source-image leakage. Also report bootstrap 95% confidence intervals for the SRCC differences.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper never specifies whether the 4:1 train/test split (Section 5.1) is performed at the level of the 1,080 source images or the 18,360 edited images. LMM4Edit explicitly takes the source image as input (Eq. 2, Figure 5). If the same source image appears in both train and test with different edited outputs from the 17 models, the model can learn source-specific priors about quality, alignment, and preservation, then exploit them at test time, inflating all three SRCCs in Table 3. Large LMMs are especially prone to memorizing image identity, so the reported margins over fine-tuned baselines (e.g., +3.28% over InternVL2.5 on perceptual quality) may be partly an artifact of this leakage. The central claim that a single LMM metric can replace per-dimension specialists requires generalization to new source images, which a leaked split does not test. This concern is more load-bearing than the viewing-resolution point because it threatens internal validity of the headline numbers, not only their external scope.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EBench-18K, a text-guided image editing benchmark with 1,080 source images, 18,360 edited images from 17 TIE models, and 1,101,600 human annotations in the form of MOSs for perceptual quality, editing alignment, and attribute preservation plus task-specific yes/no QA pairs. It also proposes LMM4Edit, a fine-tuned Qwen2.5-VL-based metric with AdaLoRA that takes source image, edited image, and prompt as input and outputs scores and QA answers. The paper reports state-of-the-art correlations with human MOSs on EBench-18K (SRCC 0.9136/0.8830/0.9048 in Table 3) and best zero-shot QA accuracy on I2EBench among compared LMMs (Figure 7).","tokens_in":28948,"tokens_out":5384,"duration_ms":59373,"significance":"If the results hold, EBench-18K is a substantial community resource: it is larger than existing TIE benchmarks, covers 21 tasks and both description- and instruction-based models, and its subjective protocol is reported in unusually specific detail (15 annotators per image, ITU-R BT.500-13 outlier rejection, Z-score rescaling). LMM4Edit is also a plausible step toward replacing per-dimension specialist metrics with a single instruction-tuned LMM, and the external I2EBench check is a genuine attempt at cross-dataset validation. However, the central generalization claim currently rests on an ambiguous train/test split and on a zero-shot evaluation that only measures QA accuracy, not score prediction on an external benchmark.","major_comments":[{"comment":"The training and testing split is not specified at the correct granularity. Section 5.1 states only that learning-based models use a 4:1 split, but it does not say whether the split is performed over the 1,080 source images or over the 18,360 edited images. Since LMM4Edit explicitly conditions on the source image (Eq. 2) and the dataset contains 17 edited images per source, a random edited-image split would place outputs of the same source in both training and test sets. A large LMM can then memorize source-specific priors about quality, alignment, and preservation, inflating the headline SRCCs in Table 3 and making the reported margins over fine-tuned baselines uninterpretable for the claimed task of scoring new source images. Please specify the split granularity; if the split is by edited image, rerun the comparison with a source-level 4:1 split and report whether the margins over InternVL2.5 remain.","section":"§5.5, Figure 7"},{"comment":"The zero-shot validation on I2EBench reports only QA accuracy, not correlation with human scores for perceptual quality, editing alignment, or attribute preservation. The abstract and conclusion describe LMM4Edit as showing strong generalization ability for evaluating TIE models, but the score-prediction dimensions—which are the core contribution—are not validated outside EBench-18K. If the claim is that LMM4Edit is an all-in-one metric that generalizes, the paper should either report zero-shot score correlations on any external benchmark with human MOSs or explicitly limit the generalization claim to QA accuracy.","section":"§3.3, Table 2"},{"comment":"The subjective experiment presents all edited images at 512×512 resolution on the display, while Table 2 lists several models that generate at 768×768 or 1024×1024 and the benchmark includes super-resolution and other low-level tasks. Downsampling to 512×512 may compress or hide quality differences that are visible at native resolution, and for super-resolution it is unclear what a human MOS at 512×512 means. The paper should provide evidence that the MOSs are robust to this presentation choice, at least for the low-level task subsets, or discuss this as a limitation of the benchmark.","section":"§4.1, Eq. (3), Table 5"},{"comment":"The overall rank used to compare metric-to-human agreement in Table 5 depends on the ad hoc weights in Eq. (3) (0.3/0.4/0.3). No sensitivity analysis is given, and different weights could change the ranking of metrics. At minimum, the authors should justify the weights and show that the main conclusion is stable across reasonable weight choices.","section":"§5.2, Table 4"}],"minor_comments":[{"comment":"There is a duplicated word in the text: 'where where r_ij is the raw rating...'","section":"§4.1"},{"comment":"The text refers to 'QwenLM2 [73]' and 'Qwen2-VL' in places, while Table 6 and the figure use Qwen2.5-VL; please make the backbone naming consistent.","section":"Table 3"},{"comment":"Please verify the TOPIQ row: the reported KRCC for editing alignment (0.6692) is much higher than the corresponding PLCC (0.3848), which is unusual and may be a typo.","section":"Table 4"},{"comment":"In the Remove column for Qwen2-VL, the reported accuracy appears as '0404' instead of '0.404'.","section":"Table 6"},{"comment":"The ablation table uses checkmarks without a clear legend; it is not obvious which model is used as the backbone in each row or which components are frozen versus trainable. Add a legend or spell out the configurations in the caption.","section":"§2.2"},{"comment":"The text cites '[25]' for EditEval, but reference [25] is listed as a survey on diffusion-based image editing; the citation for EditEval should be corrected.","section":"§5.1"},{"comment":"For learning-based methods, the paper states that 'the same training and testing split (4:1)' is used as in previous literature, but the prior literature is not identified; please cite the source of this split protocol.","section":"§3.3"},{"comment":"No inter-annotator agreement statistic is reported; given that the benchmark's value depends on MOS reliability, a brief report of agreement (e.g., ICC or Krippendorff's alpha) would strengthen the paper.","section":"§3.1"},{"comment":"The claim of being 'the first large-scale TIE benchmark' should be qualified against I2EBench and IE-Bench, which are discussed in related work; please state explicitly what EBench-18K adds beyond them (MOSs for three dimensions, 21 tasks, both prompt types).","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the train/test split granularity. If the authors confirm that the split is already source-level, this becomes a minor revision. If not, the headline comparisons must be rerun; the current text does not let a reader verify this. The viewing-resolution and ad hoc weighting concerns are secondary but should be addressed in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is genuinely useful: EBench-18K gives the field 18K edited images from 17 models across 21 tasks, with three-dimensional MOSs and QA pairs, where prior benchmarks mostly lacked MOSs or a single combined metric. Second, the central empirical claim—LMM4Edit beats fine-tuned LMM baselines by clear margins—rests on a train/test split the paper does not define. Section 5.1 says a 4:1 split but never says whether it is at the level of the 1,080 source images or the 18,360 edited images. Since LMM4Edit takes the source image as input, an edited-image split would let the model memorize source-specific priors and inflate all three SRCCs in Table 3. This is a load-bearing ambiguity, not a minor omission.\n\nWhat the paper does well: the subjective experiment is carefully described (ITU-R BT.500-13 outlier rejection, Z-score rescaling, 15 annotators per image). Table 1 honestly documents gaps in prior benchmarks. The metric design—two-stage score regression with AdaLoRA on both encoder and LLM—is reasonable, and the ablation study is informative. The zero-shot check on I2EBench is a good external sanity test, though it only covers QA accuracy, not MOS prediction.\n\nSoft spots beyond the split issue: the correlations are point estimates from a single split, with no confidence intervals or significance tests. The viewing-resolution concern raised in the reader note is real but secondary. Reproducibility is unclear; the paper says the dataset and code are available but does not say whether model weights, full MOS files, and an inference script with a commit hash are included. The overall score weighting in Eq. 3 (0.3/0.4/0.3) is somewhat arbitrary but only affects model-level ranking.\n\nThis paper deserves a serious referee, but the referee should press for a clarified split and, if it is at the edited-image level, a re-analysis. If the split is by source image, the headline numbers likely stand. As is, I would not cite the reported correlations in my own work.\n\nRecommendation: send to peer review with a major-revision request focused on split definition and release details.","headline":"A genuinely useful benchmark and a plausible metric, but the undefined train/test split undermines the headline numbers until clarified.","tokens_in":29513,"tokens_out":2002,"would_cite":false,"duration_ms":20657,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that one fine-tuned multimodal language model can replace separate specialist metrics for text-guided image editing by scoring perceptual quality, editing alignment, and attribute preservation in a single pass.","keywords":["text-guided image editing","benchmark dataset","large multimodal models","image quality assessment","human preference alignment","instruction tuning","zero-shot generalization","mean opinion score"],"falsifier":"Re-run the subjective study on the same edited images at each model's native output resolution and recompute LMM4Edit's rank correlation with the new human scores; if the numbers fall materially below the reported values, the claimed human alignment is an artifact of downsampling to 512x512.","tokens_in":28406,"feed_emoji":"🖼️","tokens_out":9465,"duration_ms":93120,"temperature":0.7,"pith_summary":"This paper claims that one fine-tuned multimodal language model can judge text-guided image edits the way pooled human raters do, across three dimensions at once: perceptual quality, editing alignment, and attribute preservation. To train and test such a model, the authors built EBench-18K, a benchmark of 18,360 edited images produced by 17 editing models on 21 tasks, with 55,080 mean opinion scores and 18,360 yes/no question-answer pairs distilled from over a million human annotations. Their model, LMM4Edit, is claimed to reach rank correlations with human scores of 0.9136 for quality, 0.8830 for alignment, and 0.9048 for preservation on this benchmark, and to generalize zero-shot to another editing benchmark where it achieves the best question-answering accuracy among tested multimodal models. If these results hold, a single automatic judge could replace most per-dimension specialist metrics and much of human evaluation when comparing editing models.","feed_headline":"One model now ranks image edits as well as human raters","feed_subtitle":"It hits 0.88–0.91 rank correlation with human scores on three dimensions and works zero-shot on new data.","key_machinery":"The carrying mechanism is a two-stage instruction-tuned LMM that always sees the source image, the edited image, and a prompt quoting the editing instruction and asking four questions: three score questions and one yes/no question. Stage one trains the model with cross-entropy loss to emit the four-answer text format, including approximate scores; stage two attaches a small quality-score decoder to the hidden state just before each score token and regresses the exact MOS with mean-squared error. AdaLoRA, adaptive low-rank adaptation that redistributes trainable parameters toward the most important layers, is applied to both the vision encoder and the language model so a pretrained vision-language backbone is adapted without full fine-tuning.","core_discovery":"The central discovery is that a single instruction-tuned multimodal language model, fed the source image, the edited image, and a prompt that asks for three scores and one yes/no answer, can predict human preference scores for text-guided image editing on all three evaluation dimensions better than any previously tested metric. The authors build EBench-18K, a large benchmark with 18,360 edited images, 55,080 MOSs, and 18,360 QA pairs from 15 raters per image, and fine-tune LMM4Edit on it. The resulting model reports rank correlations with human MOS of 0.9136 for perceptual quality, 0.8830 for editing alignment, and 0.9048 for attribute preservation, exceeding every comparison metric in Table 3, and it reaches 87.8% QA accuracy in-domain plus the best zero-shot QA accuracy among tested LMMs on the I2EBench dataset.","pith_inferences":["If the reported correlations survive full-resolution testing, editing-model releases could be screened automatically, with human review reserved for a shortlist of candidates; the paper itself does not test this workflow.","The 512x512 viewing condition in the subjective study may under-weight artifacts that only appear at native resolutions, so an immediate next experiment is to collect MOSs at 1024x1024 and check whether model rankings shift.","The two-stage training recipe appears portable to other generation-evaluation settings with per-dimension MOS data, such as text-to-video or text-to-3D quality assessment."],"forward_implications":["A single LMM4Edit forward pass gives the three scalar scores and a yes/no task check, so per-dimension specialist metrics are no longer needed for TIE evaluation on this benchmark.","The same metric can rank whole editing models: its model-level rankings agree with human rankings more closely than any baseline tested in Table 5.","Fine-tuning on EBench-18K lifts editing-alignment correlation from about 0.41 for a strong zero-shot LMM to 0.88, showing that alignment judgments can be learned from MOS data.","Zero-shot QA accuracy on I2EBench indicates that the model's edit-checking skill transfers to unseen tasks and instructions without retraining."],"supporting_citations":[{"why":"Supplies the I2EBench QA-pair dataset used for zero-shot cross-dataset generalization testing.","marker":"[46]"},{"why":"Provides Q-Align, the strongest LMM-based visual scoring baseline on perceptual quality that LMM4Edit improves upon.","marker":"[77]"},{"why":"Supplies AdaLoRA, the adaptive low-rank adaptation applied to both the vision encoder and the language model.","marker":"[90]"},{"why":"Defines the outlier-rejection and Z-score rescaling procedure that converts raw human ratings into MOS labels.","marker":"[28]"},{"why":"Introduces instruction-based image editing and is one of the 17 generation models whose outputs populate EBench-18K.","marker":"[7]"},{"why":"EditEval, a prior TIE evaluation the paper compares against, showing existing score-based benchmarks lack MOS labels and human alignment.","marker":"[25]"},{"why":"TedBench/Imagic, the small earlier TIE benchmark with 100 edited images, motivates EBench-18K's scale and annotation design.","marker":"[31]"},{"why":"InternVL2.5, the strongest fine-tuned LMM baseline, which LMM4Edit must beat on all three dimensions.","marker":"[11]"}],"fun_headline_variants":["LMM4Edit matches human judgment on image edits","AI metric predicts human preference for image editing across tasks","One LMM scores image editing: 0.91 correlation with humans","New benchmark and model rank image edits like humans do","LMM4Edit: human-level evaluation of text-guided image editing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human MOS labels, collected at 512x512 on one calibrated monitor and rescaled per subject with Z-scores, are the correct ground truth for editing quality, even though most of the 17 editing models generate images at higher resolutions.","fun_headline_variants_meta":{"raw":{"variants":["LMM4Edit matches human judgment on image edits","AI metric predicts human preference for image editing across tasks","One LMM scores image editing: 0.91 correlation with humans","New benchmark and model rank image edits like humans do","LMM4Edit: human-level evaluation of text-guided image editing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1336,"prompt_tokens":1023,"completion_tokens":313,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":230}},"tokens_in":639,"tokens_out":313,"duration_ms":3733,"temperature":1.0,"reasoning_tokens":230,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:14:28.827833+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the subjective study on the same edited images at each model's native output resolution and recompute LMM4Edit's rank correlation with the new human scores; if the numbers fall materially below the reported values, the claimed human alignment is an artifact of downsampling to 512x512.","supporting_citations":[{"cited_title":"ZONE: Zero-Shot Instruction-Guided Local Editing","cited_arxiv_id":"2312.16794","evidence_quote":"Supplies the I2EBench QA-pair dataset used for zero-shot cross-dataset generalization testing."},{"cited_title":"InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","cited_arxiv_id":null,"evidence_quote":"Provides Q-Align, the strongest LMM-based visual scoring baseline on perceptual quality that LMM4Edit improves upon."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies AdaLoRA, the adaptive low-rank adaptation applied to both the vision encoder and the language model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TedBench/Imagic, the small earlier TIE benchmark with 100 edited images, motivates EBench-18K's scale and annotation design."}],"review_version":1}