{"id":"1867858e-8bd5-42e5-bf3a-db177cf0dba3","arxiv_id":"2506.03448","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RefEdit-Bench measures referring-expression image editing; the RefEdit model, trained on 20K synthetic triplets, reports state-of-the-art results over million-scale baselines.","lead":"This paper introduces a benchmark and a model for editing images by referring to one specific object in a busy scene, such as 'the cat on the right' or 'the chef in the white shirt.' It reports that training a model on only 20,000 carefully generated examples beats models trained on millions, which could make image editors more useful in real-world multi-object photos.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'only 20,000 triplets' claim is unsupported: RefEdit is also trained on MagicBrush data and initialized from InstructPix2Pix/UltraEdit, so the experimental design does not isolate the 20K contribution.","rationale":"We read the central claim as the abstract's assertion that training on only 20K synthetic referring-expression triplets outperforms models trained on millions of samples. For that claim to hold, the 20K set must be the causally sufficient training signal. The experimental section does not isolate this signal: Section 3.3 reveals that training starts from pretrained InstructPix2Pix and includes MagicBrush data, producing 30K combined samples. The SD3 variant is initialized from UltraEdit, so its improvement over UltraEdit is a fine-tuning effect, not evidence about data scale. The reader's weakest assumption—that the synthetic triplets are correct—is plausible but not the most load-bearing point, because even with perfect triplets the current experiment cannot attribute the result to the 20K set. The missing ablation is a standard control and should be required before accepting the abstract's scaling narrative. We therefore keep the CONDITIONAL verdict, with the added condition that the authors either provide the ablation or weaken the claim to 'fine-tuned on 20K'.","tokens_in":14966,"tokens_out":8774,"duration_ms":83657,"concrete_test":"Retrain RefEdit-SD1.5 with the same hyperparameters and the same InstructPix2Pix initialization, but using only the 20K RefEdit-Data (no MagicBrush), and also train a control on the 10K MagicBrush only. If the 20K-only model's RefEdit-Bench or PIE-Bench scores fall to within noise of the baselines, or if the MagicBrush-only control matches RefEdit, the abstract's claim is not supported. For the SD3 variant, repeat by initializing from UltraEdit vs. a randomly initialized SD3; if random-init training on 20K fails to converge, the paper should be reworded to 'fine-tuned on 20K' rather than 'trained on only 20,000'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims RefEdit is 'trained on only 20,000 editing triplets', but Section 3.3 says the authors 'combine MagicBrush data with our own generated data during the training', yielding 30K combined samples, and that the SD1.5 variant starts from the pre-trained InstructPix2Pix model; the SD3 variant starts from UltraEdit. Thus the 20K synthetic set is neither the sole training signal nor the origin of the model's capability. Comparing RefEdit-SD3 to UltraEdit is especially confounded, since RefEdit-SD3 is UltraEdit fine-tuned on a small extra set. Without an ablation that removes MagicBrush or the pretrained initialization, the 'small data beats millions' claim is not established. The reader's concern about synthetic-data quality is secondary: even a perfectly labeled 20K set would not support the claim as stated.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RefEdit-Bench, a 200-image benchmark for instruction-based image editing with referring expressions, split into Easy and Hard categories. It also proposes a synthetic data pipeline: GPT-4o generates the textual components (image prompts, editing instructions, referring expressions), FLUX renders images, Grounded SAM produces masks, and FlowChef/Inpaint Anything create the edited images, yielding roughly 20,000 synthetic editing triplets. The authors fine-tune InstructPix2Pix (SD1.5) and UltraEdit (SD3) on a combination of these synthetic triplets and MagicBrush data, producing RefEdit and RefEdit-SD3. They evaluate on RefEdit-Bench and PIE-Bench using GPT-4o-based VIEScore and human A/B tests, reporting improvements over prior instruction-based models and qualitative comparisons with closed-source methods.","tokens_in":15104,"tokens_out":5894,"duration_ms":53327,"significance":"If the central claim were established—that roughly 20,000 synthetic triplets suffice to beat million-scale training for referring-expression editing—this would be a valuable efficiency result. The benchmark itself addresses a real gap, since existing editing datasets and evaluations focus mostly on single prominent objects. The data-generation pipeline is automated and the authors release data and checkpoints, which supports reproducibility. However, the headline claim is currently undercut by the training protocol and evaluation choices; with proper ablations and independent metrics, the benchmark and pipeline could still be a solid contribution.","major_comments":[{"comment":"The abstract claims RefEdit is 'trained on only 20,000 editing triplets', but Section 3.3 states that MagicBrush data is combined with the generated data, yielding 30K combined training samples, and that both variants start from pretrained checkpoints (InstructPix2Pix and UltraEdit). Therefore the 20K synthetic set is neither the sole training signal nor the origin of the model's capability. The comparison of RefEdit-SD3 with UltraEdit is especially confounded: RefEdit-SD3 is UltraEdit fine-tuned on additional data. An ablation that removes MagicBrush data, or that trains from a randomly initialized backbone, is required to support the 'small data beats millions' claim; without it, the paper's headline claim is not established.","section":"Abstract / Section 3.3"},{"comment":"Section 5.1 uses VIEScore with GPT-4o as the judge, while Section 3.2 uses GPT-4o to generate the synthetic training instructions and descriptive referring expressions. This creates a potential bias: the trained model may be rewarded for matching GPT-4o's linguistic style rather than for the actual semantic edit. In addition, the 'Modified VIEScore' for RefEdit-Bench incorporates the ground-truth mask to extract the region of interest, which may remove the referring-expression localization component from the metric. The paper should report evaluations with an independent judge (e.g., a different MLLM or human ratings with inter-annotator agreement) and should report both full-image and masked-ROI scores.","section":"Section 5.1 / Section 3.2"},{"comment":"Table 4 reports VIEScore margins of 5.21 vs 5.12 (RefEdit vs OmniEdit) and 5.70 vs 5.12 (RefEdit-SD3 vs OmniEdit) with no error bars, repeated runs, or significance tests. The human evaluation in Figure 6 reports only percentages, without the number of annotators, their recruitment, or agreement measures. Given the small margins and the self-constructed benchmark, the 'state-of-the-art' and 'significantly outperforms' claims are not statistically supported.","section":"Table 4 / Figure 6"},{"comment":"The data-generation pipeline has no described quality filtering: GPT-4o may generate ambiguous expressions, Grounded SAM may produce incorrect masks, and FlowChef/Inpaint Anything may fail to apply localized edits. If any component fails systematically, the 20K triplets could teach incorrect input-output mappings. The paper should provide a quality audit (e.g., manual inspection of a random subset, mask-edit overlap statistics, or filtering heuristics) and report how many generated triplets were discarded.","section":"Section 3.2 / Figure 3"}],"minor_comments":[{"comment":"The text says training follows Eq. (1), but the InstructPix2Pix objective is given in Eq. (2); please clarify which loss is actually used.","section":"Section 3.3"},{"comment":"The description of 'Modified VIEScore' is ambiguous: it should state explicitly whether both SC and PQ are computed on the masked region or on the full edited image with the mask used only for prompting.","section":"Section 5.1"},{"comment":"References [5] and [6] are duplicates (the InstructPix2Pix paper), as are [45] and [46] (Inpaint Anything); please consolidate them.","section":"References"},{"comment":"The meaning of the numeric entries in the 'Use of referring expressions?' column is unclear; please clarify what the values represent.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and synthetic pipeline are potentially useful resources, but the paper's central efficiency claim is not currently supported by the experimental design. I would advise the editor to request a revision that adds ablations isolating the 20K synthetic set, independent evaluation metrics, and statistical robustness measures before considering publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good: a new benchmark for referring-expression editing and a synthetic pipeline that targets it. RefEdit-Bench is a real contribution: 200 RefCOCO images with manual instructions, Easy/Hard splits, and it shows a genuine failure mode in existing instruction-based editors. The pipeline combining GPT-4o for text, FLUX for rendering, Grounded SAM for masks, and FlowChef/Inpaint Anything for the edit is a sensible integration, and the authors are transparent about the five edit types. The human preference study is also encouraging: RefEdit-SD3 wins more than it loses against each baseline.\n\nThe soft spots are in the central claim. The abstract says 'trained on only 20,000 editing triplets,' but Section 3.3 says the training data combines MagicBrush with the synthetic set (30K combined) and that both variants start from pretrained instruction-editing models (InstructPix2Pix for SD1.5, UltraEdit for SD3). So RefEdit-SD3 is UltraEdit fine-tuned on a small extra set. The comparison to UltraEdit is exactly the one that would test 'small data beats millions,' and it is confounded by initialization. The paper provides no ablation removing MagicBrush or the pretrained start, so the headline claim is not established. The paper is honest about the setup, which makes the abstract just overreach rather than fraud.\n\nAlso, GPT-4o generates the synthetic instructions and expressions and then judges the results via VIEScore. That circularity is real, though it is softened by the human evaluation and by the transfer to PIE-Bench. The benchmark is small (200 images, self-curated), there are no error bars or repeated runs, no inter-annotator agreement, and no artifact hashes or links are verified. The PIE-Bench gains are modest (5.21 vs 5.12) and could be noise.\n\nVerdict: this deserves peer review, but not as is. A referee should demand (1) an ablation that isolates the 20K synthetic set, (2) variance and inter-annotator numbers, (3) a judge-model change or an analysis showing the result does not depend on GPT-4o's phrasing, and (4) released artifacts with hashes. The benchmark itself is useful regardless of the training claim; I would cite it if I worked on instruction editing. But the paper's main quantitative assertion needs to be backed by the missing ablation.","headline":"Useful new benchmark for referring-expression editing, but the '20K triplets beat millions' headline is undercut by the MagicBrush mixing and pretrained initialization.","tokens_in":15714,"tokens_out":2153,"would_cite":true,"duration_ms":21008,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Instruction-based image editors fail when an instruction must pick one of several similar objects; the paper shows 20,000 targeted training examples fix that failure.","keywords":["referring expression image editing","instruction-based image editing","synthetic data generation","RefEdit-Bench","RefCOCO","localized editing","data efficiency","diffusion models"],"falsifier":"Take a random sample of the released 20,000 training triplets and have independent annotators mark each as correct or wrong on three criteria: the referring expression identifies exactly one intended object, the Grounded SAM mask covers that object and nothing else, and the edited image changes only inside the mask. If wrong triplets are common, the claimed advantage is not evidence for referring-expression training.","tokens_in":14708,"feed_emoji":"🎯","tokens_out":7327,"duration_ms":77799,"temperature":0.7,"pith_summary":"Instruction-based image editors follow commands easily when the image has one salient object, but they falter when the command must pick out one instance among several similar ones, such as \"the chef cutting vegetables\" or \"the rightmost person.\" The paper argues that this failure is a data problem, not just a model-size problem, and that a small, carefully constructed set of training examples can fix it. It introduces RefEdit-Bench, a benchmark built from real-world RefCOCO images with easy and hard splits, and RefEdit, a model fine-tuned on only 20,000 synthetic editing triplets. The paper reports that RefEdit outperforms baselines trained on millions of examples on referring-expression editing, improves standard editing benchmarks, and is preferred by human raters. If correct, the result says targeted synthetic data can substitute for brute-force data scale in instruction-based image editing.","feed_headline":"20K targeted edits beat million-scale image editors","feed_subtitle":"A referring-expression dataset teaches models to edit one object among many, without sacrificing general skill.","key_machinery":"The load-bearing mechanism is the synthetic data generation pipeline, whose key idea is to generate the textual editing case and the image together from one template. GPT-4o writes an image prompt containing two or more instances of the same object distinguished by color, item, or activity, and also writes the edited prompt, editing instruction, target object, a referring expression, and a more descriptive version of that expression. FLUX renders the image, Grounded SAM converts the descriptive expression into a mask over exactly the referred instance, and FlowChef (for color, object, texture, and add edits) or Inpaint Anything (for removals) produces the edited image from the masked region. Assembling pairs this way guarantees that every training instruction has a uniquely identifiable referent and a localized ground-truth edit, which is what the released RefEdit-Data provides.","core_discovery":"On the paper's own terms, the central discovery is that the referring-expression capability of instruction-based editors is bottlenecked by training data, not by backbone capacity. Existing instruction-editing datasets almost always contain a single dominant object per image, so models learn to edit \"the object\" and have no practice resolving expressions that identify one instance among several similar ones. The authors' synthetic pipeline generates 20,000 training triplets in which images deliberately contain several similar instances, and each instruction's referring expression uniquely identifies the target; the mask isolates that target, and the edit changes only it. Fine-tuning InstructPix2Pix and an SD3-based editor on this data, mixed with existing MagicBrush data, yields a model that the paper shows beats million-example baselines on RefEdit-Bench and PIE-Bench and wins human A/B comparisons. The paper frames this as evidence that quality-focused synthetic data, not scale alone, is what unlocks complex real-world editing.","pith_inferences":["Editorial inference: If targeted data composition is the active ingredient, the same pipeline should transfer to other referring-expression benchmarks, and the performance gap between RefEdit and generic editors should widen as the number of visually similar entities per image grows.","Editorial inference: The reported results do not isolate which pipeline stage matters; ablating the LLM-expression generator, the masker, and the editing model on a fixed evaluation subset would reveal whether the gains come from instruction diversity, mask precision, or edit fidelity.","Editorial inference: Because RefEdit-Bench scores are produced by an MLLM, a natural stress test is to re-rank the same model outputs with different MLLM judges and prompt templates to confirm the benchmark measures referring-expression skill rather than evaluator preference.","Editorial inference: Separating edit leakage from semantic correctness would be a useful extension; the reported overall score combines instruction alignment and perceptual quality, so it does not by itself say whether edits stay inside the referred object."],"forward_implications":["A model trained on 20,000 targeted referring-expression triplets can beat instruction-editing models trained on millions of examples, on both the new benchmark and PIE-Bench.","The easy/hard split of RefEdit-Bench provides a reusable probe: methods that score well on generic editing benchmarks drop sharply on referring-expression tasks, so the benchmark isolates a distinct capability.","Fine-tuning a larger SD3-based editor on the same synthetic data pushes performance further, suggesting the data helps across backbone scales rather than only small models.","Mixing MagicBrush data with RefEdit-Data preserves general editing ability, so the synthetic dataset works as a supplement rather than a replacement for existing instruction-editing data."],"supporting_citations":[{"why":"Supplies the real-world RefCOCO images and referring expressions used to build RefEdit-Bench.","marker":"[44]"},{"why":"Generates the structured text (image prompt, edited prompt, instruction, referring expression) and serves as the MLLM judge for VIEScore.","marker":"[29]"},{"why":"Renders the 1024x1024 base images on which the synthetic editing triplets are built.","marker":"[19]"},{"why":"Grounds descriptive referring expressions as object masks that isolate the intended instance.","marker":"[36]"},{"why":"Performs the inversion-free localized edits used for most training triplets.","marker":"[33]"},{"why":"Produces the object-removal training pairs in the synthetic dataset.","marker":"[46]"},{"why":"Supplies the pretrained InstructPix2Pix base model that RefEdit (SD1.5) is fine-tuned from.","marker":"[6]"},{"why":"Provides the fine-tuning configuration and the mixed baseline data that preserves general editing ability.","marker":"[47]"},{"why":"Serves as the million-scale baseline and the starting checkpoint for the RefEdit-SD3 variant.","marker":"[49]"},{"why":"Defines the training-free MLLM-based VIEScore metric used to measure semantic consistency and perceptual quality.","marker":"[18]"}],"fun_headline_variants":["Small data, big gains: 20K edits top million-scale editors","Quality over quantity: 20K edits outperform million-scale training","Referring expressions unlock multi-object editing with 20K samples","20K triplets beat millions: a new benchmark and method for image editing","Editing complex scenes: 20K targeted examples outdo massive datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The advantage rests on the synthetic pipeline producing correct training pairs: each instruction's referring expression truly picks out one intended object, the mask covers exactly that object, and the edited image changes nothing outside it.","fun_headline_variants_meta":{"raw":{"variants":["Small data, big gains: 20K edits top million-scale editors","Quality over quantity: 20K edits outperform million-scale training","Referring expressions unlock multi-object editing with 20K samples","20K triplets beat millions: a new benchmark and method for image editing","Editing complex scenes: 20K targeted examples outdo massive datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1508,"prompt_tokens":874,"completion_tokens":634,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":543}},"tokens_in":490,"tokens_out":634,"duration_ms":7206,"temperature":1.0,"reasoning_tokens":543,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:03:02.598514+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the released 20,000 training triplets and have independent annotators mark each as correct or wrong on three criteria: the referring expression identifies exactly one intended object, the Grounded SAM mask covers that object and nothing else, and the edited image changes only inside the mask. If wrong triplets are common, the claimed advantage is not evidence for referring-expression training.","supporting_citations":[{"cited_title":"Berg, and Tamara L","cited_arxiv_id":null,"evidence_quote":"Supplies the real-world RefCOCO images and referring expressions used to build RefEdit-Bench."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Generates the structured text (image prompt, edited prompt, instruction, referring expression) and serves as the MLLM judge for VIEScore."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Renders the 1024x1024 base images on which the synthetic editing triplets are built."},{"cited_title":"Grounded sam: Assembling open-world models for diverse visual tasks, 2024","cited_arxiv_id":null,"evidence_quote":"Grounds descriptive referring expressions as object masks that isolate the intended instance."},{"cited_title":"Metaxas, and Yezhou Yang","cited_arxiv_id":null,"evidence_quote":"Performs the inversion-free localized edits used for most training triplets."},{"cited_title":"Inpaint anything: Segment anything meets image inpainting, 2023","cited_arxiv_id":null,"evidence_quote":"Produces the object-removal training pairs in the synthetic dataset."},{"cited_title":"Magicbrush: A manually annotated dataset for instruction- guided image editing, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the fine-tuning configuration and the mixed baseline data that preserves general editing ability."},{"cited_title":"score\" : [...],","cited_arxiv_id":null,"evidence_quote":"Serves as the million-scale baseline and the starting checkpoint for the RefEdit-SD3 variant."}],"review_version":1}