{"id":"36d97ff2-59f7-4655-b648-3ad78967fbe2","arxiv_id":"2508.06905","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MultiRef-bench shows that current image generators that accept multiple visual references still fail to combine them reliably, with the best tested model OmniGen reaching only 66.6% synthetic and 79.0% real-world alignment versus golden answers.","lead":"This paper introduces MultiRef-bench, a benchmark of 990 synthetic and 1,000 real-world tasks that require generating an image from multiple reference images, plus a 38k-image synthetic dataset built by a reference-blending engine. It reports that nine tested image generation systems, including unified models like OmniGen and agentic pipelines, all lose fidelity and instruction-following accuracy once they must combine several visual references at once.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Primary judge (GPT-4o-mini) contradicts paper's own Table 3/10: GPT-4o has higher IF/SF correlation, so rankings and 66.6%/79.0% are not numerically supported.","rationale":"The paper is genuinely useful: it releases a concrete dual-pipeline benchmark, uses rule-based metrics for spatial conditions, and reports extensive ablations. The central scientific claim, however, is a negative result about current models, and that claim is quantified through the MLLM judge. The paper's own validation tables contain a direct contradiction that undermines the stated basis for judge choice. Because the differences used to rank models (e.g., IF 0.496 vs 0.589) are comparable to the judge's error/correlation gap, the reported ordering is not yet established. This is not a disagreement with consensus; it is an internal inconsistency plus insufficient validation. The reader's verdict of CONDITIONAL is appropriate; my concern reinforces it rather than moving it. A re-scoring with the better-ranked judge and, ideally, a human preference sample, would settle it.","tokens_in":27344,"tokens_out":8287,"duration_ms":83705,"concrete_test":"Re-score all model outputs and ground truth in Tables 4/11 with GPT-4o (the judge the paper's own Table 3 ranks higher) using the same IF/SF rubrics, then recompute the model ordering and the 66.6%/79.0% headline values. If OmniGen versus Claude+SD3.5 (or other pairwise rankings) flips, or if the headline percentages shift materially, the published rankings are judge-dependent and the benchmark's core conclusion must be treated as conditional on judge choice and human validation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central conclusions are carried by overall Instruction Following (IF) and Source Fidelity (SF) scores, which Section 2.3 assigns to GPT-4o-mini 'for its superior alignment with human judgment shown in Table 3'. The paper's own Table 3 contradicts this: on realistic data GPT-4o's Pearson vs humans is 0.624 IF / 0.613 SF vs GPT-4o-mini's 0.530 / 0.514; on synthetic data it is 0.668 / 0.659 vs 0.632 / 0.616. Appendix C.3 repeats the claim that GPT-4o-mini surpasses other models, also contrary to Table 10. Thus the judge was not chosen by the reported criterion. Furthermore, even the best available correlations are moderate relative to human-human agreement (e.g., realistic IF 0.624 vs 0.665; synthetic IF 0.668 vs 0.721). The gaps between model scores are of the same size as the judge's disagreement with humans (e.g., OmniGen IF 0.496 vs Claude+SD3.5 IF 0.589), so the model ordering in Tables 4 and 11 could be an artifact of the particular judge. The real-world part additionally treats one Reddit edit as the unique golden output, although multiple valid outputs can satisfy the same request. The abstract's 66.6% and 79.0% figures are not derived in any stated aggregation from Tables 4/11, so the exact headline numbers lack an audit trail.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MultiRef, a benchmark and dataset for multi-reference image generation. It combines 990 synthetically generated samples produced by the RefBlend engine (10 reference types, 33 reference combinations) with 1,000 real-world tasks sampled from Reddit's r/PhotoshopRequest community. The authors evaluate three open-source unified models (OmniGen, ACE, Show-o) and six agentic frameworks (e.g., ChatDiT, LLM+SD) using rule-based metrics for spatial conditions and an MLLM-based judge for semantic-level assessments. They report that even the best existing system, OmniGen, performs far below the golden answer, with, e.g., 0.496 instruction-following on real-world tasks, and they release a 38k-image synthetic dataset.","tokens_in":27782,"tokens_out":8700,"duration_ms":84278,"significance":"The contribution is potentially valuable. A reproducible benchmark targeting multi-reference conditioning is missing, and the paper assembles a large dataset, a data engine, detailed reference definitions, and publicly available materials. The rule-based spatial metrics (IoU, MSE, mAP) are well matched to the conditions they measure. However, the quantitative headline results are not currently trustworthy: the choice of primary judge contradicts the paper's own validation tables, and the abstract's aggregate percentages are not derivable from the reported tables. The real-world part also assumes a single golden output for edits that often admit multiple valid solutions. If these issues are addressed, the benchmark could become a useful community resource.","major_comments":[{"comment":"The stated reason for choosing GPT-4o-mini as the primary judge is contradicted by the paper's own validation. Section 2.3 says GPT-4o-mini is selected for 'superior alignment with human judgment shown in Table 3', but Table 3 reports GPT-4o as having higher Pearson correlation than GPT-4o-mini on IF and SF (Realistic: 0.624 vs 0.530 for IF, 0.613 vs 0.514 for SF; Synthetic: 0.668 vs 0.632, 0.659 vs 0.616). Appendix C.3 repeats the claim, contrary to Table 10. Since the IF/SF scores in Tables 4 and 11 are produced by GPT-4o-mini, the reported model ordering and performance gaps are not numerically supported by the criterion the paper itself states. Re-evaluation with GPT-4o, or an explicit justification based on the dimensions where GPT-4o-mini actually wins, is needed before the central conclusions can be relied upon.","section":"§2.3, Table 3, Appendix C.3"},{"comment":"The headline numbers 66.6% and 79.0% have no stated aggregation. OmniGen's scores in Table 4 (Synthetic: IQ 0.730, IF 0.532, SF 0.438) and Table 11 (Real-world: IQ 0.555, IF 0.496, SF 0.421) do not directly produce 0.666 or 0.790 under any formula described in Section 2.3. If these are ratios to the golden answer, the conversion is not specified. Please provide the exact computation or remove the percentages from the abstract and Section 1.","section":"Abstract and Section 2.3/Tables 4 and 11"},{"comment":"The real-world benchmark treats one selected Reddit output as the unique golden answer. Section 2.1 and C.1 say that when multiple outputs exist, annotators select only one, and Table 11 computes IF/SF against this single ground truth. But many editing requests admit several valid output images; a model producing a different but correct result is scored as failing. This can inflate reported failure rates and may reorder models on the real-world subset. The paper should include multiple accepted outputs per task, or report reference-free / rule-based metrics alongside the comparison to the single gold.","section":"§2.1, §C.1, Table 11"},{"comment":"The evaluation pipeline has a self-referential component and the human-alignment correlations are moderate. The fine-tuned judge (Qwen-2.5-VL-7B-FT) is trained on the authors' own human annotations (B.5), used to filter MultiRef in Step 4, and used in evaluation, while GPT-4o-mini is used for overall IF/SF. Table 3 shows human-human agreement is 0.665 (Realistic IF) and 0.721 (Synthetic IF), versus 0.530 and 0.632 for GPT-4o-mini. The score differences between evaluated systems are comparable to the judge-human disagreement (Table 11: OmniGen IF 0.496 vs Claude+SD3.5 0.589). The reported ordering could therefore change with a different validated judge. Please report confidence intervals, a human-evaluated subset for the main comparisons, or a robustness analysis across judges.","section":"§2.2 Step 4, §2.3, Appendix B.4/B.5"}],"minor_comments":[{"comment":"Section 2.2 says RefBlend produces samples 'across 34 reference combinations', while the abstract and C.2 say 33 combinations. Please correct the discrepancy.","section":"§2.2 vs §C.2"},{"comment":"Section 2.1 reports collecting 2,300 queries and keeping 45% (1,000 examples), but 45% of 2,300 is approximately 1,035. Clarify the quality-control criterion or the exact counts.","section":"§2.1"},{"comment":"The legend lists 'Semantic Content Specification' twice (Green and Pink); one of these categories is presumably different. Also define the symbols (✓, ➞, and the blank/other marker) in the caption.","section":"Table 1"},{"comment":"Several metrics are labeled '↓' (lower better), but the AVG column seems to combine them with '↑' metrics. State the normalization formula (e.g., 1 - normalized error) so that AVG is interpretable.","section":"Table 4"},{"comment":"The appendix says the judge validation used 'a selected test set of 300 samples for either Realistic and Synthetic dataset'. Report the exact test-set sizes and whether Table 3 and Table 10 are based on the same sets, ideally with confidence intervals.","section":"Appendix C.3"},{"comment":"Reference [2] is titled 'Claude 3.5: A Sonnet', but the text mentions 'Claude-3.7-Sonnet' in the experimental setup. Please align the reference and the model version.","section":"References"},{"comment":"The phrase 'state-of-the-art systems' is broader than the evaluated set (three open-source unified models and six agentic frameworks). Recommend scoping the claim to the evaluated models, since Emu2-Gen was omitted and proprietary generators are not exhaustive.","section":"Abstract/Introduction"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and dataset are valuable, but the judge-selection inconsistency should be resolved before acceptance. I would ask the authors to rerun the overall assessments with the model that actually maximizes human alignment (GPT-4o) or provide a rigorous defense for GPT-4o-mini, and to give a derivation for the aggregate percentages. The paper does not need additional models if the claims are properly scoped."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know first. The empirical and data contributions are real: MultiRef-bench has 990 synthetic samples with 33 reference combinations and 1,000 real-world Reddit editing tasks, plus a 38k-image dataset built by the RefBlend compatibility-graph engine. The rule-based metrics for depth, mask, bbox, sketch, and semantic map are sensible, and the qualitative finding that current generators struggle to combine several visual references is likely correct. The paper earns credit for constructing a first-of-its-kind evaluation resource.\n\nThe soft spots are concentrated in the evaluation layer. Section 2.3 says GPT-4o-mini was chosen as the judge 'for its superior alignment with human judgment shown in Table 3,' and Appendix C.3 repeats that. But Table 3 and Table 10 show GPT-4o outperforming GPT-4o-mini on the two headline metrics, Instruction Following and Source Fidelity, on both realistic and synthetic splits (0.624 vs 0.530 and 0.668 vs 0.632 for IF). Since the central rankings in Tables 4 and 11 ride on IF/SF, this internal contradiction is not cosmetic. The judge-human correlations are also moderate (0.53–0.67) relative to human-human agreement (0.66–0.72), so small score gaps between models are not a solid basis for ordering. The abstract's 66.6% and 79.0% figures are not derived anywhere in the stated aggregation; they lack an audit trail.\n\nThe real-world part also assumes a single golden output per Reddit task, though the same request can have several valid results. That matters for absolute scores, though less for relative model comparisons on the same ground truth. One concern from the stress-test is overstated: the fine-tuned Qwen model is used to filter the synthetic dataset, not to evaluate models, so the claimed self-referential loop is not present. Dataset filtering could still introduce bias, but that is a different issue.\n\nThis is a paper a serious referee should see, with the explicit request to fix the judge justification and re-derive the headline numbers. The benchmark and dataset themselves are useful; I'd cite the resource while not quoting the percentages until the evaluation layer is cleaned up.","headline":"Real multi-reference benchmark and dataset, but the judge choice and headline numbers contradict the paper's own tables.","tokens_in":28250,"tokens_out":5943,"would_cite":true,"duration_ms":58905,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current image generators fail to combine multiple visual references, a new 1,990-sample benchmark shows.","keywords":["controllable image generation","multi-reference conditioning","benchmark","dataset","RefBlend","image editing","MLLM-as-a-judge","unified image generation models"],"falsifier":"Re-run human scoring on the benchmark's spatial-condition subsets—pose and bounding box—where the fine-tuned judge's Pearson correlation is only around 0.26 and 0.40; if human rankings of the tested models on those subsets differ from the GPT-4o-mini-based rankings, the claimed performance ordering would not hold on the conditions the benchmark most emphasizes.","tokens_in":1655,"feed_emoji":"🎨","tokens_out":2512,"duration_ms":78127,"temperature":0.7,"pith_summary":"The paper tries to establish that today's image generation systems cannot reliably follow multiple visual references at once—even though real creative work often blends several source images. It introduces MultiRef-bench, a benchmark of 990 synthetic and 1,000 real-world tasks, built with a new synthetic data engine called RefBlend and a 38k-image dataset of the same name. Across three unified models and six agentic frameworks, the best tested model, OmniGen, reaches only 66.6% on synthetic samples and 79.0% on real-world cases relative to the golden answer. The wider point is that multi-reference conditioning is a distinct, measurable capability that current single-source or loosely chained systems do not possess.","feed_headline":"Best image generator scores just 66.6% on multi-reference test","feed_subtitle":"A new benchmark pairs Reddit edit requests with synthetic controls; no tested model reliably blends several references at once.","key_machinery":"The argument is carried by a two-part measurement apparatus. RefBlend is the synthetic data engine: it extracts ten reference types (depth, Canny edges, masks, bounding boxes, poses, sketches, semantic maps, captions, subjects, styles) from source images and combines them under a reference compatibility graph that forbids contradictory pairings, yielding 33 valid reference combinations and a filtered dataset of 38,076 samples. MultiRef-bench then pairs this synthetic test split with 1,000 human-filtered Reddit editing tasks, and evaluates outputs through rule-based metrics (IoU, MSE, mAP, CLIP scores) plus a fine-tuned MLLM-as-a-judge that the paper validates against human scores for instruc","core_discovery":"The central claim is that no currently available generator reliably integrates content from multiple visual references, and that this failure can be observed and quantified. On the synthetic half of MultiRef-bench, the best unified model, ACE, averages 0.553 reference fidelity against a ground-truth average of 0.709, while OmniGen averages 0.464; on the real-world half, OmniGen scores 0.496 instruction following and 0.421 source fidelity, far below the golden answer's 0.767 and 0.706. Compositional pipelines such as Claude + SD3.5 produce competitive image quality but degrade instruction adherence and source fidelity, suggesting that separating a perceiver from a generator fundamentally hurt","pith_inferences":["If the judge-based ordering is sound, the 38k synthetic RefBlend samples become a ready training resource: a model fine-tuned on multi-reference pairings could close a large share of the reported gap, and MultiRef-bench's synthetic/real split would directly measure that gain.","The paper's own judge validation shows low human correlation on pose (Pearson 0.257) and bounding boxes (0.404) in its fine-tuned model; model rankings on those spatial conditions may be noisier than rankings on style or subject, and a human re-scoring of those subsets would be a cheap robustness test.","Because the real-world ground truth is a single selected Reddit output, tasks with multiple valid edits may underestimate model ability; a version of the benchmark that scores against several accepted outputs would be a natural extension.","The task taxonomy—replacement, addition, style, spatial manipulation, attribute transfer—could transfer to video or 3D editing, where multi-source conditioning is similarly untested."],"forward_implications":["MultiRef-bench gives any claimed general-purpose generator a concrete test: 1,990 multi-reference tasks with per-condition metrics for spatial, semantic, and style control.","Unified end-to-end architectures are the more promising direction for multi-reference generation, since separated perceiver-plus-generator frameworks consistently lose instruction following and source fidelity.","Captions are load-bearing for current models: removing a caption degrades depth fidelity and aesthetics across models, while sometimes improving sketch and semantic-map fidelity.","Input reference order is not commutative: changing the order of reference images shifts condition-specific fidelity more than global image quality, so prompt or input design has real effects on multi-reference success.","Spatial constraints are the weakest reference types across nearly all models, making bounding-box and pose fidelity a natural first target for future multi-reference training."],"supporting_citations":[{"why":"OmniGen is the best-performing unified model in the comparison and supplies the headline 66.6% synthetic / 79.0% real-world gap claim.","marker":"[56]"},{"why":"ACE is the unified-model baseline with the highest average reference fidelity on synthetic tasks, setting the bar the paper shows is still far below ground truth.","marker":"[19]"},{"why":"Show-o is evaluated through multi-turn dialogue as another unified image-text model, establishing that the failure is not specific to one architecture.","marker":"[57]"},{"why":"ChatDiT serves as the agentic framework baseline, representing training-free LLM-plus-diffusion pipelines in the comparison.","marker":"[26]"},{"why":"RealEdit is the source of the real-world benchmark design, supplying the Reddit r/PhotoshopRequest tasks with multiple input images and human-verified outputs.","marker":"[51]"},{"why":"MLLM-as-a-Judge is the method the paper fine-tunes for filtering and semantic-level scoring; its human-correlation validation underlies the evaluation protocol.","marker":"[5]"},{"why":"ControlNet is the canonical single-condition control method that the paper contrasts with the harder multi-reference setting.","marker":"[62]"},{"why":"IDEA-Bench is the prior design-oriented benchmark that the paper distinguishes from true multi-reference evaluation, motivating the new benchmark.","marker":"[33]"}],"fun_headline_variants":["AI image models fail to blend multiple visual references","Best AI image generator scores just 66.6% on multi-reference test","No AI image model reliably blends multiple references","Multi-reference image generation: even best models fall short","Even best AI image model scores 66.6% on MultiRef test"],"cache_read_input_tokens":29952,"weakest_assumption_plain":"The reported model rankings rest on trusting the MLLM judge's correlation with human judgment for instruction following and source fidelity, and on treating each Reddit output image as the single correct answer to its editing request.","fun_headline_variants_meta":{"raw":{"variants":["AI image models fail to blend multiple visual references","Best AI image generator scores just 66.6% on multi-reference test","No AI image model reliably blends multiple references","Multi-reference image generation: even best models fall short","Even best AI image model scores 66.6% on MultiRef test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001144,"raw_usage":{"total_tokens":4599,"prompt_tokens":779,"completion_tokens":3820,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":3737}},"tokens_in":523,"tokens_out":3820,"duration_ms":26131,"temperature":1.0,"reasoning_tokens":3737,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:25:32.531236+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run human scoring on the benchmark's spatial-condition subsets—pose and bounding box—where the fine-tuned judge's Pearson correlation is only around 0.26 and 0.40; if human rankings of the tested models on those subsets differ from the GPT-4o-mini-based rankings, the claimed performance ordering would not hold on the conditions the benchmark most emphasizes.","supporting_citations":[{"cited_title":"REALEDIT: Reddit Edits As a Large-scale Empirical Dataset for Image Transformations","cited_arxiv_id":"2502.03629","evidence_quote":"Show-o is evaluated through multi-turn dialogue as another unified image-text model, establishing that the failure is not specific to one architecture."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ChatDiT serves as the agentic framework baseline, representing training-free LLM-plus-diffusion pipelines in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RealEdit is the source of the real-world benchmark design, supplying the Reddit r/PhotoshopRequest tasks with multiple input images and human-verified outputs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ControlNet is the canonical single-condition control method that the paper contrasts with the harder multi-reference setting."},{"cited_title":"In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"IDEA-Bench is the prior design-oriented benchmark that the paper distinguishes from true multi-reference evaluation, motivating the new benchmark."}],"review_version":1}