{"id":"8a1ec32c-c60c-4f7e-9120-4819fc7a9e0e","arxiv_id":"2411.09449","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces Image Regeneration, an evaluation benchmark where text-to-image models must reproduce a reference image from MLLM-generated prompts, along with the ImageRepainter framework and two new datasets.","lead":"Researchers propose evaluating text-to-image AI models by having them recreate a reference image, with GPT-4V converting the image into detailed text prompts and an iterative refinement loop. They report that this 'image regeneration' ranking matches human judgments more closely than standard text-based scores, though the evaluation design uses the same metrics for selection and scoring.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Selection-oracle circularity is the load-bearing risk: the same CLIP/DINOv2/GPT-4V metrics choose the best candidate and then produce the final score, so reported rankings may reward search effort and output diversity rather than generative ability.","rationale":"The reader correctly identifies the selection/evaluation circularity as the load-bearing risk. My reading of the manuscript confirms that the 'Image selection' step uses CLIP, DINOv2, and GPT-4V to choose the best candidate, and the 'Evaluation Metrics' section uses the same three measures for the final table. The final score is thus an order statistic of a model-specific candidate distribution, which means the reported numbers conflate the T2I model's output distribution with the framework's search process. The paper's own iteration ablation is direct evidence that the search contribution is non-uniform: weaker or more seed-sensitive models benefit more from repeated attempts, so the gap between the selected maximum and the model's typical output varies across models. The human user study is real evidence, but it evaluates standard text-image alignment rather than regeneration-versus-reference similarity, so it does not establish that the selected-maximum scores are what humans would endorse. This is a testable methodological confound rather than a reason to reject the proposal: the task and framework are novel, and the concern can be settled by re-running without selection or by reporting selection gains. Since the reader's conditional verdict already accounts for this weakness, my independent read does not move the verdict.","tokens_in":843,"tokens_out":1006,"duration_ms":52646,"concrete_test":"Recompute Table 1 with selection disabled: for each model and each reference image, generate one image from a single initial IUT prompt (T=1, N=1) and score it with CLIP, DINOv2, and GPT-4V; also compute the mean score over all candidates in the final iteration. If the model ranking by single-sample or mean scores differs from the ranking by selected maxima (for example, SD1.5-DPO, SD2.0, SDXL1.0, or Juggernaut reorder), the reported result is driven by the selection oracle. Alternatively, measure the selection gain delta = max_score minus mean_score per model; large and varying delta across models would confirm the confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ImageRepainter 'aligns with human perception' rests on the scores in Table 1. In the 'Image selection' paragraph, CLIP, DINOv2, and GPT-4V are used to select the highest-scoring image from N_t candidates; in 'Evaluation Metrics', the same CLIP, DINOv2, and GPT-4V scores are then reported as the evaluation result. The final score is therefore a maximum over a model-dependent candidate set, not a typical sample from the model. Models with higher output diversity, or models whose outputs are more easily steered by GPT-4V-based prompt revision, can obtain larger search gains even if their average single-image regeneration quality is lower. The paper's own iteration ablation (Figure 9) states that iteration helps weaker or more seed-sensitive models more, so this gain is not uniform across models. No analysis separates selection gain from generative capability, and the human user study rates standard text-image pairs rather than regeneration outputs against reference images, so it does not directly validate that these selected maxima align with human perception. Without such separation, the reported rankings cannot be attributed to the T2I model's generative ability alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new evaluation protocol for text-to-image (T2I) models, called image regeneration: given a reference image, an MLLM (GPT-4V) converts it into a text prompt (organized as an Image Understanding Tree, IUT), the T2I model generates an image, and the regenerated image is compared with the reference using CLIP, DINOv2, and GPT-4V scores. The authors introduce ImageRepainter, an iterative framework that revises prompts and selects the highest-scoring candidate across iterations, and two benchmarks (content-diverse and style-diverse) for evaluation. They report that their method ranks models more consistently with a user study than existing text-image alignment metrics such as T2I-CompBench, HPSv2, and PickScore, and claim that visual-to-visual assessment better aligns with human perception.","tokens_in":12913,"tokens_out":2733,"duration_ms":25774,"significance":"If validated, the image regeneration task would provide a genuinely new paradigm for T2I evaluation that removes the cross-modal asymmetry of text-image matching, and the IUT-based prompt construction plus iterative refinement is a reasonable way to operationalize the task. The paper also proposes two diverse benchmarks and compares a range of recent diffusion models, including community fine-tunes, which is useful for the community. The central weakness is that the same metrics used to select the best candidate are then used as the final evaluation score, so the reported rankings may partly reflect the search effort of the framework rather than the generative capability of the T2I model; furthermore, the claimed human alignment is not directly supported by the described user study. The paper does not release datasets or code, which limits reproducibility.","major_comments":[{"comment":"The selection step and the evaluation step use exactly the same three metrics: CLIP, DINOv2, and GPT-4V. In the 'Image selection' paragraph, the framework selects the highest-scoring image among N_t candidates using these metrics; in 'Evaluation Metrics', the final score for a model is computed with the same metrics on the selected image. Consequently, the reported score is a best-of-N maximum over a model-dependent candidate set, not a typical sample from the model. For a model with higher output diversity, or with outputs that are more easily improved by the GPT-4V-based prompt revision loop, the selection gain will be larger regardless of average single-image regeneration quality. The paper's own iteration ablation (Figure 9) states that weaker or more seed-sensitive models benefit more from iterations, so the gain is not uniform across models. The authors should report the score at the first iteration, the average score over all candidates, or a held-out metric not used in selection, and explicitly quantify the selection gain. Without this separation, the rankings in Table 1 cannot be attributed to the T2I model's generative ability alone.","section":"Methodology, 'Image selection' and 'Evaluation Metrics'"},{"comment":"The user study is described as rating content consistency and perceptual quality of 'text-image pairs' following the ImagenHub template, i.e., standard text-to-image alignment judgments. This does not directly evaluate the regeneration outputs against the reference images, which is the actual claim of the paper. The assertion that 'image regeneration for evaluation aligns more closely with human annotations' (Section 'Evaluating T2I Models') is therefore not substantiated by the reported experiment. The authors should either conduct a human study in which annotators compare regenerated images with the reference image, or at least compute a correlation (e.g., Spearman rank correlation) between the regeneration scores and human preference scores on the same set of model outputs. A qualitative statement in the text and a visual inspection of a few cases are not sufficient to support the human-alignment claim.","section":"Experiments, 'Human evaluation' and Table 1"},{"comment":"No variance or statistical significance is reported for any of the main evaluation tables. The datasets contain 100 or 200 samples, so the CLIP/DINO/GPT-4V scores presumably have non-negligible sampling error; moreover, the reported differences between some models are small (e.g., SD1.5 vs SD2.0 in Table 1). The authors should report standard deviations, confidence intervals, or significance tests (e.g., bootstrap or paired tests) to establish that the observed rankings are not noise. This is particularly important because the paper's central claim is that its method can distinguish T2I models more effectively than existing metrics.","section":"Table 1, Table 2, Table 3"},{"comment":"The two benchmarks are described only briefly and are not released, and no code is provided for IUT construction or the iterative selection procedure. The paper states that the content-diverse benchmark is 'manually collected through an open-source creation platform' and normalized using ChatGPT, but gives no URLs, license information, or detailed statistics beyond Figure 5. Without the actual datasets or a detailed construction protocol, the quantitative results cannot be reproduced or compared against in future work. The authors should release the benchmarks and code, or at least provide full prompt templates, sample images, and the complete list of text prompts used.","section":"Evaluation Datasets"}],"minor_comments":[{"comment":"There are several typos: 'multimodel' should be 'multimodal', 'T2M' should be 'T2I', and 'mutimodel' in the contributions list should be 'multimodal'.","section":"Abstract and Contributions"},{"comment":"The phrase 'more insensitive to fine-grained information' is awkward and presumably means 'less sensitive to fine-grained information' or 'more robust to fine-grained variations'; please rephrase.","section":"Conclusion"},{"comment":"The text says 'perceputual information', which should be 'perceptual information'.","section":"Methodology, 'Feedback generation'"},{"comment":"The authors state they use 'ChatGPT' for pure text tasks and specifically 'text-davinci-003'; text-davinci-003 is an InstructGPT model, not ChatGPT, and the terminology should be corrected for accuracy.","section":"Implementation Details"},{"comment":"The DINO score for JuggerXL v9 'Direct' in Table 3 is 94.83, but the text and Figure 7 plots would benefit from a consistent formatting of decimal points (the table uses 94.83 and the figure uses 94.83%, which is fine); more importantly, the caption of Figure 7 says 'a significant improvement', but no statistical test is provided to support 'significant'.","section":"Figure 7 and Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper presents an interesting evaluation idea, but the current evidence for the central claim is undermined by the selection-evaluation circularity and by the mismatch between the user study and the regeneration setting. The missing release of data and code also raises a reproducibility concern that the editor may want to weigh, especially for a benchmark-oriented submission. I would lean toward giving the authors the opportunity to address these issues in a revision rather than rejecting outright, because the core task formulation has value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nOne thing to know first: the image-regeneration evaluation idea is genuinely new. Instead of scoring text-image alignment, you give the T2I model a reference image, use GPT-4V to turn it into a structured prompt (the Image Understanding Tree), let the model generate, iterate, and pick the best candidate, then score the chosen image against the reference. Same-modality comparison is intuitive, and the IUT prompt structure is a reasonable mechanism.\n\nWhat the paper does well: the two benchmarks (content-diverse, style-diverse) are sensible if released. The comparison against CLIP-interrogator shows the IUT-based prompting helps on DINOv2 and GPT-4V metrics. Table 2 gives some evidence that regeneration separates models better than direct GPT-4V text-image matching.\n\nThe soft spot is load-bearing. The image-selection step uses CLIP, DINOv2, and GPT-4V to pick the highest-scoring candidate, and the evaluation metrics are the same three scores. So the reported number is a best-of-N maximum, not a typical sample. The paper's own iteration ablation (Figure 9) says weaker, more seed-sensitive models benefit more from iteration, so the search gain is not uniform. The rankings in Table 1 could reflect output diversity and searchability as much as generative ability. The human user study does not resolve this: annotators rate standard text-image pairs, not the regenerated selections against the reference, so the 'aligns with human perception' claim is not directly validated. Smaller issues: no variance, no correlation coefficient, unspecified aggregation of the three scores, and no code or data.\n\nThe central idea is not dead, and the direction is worth pursuing. But as written, the evaluation is confounded and the paper overclaims. This deserves peer review because the protocol could become a useful benchmark, but the authors need to separate selection from evaluation or also report typical samples, release the benchmarks and prompts, and run a user study on the actual regeneration outputs with a quantitative correlation.\n\nFor you: worth a reading-group slot, but I would not cite it yet.\n\nBest,","headline":"The regeneration-task idea is genuinely new and worth discussing, but the selection-and-score loop uses the same metrics twice, so the reported rankings don't cleanly support the human-alignment claims.","tokens_in":13470,"tokens_out":3497,"would_cite":false,"duration_ms":30843,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a text-to-image model is best judged by asking it to reproduce a reference image and comparing image to image, rather than matching output text to prompt.","keywords":["image regeneration","text-to-image evaluation","multimodal large language models","GPT-4V","ImageRepainter","Image Understanding Tree","diffusion models","image generation benchmark"],"falsifier":"Run the ImageRepainter pipeline on a fixed set of reference images but replace the image-selection step with random selection from the same candidates; if the resulting model rankings roughly match the reported ones, selection is not inflating scores, but if they diverge, the metric is measuring search effort. Additionally, a larger pairwise human study asking which of two regenerations better matches the reference would either confirm or weaken the claimed alignment with human perception.","tokens_in":12493,"feed_emoji":"🎨","tokens_out":6331,"duration_ms":55279,"temperature":0.7,"pith_summary":"The paper is trying to establish that text-to-image models should be evaluated by giving each model a reference image and asking it to generate a matching image, then scoring the regenerated image against the reference, rather than by matching the generated image to the original text prompt. Because both sides of the comparison are images, the authors argue, the evaluation avoids the information asymmetry of cross-modal text-image matching and mimics how humans judge a painting reproduction. They implement this as ImageRepainter, which uses GPT-4V to turn the reference into a structured prompt and iteratively revises and selects candidate images, and they report rankings of current diffusion models that agree with human Likert ratings more closely than CLIP score, T2I-CompBench, HPSv2, or PickScore. The practical stakes are that model leaderboards could become more trustworthy and interpretable without relying on brittle text alignment or expensive human studies.","feed_headline":"Judge text-to-image models by how they repaint a reference","feed_subtitle":"The iterative GPT-4V pipeline ranks diffusion models closer to human ratings than CLIP or PickScore.","key_machinery":"ImageRepainter's load-bearing component is the image understanding tree (IUT): a hierarchical JSON-like description of the reference image extracted by GPT-4V, containing the overall caption, global features, objects, object relations, and per-object detailed attributes. The framework then runs an iterative loop of prompt expansion, image generation, candidate selection, feedback generation, and prompt revision, using CLIP and DINOv2 for coarse semantic and visual similarity and GPT-4V for fine-grained content and perceptual scoring. IUT supplies the richer, more accurate text that lets a T2I model approach the reference, while the selection and feedback cycle stabilizes generation and concentrates each revision on one aspect such as style, color, or detail per iteration.","core_discovery":"The central claim is that a reference image, not a text prompt, is the right probe for measuring what a text-to-image model can generate: a good model should be able to reproduce the content and style of a given image after that image is translated into text through a multimodal large language model. The paper supports this with the ImageRepainter framework, whose Image Understanding Tree (IUT) organizes reference-image details into a hierarchy of overall features, objects, relationships, and fine attributes; an iterative loop then generates multiple synonymous prompts, produces candidate images, selects the best by CLIP, DINOv2, and GPT-4V similarity, and revises the prompt based on observed differences. On two new benchmarks, content-diverse and style-diverse, the paper reports that this visual-to-visual regeneration ranks models such as SD1.4, SD1.5, SD2.0, SDXL1.0, and JuggernautXLv1/v9 in closer agreement with human perception than existing text-image alignment metrics.","pith_inferences":["Beyond the paper: because the selection step uses the same CLIP, DINOv2, and GPT-4V metrics as the final score, a model with high output diversity may be advantaged even if a single random sample is mediocre; a paired experiment with fixed inference budget could separate generation ability from search efficiency.","Beyond the paper: the IUT prompt-revision loop could double as a benchmark for prompt engineering itself, since a robust T2I model should recover a reference image under many synonymous phrasings of the same content.","Beyond the paper: the claimed visual-to-visual advantage could be stress-tested by injecting small edits into the reference image and measuring whether the regeneration scores respond proportionally, which would reveal how sensitive the protocol really is to fine-grained detail."],"forward_implications":["If the image-regeneration protocol is accepted, T2I models can be ranked by reference-image reproduction without relying on any text-image alignment metric.","The iterative ImageRepainter pipeline yields higher CLIP, DINOv2, and GPT-4V regeneration scores than using CLIP-interrogator captions, for both content and style.","The content-diverse benchmark ranks the evaluated diffusion models in closer agreement with user-study Likert ratings than T2I-CompBench or PickScore do.","Iterative revision matters more for weaker models, while stronger models like JuggernautXLv9 can produce similar-quality images with fewer iterations, suggesting the protocol can also expose generation consistency."],"supporting_citations":[{"why":"Supplies the CLIP similarity metric used both for candidate selection and for final image-to-image scoring.","marker":"(Radford et al. 2021)"},{"why":"Supplies the DINOv2 visual-similarity metric, which the paper uses because it is more sensitive to lighting and color tone than CLIP.","marker":"(Oquab et al. 2024)"},{"why":"Provides the GPT-4V multimodal model that builds the Image Understanding Tree, generates feedback, and scores content consistency and perceptual quality.","marker":"(OpenAI et al. 2024)"},{"why":"Provides T2I-CompBench, the text-image compositionality baseline the proposed method is compared against.","marker":"(Huang et al. 2023)"},{"why":"Supplies SDXL1.0, a state-of-the-art T2I model whose ranking differs between existing metrics and the proposed image-regeneration method.","marker":"(Podell et al. 2023)"},{"why":"Supplies SD1.5-DPO, a human-preference-finetuned model included in the model rankings.","marker":"(Wallace et al. 2023)"},{"why":"Supplies the ImagenHub Likert-scale human evaluation template used to validate the alignment of the proposed rankings with human perception.","marker":"(Ku et al. 2024)"},{"why":"Provides the CLIP-interrogator baseline image-understanding method that ImageRepainter is quantitatively compared against in the regeneration experiments.","marker":"(Li et al. 2022)"}],"fun_headline_variants":["Repaint a reference to judge text-to-image models","Image regeneration: new benchmark ranks diffusion models","Test T2I models by making them regenerate an image","GPT-4V grades diffusion models via iterative re-painting","Reference image as prompt: better evaluate image generators"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that selecting the best of several candidate images and then scoring it with the same metrics measures the generation model's capability, not the evaluator's search effort.","fun_headline_variants_meta":{"raw":{"variants":["Repaint a reference to judge text-to-image models","Image regeneration: new benchmark ranks diffusion models","Test T2I models by making them regenerate an image","GPT-4V grades diffusion models via iterative re-painting","Reference image as prompt: better evaluate image generators"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1462,"prompt_tokens":992,"completion_tokens":470,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":394}},"tokens_in":608,"tokens_out":470,"duration_ms":5436,"temperature":1.0,"reasoning_tokens":394,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:37:36.409985+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the ImageRepainter pipeline on a fixed set of reference images but replace the image-selection step with random selection from the same candidates; if the resulting model rankings roughly match the reported ones, selection is not inflating scores, but if they diverge, the metric is measuring search effort. Additionally, a larger pairwise human study asking which of two regenerations better matches the reference would either confirm or weaken the claimed alignment with human perception.","supporting_citations":[],"review_version":1}