{"id":"e72f63ce-1792-4680-bf36-ba074f769dad","arxiv_id":"2610.32120","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ImageRSI couples verification-driven repair, cross-task experience library, and judge self-evolution to achieve higher checklist accuracy on complex image-generation requests.","lead":"ImageRSI is a framework that improves text-to-image generation by verifying and repairing each output, storing successful repair strategies for future requests, and updating its own judge model from past judgments. It achieves state-of-the-art scores on two complex image-generation benchmarks, showing that a frozen generator can be made more reliable through structured feedback and learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Experience library is constructed using the same evaluator (Gemini 3.1 Pro) used for final CSA evaluation, raising concern that gains reflect overfitting to evaluator biases rather than genuine improvement in image generation.","rationale":"The Reader identified the reliance on a single external evaluator without human validation as the weakest assumption, which is correct. My stress-test sharpens this into a specific technical concern: the experience library admission protocol uses the same evaluator (Gemini 3.1 Pro) that produces the final CSA numbers, introducing a risk of indirect overfitting that the Reader did not explicitly articulate. The Reader's verdict (CONDITIONAL) already captures the need for validation, but the concrete test I propose is distinct (focusing on the experience library pipeline rather than general evaluator bias). I recommend keeping the verdict CONDITIONAL because the paper's ablations and design are otherwise thorough, and the concern is testable. I do not call for rejection because the paper's methodology (frozen generator, controlled budgets, A/B validation) is sound in principle; only the evaluator dependence needs external confirmation.","tokens_in":20913,"tokens_out":4168,"duration_ms":37411,"concrete_test":"Randomly sample 100 test-set requests (50 from each benchmark). Have two independent human raters and a different VLM (e.g., GPT-5-Vision) re-judge all checklist items for both (a) the ImageRSI output and (b) the strongest baseline output (GEMS for MindBench, Nano Banana for ImageReq-Bench). Compute CSA with these new verdicts and compare the gap to the reported gap with Gemini 3.1 Pro. If the gap shrinks by more than 3 pp (one-tailed), then the experience library's gains are partly evaluator-specific and the claim of genuine improvement is weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim rests on CSA gains from two components: localized repair and cross-task experience reuse. The experience library (§3.3) admits candidates based on paired A/B validation where the evaluator, 'u' in Equation 2, is the score assigned by a blinded evaluator. In all experiments, this evaluator is Gemini 3.1 Pro—the same model used for the final CSA evaluation on the test set (§4.1.1). While the paper states that the external evaluator is independent of the internal judge (Appendix B.2), it does not state that the experience-admission evaluator is a different model. The calibration and holdout phases (§A.4) select experiences that produce positive Δe(q) on the experience set according to Gemini scores. If Gemini has systematic biases—e.g., rewarding certain prompt structures, image layouts, or text rendering strategies—then experiences that exploit those biases on the experience set will also inflate CSA on the test set, even if the images are not genuinely more faithful to the user request. The 10.0 pp gain from experience reuse (Figure 3b) could therefore be partially spurious. The paper provides no human agreement study or cross-evaluation with a different VLM to rule out this form of evaluator overfitting. This is the most load-bearing concern because it threatens the validity of the headline numerical claims and the claim that cross-task experience 'improves performance' (§4.3). The concern is specific: the admission evaluator and the final evaluator are the same model, and no evidence is given that gains transfer to a different judge.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes ImageRSI, a framework for agentic image generation that keeps the generator frozen and combines three components: (i) within-task five-dimensional verification and targeted repair, (ii) a cross-task experience library with paired A/B validation before admission, and (iii) offline judge self-evolution via SFT and GRPO on accumulated image-judgment records. The authors also introduce ImageReq-Bench, a benchmark of 961 complex requests with evaluation checklists. Experimental results report state-of-the-art Checklist Strict Accuracy (CSA) on MindBench (52.6% with Nano Banana Pro) and ImageReq-Bench (65.66%), surpassing strong baselines by 10.2 and 9.28 percentage points respectively. Ablations demonstrate the contributions of localized repair, requirement locking, increasing experience library size, and judge training.","tokens_in":21462,"tokens_out":3176,"duration_ms":29287,"significance":"If the reported results are robust, the paper makes a meaningful contribution to agentic image generation by showing that a frozen generator can be made significantly more reliable through a combination of localized verification, cross-task experience reuse with rigorous admission, and self-evolution of the judge. The introduction of ImageReq-Bench fills a gap for evaluating complex multi-constraint requests. The methodology is modular and the ablations (repair strategies, library size, judge training) are thorough, providing clear evidence for the source of gains. The central idea of validating candidate experience via paired A/B testing before reuse is principled and could influence future work on experience-driven improvement.","major_comments":[{"comment":"The paper does not state whether the blinded evaluator 'u' used for paired admission of candidate experiences (Equation 2) is the same model as the external evaluator (Gemini 3.1 Pro) used for the final CSA metric. If they are the same model, the experience library is constructed using judgments from the same evaluator that later certifies the test-set performance. This creates a risk that admitted experiences exploit systematic biases of Gemini 3.1 Pro rather than producing genuinely better images, and the reported gains from experience reuse (10.0 pp, Figure 3b) could be partially spurious. The paper should clarify the identity of the admission evaluator; if it is the same model, provide a human agreement study or cross-validation with a different VLM to demonstrate that the gains are not driven by evaluator overfitting.","section":"§3.3, Eq. (2); §4.1.1; §4.3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The judge self-evolution trains on records generated by earlier versions of the judge itself (including SFT and GRPO targets derived from model-generated annotations). This could lead to confirmation bias where the judge reinforces its own scoring patterns. Although the paper states that external test verdicts are not fed back, the training data are derived from the system's own evaluation trajectories. The paper does not analyze how much of the improvement in CSA (from 38.6% to 48.2% in Table 3) is due to genuine better alignment with human preferences versus adaptation to the internal scoring style. A human evaluation on a subset of test-set images or a correlation study of judge scores with human ratings would strengthen this analysis.","section":"§3.4; Table 3","authors_acknowledge":"How- ever, a single successful repair does not establish that a strategy is reliable or transferable: stochastic generation can produce improvements by chance, redundant trajectories can overstate the evidence, and judge errors can reinforce incorrect diagnoses.","authors_acknowledge_p":0.61}],"minor_comments":[{"comment":"The title of Section 3.4 uses 'Judger Self-Evolution' while the rest of the paper uses 'judge'. Please standardize.","section":"Throughout","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The calibration and holdout thresholds (e.g., mean difference >=0.05) are given without justification for their values. A brief note on how these thresholds were chosen would improve reproducibility.","section":"Appendix A.4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The acceptance threshold (aggregate score >=4.5) and weight vector are introduced without sensitivity analysis. While the offline ablation in Figure 5 is helpful, a brief discussion of how sensitive the main results are to the threshold value would be informative.","section":"§4.1.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The definition of CSA counts evaluator errors as failures. It would be helpful to report the number of evaluator errors per experiment to ensure comparability across methods.","section":"§B.2, Eq. (3)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The description of the GRPO reward mentions a 'separate lower band' for severe false accepts and critical fabrications but does not specify the numerical reward values or the size of the ranking adjustment. Providing these details would aid reproducibility.","section":"Appendix D.2","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper presents a sensible and well-ablated framework for improving text-to-image generation by combining within-task verification/repair, cross-task experience reuse, and offline judge fine-tuning. The 10–27 pp CSA gains over strong baselines are substantive, and the ablation studies cleanly isolate the contribution of each component. The experience library with paired A/B admission is a careful design choice. The judge self-evolution, though limited to one cycle, shows a clear improvement over the base model.\n\nThe main weakness—and the one that keeps me from fully trusting the numbers—is the complete reliance on Gemini 3.1 Pro as the external evaluator for both CSA and experience admission. The stress test point is valid: the same model that admits experiences is also the oracle for the final metric. If Gemini has systematic biases (e.g., toward certain prompt structures or layout styles), the reported gains could partly reflect overfitting to those biases. The paper acknowledges the need for human validation only implicitly, and no inter-rater reliability is reported. Without at least a small human agreement study or a cross-check with a different VLM, the central claim is conditional.\n\nMinor points: the repeated self-evolution cycle is deferred, so the 'recursive' part is not yet demonstrated. The ImageReq-Bench categories are acknowledged as organizational rather than a validated taxonomy. The judge training uses model-generated annotations, which introduces a circularity risk, though the end-to-end metric is external.\n\nOverall, this is serious work with transparent experiments. The weaknesses are acknowledged in the Limitations section, which is to the authors' credit. I think the paper deserves a proper peer review; the authors should be asked to address the evaluator-dependence with human validation or a second evaluator. For a reader interested in agentic image generation, this is worth reading and engaging with critically.","headline":"A well-engineered framework with strong CSA gains, but the reliance on a single external evaluator (Gemini 3.1 Pro) without human validation makes the headline numbers fragile.","tokens_in":21920,"tokens_out":1214,"duration_ms":17695},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"ImageRSI, a framework that keeps the generator frozen and combines within-task repair, cross-task experience, and judge self-evolution, achieves state-of-the-art Checklist Strict Accuracy on MindBench (52.6%) and ImageReq-Bench (65.66%)…","keywords":["text-to-image generation","agentic image generation","verification and repair","cross-task experience","judge self-evolution","frozen generator","checklist strict accuracy","complex instruction following"],"falsifier":"A human agreement study on 100 randomly sampled test cases from ImageReq-Bench, comparing human verdicts to Gemini 3.1 Pro verdicts on each checklist item, would determine whether the reported CSA improvements reflect genuine requirement satisfaction or evaluator artifacts. If Cohen's kappa falls below 0.8, the reported numbers should be interpreted with caution.","tokens_in":22461,"feed_emoji":"🎨","tokens_out":6649,"duration_ms":57650,"temperature":0.7,"pith_summary":"The paper presents ImageRSI, a system that makes a frozen image generator more reliable by adding a verification-and-repair loop, a validated experience library, and an offline self-evolving judge. Rather than training the generator, it improves the process: five-dimensional scoring localizes defects, prompts are rewritten while locking satisfied requirements, effective strategies from past tasks are admitted only after A/B testing, and the judge is periodically updated with SFT and GRPO on its own accumulating records. The key claim is that these three mechanisms compound to yield substantial gains: ImageRSI achieves 52.6% CSA on MindBench (with Nano Banana Pro) and 65.66% on the new ImageReq-Bench, outperforming both generative models and prior agent systems by wide margins. The paper also introduces ImageReq-Bench, a 961-request benchmark with per-request checklists for evaluating joint requirement satisfaction.","feed_headline":"Self-improving verification lifts image accuracy by 10+ points","feed_subtitle":"ImageRSI combines targeted repair, cross-task experience, and offline judge training to set state-of-the-art results on two benchmarks.","key_machinery":"The key machinery is a three-component architecture operating on a frozen generator: (a) an online verification-repair loop that uses a five-dimensional judge (Alignment, Faithfulness, Quality, Aesthetic, Physics) to score images, localize defects, and rewrite the prompt while locking parsed requirements; (b) a cross-task experience library that stores candidate strategies with trigger conditions, admits them only after paired A/B calibration and holdout, and activates them for future tasks; (c) an offline judge self-evolution cycle that collects trajectory image-judgment pairs, performs SFT on score-and-reason outputs, then GRPO with a reward combining pointwise accuracy and within-group ranking, and redeploys the judge only after evaluation. The generator remains frozen throughout.","core_discovery":"ImageRSI's central discovery is that a frozen text-to-image generator can be made substantially more reliable by coupling three process-level improvements: 1) five-dimensional verification (Alignment, Faithfulness, Quality, Aesthetic, Physics) that localizes defects and guides prompt repair while locking already-satisfied requirements; 2) a cross-task experience library that admits candidate repair strategies only after paired A/B validation, ensuring reused strategies are genuinely beneficial; and 3) offline judge self-evolution that collects trajectory images, trains with supervised fine-tuning and GRPO on accumulated judgment records, and redeploys the judge only after evaluation. The generator itself is never fine-tuned. The system achieves state-of-the-art Checklist Strict Accuracy on MindBench (52.6% with Nano Banana Pro) and ImageReq-Bench (65.66%), outperforming the strongest baselines by 10.2 and 9.28 percentage points respectively. Ablations confirm that localized repair with requirement locking contributes 12.3 pp, the experience library adds up to 10.0 pp, and judge post-training (SFT+GRPO) adds 9.6 pp over the untrained base judge.","pith_inferences":["The reliance on Gemini 3.1 Pro as the external evaluator is a critical dependency; if the evaluator has systematic biases, the reported CSA numbers may not align with human judgment. A human evaluation on a subset would clarify the practical significance.","The three-component framework (verify, reuse, evolve) could be adapted to other constrained generation domains such as video generation, 3D scene assembly, or document layout, where verification criteria can be explicitly specified.","The paper does not analyze the computational cost of verification and repair cycles; in high-throughput applications, the added inference cost might outweigh the quality gains, and a cost-benefit analysis would be useful."],"forward_implications":["Frozen generators can be made more reliable through process improvements alone, reducing the need for costly retraining or fine-tuning.","The experience library's monotonic gain with size suggests that continued collection across diverse tasks would yield further improvements, potentially enabling a compounding effect.","The judge self-evolution method (SFT+GRPO) on the system's own records outperforms off-the-shelf larger judges, indicating that task-specific training on generated data is valuable.","The ImageReq-Bench benchmark provides a reusable evaluation suite for multi-constraint image generation, enabling standardized comparison of agentic approaches.","The methodology is generator-agnostic; the paper demonstrates similar gains across three different generators (Qwen-Image-2.0, Wan2.7-Image, Nano Banana Pro)."],"supporting_citations":[{"why":"Provides the MindBench benchmark and the basline results for comparison.","marker":"(He et al., 2026b)"},{"why":"Supplies the strongest agent baseline (GEMS) that ImageRSI surpasses.","marker":"(He et al., 2026c)"},{"why":"Provides the GRPO optimization method used for judge post-training.","marker":"(Shao et al., 2024)"},{"why":"Provides the Qwen-Image-2.0 generator used in MindBench experiments and as one of the frozen generators.","marker":"(Zhao et al., 2026)"},{"why":"Provides the WISE taxonomy and prompt design that guide construction of additional training requests for judge training.","marker":"(Niu et al., 2025)"}],"fun_headline_variants":["Frozen image generator gains 10+ points via self-improving verification","ImageRSI boosts text-to-image reliability by 10 points","Three process improvements yield 10-point reliability gain in image generation","Verification-driven self-improvement lifts image accuracy by 10 points","Frozen generator + cross-task experience + judge training = 10-point gain"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The entire evaluation pipeline relies on Gemini 3.1 Pro as the external evaluator to provide accuracy binary verlits for each checklist item; if this evaluator is biased or inconsistent, the reported CSA numbers and the experience distillation from the experience set could be unreliable.","fun_headline_variants_meta":{"raw":{"variants":["Frozen image generator gains 10+ points via self-improving verification","ImageRSI boosts text-to-image reliability by 10 points","Three process improvements yield 10-point reliability gain in image generation","Verification-driven self-improvement lifts image accuracy by 10 points","Frozen generator + cross-task experience + judge training = 10-point gain"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000922,"raw_usage":{"cost":0.000922348,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000922348,"upstream_inference_prompt_cost":0.000105588,"upstream_inference_completions_cost":0.00081676},"total_tokens":3876,"prompt_tokens":959,"completion_tokens":2917,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":2824}},"tokens_in":959,"tokens_out":2917,"cost_source":"openrouter_usage","duration_ms":25044,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":2824,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-26T13:15:56.450897+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"A human agreement study on 100 randomly sampled test cases from ImageReq-Bench, comparing human verdicts to Gemini 3.1 Pro verdicts on each checklist item, would determine whether the reported CSA improvements reflect genuine requirement satisfaction or evaluator artifacts. If Cohen's kappa falls below 0.8, the reported numbers should be interpreted with caution.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}