{"id":"c6d4b5af-913f-47c2-a8ba-56a0c02b653a","arxiv_id":"2608.11907","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"By asking a unified multimodal model to caption an image, regenerate it, and answer questions about the regenerated image, SGU consistently exposes accuracy drops that separate benchmarks do not reveal.","lead":"Unified AI models that both understand and generate images usually get tested on each skill separately. This paper introduces a closed loop: the model describes an image, redraws it from its own description, then answers questions about its redrawn picture, exposing gaps that separate benchmarks miss.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SGU's final VQA may answer from text rendered into the reconstructed image rather than from semantic visual understanding; the image-swap test does not control for this.","rationale":"The central claim requires that the final answer reflects genuine visual understanding of the self-generated context. The paper's only dedicated control, the image-swap test, is limited to a high-scoring MMBench subset and only checks model-specific self-consistency cues; it cannot detect a text round-trip shortcut in which both models read the same caption-derived text rendered into the image. This is exactly the 'text remnants leaked from the caption' scenario named in the reader's weakest assumption, so I agree with the reader's identification of the load-bearing risk. The concern is concrete: OCR-VQA is one of the four benchmarks, SGU gaps are largest there, and a headline example (UniWorld vs OmniGen2) is drawn from it, so a positive text-masking result would remove support for interpreting SGU as a semantic visual-reasoning metric on that benchmark. It would not, however, invalidate the framework entirely: MMStar, MMBench, and MathVista results, as well as the consistent drops across models, still support the weaker claim that UMMs lose performance through the generation loop. The omission is fixable by adding the masking control, which is why the reader's CONDITIONAL verdict remains appropriate rather than moving to ACCEPT or REJECT.","tokens_in":17557,"tokens_out":13245,"duration_ms":142591,"concrete_test":"Run the SGU loop on a fixed subset of OCR-VQA (e.g., the paper's 200-sample subset) and then rerun the final VQA stage with detected text regions in each reconstructed image masked (using an off-the-shelf OCR detector plus inpainting), keeping stages 1-2 and the question set fixed. For each model compute Δ_umm = s_umm - s_umm_masked and compare with Δ_base = s_base - s_base_masked, where s_base_masked is direct VQA on original images with text regions masked. If Δ_umm significantly exceeds Δ_base (e.g., >10 points) for any model, SGU on OCR-VQA is substantially driven by reading text embedded in the generated image; extend the same text-masking check to MMBench and MathVista charts to assess the generality of the shortcut.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (4) treats final VQA accuracy on the reconstructed image as the SGU score, but that image is generated from the model's own textual description. On OCR-VQA and other text-containing benchmarks, the captioning stage transcribes text from the original image, and the generation stage may render that text (legibly or partially) into the reconstructed image. The final VQA stage can then answer by reading the embedded text (e.g., recognizing a book title printed in the generated image) instead of performing semantic visual reasoning about the regenerated scene. This is a text round-trip shortcut: the answer is laundered through the image rather than derived from visual understanding. The image-swap test (Table 4) does not settle this. It is run only on a high-scoring MMBench subset (198 samples, general perception) where text rendering is rarely the answer channel, and comparing two models on the same rendered images leaves cross/self differences small even if both models read the same embedded text. The paper also reports no language-prior control or text-masking control for the final VQA stage. If this shortcut operates, SGU's 'integrated semantic understanding-generation' interpretation fails on text-heavy tasks, and model ranking differences (e.g., UniWorld 28.67 vs OmniGen2 56.59 on OCR-VQA) may reflect text-rendering fidelity rather than holistic UMM capability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Self-Generative-Understanding (SGU), an evaluation framework for unified multimodal models (UMMs) that runs a three-stage closed loop: the model generates a textual description of an input image, generates a reconstructed image from that description, and then answers the original VQA question from the reconstructed image. The SGU score is the final VQA accuracy, optionally normalized by direct VQA accuracy on the original image. The authors evaluate six UMMs on four benchmarks (MMStar, MMBench, MathVista, OCR-VQA) and report a consistent drop from direct VQA accuracy to SGU accuracy, plus stage-wise replacement experiments, a prompt-sensitivity study, an image-swap shortcut check, and auxiliary CLIP-based similarities. The central claim is that SGU provides a complementary, annotation-free, system-level evaluation that exposes integration failures not captured by separate understanding or generation metrics.","tokens_in":17882,"tokens_out":5002,"duration_ms":52767,"significance":"If the SGU score measures what it claims, it is a useful complementary evaluation axis: it is annotation-free, reuses existing VQA benchmarks, is model-internal with no external judge, and its stage-wise diagnostics can localize bottlenecks. The empirical pattern across six models and four datasets is consistent and the stateless protocol is carefully described. However, the validity of the metric is not yet established for text-centric benchmarks because of a possible text round-trip shortcut, and the statistical support is limited to single-run point estimates. The proposition in Appendix C.2 is circular and formally under-specified. With additional controls and variance reporting, the framework could become a valuable benchmarking tool, but the current evidence is insufficient to support the strong interpretation placed on the OCR-VQA results.","major_comments":[{"comment":"The text round-trip shortcut is not controlled. On OCR-VQA, Phase 2 can render text from the caption into the reconstructed image, and Phase 3 can then answer by reading embedded text rather than by semantic visual understanding of the regenerated scene. The image-swap test is run only on a high-scoring MMBench subset of 198 general-perception samples and compares models on identical rendered images, so it cannot detect this channel; it also does not include OCR-VQA. I request a text-masking or text-removal control (e.g., blanking detected text regions of the reconstructed image before final VQA), a caption-only control without any image, and an OCR-VQA-specific breakdown or image-swap test. Without these, the claim that SGU measures integrated semantic understanding-generation on text-centric benchmarks is not established.","section":"§4.2 (Q4) and Table 4"},{"comment":"Every reported s_base and s_umm is a single-run point estimate with no confidence intervals or significance tests. The headline example (OCR-VQA: UniWorld 81.33→28.67 vs OmniGen2 79.39→56.59) is presented as a robust cross-model difference, but generation is stochastic (temperature 1.0 for autoregressive models, diffusion sampling for others), and no seed sweep or variance estimate is reported. Table 3's prompt-sensitivity study covers one model (OmniGen2) on one 198-sample MMBench subset, so it does not establish stability across the six models and four datasets in Table 1. Please report means and standard deviations over at least 3–5 seeds for the full protocol, or at least for the constructed subsets in Appendix E, and state the sample sizes of the full splits.","section":"§4.1 (Evaluation setup) and Table 1"},{"comment":"The proposition's proof is circular and formally under-specified. The assumption states that generation cannot introduce new task-solving semantic evidence, and Eq. (17) is essentially this assumption restated; the proof then derives G_SGU ⊆ G_base from it. Additionally, B_d is defined as a set of subsets s ⊆ H_v such that Match(M_U(s,q),a)=1, but M_U is defined on images, not on semantic subsets, so Eqs. (11)–(20) are type-inconsistent. The text immediately after the proof concedes that 'rare cases may occur' where s_umm > s_base, contradicting the universal inequality in Eq. (10). I recommend removing the proposition label and presenting s_base only as a heuristic model-specific reference, or supplying a correct proof under explicit conditions under which the inequality actually holds.","section":"Appendix C.2, Eq. (10)"}],"minor_comments":[{"comment":"Calling SGU 'zero-cost' is misleading: the pipeline runs six models on four benchmarks and includes image generation, which is computationally expensive. 'Annotation-free' is the accurate and sufficient claim.","section":"Abstract and §1"},{"comment":"The row labeled UPPERBOUND is never defined in the text; specify that it is the maximum s_base across models per dataset, and note that it is not an upper bound on s_umm in general.","section":"Table 1"},{"comment":"The header 'Und./Gen. Promptdenote' is missing a space; it should read 'Und./Gen. Prompts' or 'Und./Gen. Prompt denotes'.","section":"Table 3 header"},{"comment":"The caption says 'We reports Δ'; it should be 'We report Δ'.","section":"Table 4 caption"},{"comment":"For open-form answers, specify the matching procedure in more detail, including whether numerical equivalence and paraphrases are handled; the current description ('parsing and normalization') may not cover semantically equivalent free-form answers.","section":"Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable evaluation-paper submission for a vision-language venue. The main risk is that the SGU metric is validated on text-heavy benchmarks without controlling for the text round-trip shortcut, and the statistical evidence is thin. I would require the additional controls and variance reporting before publication. The Appendix C.2 proof should be removed or replaced with a correct statement, since it currently restates its assumption."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zhang et al. propose SGU: describe an image, regenerate it from your own caption, then answer the original VQA question on the regenerated image. The closed-loop design is new, and the consistent drop from direct VQA to SGU across all six models and four datasets is a real empirical pattern. This is a useful complementary evaluation axis, and the authors deserve credit for the stateless protocol, the stage-wise replacement experiments, and the prompt-sensitivity check. They also acknowledge the text bottleneck and that SGU is not a standalone theoretical criterion.\n\nThe soft spots are mostly about interpretation. On OCR-VQA the captioning stage can transcribe text, the generator can render it back into the image, and the final VQA can answer by reading the embedded text. That is a text round-trip shortcut, and it undercuts the claim that SGU measures 'semantic understanding of self-generated contexts' on text-heavy benchmarks. The image-swap test does not close this gap: it is run on a high-scoring MMBench subset (general perception, 198 samples) where text rendering is rarely the answer channel, and it only compares two models on the same rendered images. A text-masking control or an OCR-VQA-specific swap would be needed. Also, all headline numbers come from single runs with no error bars, and the prompt-robustness ablation is one model on one subset. No code or data is released, so the numbers are not independently verifiable.\n\nThe Appendix C.2 upper-bound proposition is circular: it assumes generation cannot introduce new task-solving semantic evidence and then concludes s_umm ≤ s_base. The proof essentially asserts the premise at Eq. (17). This matters less because s_base is only called a reference, not an unconditional guarantee, but it should be reframed.\n\nThe abstract's 'not captured by separate evaluations' is also stronger than the paper's own evidence: Figure 5 shows the SGU ranking is broadly consistent with an isolated average. SGU is a complementary signal, not a new capability that component metrics miss entirely.\n\nWho is this for? People building or benchmarking unified multimodal models. It deserves a serious referee: the idea is simple, the protocol is reusable, and the negative result (models drop sharply on their own regenerated context) is worth publishing. But the authors need to add uncertainty quantification, address the text-rendering shortcut on OCR-VQA, and either fix or soften Appendix C.2 before I'd treat the numbers as reliable.","headline":"SGU is a genuinely new closed-loop evaluation axis for unified multimodal models; the OCR-VQA numbers may partly measure text-rendering fidelity, and the paper needs tighter controls before the headline claim is safe.","tokens_in":18371,"tokens_out":3427,"would_cite":true,"duration_ms":30217,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes SGU, a closed-loop evaluation score for unified multimodal models: the model describes an image, redraws it from its own caption, and answers the original questions from the redrawn image, with accuracy on that loop…","keywords":["unified multimodal models","self-generative understanding","closed-loop evaluation","visual question answering","image generation","system-level evaluation","semantic preservation","multimodal reasoning"],"falsifier":"Take a set of SGU samples and, after the model generates its caption, corrupt the regenerated image—for example, blur or crop exactly the region that contains the answer—while keeping the caption unchanged. If the final question-answering accuracy stays close to direct VQA accuracy on the original image, the loop is being solved from the caption or from language priors rather than from visual understanding of the regenerated image; that outcome would falsify the claim that SGU measures integrated visual understanding-and-generation.","tokens_in":17356,"feed_emoji":"🎨","tokens_out":17882,"duration_ms":162350,"temperature":0.7,"pith_summary":"Unified multimodal models are meant to combine visual understanding and image generation in one system, but existing benchmarks grade those two abilities separately. This paper argues that a model can score well on both yet still fail when the two must work together, and proposes Self-Generative-Understanding (SGU) to test the integrated loop. SGU is annotation-free: it reuses standard visual question-answering datasets, asking the model to caption an image, regenerate the image from its own caption, and then answer the original questions about the regenerated image. In experiments across four benchmarks and six models, accuracy under the loop drops far below direct question-answering accuracy, and models with similar base understanding diverge sharply on the looped score. The case the paper makes is that SGU exposes integration failures—information lost between seeing, describing, redrawing, and re-seeing—that separate component scores do not reveal.","feed_headline":"Image AI models lose accuracy when quizzed on their own redrawings","feed_subtitle":"A closed-loop score sends models through caption, redraw, and answer—revealing failures separate tests miss.","key_machinery":"The machinery is the semantic closed-loop composition $M_U \\circ M_G \\circ M_U$ evaluated on VQA triplets, executed statelessly so each stage sees only the explicit intermediate artifact (caption or regenerated image) and no hidden state or memory is carried across stages. The supporting identity is a semantic-preservation upper bound: under the assumption that generation cannot introduce new task-solving evidence beyond what the understanding pathway extracted, direct VQA accuracy $s_\\mathrm{base}$ bounds the SGU score from above, $s_\\mathrm{umm}\\le s_\\mathrm{base}$. This makes $s_\\mathrm{base}$ a model-specific reference, and the ratio $s_\\mathrm{umm,r}=s_\\mathrm{umm}/s_\\mathrm{base}$ measures how much integrated capability survives the loop. The framework also defines answer matching for multiple-choice and open-form questions so the score can be instantiated on any existing VQA benchmark without new annotations.","core_discovery":"The central claim is that the SGU score, defined as $s_\\mathrm{umm} = \\mathbb{E}_{(v,q,a)\\sim D}[\\mathbb{I}(\\mathrm{Match}(M_U(M_G(M_U(v)), q), a))]$, measures whether a unified multimodal model can carry task-relevant visual information through its own understanding-and-generation loop. The pipeline is a three-stage composition: the understanding function $M_U$ produces a caption $t_g$ from the original image $v$; the generation function $M_G$ reconstructs an image $\\hat{v}$ from that caption alone; and $M_U$ again answers the original question $q$ using only $\\hat{v}$, with the score being the expected match accuracy against ground-truth answers. A relative score $s_\\mathrm{umm,r}=s_\\mathrm{umm}/s_\\mathrm{base}$ is used to compare how much performance survives the loop across models. The paper's key finding is that this closed-loop accuracy is consistently and often dramatically lower than direct VQA accuracy $s_\\mathrm{base}$ on the original image, with the largest drops on visually grounded math and text-in-image questions. Stage-wise replacement experiments indicate the image-generation stage is a major bottleneck, and caption-only analyses show the intermediate textual description also loses information. The contribution is therefore an outcome-based, system-level evaluation signal that complements—rather than replaces—component-wise metrics for understanding and generation.","pith_inferences":["The authors leave implicit that the same closed-loop protocol could measure information flow quantitatively: comparing how much question-relevant evidence survives in the caption and in the regenerated image would decompose the SGU gap into understanding loss, generation loss, and reasoning loss.","A testable extension of the paper's logic is question-conditioned captioning: if the caption stage is allowed to see the question and SGU still does not approach $s_\\mathrm{base}$, the bottleneck lies downstream in generation or self-reasoning rather than in the textual representation.","If SGU were adopted as a training objective, models might learn to regenerate images tuned for their own VQA success rather than for human-perceived fidelity; that reward-hacking risk is not addressed in the paper and would need to be checked against external human-judged generation benchmarks.","The image-swap shortcut check is run on one high-scoring subset; extending it across all four datasets and more model pairs would give stronger evidence that SGU images do not carry model-specific self-consistency cues."],"forward_implications":["A model's ranking on SGU can differ substantially from its ranking on direct VQA; in the reported data, models with similar direct accuracy differ by roughly 28 points on OCR-VQA, so system-level comparisons need a looped score rather than an aggregate of component scores.","Because replacing the generation stage with a stronger generator yields larger gains than replacing the captioning stage, current unified multimodal models' visual reconstruction is a principal bottleneck, and improving generation fidelity should directly raise SGU scores.","Since SGU is annotation-free and reuses standard VQA datasets, system-level evaluation can be scaled to any existing benchmark without human labeling or external judge models.","The relative score $s_\\mathrm{umm}/s_\\mathrm{base}$ separates the cost of the loop from base understanding ability, allowing practitioners to identify whether a model's failure comes from weak perception or from information loss in the integrated process.","Caption-length and caption-only analyses show the text bottleneck is not merely a token limit: increasing the caption limit from 256 to 512 tokens barely changes caption-only QA accuracy, so how the model organizes visual information matters more than caption verbosity."],"supporting_citations":[{"why":"Supplies the MMStar vision-dependent QA benchmark used in the main SGU evaluation.","marker":"[3]"},{"why":"CLIPScore serves as the text-image alignment component in the isolated average comparison SGU is contrasted with.","marker":"[11]"},{"why":"FID serves as the distributional generation-quality component in the isolated average comparison.","marker":"[12]"},{"why":"VQAScore established QA accuracy as a proxy for generation quality, the idea SGU extends into a full self-loop.","marker":"[16]"},{"why":"Supplies MMBench, the general visual reasoning QA benchmark used for the SGU score and the image-swap check.","marker":"[19]"},{"why":"Supplies MathVista, the visually grounded math benchmark where SGU shows large drops and generation-bottleneck diagnosis.","marker":"[20]"},{"why":"Supplies OCR-VQA, the text-in-image benchmark showing the largest SGU gaps and the caption-bottleneck analysis.","marker":"[21]"},{"why":"Used as the replacement understanding module in stage-wise experiments to isolate captioning effects.","marker":"[1]"},{"why":"Used as the replacement generation module in stage-wise experiments to isolate the reconstruction bottleneck.","marker":"[30]"}],"fun_headline_variants":["AI models fail when quizzed on their own redrawings","Closed-loop test: AI's own redraws break its reasoning","Self-generated images trip up unified multimodal AI","AI's own redrawings torpedo its performance in closed-loop test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire score depends on the assumption that correct answers on the reconstructed image genuinely require the model to have preserved and re-rendered the question-relevant visual evidence, rather than being obtained from language priors, information leaked through the caption, or self-consistency cues.","fun_headline_variants_meta":{"raw":{"variants":["AI models fail when quizzed on their own redrawings","Closed-loop test: AI's own redraws break its reasoning","Self-generated images trip up unified multimodal AI","AI's own redrawings torpedo its performance in closed-loop test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000559,"raw_usage":{"total_tokens":2713,"prompt_tokens":1055,"completion_tokens":1658,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":1588}},"tokens_in":671,"tokens_out":1658,"duration_ms":13678,"temperature":1.0,"reasoning_tokens":1588,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:22:29.311196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of SGU samples and, after the model generates its caption, corrupt the regenerated image—for example, blur or crop exactly the region that contains the answer—while keeping the caption unchanged. If the final question-answering accuracy stays close to direct VQA accuracy on the original image, the loop is being solved from the caption or from language priors rather than from visual understanding of the regenerated image; that outcome would falsify the claim that SGU measures integrated visual understanding-and-generation.","supporting_citations":[{"cited_title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","cited_arxiv_id":null,"evidence_quote":"FID serves as the distributional generation-quality component in the isolated average comparison."}],"review_version":1}