{"id":"96e1e1e8-68e4-4b45-a881-72c4f6cea689","arxiv_id":"2505.22525","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning a unified multimodal model to generate intermediate images and self-critiques improves text-to-image generation on GenEval and DPG-Bench.","lead":"This paper teaches a multimodal AI model to create its own intermediate images while solving image-generation tasks, rather than only using images given by a user. The authors show this visual thinking can improve performance on benchmarks for generating scenes with multiple objects.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are not attributable to visual thoughts: no direct-SFT control exists, and the self-critique comparison changes CFG and conditions on a real prior image, so a same-data direct-generation baseline is required.","rationale":"The reader's verdict is CONDITIONAL with moderate confidence, and I agree. The paper's strongest claim is causal: intermediate visual thoughts cause better generation. To establish that, one needs a control that keeps data, loss, and inference protocol fixed while removing the visual-thought tokens. The paper lacks it. For subgoals, the baseline is only JourneyDB-continued Anole, so the 5k synthetic SFT samples plus reconstruction loss plus tuned CFG could explain the Two Obj gain. For self-critique, the internal comparison (visual hypo vs final) is also confounded: different CFG scales (Table 5), conditioning on a real first image, and a second autoregressive pass. The paper's own Fig. 9 failure cases show correct critique with unchanged wrong output, demonstrating that the critique reasoning is not sufficient for improvement—so positive cases cannot be attributed to the reasoning without a control. Additionally, Table 1 omits Overall and several categories for TwGI-Obj., weakening the 'consistently outperforms' claim. Given these confounds, the appropriate verdict remains CONDITIONAL: the method is plausible and the release of artifacts is promised, but the central mechanism is unverified. The concrete test above would settle whether the concern actually lands.","tokens_in":14062,"tokens_out":4223,"duration_ms":48814,"concrete_test":"Train a control Anole-7b on the same 40k self-critique synthetic prompts and the same Lmm+Lrec objective, with the final image as the only target (no initial hypothesis, no critique tokens), using the same 26K steps and the same CFG grid search; then compare DPG-Bench and GenEval. Also rerun TwGI-Anole-7b-Crit. (final) with the visual-hypothesis CFG weights (full 2.0, image 1.2) to separate CFG effects. If the direct control matches or exceeds the reported final scores, or if the CFG change alone explains the gap, the central attribution to intermediate visual thoughts fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim—that interleaved visual thoughts, not extra SFT data or inference-time conditioning, drive the gains—is not tested. Table 1 compares TwGI-Anole-7b-Obj. only against Anole-7b continued-pretrained on JourneyDB; no model is fine-tuned on the same curated prompts with the final image as the only target. Table 1 also omits Overall and most GenEval categories for the subgoal model, though the text says it 'consistently outperforms.' For self-critique, 'visual hypo' vs 'final' (DPG 62.83→67.14, GenEval 0.45→0.48) changes CFG scales (Table 5: full conditions 2.0→1.5, image conditions 1.2→0.8) and the final pass conditions on a real first image plus critique text, so the gain could come from having a second chance conditioned on a good prior image rather than from the learned critique reasoning. Stage-2 data also differs (5k vs 40k samples), and no control isolates the reconstruction loss or the QVQ/Flux-sourced training targets. The paper's own failure cases (Fig. 9) admit correct critique with unchanged wrong output, showing the critique reasoning is not sufficient for improvement. Without a same-prompt, same-loss, same-CFG direct-generation SFT control, the attribution to 'intermediate visual thoughts' is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'Thinking with Generated Images', a paradigm in which a unified autoregressive large multimodal model (Anole-7b) is fine-tuned to generate intermediate visual thoughts as part of its reasoning process. Two mechanisms are instantiated: (1) vision generation with intermediate visual subgoals, where the model decomposes a prompt into sub-images and then composes a final image, and (2) vision generation with self-critique, where the model generates an initial visual hypothesis, critiques it in text, and produces a refined image. The authors construct synthetic SFT data using external models (QVQ, Flux, etc.), train on two stages, and report gains on GenEval and DPG-Bench, including a 0.38-to-0.57 improvement on GenEval Two Obj for the subgoal model and a 62.83-to-67.14 improvement on DPG-Bench Overall for the self-critique final output. The paper claims that this enables spontaneous, iterative multimodal reasoning and test-time scaling.","tokens_in":14417,"tokens_out":3415,"duration_ms":35809,"significance":"If the causal claim is established, this is a useful step toward multimodal cognition and test-time scaling for unified LMMs. The paper is clearly written, releases an open-source suite, and provides a detailed synthetic data pipeline. The core scientific value, however, depends on isolating whether the gains come from the intermediate visual thoughts themselves or from confounded factors such as additional SFT data, different CFG settings, and hyperparameters tuned on the evaluation benchmark. The paper does not currently provide that isolation, so the significance is conditional on additional controlled experiments.","major_comments":[{"comment":"The central causal claim—that intermediate visual thoughts drive the improvements—is not tested. The baseline Anole-7b is only continued-pretrained on JourneyDB; no model is fine-tuned on the same curated synthetic prompts with only direct image-to-image generation as the target. Consequently, the observed gains (e.g., GenEval Two Obj 0.38→0.57, DPG-Bench Overall 58.32→68.44 for the subgoal model) could be due to the additional SFT data or the specific prompt distribution rather than to the thinking process. A same-prompt, same-loss, same-CFG direct-generation SFT control is required before the improvement can be attributed to visual thoughts.","section":"§3.6, Tables 1 and 2"},{"comment":"The self-critique comparison between TwGI-Anole-7b-Crit. (visual hypo.) and TwGI-Anole-7b-Crit. (final) changes both the CFG scales and the conditioning. The final pass uses different guidance weights (full conditions 2.0→1.5, image conditions 1.2→0.8) and additionally conditions on the real first-round image plus the critique text. The DPG-Bench gain from 62.83 to 67.14 and the GenEval gain from 0.45 to 0.48 could therefore stem from having a second chance conditioned on a good prior image and from the retuned CFG, not from the learned critique reasoning. An ablation that holds CFG and conditioning fixed, or that removes the critique text while keeping the same prior-image condition, is needed.","section":"§3.6 and Appendix A.3 (Table 5)"},{"comment":"Several hyperparameters were selected on the same benchmarks used for the main results. The reconstruction loss weight λ was tuned on GenEval (Table 3) and then used in the reported GenEval results, and the CFG scales in Table 5 were presumably chosen on the same benchmarks. This introduces selection bias. In addition, the two mechanisms use very different stage-2 dataset sizes (5k vs. 40k samples), and no error bars or multiple-seed runs are reported. At minimum, a held-out validation set for hyperparameter selection and repeated evaluation runs are necessary to support the quantitative claims.","section":"Appendix A.1, A.2, A.3"},{"comment":"The paper's own failure case for self-critique shows that the model can produce a correct critique and still fail to generate the corresponding corrected image (the TV remote example). This directly qualifies the central claim that models can 'critique their own visual hypotheses and refine them as integral components of their reasoning process.' The mechanism is not sufficient for consistent improvement; the paper should either temper the claim or provide a more detailed analysis of when the critique leads to correction and when it does not.","section":"§B.3, Figure 9"}],"minor_comments":[{"comment":"The row for TwGI-Anole-7b-Obj. omits GenEval Overall and several categories (Single Obj, Counting, Colors), even though the text says it 'consistently outperforms.' Please report the full table or explicitly justify the omission, as the missing entries make it difficult to verify the claim.","section":"§3.6, Table 1"},{"comment":"The DPG-Bench 'Other' category declines from 43.60 (Anole-7b) to 42.00 (TwGI-Anole-7b-Obj.). The text highlights only the improvements in Entity, Attribute, and Relation; the decline in a major category should be acknowledged and discussed.","section":"§3.6, Table 2"},{"comment":"The text interchangeably uses 'Anole-7b-Obj' and 'TwGI-Anole-7b-Obj'; please standardize the model naming to avoid confusion.","section":"§3.7"},{"comment":"The reconstruction loss uses f_vq to obtain codebook features, but the manuscript does not state whether this VQ-VAE encoder is frozen from Anole or fine-tuned. Please clarify the training status of the visual encoder.","section":"Appendix A.1"},{"comment":"There are minor typographical issues, including inconsistent capitalization in 'Thinking with Generated images' in Section 2.2 and equation references that appear as 'equation 1' rather than a numbered citation. A final proofread would be beneficial.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would be strengthened substantially by adding the direct-SFT control and by reporting results with fixed CFG and hyperparameters selected on a validation split. The current evidence is suggestive but not yet conclusive for the central causal claim. The paper may be within the scope of the journal, but the experimental design needs to be tightened before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes something worth taking seriously: a single unified autoregressive LMM that generates intermediate images as thinking steps before producing a final image, with two variants—subgoal decomposition and self-critique. Earlier single-model work was maze-only, and agentic systems use separate passes; here it's end-to-end on general text-to-image benchmarks. The authors also release their suite, and the failure cases in Fig. 9 show they're aware of the limits. Credit where due: the data-curation pipeline is carefully described, and the improvements are consistent across many GenEval and DPG-Bench categories.\n\nBut the central attribution—that the visual thoughts, not the extra SFT data or inference-time conditioning, drive the gains—is not established. There is no same-prompt, same-loss, same-CFG baseline trained to produce only the final image. The self-critique comparison changes CFG scales (Table 5) and conditions the final pass on a real first image plus critique text; that alone could explain part of the gain. Stage-2 data sizes also differ (5k vs 40k samples), and hyperparameters were tuned on the same benchmarks used for reporting. No error bars. So the improvements are suggestive, not conclusive.\n\nThe paper's own failure case—correct critique but unchanged wrong output—is honest but also symptomatic: it shows the critique mechanism can be right and still fail to help, meaning the causal link is fragile.\n\nWho gets value: researchers working on multimodal reasoning, test-time scaling, and unified LMMs will find this a useful reference point and a reasonable baseline. It deserves a serious referee, but the review should demand a direct control without visual thoughts, plus ablations that separate CFG effects and data-size effects. I'd engage with it; with those controls it could be a solid paper. As is, treat the visual-thought attribution as an open question.","headline":"A genuinely novel mechanism for letting LMMs think in images, but the reported gains aren't yet attributable to the visual thoughts because a direct same-data SFT control is missing.","tokens_in":14922,"tokens_out":1891,"would_cite":true,"duration_ms":24822,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single unified model can improve image generation by spontaneously generating intermediate visual thoughts, either as subgoals or as self-critiqued hypotheses.","keywords":["thinking with generated images","large multimodal models","visual chain-of-thought","test-time scaling","autoregressive image generation","self-critique","intermediate visual subgoals","visual reasoning"],"falsifier":"Take the same synthetic prompt set used for TwGI-Anole-7b-Obj. and TwGI-Anole-7b-Crit., and fine-tune Anole-7b on the same data but with all intermediate visual thought tokens removed (i.e., direct prompt-to-final-image). If GenEval Two Obj and DPG-Bench scores match or exceed those of the thought models, the claimed benefit of generated visual thoughts is not causal. Conversely, if the models without visual thoughts score lower by the same margins, the claim gains support.","tokens_in":13911,"feed_emoji":"🧠","tokens_out":2567,"duration_ms":29182,"temperature":0.7,"pith_summary":"The paper claims that large multimodal models can reason across text and vision by generating their own intermediate images, not just by processing user-provided ones. This 'thinking with generated images' is implemented as a native long-multimodal thought process on a unified autoregressive LMM (Anole-7b), trained with curated synthetic data. On vision generation benchmarks, the approach improves multi-object accuracy (GenEval 'Two Obj' from 0.38 to 0.57) and overall image-prompt alignment (DPG-Bench from 62.83 to 67.14 after self-critique). If true, it opens a new axis of test-time scaling where models explore visual hypotheses as part of reasoning. The central move is to treat the generated image itself as a thinking step, not as a final output.","feed_headline":"Generated images become visual thoughts that lift generation scores","feed_subtitle":"Fine-tuned Anole lifts GenEval Two Obj from 38% to 57% and DPG-Bench from 62.83 to 67.14.","key_machinery":"The central object is the native long-multimodal thought process: a sequence of interleaved tokens from a unified vocabulary that includes both text tokens and image tokens (1024 VQ-VAE codebook tokens per image, delimited by special markers). It is instantiated on Anole-7b, a unified autoregressive next-token-prediction LMM, which is fine-tuned on synthetic data built with external generators (Flux for images, QVQ for critique reasoning). Training adds a visual reconstruction loss (MSE between projected hidden states and ground-truth codebook features) to the standard cross-entropy loss, and inference uses tuned classifier-free guidance with separate conditioning scales for the original prompt, the intermediate subgoal images, and the critique text.","core_discovery":"The paper establishes that a fine-tuned unified autoregressive LMM can be trained to interleave text and image tokens in a single autoregressive pass, so that it spontaneously generates intermediate visual thoughts while solving a vision generation task. Two complementary thought processes are demonstrated: vision generation with intermediate visual subgoals, where the model renders objects separately and then combines them, and vision generation with self-critique, where the model generates an initial visual hypothesis, reflects on it in text, and refines it. The resulting models, TwGI-Anole-7b-Obj. and TwGI-Anole-7b-Crit., outperform the base Anole-7b on GenEval and DPG-Bench, with the subgoal model raising Two Obj accuracy from 0.38 to 0.57 and the self-critique model raising DPG-Bench overall from 62.83 to 67.14. The paper argues that these gains come from the visual thinking process itself, which enables test-time scaling across modalities.","pith_inferences":["The paper does not isolate the causal effect of the visual thoughts themselves: the supervised fine-tuning data, the reconstruction loss, and the custom CFG schedules could account for part or all of the gains. A control baseline trained on the same prompts with direct image-to-image generation (no intermediate visual tokens) would settle this, but no such baseline is reported.","The self-critique gains (DPG-Bench 62.83 to 67.14) are measured on the same model's own initial hypothesis, so they might partly reflect test-time compute rather than a learned reasoning ability; a fair comparison would pair the critique model's final output against a direct generation at similar token budget.","The choice of benchmarks (GenEval, DPG-Bench) is limited to standard image generation; the paper argues for real-world tasks like design or 3D modeling, but does not evaluate those, so the general 'visual imagination' claim remains a hypothesis.","The data pipeline relies on strong external generators (Flux, QVQ) to produce the thought chains, which raises a question of whether the fine-tuned model learns to think or merely imitates the annotations; a test without such supervision would clarify the emergence of spontaneous visual thinking."],"forward_implications":["If the mechanism works as claimed, future unified LMMs could be trained to decompose complex visual tasks into subgoals, improving compositional image generation without needing external tools or multi-agent pipelines.","The self-critique loop implies that a single model can act as its own critic and refiner, so visual hypotheses can be iteratively improved at inference time, effectively enabling test-time scaling across both text and image tokens.","The same native long-multimodal thought process should extend to other modalities (audio, video, 3D), since the vocabulary is defined as a union of per-modality token spaces.","The approach suggests that stronger base unified LMMs will likely amplify the benefit, since the observed failure cases (e.g., failing to combine subgoals, failing to execute its own critique) are attributed to base-model limitations.","The reconstruction loss and CFG tuning are load-bearing components; the paper reports that these contribute roughly 3 points on GenEval during the JourneyDB stage, so they are part of the claimed improvement mechanism."],"supporting_citations":[{"why":"Anole-7b is the base unified autoregressive LMM used for the native long-multimodal thought process; it supplies the interleaved text-image token modeling and the 1024-token image representation.","marker":"[5]"},{"why":"GenEval provides the object-focused benchmark used to measure multi-object, counting, color, and position alignment, including the Two Obj metric central to the subgoal claim.","marker":"[9]"},{"why":"DPG-Bench (via ELLA) provides the image-prompt alignment benchmark used to evaluate entity, attribute, relation, and overall generation quality.","marker":"[14]"},{"why":"Flux1-dev and Flux1-Redux are used to generate and refine intermediate visual thoughts in the synthetic data pipeline, grounding the training examples.","marker":"[18]"},{"why":"QVQ-72B-Preview generates the high-quality reflection-reasoning chains (critique and refinement) that constitute the self-critique training signal.","marker":"[31]"},{"why":"Chameleon is the reference architecture for mixed-modal early-fusion foundation models that the paper's unified LMM approach builds on.","marker":"[30]"},{"why":"Classifier-free diffusion guidance is the mechanism adapted for inference-time conditioning on multiple thought-related conditions.","marker":"[13]"}],"fun_headline_variants":["Visual thoughts in one pass: model renders, critiques, refines its own images","Models generate their own visual subgoals to beat multi-object benchmarks","Self-critiquing vision: models render, reflect, and refine in one pass","Visual thinking lifts multi-object generation by up to 50%","From 38 to 57: visual subgoals sharpen multi-object generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The observed improvements are caused by the intermediate visual thoughts themselves, rather than by the extra supervised fine-tuning data, the reconstruction loss, or the tuned guidance schedules — yet no baseline is trained on the same prompts with direct image-to-image generation, so the causal role of visual thinking is not established.","fun_headline_variants_meta":{"raw":{"variants":["Visual thoughts in one pass: model renders, critiques, refines its own images","Models generate their own visual subgoals to beat multi-object benchmarks","Self-critiquing vision: models render, reflect, and refine in one pass","Visual thinking lifts multi-object generation by up to 50%","From 38 to 57: visual subgoals sharpen multi-object generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001538,"raw_usage":{"total_tokens":6199,"prompt_tokens":1037,"completion_tokens":5162,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":5063}},"tokens_in":653,"tokens_out":5162,"duration_ms":42841,"temperature":1.0,"reasoning_tokens":5063,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:04:44.103230+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same synthetic prompt set used for TwGI-Anole-7b-Obj. and TwGI-Anole-7b-Crit., and fine-tune Anole-7b on the same data but with all intermediate visual thought tokens removed (i.e., direct prompt-to-final-image). If GenEval Two Obj and DPG-Bench scores match or exceed those of the thought models, the claimed benefit of generated visual thoughts is not causal. Conversely, if the models without visual thoughts score lower by the same margins, the claim gains support.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"QVQ-72B-Preview generates the high-quality reflection-reasoning chains (critique and refinement) that constitute the self-critique training signal."}],"review_version":1}