{"id":"8d23ff6c-4d99-42e4-a924-f922698f7c8a","arxiv_id":"2505.14682","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"UniGen shows a 1.5B model trained on open data can beat larger systems on image understanding and generation once it verifies its own outputs with chain-of-thought and Best-of-N selection.","lead":"A compact 1.5B unified model, UniGen, both understands and generates images, and at test time it uses its own reasoning to check its generated images and keep the best ones. Its chain-of-thought verifier lifts text-to-image alignment scores (GenEval 0.78, DPG-Bench 85.19) using only open-source training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline GenEval/DPG-Bench margins are computed with Best-of-20 top-4 selection while baselines are single-sample; the 'state-of-the-art' claim conflates inference budget with model quality.","rationale":"The reader's weakest assumption identifies the single most load-bearing concern: the headline generation numbers are not comparable across models because UniGen is evaluated with an expensive Best-of-20 top-4 selection while the cited baselines are single-sample. This directly threatens the central 'state-of-the-art' claim in the abstract. The concern is concrete and verifiable, and it is independent of the paper's useful ablations showing that CoT-V helps UniGen itself; even if the mechanism works, the reported margins against baselines are unsupported under the current protocol. I considered a second concern, that the GenEval metric is used to label the preference data (Section 3.4) and the CoT-V post-training data (Section 3.5.1), which risks training the evaluation metric into the model. That is a real risk worth checking, but it is secondary because the paper's internal ablations (Table 3) still show CoT-V improving DPG-Bench, which is less directly optimized, and because the primary comparison claim is already invalidated by the budget mismatch alone. The correct remedy is to report matched-budget baselines or clearly frame the contribution as 'test-time scaling with CoT-V,' not as unconditional SOTA. The reader's verdict of CONDITIONAL is appropriate; my read does not move it.","tokens_in":21797,"tokens_out":11285,"duration_ms":101333,"concrete_test":"Reproduce Table 2 with Janus-Pro-1.5B and Show-o-1.3B under the same protocol: sample N=20 candidate images per prompt using their released inference code, select top-K=4 with a verifier of the same class (e.g., the released UniGen CoT-V model or an off-the-shelf CLIP-based verifier), and score with the official GenEval and DPG-Bench repositories. If either baseline reaches or exceeds UniGen's 0.78/85.19 under the matched budget, the claimed SOTA margins are an artifact of test-time scaling rather than model quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The generation comparisons in Tables 1-2 report UniGen's GenEval (0.78) and DPG-Bench (85.19) scores using a Best-of-20 top-4 protocol with the CoT-V verifier, whereas the published baseline numbers for Janus-Pro, Show-o, and other models were obtained without any test-time selection budget. The abstract's 'state-of-the-art' claim therefore depends on an asymmetric evaluation: with 20 candidates and a learned verifier, even a weaker generator can surpass a stronger single-sample model, particularly on the Counting category (UniGen 0.78 vs Janus-Pro 0.51) where selecting the correct count among many samples is easy. The paper does not report baselines under the same Best-of-20 protocol, nor does it report its own single-sample score for comparison, so the reported margins are not evidence that UniGen's underlying generation quality is better than Janus-Pro's or Show-o's. This is an evaluation-protocol premise, and it is load-bearing for the headline 'SOTA' assertion on generation benchmarks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents UniGen, a 1.5B unified multimodal LLM that handles both image understanding and text-to-image generation. The authors describe a three-stage training pipeline (pre-training, SFT, DPO) built from open-source datasets, and propose Chain-of-Thought Verification (CoT-V), a test-time scaling method in which the model generates N candidate images, verifies each in a step-by-step manner, and selects top-K. On GenEval and DPG-Bench they report scores of 0.78 and 85.19, claiming state-of-the-art among unified models, and they provide extensive ablations of each training stage and design choice.","tokens_in":22046,"tokens_out":5172,"duration_ms":40199,"significance":"If the results withstand scrutiny, the paper makes a useful contribution: it demonstrates a clean open-data recipe for unified MLLMs, provides careful ablations of each training stage, and shows that a lightweight CoT-V post-training (500 steps) can turn the model into an effective self-verifier, with transfer to Show-o. The internal ablations in Tables 3 through 6 are consistent and well-controlled. However, the headline SOTA claim on generation is currently not supported because the evaluation protocol is asymmetric and partially circular. The CoT-V mechanism itself remains promising but needs re-evaluation under a fair protocol.","major_comments":[{"comment":"The reported GenEval score of 0.78 and DPG-Bench score of 85.19 are obtained with N=20 candidate images per prompt and top-K=4 selection by the CoT-V verifier (Sec. 4.1), while all baseline numbers in Table 2 (Janus-Pro, Show-o, etc.) are single-sample published results. This asymmetry is load-bearing: Table 3 shows UniGen-DPO without CoT-V already reaches 0.73 on GenEval, identical to Janus-Pro's 0.73 in Table 2, so the headline +0.05 margin on GenEval is entirely produced by the test-time selection budget, not by model quality. The paper neither reports UniGen's single-sample GenEval/DPG-Bench scores in Table 2 nor runs baselines under the same Best-of-20 protocol. Moreover, it is not specified how the K selected images are aggregated into the official metric (best-of-K vs. averaging); if each selected image is submitted and the maximum is taken, this further inflates the score. The abstract's 'state-of-the-art' claim therefore conflates inference budget with model quality. Please report single-sample numbers, specify the aggregation rule, and, ideally, evaluate baselines under the same selection budget.","section":"Sec. 4.1-4.2, Tables 2-3"},{"comment":"The GenEval evaluation is partially circular with the training data. For short prompts, DPO preference pairs are selected using GenEval metrics (Sec. 3.4), and the CoT-V post-training labels for PARM prompts are derived from GenEval's templated sub-questions and rules (Appendix E.0.5). The same GenEval benchmark is then used for the main evaluation in Table 2. This means the model is in part trained to reproduce GenEval's scoring criteria, so the GenEval improvements (0.73 to 0.78) cannot be taken as evidence of general text-to-image alignment. The authors should demonstrate that the CoT-V gain persists on a benchmark not used in any training or data-labeling stage, or at least analyze per-category results after excluding categories whose rules were used for labeling.","section":"Sec. 3.4 and Appendix E.0.5"}],"minor_comments":[{"comment":"The symbol X_U is used both for the input image and its encoded feature sequence; please use distinct notation (e.g., X_U and Z_U) to avoid confusion.","section":"Sec. 3.1"},{"comment":"The scores in panels (b) and (c) are not clearly mapped to the formula in Eq. (2); please add a caption explaining how 0.66 is computed.","section":"Fig. 4"},{"comment":"The phrase 'select top-K (K=4) out of it, sending for evaluation' is ambiguous; state explicitly whether the reported GenEval and DPG-Bench numbers are the average over the K selected images or the best of them.","section":"Sec. 4.1"},{"comment":"The acknowledged failure of CoT-V on free-form complex prompts (also shown in Fig. 9) is in tension with the DPG-Bench claim; consider tempering the SOTA wording or reporting a DPG-Bench breakdown for prompts of varying complexity.","section":"Sec. 5 (Limitations)"},{"comment":"The 'Color Attri.' column for UniGen (0.54) is lower than Janus-Pro's 0.56, yet the text claims an 'overwhelming advantage' on Counting; please make the per-category discussion balanced.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a large number of ablations and useful practical lessons, but the evaluation protocol issue is serious. The authors should be asked to provide single-sample results and re-run baselines with the same test-time budget, or explicitly reframe the SOTA claim as 'under Best-of-20 selection'. Additionally, the GenEval label leakage should be addressed with an out-of-distribution benchmark. These issues are fixable within the manuscript's scope and do not, in my view, warrant rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi — quick take on arXiv:2505.14682. The real contribution is CoT-V: a 500-step post-training trick that turns a 1.5B unified MLLM into its own image verifier, and the ablations make a decent case that it works. The open-data recipe is also useful. But the abstract's SOTA claim does not survive contact with the evaluation protocol. UniGen's GenEval 0.78 and DPG-Bench 85.19 come from Best-of-20, top-4 selection with the learned verifier; the published Janus-Pro and Show-o numbers are single-sample. That is an apples-to-oranges comparison. The Counting gap especially (0.78 vs 0.51) is exactly what test-time selection inflates — pick the one that has the right count out of 20 rolls and you beat a stronger model that only got one roll. The paper never reports its own single-sample score, so we can't isolate how much of the margin is the verifier and how much is plain Best-of-N.\n\nSecond real issue is circularity. DPO preference pairs are selected by GenEval rules (Sec 3.4), CoT-V post-training labels use GenEval sub-questions (Sec E.0.5), and then GenEval is the headline number. The generation itself isn't hardcoded to the metric, so it's not a fatal flaw, but it does mean part of the reported gain is the model being trained to produce images that the same rule-based checker likes. That's a real confound for the \"state-of-the-art\" language.\n\nWhat's genuinely good: the CoT-V ablation is clean — outcome verification gives nothing, rule-based gives a bump, CoT gives the largest gain, and the CoT-V post-train is necessary. CoT-V also transfers to Show-o. The DPO data ablation and the stage-by-stage table (Table 3) are honest and useful. The failure cases in Fig 9 and the acknowledged limitation about noisy Qwen labels add credibility.\n\nMy read: the core mechanism is plausible and worth building on. The headline numbers need matched-budget baselines and a single-sample row before they can be taken at face value. This deserves a serious referee — the systems work is competent and the recipe is reproducible in principle — but the SOTA claim needs heavy revision, and the circularity at least needs explicit acknowledgement and ideally an eval on a held-out benchmark. Send to review.","headline":"CoT-V is a solid and transferable test-time scaling idea, but the SOTA claims rest on a Best-of-20 vs single-sample comparison and a partially trained GenEval metric.","tokens_in":22609,"tokens_out":2319,"would_cite":true,"duration_ms":21172,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"UniGen claims that a 1.5B unified multimodal model reaches state-of-the-art text-to-image results by verifying its own generated images step-by-step, scoring 0.78 on GenEval and 85.19 on DPG-Bench.","keywords":["unified multimodal large language model","text-to-image generation","image understanding","chain-of-thought verification","test-time scaling","Best-of-N selection","direct preference optimization","masked token prediction"],"falsifier":"Run GenEval and DPG-Bench under a matched protocol: give Janus-Pro, Show-o, and Janus the same 20-sample Best-of-N budget (using their own confidence scores or a shared verifier) and also report UniGen with a single sample per prompt; if UniGen's margin over baselines disappears, the state-of-the-art claim rests on extra inference compute rather than on CoT-V. A second direct check is to compare CoT-V's top-4 selection with random selection among the same 20 samples on DPG-Bench; if random selection scores equally well, the verifier itself is not the cause of the gain.","tokens_in":21610,"feed_emoji":"🖼️","tokens_out":7413,"duration_ms":62649,"temperature":0.7,"pith_summary":"UniGen is a 1.5B unified multimodal language model that performs both image understanding and text-to-image generation, trained entirely on open-source data across two pre-training stages, supervised fine-tuning, and direct preference optimization. The paper claims this fully open-data recipe reaches state-of-the-art generation scores of 0.78 on GenEval and 85.19 on DPG-Bench while staying competitive on understanding benchmarks. The central new mechanism is Chain-of-Thought Verification (CoT-V): at test time the model generates 20 candidate images per prompt, then acts as its own verifier by decomposing the prompt into atomic visual questions, answering each yes/no in a chain-of-thought format, and keeping the highest-scoring candidates. Ablations show each training stage adds a measurable gain and that CoT-V adds a further margin without sacrificing understanding capability. If correct, the work demonstrates that generation quality can be improved by test-time self-verification in a single model, using compute rather than parameter count or proprietary data.","feed_headline":"A 1.5B model hits 0.78 on GenEval by verifying its own drafts","feed_subtitle":"Test-time self-check lifts image generation quality, while open-data training matches larger unified models.","key_machinery":"The central object is CoT-V (Chain-of-Thought Verification), a test-time scaling strategy in which UniGen acts simultaneously as image generator and image verifier. Given a text prompt, the model first produces N candidate images (N=20 in the main experiments), then for each candidate it is prompted to break the prompt into atomic visual questions, answer each question with yes or no inside a chain-of-thought block, and the final quality score is the fraction of 'yes' answers; the top K=4 candidates by score are selected. The machinery also includes the training recipe that makes this possible: masked token prediction with a cosine masking schedule for generation, decoupled SigLIP and MAGVIT-v2 encoders, a DPO stage for preference alignment, and the 500-step CoT-V post-training that teaches the model to emit the verification format using pseudo-labeled question-answer triplets.","core_discovery":"The paper's central claim is that a single autoregressive LLM with decoupled visual encoders—a continuous encoder for understanding and a discrete tokenizer for generation—can, with the right open-data training recipe, surpass larger and more specialized text-to-image models on GenEval and DPG-Bench while matching strong understanding-focused MLLMs. The key discovery is that letting the same model verify its own outputs, in a step-by-step chain-of-thought manner, produces reliable enough quality scores to make Best-of-N selection work: UniGen goes from 0.74 to 0.78 on GenEval and 85.02 to 85.19 on DPG-Bench when CoT-V is applied, whereas a single yes/no outcome question gives no improvement. The paper also establishes that this verification ability can be instilled with only 500 steps of lightweight post-training and that the resulting CoT-V post-trained model preserves the model's general understanding performance almost unchanged.","pith_inferences":["Because CoT-V's score is an average over atomic yes/no answers, selection quality is bounded by the question decomposition; using a stronger decomposer or human-filtered CoT labels should raise the ceiling, and the paper's own failure cases on long prompts are consistent with this.","The headline margins over Janus-Pro and Show-o are reported with UniGen using 20 samples per prompt and top-4 selection, while baselines are quoted without that budget; until matched-budget comparisons exist, the margin should be read as including extra test-time compute.","The same CoT-V scorer could be recycled as a reward model for another round of preference optimization or reinforcement learning, since it scores arbitrary image-prompt pairs; the paper does not explore this closed loop."],"forward_implications":["A model can improve its own generation quality by spending more test-time compute in a verifier loop, so unified understanding capability becomes a direct asset for generation rather than a separate head.","Open-source training data plus careful stage-wise recipes can bring a 1.5B unified model to benchmark levels previously reached by much larger or proprietary-data models.","CoT-V transfers: fine-tuning Show-o with the same DPO and CoT-V data raises its GenEval from 0.56 to 0.64 and DPG-Bench from 71.70 to 76.32.","The per-stage gains (pre-training, SFT, DPO, CoT-V) are additive on both GenEval and DPG-Bench, so each pipeline component earns its keep."],"supporting_citations":[{"why":"Supplies the Show-o architecture, MAGVIT-v2 tokenizer, and masked token prediction recipe that UniGen builds on, and serves as a primary unified-model baseline.","marker":"[76]"},{"why":"Janus-Pro is the strongest unified baseline UniGen must beat on both understanding and generation benchmarks.","marker":"[8]"},{"why":"Janus provides the decoupled visual encoding design that motivates UniGen's separate understanding and generation encoders.","marker":"[74]"},{"why":"GenEval supplies the object-focused text-to-image alignment benchmark and the rule-based metric used to build DPO preference pairs.","marker":"[15]"},{"why":"DPG-Bench supplies the dense-prompt generation benchmark used for evaluation and for measuring compositional alignment.","marker":"[21]"},{"why":"Direct preference optimization provides the loss used in UniGen's alignment stage.","marker":"[53]"},{"why":"PARM contributes the short-prompt preference data and the CoT-based verification idea that CoT-V generalizes.","marker":"[18]"},{"why":"MaskGIT defines the masked token prediction objective and cosine masking inference schedule used for image generation.","marker":"[5]"}],"fun_headline_variants":["UniGen: self-verifying drafts lift GenEval to 0.78","CoT verification at test time lifts UniGen to 0.78 GenEval","Open-data unified MLLM self-verifies to top GenEval and DPG-Bench","UniGen: 500-step CoT post-training lifts GenEval to 0.78","Small unified model beats specialists by checking its own images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported state-of-the-art comparison assumes that a test-time-scaled UniGen, which sees 20 generated candidates per prompt and keeps the top 4 by its own verifier's score, can be directly compared with published baseline scores obtained without that same selection budget.","fun_headline_variants_meta":{"raw":{"variants":["UniGen: self-verifying drafts lift GenEval to 0.78","CoT verification at test time lifts UniGen to 0.78 GenEval","Open-data unified MLLM self-verifies to top GenEval and DPG-Bench","UniGen: 500-step CoT post-training lifts GenEval to 0.78","Small unified model beats specialists by checking its own images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001223,"raw_usage":{"total_tokens":5028,"prompt_tokens":944,"completion_tokens":4084,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":3973}},"tokens_in":560,"tokens_out":4084,"duration_ms":26065,"temperature":1.0,"reasoning_tokens":3973,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:29:26.940153+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run GenEval and DPG-Bench under a matched protocol: give Janus-Pro, Show-o, and Janus the same 20-sample Best-of-N budget (using their own confidence scores or a shared verifier) and also report UniGen with a single sample per prompt; if UniGen's margin over baselines disappears, the state-of-the-art claim rests on extra inference compute rather than on CoT-V. A second direct check is to compare CoT-V's top-4 selection with random selection among the same 20 samples on DPG-Bench; if random selection scores equally well, the verifier itself is not the cause of the gain.","supporting_citations":[],"review_version":1}