{"id":"a5117139-402b-4345-b9ad-b92a14721440","arxiv_id":"2501.12327","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"VARGPT combines LLaVA-style next-token visual understanding with VAR-style next-scale visual generation in one autoregressive multimodal model trained in three stages.","lead":"This paper introduces VARGPT, an AI model that handles both understanding images and generating new images in one system, using different prediction rules for each task. A reader might care because it tests whether a single large language model can naturally switch between answering questions about images and creating images from text.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed comprehension superiority over LLaVA-1.5 is confounded by LLaVA-OneVision data; Table 5 shows VARGPT without it falls below LLaVA-1.5 on MMMU and MME.","rationale":"The reader's verdict of CONDITIONAL is appropriate, and the identified concerns are real. However, I find the understanding-data confound more load-bearing than the generation-evaluation-on-training-distribution issue. The paper's strongest quantitative claim is comprehension superiority over LLaVA-1.5 and other MLLMs. The ablation in Table 5 demonstrates that removing LLaVA-OneVision drops VARGPT below LLaVA-1.5 on both MMMU and MME, so the claimed superiority is not a property of the architecture or the unified training; it comes from a newer, larger dataset. This directly undermines the central claim as phrased in the abstract and introduction. The generation evaluation issue is secondary: the paper itself admits generation quality lags dedicated models, and the qualitative examples provide some evidence of instruction following even if the FID/CLIP numbers are weakened by train-set evaluation. I give credit for the clear architectural description, the three-stage training design, and the inclusion of ablations that partially expose the data dependence. The most decisive check is already present in Table 5; re-running that condition and reporting a controlled baseline would settle the question. Therefore I recommend keeping the verdict CONDITIONAL, with the explicit condition that the authors either provide a controlled comparison on matched data or temper the comprehension superiority claim.","tokens_in":27361,"tokens_out":5664,"duration_ms":52129,"concrete_test":"Use the ablation already reported in Table 5 as the decisive test: reproduce stage-2 training of VARGPT with only LLaVA-1.5-665K plus 5K generation samples (the '- LLaVA-OneVision-502K' condition) and evaluate on MMMU and MME. If the reproduced scores are at or below the published LLaVA-1.5-hf scores (MMMU 35.24, MME 1480.1), then the comprehension superiority in the abstract is entirely due to the OneVision data, and the unified architecture provides no measurable benefit. To isolate architecture benefit, additionally train a LLaVA-1.5-style model on the same combined 1.17M mixed dataset (LLaVA-1.5 + OneVision) without any generation tokens; if that model matches or exceeds VARGPT on understanding benchmarks, the unified next-scale generation training is not the source of the improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim asserts that VARGPT surpasses numerous MLLMs of comparable scale in comprehension. The paper's stage-2 training uses 665K LLaVA-1.5 samples plus 508K LLaVA-OneVision samples, which is a larger and newer dataset than LLaVA-1.5 alone. The ablation in Table 5, row '- LLaVA-OneVision-502K', removes the OneVision contribution and retains LLaVA-1.5 plus 5K generation data; this yields MMMU 33.56 and MME 1403.5, both below the reported LLaVA-1.5-hf scores of 35.24 and 1480.1 respectively. Therefore, the claimed superiority over LLaVA-1.5 is entirely attributable to the additional OneVision data, not to the proposed next-token/next-scale unified architecture. The abstract's phrasing 'Despite its LLAVA-based architecture... significantly outperforms LLaVA-1.5' implies the architecture drives the gain, but the ablation contradicts this. This is more load-bearing than the generation evaluation on the training distribution because the comprehension benchmarks are the paper's primary quantitative evidence for the headline claim. Without a controlled comparison—either a LLaVA-1.5 baseline trained on the same combined data, or a VARGPT variant trained only on LLaVA-1.5 data—the comprehension superiority claim remains unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"VARGPT proposes a single autoregressive multimodal model that performs visual understanding through next-token prediction and visual generation through next-scale prediction, extending the LLaVA-1.5 architecture with a 2B visual decoder and two generation projectors. The model is trained in three stages on a curated mixture of LLaVA-1.5, LLaVA-OneVision, and ImageNet-based instruction data, and it can output mixed text-and-image responses in a single conversation. The paper reports comprehension scores on eleven benchmarks and generation metrics (FID, CLIP score) on a 50,000-instruction ImageNet evaluation set, together with ablations of the training stages and data composition.","tokens_in":27652,"tokens_out":5898,"duration_ms":57626,"significance":"If the central claims hold, the architecture is a useful step toward unified multimodal models: it demonstrates that a next-scale visual generator can be attached to a next-token understanding LLM in a single model, and the qualitative mixed-modal outputs in Figures 2, 9, 13, and 14 show a genuine capability. The paper also provides a clearly described three-stage training recipe and ablation studies on both generation stages and understanding data, which are valuable for follow-up work. However, the headline comprehension claim is confounded by the addition of LLaVA-OneVision data, and the generation evaluation is performed on the training distribution without comparison to existing unified models or text-to-image baselines. The significance of the numerical results is therefore not yet established; the architectural contribution is plausible but requires stronger controlled evidence.","major_comments":[{"comment":"The claimed comprehension superiority over LLaVA-1.5 is not supported by a controlled comparison. The stage-2 training mixes 665K LLaVA-1.5 samples with 508K LLaVA-OneVision samples, and the ablation in Table 5 (row '- LLaVA-OneVision-502K') shows that after removing the OneVision portion the model attains MMMU 33.56 and MME 1403.5, below the reported LLaVA-1.5-hf scores of 35.24 and 1480.1. Since the abstract states that 'Despite its LLAVA-based architecture... significantly outperforms LLaVA-1.5,' the improvement is likely attributable to the addition of a newer, larger data mixture rather than to the next-token/next-scale architecture. The paper needs a matched comparison: either train LLaVA-1.5 on the same combined data, or train VARGPT only on the LLaVA-1.5 portion, and report both understanding and generation metrics.","section":"Section 5.1, Table 5, Section 4.2"},{"comment":"The generation evaluation does not measure generalization to novel instructions. The 50,000 test instructions are constructed from ImageNet-1K captions with 50 samples per category, the model is trained on ImageNet-1K, and the FID is computed against the ImageNet-1K dataset itself; CLIP score is computed between instructions and images drawn from the same caption distribution used to build the training data. These numbers cannot distinguish genuine instruction following from memorization of the training distribution. The authors should evaluate on held-out categories, on captions from a different source, or with human raters, and should compare against at least one text-to-image or unified-model baseline under identical conditions.","section":"Section 5.1, Evaluation on Instruction-to-image Task"},{"comment":"The generation quality claim is never benchmarked against the unified models listed in Table 1 (Chameleon, SEEDLLaMA, Show-o, VILA-U, Liquid) or against the VAR backbone used to initialize the visual decoder. Table 4 only ablates the training stages; without a comparison model, the reader cannot judge whether the reported FID 12.6 / CLIP 27.4 is strong or weak. The limitation section asserts that VARGPT's generation lags behind diffusion models and other unified models, but no quantitative comparison is provided to support this. Adding a standard text-to-image benchmark with matched baselines would substantiate the 'instruction-to-image' claim.","section":"Section 5, Tables 1 and 4; Section 6, Limitation"}],"minor_comments":[{"comment":"There are several typos and grammatical errors: 'multimodel understanding' should be 'multimodal understanding' in the abstract, and 'The unified training strategy are designed' should be 'is designed'.","section":"Abstract and Section 3.2"},{"comment":"The ablation row '- LLaVA-OneVision-502K' is inconsistent with the data description, which states that 508K samples were sampled from LLaVA-OneVision; please reconcile the numbers or clarify what exactly was removed.","section":"Table 5 vs Section 4.2 and Figure 8b"},{"comment":"The legend 'We gray out the model has trained on the dataset' is ungrammatical and unclear; please specify which models were trained on which evaluation datasets and what the gray highlighting indicates.","section":"Table 3"},{"comment":"The CFG equation uses the same symbol R_t on both sides of the final line; please write the combination with distinct notation, e.g., \\hat{R}_t = (1+\\lambda) R_t^{cond} - \\lambda R_t^{uncond}, to avoid confusion between the conditional distribution, the unconditional distribution, and the combined result.","section":"Appendix 7.6, Eq. (3)"},{"comment":"Sampling hyperparameters (top-k=900, top-p=0.95, CFG=1.5) are reported without sensitivity analysis or a citation to the source of these defaults; a short sensitivity table or a reference to the VAR settings would make the results more reproducible.","section":"Section 5, Details"}],"recommendation":"major_revision","confidential_remarks":"The core architectural idea is worth publishing if the evaluation is made rigorous. The main risk is that the paper's headline numbers overstate the architecture's contribution to understanding (the OneVision data confound) and its generation generalization (in-distribution evaluation). A matched-data control and out-of-distribution generation evaluation are feasible within the manuscript's scope. I would also encourage the authors to state whether code and checkpoints will be released; the paper currently does not mention this, which limits reproducibility verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's new: VARGPT is the first MLLM I know of that unifies next-token understanding with next-scale generation in a single autoregressive pass, using two learned projectors and a separate 2B VAR-initialized visual decoder. That architecture is genuinely absent from the cited prior work. The three-stage training—projector pretraining, mixed SFT for understanding, then generation SFT—is clearly described, and the ablations (Tables 4 and 5) show each stage matters. The qualitative mixed-modal outputs look real, and the paper is honest in its limitations section.\n\nNow the soft spots. The stress-test is right. Table 5's ablation removing LLaVA-OneVision-502K drops VARGPT to MMMU 33.56 and MME 1403.5, both below the reported LLaVA-1.5-hf scores of 35.24 and 1480.1. That means the paper's claimed comprehension superiority over LLaVA-1.5 is entirely attributable to the larger/newer OneVision data, not to the unified architecture. The abstract's phrasing \"Despite its LLaVA-based architecture... significantly outperforms LLaVA-1.5\" implies the architecture drives the gain, and that is misleading. A controlled comparison is missing. This is not a minor omission—it is the paper's main quantitative support for its headline claim.\n\nThe generation evaluation has a related weakness: the 50,000 test instructions are built from ImageNet-1K captions, the model was trained on ImageNet-1K, and FID is computed against ImageNet-1K. High FID/CLIP scores on that distribution don't demonstrate instruction-following generalization; they could reflect memorization. A held-out dataset or zero-shot transfer would be needed.\n\nThe paper also does not release code, weights, or data, which limits reproducibility, though that is common for system papers.\n\nThese flaws are addressable. A controlled baseline (LLaVA-1.5 trained on the same combined data, or VARGPT trained only on LLaVA-1.5 data) and a held-out generation evaluation would settle the main questions. The architecture itself is interesting and the system is plausible.\n\nFinal take: this deserves a serious peer review, not a desk reject. The reviewers should concentrate on the confound in the comprehension claim and the in-distribution generation eval. If those are fixed, the paper would be a solid contribution to unified MLLMs.","headline":"A clearly described unified MLLM whose comprehension-over-LLaVA-1.5 claim is undercut by its own ablation data; still worth a serious referee.","tokens_in":28231,"tokens_out":3622,"would_cite":true,"duration_ms":33154,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VARGPT is the first unified model to support next-token visual understanding and next-scale visual generation in one autoregressive framework.","keywords":["VARGPT","visual autoregressive generation","next-scale prediction","multimodal large language model","visual understanding","text-to-image generation","unified model","instruction tuning"],"falsifier":"Evaluate instruction-to-image generation on prompts built from a different distribution, such as COCO captions or ImageNet categories held out from training, while keeping VARGPT frozen; if FID and CLIP scores drop far below the ImageNet-derived results, the reported instruction-following is mostly in-distribution recall.","tokens_in":27150,"feed_emoji":"🎨","tokens_out":6894,"duration_ms":65808,"temperature":0.7,"pith_summary":"VARGPT claims that one autoregressive multimodal model can do both visual understanding and image generation by using two different prediction rules: predict the next text token when answering questions, and predict the next visual scale when drawing an image. The paper builds this by extending the LLaVA-1.5 architecture with a 2-billion-parameter visual decoder and two visual feature projectors, then training the combined model in three stages on roughly 3.86 million instruction samples. On eleven visual-understanding benchmarks the model beats LLaVA-1.5 and several comparable unified models, while also producing 256x256 images from text instructions in the same conversation. A sympathetic reader would take the central claim to be that a single weight set can host both capabilities without either degrading the other.","feed_headline":"One model answers image questions and draws pictures in a single pass","feed_subtitle":"VARGPT pairs next-token comprehension with next-scale image generation, beating LLaVA-1.5 on understanding benchmarks.","key_machinery":"The central mechanism is dual-paradigm autoregression inside one Transformer. For understanding, the model predicts the next text token under a standard causal mask. For generation, the special <image_gen_start> token switches on a second autoregressive loop where the LLM's hidden states pass through a generation projector into a 2B visual decoder that predicts next-scale tokens under a block-causal attention mask; a multi-scale VQ-VAE then decodes the tokens into an image. The special tokens used to mark generation positions and the staged training schedule are the load-bearing connectors that allow one model to answer a question and then draw a picture in a single mixed-modal response.","core_discovery":"The paper's central claim is that VARGPT is the first unified model to support predicting the next token for understanding and predicting the next scale for generation, and that this single model can outperform LLaVA-1.5-class multimodal LLMs on comprehension while adding autoregressive text-to-image generation. Concretely, the LLM emits text tokens normally; once it predicts the special <image_gen_start> token, the image-related features are projected into a separate 2B visual decoder that predicts multi-scale image tokens in block-causal order, and a multi-scale VAE decoder turns them into a 256x256 image. The three-stage training recipe, consisting of pretraining the generation projectors, mixed instruction tuning for understanding plus a small generation subset, and then generation-focused instruction tuning, is what lets the two paradigms coexist. Reported results include higher scores than LLaVA-1.5 on benchmarks such as MMBench, SEED-Bench, MME, POPE, GQA, TextVQA, VQAv2, and ScienceQA, with an FID of 12.6 and a CLIP score of 27.4 on the constructed instruction-to-image evaluation.","pith_inferences":["An untested implication is that next-scale generation may scale with data the way text autoregression does, potentially letting a single unified model approach diffusion-level image quality while retaining understanding.","A sharper test than the paper runs would evaluate instruction-to-image generation on a held-out distribution, since the test prompts are built from ImageNet-1K captions and FID is measured against ImageNet-1K, which the model saw during training.","The decoupling of the visual decoder from the LLM suggests a broader design principle: understanding and generation can share a language model only when generation has its own parameter space, and future unified models may vary how much of the generation burden the LLM itself carries.","A practical consequence the authors do not spell out is that mixed-modal conversations in one model remove the need to coordinate separate question-answering and image-generation systems, which could simplify deployment and reduce latency."],"forward_implications":["A single model can hold a conversation, answer visual questions, and then generate an image in the same response, without routing generation to an external diffusion model.","At the 7B-plus-2B scale, unified understanding and generation does not have to come at a comprehension cost; VARGPT reports higher scores than LLaVA-1.5 on the understanding benchmarks it evaluates.","The three-stage recipe and the 5K generation samples mixed into stage-2 instruction tuning teach the model when to emit generation tokens, so task routing emerges from instruction following rather than from a separate controller.","Generation quality is tied to data: the paper's stated limitation is that ImageNet-only training and 256x256 resolution leave a quality gap to large diffusion models, so scaling data and resolution is the explicit next step."],"supporting_citations":[{"why":"Supplies the base LLaVA-1.5 architecture and initial weights for the LLM, visual encoder, and understanding projector.","marker":"[50]"},{"why":"Defines the next-scale prediction paradigm, the multi-scale tokenizer, the block-causal attention layout, and the visual decoder initialization.","marker":"[84]"},{"why":"Provides the enriched ImageNet-1K captions from which the generation instruction datasets are built.","marker":"[34]"},{"why":"Generates the prompt and answer seeds and templates used to construct ImageNet-Instruct-130K.","marker":"[10]"},{"why":"Supplies the multi-scale VQ-VAE tokenizer that turns images into the scale-wise tokens the model predicts.","marker":"[14]"},{"why":"Supplies the 665K LLaVA-1.5 instruction-following samples used in stage-2 understanding supervised fine-tuning.","marker":"[51]"},{"why":"Supplies the 508K LLaVA-OneVision instruction samples mixed into stage-2 training.","marker":"[41]"},{"why":"ImageNet-1K is the image source for all generation training data and the reference distribution for the FID evaluation.","marker":"[11]"}],"fun_headline_variants":["VARGPT: one model, next-token understanding, next-scale generation","VARGPT unifies visual understanding and generation in one autoregressive pass","Beats LLaVA-1.5 on understanding and also generates images","A single MLLM that reads and draws: next-token and next-scale","From Q&A to image generation: VARGPT does both autoregressively"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the generation benchmarks measure generalization rather than memorization, because the test instructions and the FID reference both come from ImageNet-1K, the same dataset the model was trained on, so high scores could instead reflect recall of training images and captions.","fun_headline_variants_meta":{"raw":{"variants":["VARGPT: one model, next-token understanding, next-scale generation","VARGPT unifies visual understanding and generation in one autoregressive pass","Beats LLaVA-1.5 on understanding and also generates images","A single MLLM that reads and draws: next-token and next-scale","From Q&A to image generation: VARGPT does both autoregressively"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002056,"raw_usage":{"total_tokens":8036,"prompt_tokens":1011,"completion_tokens":7025,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":6924}},"tokens_in":627,"tokens_out":7025,"duration_ms":47397,"temperature":1.0,"reasoning_tokens":6924,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:16:00.815005+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate instruction-to-image generation on prompts built from a different distribution, such as COCO captions or ImageNet categories held out from training, while keeping VARGPT frozen; if FID and CLIP scores drop far below the ImageNet-derived results, the reported instruction-following is mostly in-distribution recall.","supporting_citations":[{"cited_title":"Improved baselines with visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the base LLaVA-1.5 architecture and initial weights for the LLM, visual encoder, and understanding projector."},{"cited_title":"Visual autoregressive modeling: Scalable image generation via next-scale prediction","cited_arxiv_id":null,"evidence_quote":"Defines the next-scale prediction paradigm, the multi-scale tokenizer, the block-causal attention layout, and the visual decoder initialization."},{"cited_title":"Datasets: imagenet-1k-vl-enriched","cited_arxiv_id":null,"evidence_quote":"Provides the enriched ImageNet-1K captions from which the generation instruction datasets are built."},{"cited_title":"Improved baselines with visual instruction tuning, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the 665K LLaVA-1.5 instruction-following samples used in stage-2 understanding supervised fine-tuning."},{"cited_title":"Llava-onevision: Easy visual task transfer, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the 508K LLaVA-OneVision instruction samples mixed into stage-2 training."}],"review_version":1}