Pith. sign in

REVIEW 2 major objections 5 minor 111 references

Argus-Unified shows that a unified image understanding-and-generation model can be trained for about $2,000 on 15.6 million public examples by bootstrapping a pretrained vision-language model with hybrid continuous/discrete visual tokens, r

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:09 UTC pith:B7Z77X67

load-bearing objection Argus-Unified is a genuinely cheap recipe for unified multimodal modeling, but its SOTA understanding claim is unanchored without the base VLM baseline. the 2 major comments →

arxiv 2607.25527 v1 pith:B7Z77X67 submitted 2026-07-28 cs.CV cs.AI

Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation

Weiming Zhuang , Jiabo Huang , Jingtao Li , Zhizhong Li , Chen Chen , Sina Sajadmanesh , Lingjuan Lyu This is my paper
classification cs.CV cs.AI
keywords unified multimodal modelsimage understandingimage generationhybrid visual tokenspretrained vision-language modelsautoregressive generationvector quantizationefficient training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Argus-Unified claims that a unified image understanding and generation model can be trained for roughly $2,000 on 15.6 million public examples—about 10 times less compute and 5 times less data than comparable unified models—by starting from a pretrained vision-language model instead of aligning modalities from scratch. Its central idea is a hybrid token design: the same frozen vision encoder feeds continuous tokens to the language model for understanding and discrete tokens to a learned quantizer/decoder for generation. On GQA, POPE, and VQAv2 the 1.5B model beats unified models with dedicated generation encoders and larger backbones, while matching their generation quality on GenEval and MJHQ-30K. If correct, this shows the main cost driver for unified models is re-learning vision-language alignment, which can be inherited from existing VLMs.

Core claim

The paper claims that pretrained vision-language models can be turned into unified multimodal models with a two-stage recipe. Stage 1 trains a vector-quantized tokenizer (quantizer plus image decoder) on top of a frozen vision encoder using only reconstruction losses; Stage 2 initializes the LLM from a pretrained VLM and fine-tunes it with a next-token objective on mixed understanding and generation data. The hybrid token scheme keeps continuous features for understanding and quantized features for generation, avoiding the semantic loss of discrete-only models and the parameter overhead of dual-encoder designs. The authors report state-of-the-art understanding on GQA, POPE, and VQAv2 and gen

What carries the argument

Hybrid visual tokens from a single frozen vision encoder. Continuous tokens (raw encoder features) are projected into the LLM for understanding; the same features are mapped through a multi-codebook vector quantizer into discrete codes for generation, and an image decoder reconstructs pixels from those codes. A two-stage training pipeline first fits the quantizer/decoder with reconstruction, perceptual, adversarial, and vector-quantization losses, then fine-tunes the VLM-initialized LLM with next-token prediction on mixed data. The load-bearing mechanism is that freezing the encoder and inheriting VLM weights removes the need for large-scale vision-language alignment.

Load-bearing premise

The load-bearing premise is that a frozen semantic vision encoder retains enough fine-grained visual detail for a learned quantizer and decoder to reconstruct high-quality images, and the paper only claims this in the compact model regime.

What would settle it

Swap the frozen vision encoder for one whose features are global-pooled (no spatial detail) and retrain only the Stage-1 quantizer and decoder; if the ImageNet reconstruction FID degrades sharply and Stage-2 generation benchmarks (GenEval, MJHQ-30K) collapse while understanding stays high, the frozen-encoder information ceiling—not the LLM—is what bounds generation.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Unified multimodal training can be done at micro-budget; the cost and data barrier drops by an order of magnitude relative to current unified models.
  • A VLM's continuous visual features are rich enough to also drive generation through a learned quantizer, so dedicated generation encoders are not necessary.
  • Hybrid tokens substantially improve understanding over discrete-only tokens while preserving generation quality, as shown by the ablation.
  • A 0.5B-LLM variant outperforms several 7B+ unified models on understanding benchmarks, suggesting LLM scale is not the main driver in this regime.
  • Data scaling behaves differently when starting from a pretrained VLM: more supervised fine-tuning data helps, but more pretraining data can hurt.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If this recipe holds, any strong open vision-language model could be converted into a unified image model at modest cost, making unified image models accessible to small labs and startups.
  • The frozen-encoder assumption implies a ceiling on generation: the paper's own ablation with a different encoder and codebook suggests that encoder choice matters, and future work may close the gap to dedicated tokenizers.
  • The finding that more pretraining data can degrade performance when starting from a pretrained VLM may generalize to other VLM-initiated fine-tuning, inverting the usual from-scratch data scaling recipe.
  • The method's scope as stated is compact models; extending to image editing or video generation would stress the frozen-encoder information ceiling and could reveal where the hybrid token design breaks down.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. Argus-Unified proposes a two-stage recipe for turning a pretrained VLM into a compact unified image-understanding and image-generation model. A frozen vision encoder is retained from a pretrained VLM; Stage 1 trains a quantizer and an image decoder on top of this frozen encoder, and Stage 2 trains the LLM (initialized from the VLM) jointly on understanding and generation data with a hybrid token design: continuous tokens for understanding and discrete tokens for generation. The paper reports state-of-the-art understanding results on GQA, POPE, and VQAv2 among the compared unified multimodal models, competitive generation results, and a total training cost of about $2,000 using 15.6M public data samples.

Significance. The main contribution is the empirical claim that a unified multimodal model can be obtained economically by adapting an existing pretrained VLM rather than training components from scratch. If the claim survives scrutiny, the hybrid-token design and the two-stage recipe are practically valuable, and the paper's emphasis on public data and low compute is a useful counterpoint to the large-scale trend in this area. The paper includes multiple ablations (initialization, token types, data volume, CFG, tokenizer data, resolution), which strengthen the evidence for the proposed pipeline. The absence of the base-VLM baseline, however, makes the headline 'state-of-the-art understanding' claim insufficiently supported.

major comments (2)
  1. [§4.2, Tab. 1, Tab. 8; §3.2] The headline understanding claim is not anchored to the pretrained VLM used for initialization. Argus-Unified-1.5B is initialized from InternVL3-2B, and Stage 2 SFT includes standard understanding data (ShareGPT4V, WiT, VFLAN, MGM-Instruct). The paper never reports the performance of untouched InternVL3-2B on GQA, POPE, VQAv2, or MME-P under the same evaluation protocol. Therefore the observed numbers could reflect the base VLM's inherited ability, the added SFT data, or degradation from joint generation training. Table 5 only contrasts initialization from a pretrained VLM against initialization from separate pretrained ViT and LLM; it does not isolate the effect of unified training relative to the base model. This is load-bearing for the abstract's 'state-of-the-art multimodal understanding' claim and for the paper's central economic argument. I request adding the untouched InternVL3-2B
  2. [§4.2, Tab. 1 and Claims in Abstract] The comparison set for 'state-of-the-art multimodal understanding' omits current strong understanding-only VLMs beyond the base model. Table 8 includes MobileVLM, LLaVA, InstructBLIP, Qwen-VL-Chat, and IDEFICS, but not more recent and stronger compact VLMs such as InternVL3-2B itself, Qwen2.5-VL, or SmolVLM. The paper's SOTA language is broad ('state-of-the-art multimodal understanding on GQA, POPE, and VQAv2'), but the evidence only supports 'best among the UMMs tabulated and among the older understanding-only models tabulated.' Adding the base VLM and at least one or two contemporary compact VLMs would let the reader assess whether the unified training preserves the pretrained understanding ability, which is the central claim.
minor comments (5)
  1. [§4.3, Tab. 3] The table heading says 'training with (w/) and without CFG', but classifier-free guidance is typically an inference-time mechanism. Please clarify whether CFG dropout is used during training, and specify the guidance scale used for the main results in Table 1, since reproduction of generation scores depends on this.
  2. [§4.1, Tab. 2 and footnote 1] The cost arithmetic should be stated unambiguously. Table 2 says '800×H100' under 'GPU Hours', while the footnote quotes an 8×H100 node rate of $20.3/hour. 800 GPU-hours at the per-GPU rate would be about $16,000; 100 node-hours at $20.3/hour gives about $2,032. If the latter is the intended calculation, say so explicitly and label the table column 'GPU-hours' or 'node-hours' consistently.
  3. [Tab. 1] Janus-Pro has no MJHQ-30K or VQAv2 entry, so the statement that generation quality is 'competitive compared to ... Janus-Pro' rests on GenEval alone. Please state this limitation in the comparison or fill in the missing values if available.
  4. [Fig. 1 caption] The inverted axis for MJHQ-30K is explained but the caption could be clearer that only the axis is reversed and that lower original FID is better. Consider labeling the axis with the original FID scale directly instead of an inverted scale.
  5. [General] The paper does not mention whether code, model checkpoints, or the exact training/evaluation scripts will be released. Given the paper's reproducibility emphasis and public data usage, stating the release plan would strengthen the contribution.

Circularity Check

0 steps flagged

No significant circularity: the benchmark results are empirical measurements from a concrete training pipeline, not derived predictions that reduce to the paper's own inputs.

full rationale

The paper's central claims are empirical: Stage 1 trains a quantizer and image decoder on a frozen vision encoder with a reconstruction objective (Eq. 1), and Stage 2 initializes the LLM from InternVL3-2B and trains it with next-token prediction on 9.7M image-text pairs. The reported GQA, POPE, VQAv2, MJHQ-30K, and GenEval numbers are measurements of the resulting model. No equation or training objective is shown to be equivalent to a benchmark score by construction, and no fitted parameter is renamed as a prediction. The self-citations (Argus, Huang et al. 'Empirical recipes') appear only in related-work or contextual sentences and do not carry the load of the state-of-the-art claim. The absence of an untouched InternVL3-2B baseline is a legitimate attribution/soundness concern, because the understanding scores may be inherited from the pretrained VLM, but that is not circularity: the unified model's scores are still empirical and are not derived from the conclusion. Similarly, the tokenizer-ceiling concern expresses an architectural dependency, not a circular reduction, and the authors explicitly test it in Table 11 by showing that better reconstruction (lower rFID) does not translate into better unified-model performance. There is no self-citation chain invoked to force a design choice, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The paper is therefore self-contained with respect to circularity, even though its understanding claim would be strengthened by reporting the base VLM baseline.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper's contributions are architectural and empirical. It introduces no new physical or mathematical entities, but it relies on several domain assumptions about the sufficiency of frozen VLM features for both understanding and generation. The main free parameters are the tokenizer loss weights, codebook configuration, inference CFG scale, and data mix, none of which are fully specified.

free parameters (5)
  • Tokenizer loss weights λ_P, λ_G, λ_VQ
    Weights in Eq. (1) for perceptual, GAN, and VQ losses are not reported; they are standard hyperparameters likely tuned by the authors.
  • Codebook size and number of codebooks in multi-codebook quantizer = not stated; ablation doubles codebook size
    The quantizer design is described as 'multi-codebook inspired by UniTok', but the actual size and number of codebooks are omitted. This is a critical detail for reproducing Stage 1.
  • Inference CFG guidance scale
    CFG is reported to improve MJHQ-30K FID (Table 3), but the guidance scale used at inference is not given.
  • Number of image tiles n=6 for understanding = 6
    Chosen to match InternVL preprocessing; affects the number of continuous tokens fed to the LLM.
  • Stage 2 data mix proportions = PT 5.35M (JourneyDB 4.1M + ShareGPT4V 1.25M); SFT 9.7M
    The final data quantities were selected based on ablations (Table 10) showing that more SFT data helps and more PT data hurts. This is a recipe-level tuning choice.
axioms (4)
  • domain assumption Continuous tokens from a pretrained VLM preserve sufficient semantic information for visual understanding.
    Used throughout; the understanding pathway feeds the frozen encoder's continuous tokens directly into the LLM (Sec. 3.2).
  • domain assumption A frozen vision encoder's features contain enough spatial detail to reconstruct images through a learned quantizer/decoder.
    Stage 1 relies solely on reconstruction loss with the encoder frozen (Eq. 1). If the frozen features discard high-frequency detail, generation quality is capped.
  • domain assumption Discrete visual tokens can serve as effective next-token prediction targets for an autoregressive LLM.
    Stage 2 trains the LLM to predict discrete visual token indices with NLL loss, a standard assumption in AR image generation (Sec. 3.2).
  • domain assumption Pretrained VLM initialization reduces or eliminates the need for large-scale vision-language alignment data in UMM training.
    This is the core economics claim: the authors assume that starting from InternVL3-2B gives the model enough alignment that only a modest warm-up and SFT are needed (Sec. 3.2, Sec. 4.3).

pith-pipeline@v1.3.0-alltime-deepseek · 23133 in / 14391 out tokens · 143162 ms · 2026-08-01T02:09:06.088204+00:00 · methodology

0 comments
read the original abstract

Unifying visual understanding and generation in one model holds immense promise, but remains challenging and expensive due to heavy compute and data demands and conflicts between the visual features needed for these two capabilities. To address these challenges, we present Argus-Unified, a compact, effective and unified multimodal model built with low demand on computation and data. Instead of aligning modalities from scratch, Argus-Unified effectively leverages pretrained vision-language models (VLMs) that provide strong multimodal priors. Specifically, we introduce hybrid visual tokens that preserve continuous tokens for understanding while learning discrete tokens for generation from a frozen unified vision encoder. Our training pipeline includes two stages: the first stage learns a quantizer and image decoder on top of the frozen vision encoder, the second stage trains the LLM initialized from a pretrained VLM for the unified multimodal modeling. Using by far the least amount of data (15.6M) and the lowest cost (~$2,000), we demonstrate that unified multimodal models can be trained economically while achieving strong performance in both understanding and generation. Notably, our model attains state-of-the-art multimodal understanding on GQA, POPE, and VQAv2, and competitive generation quality compared to models with dedicated vision encoders (e.g., Janus, Janus-Pro), all at ~10x lower cost and with ~5x less data. We envision Argus-Unified as a useful baseline that lowers the development barrier for unified models.

Figures

Figures reproduced from arXiv: 2607.25527 by Chen Chen, Jiabo Huang, Jingtao Li, Lingjuan Lyu, Sina Sajadmanesh, Weiming Zhuang, Zhizhong Li.

Figure 1
Figure 1. Figure 1: Argus-Unified achieves strong performance [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustrations of model architectures of autoregressive vision-language model (VLM) and unified multimodal [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of Argus-Unified. Argus-Unified [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Image understanding examples produced by Argus-Unified-1.5B. Area of Effect Healing treerealism ver M Man Monkey with headphone [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Images generated by Argus-Unified-1.5B. More examples are provided in the Fig. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Loss curves over the course of Stage 2 train [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of reconstructed images on ImageNet. Our frozen-encoder design produces high-fidelity [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Images generated by Argus-Unified from the corresponding text prompts. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

111 extracted references · 52 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    Pith/arXiv arXiv

  2. [2]

    2018 , publisher=

    Improving language understanding by generative pre-training , author=. 2018 , publisher=

    2018

  3. [3]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  4. [4]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2103.10360 , year=

    Glm: General language model pretraining with autoregressive blank infilling , author=. arXiv preprint arXiv:2103.10360 , year=

    Pith/arXiv arXiv

  6. [6]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  7. [7]

    Advances in neural information processing systems , volume=

    Visual instruction tuning , author=. Advances in neural information processing systems , volume=

  8. [8]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Improved baselines with visual instruction tuning , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  9. [9]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Vila: On pre-training for visual language models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  10. [10]

    5-vl technical report , author=

    Qwen2. 5-vl technical report , author=. arXiv preprint arXiv:2502.13923 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2412.05271 , year=

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling , author=. arXiv preprint arXiv:2412.05271 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2504.10479 , year=

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models , author=. arXiv preprint arXiv:2504.10479 , year=

    Pith/arXiv arXiv

  13. [13]

    arXiv preprint arXiv:2403.05525 , year=

    Deepseek-vl: towards real-world vision-language understanding , author=. arXiv preprint arXiv:2403.05525 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2412.10302 , year=

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding , author=. arXiv preprint arXiv:2412.10302 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv preprint arXiv:2504.05299 , year=

    Smolvlm: Redefining small and efficient multimodal models , author=. arXiv preprint arXiv:2504.05299 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:2406.06525 , year=

    Autoregressive model beats diffusion: Llama for scalable image generation , author=. arXiv preprint arXiv:2406.06525 , year=

    Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:2410.10812 , year=

    Hart: Efficient visual generation with hybrid autoregressive transformer , author=. arXiv preprint arXiv:2410.10812 , year=

    Pith/arXiv arXiv

  18. [18]

    Advances in neural information processing systems , volume=

    Visual autoregressive modeling: Scalable image generation via next-scale prediction , author=. Advances in neural information processing systems , volume=

  19. [19]

    Proceedings of machine learning and systems , volume=

    Efficiently scaling transformer inference , author=. Proceedings of machine learning and systems , volume=

  20. [20]

    Proceedings of the 29th symposium on operating systems principles , pages=

    Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=

  21. [21]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Janus: Decoupling visual encoding for unified multimodal understanding and generation , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  22. [22]

    arXiv preprint arXiv:2409.04429 , year=

    Vila-u: a unified foundation model integrating visual understanding and generation , author=. arXiv preprint arXiv:2409.04429 , year=

    Pith/arXiv arXiv

  23. [23]

    arXiv preprint arXiv:2409.18869 , year=

    Emu3: Next-token prediction is all you need , author=. arXiv preprint arXiv:2409.18869 , year=

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2412.04332 , year=

    Liquid: Language models are scalable and unified multi-modal generators , author=. arXiv preprint arXiv:2412.04332 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2506.18898 , year=

    Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations , author=. arXiv preprint arXiv:2506.18898 , year=

    Pith/arXiv arXiv

  26. [26]

    Advances in neural information processing systems , volume=

    Neural discrete representation learning , author=. Advances in neural information processing systems , volume=

  27. [27]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Taming transformers for high-resolution image synthesis , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  28. [28]

    arXiv preprint arXiv:2502.20321 , year=

    Unitok: A unified tokenizer for visual generation and understanding , author=. arXiv preprint arXiv:2502.20321 , year=

    arXiv

  29. [29]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  30. [30]

    arXiv preprint arXiv:2501.17811 , year=

    Janus-pro: Unified multimodal understanding and generation with data and model scaling , author=. arXiv preprint arXiv:2501.17811 , year=

    Pith/arXiv arXiv

  31. [31]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  32. [32]

    International conference on machine learning , pages=

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  33. [33]

    International conference on machine learning , pages=

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  34. [34]

    arXiv preprint arXiv:2205.01917 , year=

    Coca: Contrastive captioners are image-text foundation models , author=. arXiv preprint arXiv:2205.01917 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2505.20147 , year=

    Fudoki: Discrete flow-based unified understanding and generation via kinetic-optimal velocities , author=. arXiv preprint arXiv:2505.20147 , year=

    Pith/arXiv arXiv

  36. [36]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Dual diffusion for unified image generation and understanding , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  37. [37]

    arXiv preprint arXiv:2505.23606 , year=

    Muddit: Liberating generation beyond text-to-image with a unified discrete diffusion model , author=. arXiv preprint arXiv:2505.23606 , year=

    Pith/arXiv arXiv

  38. [38]

    arXiv preprint arXiv:2503.10772 , year=

    Flowtok: Flowing seamlessly across text and image tokens , author=. arXiv preprint arXiv:2503.10772 , year=

    arXiv

  39. [39]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  40. [40]

    arXiv preprint arXiv:2408.12528 , year=

    Show-o: One single transformer to unify multimodal understanding and generation , author=. arXiv preprint arXiv:2408.12528 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2506.15564 , year=

    Show-o2: Improved Native Unified Multimodal Models , author=. arXiv preprint arXiv:2506.15564 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2412.14164 , year=

    Metamorph: Multimodal understanding and generation via instruction tuning , author=. arXiv preprint arXiv:2412.14164 , year=

    Pith/arXiv arXiv

  43. [43]

    arXiv preprint arXiv:2504.06256 , year=

    Transfer between modalities with metaqueries , author=. arXiv preprint arXiv:2504.06256 , year=

    Pith/arXiv arXiv

  44. [44]

    arXiv preprint arXiv:2405.09818 , year=

    Chameleon: Mixed-modal early-fusion foundation models , author=. arXiv preprint arXiv:2405.09818 , year=

    Pith/arXiv arXiv

  45. [45]

    arXiv preprint arXiv:2410.01756 , year=

    Imagefolder: Autoregressive image generation with folded tokens , author=. arXiv preprint arXiv:2410.01756 , year=

    Pith/arXiv arXiv

  46. [46]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Maskgit: Masked generative image transformer , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  47. [47]

    arXiv e-prints , pages=

    The llama 3 herd of models , author=. arXiv e-prints , pages=

  48. [48]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Gqa: A new dataset for real-world visual reasoning and compositional question answering , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  49. [49]

    arXiv preprint arXiv:2306.13394 , volume=

    Mme: A comprehensive evaluation benchmark for multimodal large language models , author=. arXiv preprint arXiv:2306.13394 , volume=

    Pith/arXiv arXiv

  50. [50]

    Proceedings of the IEEE international conference on computer vision , pages=

    Vqa: Visual question answering , author=. Proceedings of the IEEE international conference on computer vision , pages=

  51. [51]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  52. [52]

    Advances in Neural Information Processing Systems , volume=

    Geneval: An object-focused framework for evaluating text-to-image alignment , author=. Advances in Neural Information Processing Systems , volume=

  53. [53]

    arXiv preprint arXiv:2305.10355 , year=

    Evaluating object hallucination in large vision-language models , author=. arXiv preprint arXiv:2305.10355 , year=

    Pith/arXiv arXiv

  54. [54]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  55. [55]

    Proceedings of the national academy of sciences , volume=

    Overcoming catastrophic forgetting in neural networks , author=. Proceedings of the national academy of sciences , volume=. 2017 , publisher=

    2017

  56. [56]

    European Conference on Computer Vision , pages=

    Sharegpt4v: Improving large multi-modal models with better captions , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  57. [57]

    Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval , pages=

    Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning , author=. Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval , pages=

  58. [58]

    arXiv preprint arXiv:2402.11684 , year=

    Allava: Harnessing gpt4v-synthesized data for lite vision-language models , author=. arXiv preprint arXiv:2402.11684 , year=

    Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:2403.18814 , year=

    Mini-gemini: Mining the potential of multi-modality vision language models , author=. arXiv preprint arXiv:2403.18814 , year=

    Pith/arXiv arXiv

  60. [60]

    Advances in Neural Information Processing Systems , volume=

    Learn to explain: Multimodal reasoning via thought chains for science question answering , author=. Advances in Neural Information Processing Systems , volume=

  61. [61]

    Advances in neural information processing systems , volume=

    Journeydb: A benchmark for generative image understanding , author=. Advances in neural information processing systems , volume=

  62. [62]

    arXiv preprint arXiv:2505.09568 , year=

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset , author=. arXiv preprint arXiv:2505.09568 , year=

    Pith/arXiv arXiv

  63. [63]

    arXiv preprint arXiv:2508.09987 , year=

    Echo-4o: Harnessing the power of gpt-4o synthetic images for improved image generation , author=. arXiv preprint arXiv:2508.09987 , year=

    Pith/arXiv arXiv

  64. [64]

    Proceedings of ACL , year =

    Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning , author =. Proceedings of ACL , year =

  65. [65]

    arXiv preprint arXiv:2210.14896 , year=

    Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models , author=. arXiv preprint arXiv:2210.14896 , year=

    Pith/arXiv arXiv

  66. [66]

    5: Three insights towards enhancing aesthetic quality in text-to-image generation , author=

    Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation , author=. arXiv preprint arXiv:2402.17245 , year=

    Pith/arXiv arXiv

  67. [67]

    5 technical report , author=

    Qwen2. 5 technical report , author=. arXiv preprint , year=

  68. [68]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  69. [69]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Vitamin: Designing scalable vision models in the vision-language era , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  70. [70]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Autoregressive image generation using residual quantization , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  71. [71]

    arXiv preprint arXiv:2505.15809 , year=

    Mmada: Multimodal large diffusion language models , author=. arXiv preprint arXiv:2505.15809 , year=

    Pith/arXiv arXiv

  72. [72]

    arXiv preprint arXiv:2503.21979 , year=

    Harmonizing visual representations for unified multimodal understanding and generation , author=. arXiv preprint arXiv:2503.21979 , year=

    Pith/arXiv arXiv

  73. [73]

    arXiv preprint arXiv:2506.17202 , year=

    UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation , author=. arXiv preprint arXiv:2506.17202 , year=

    Pith/arXiv arXiv

  74. [74]

    arXiv preprint arXiv:2404.14396 , year=

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation , author=. arXiv preprint arXiv:2404.14396 , year=

    Pith/arXiv arXiv

  75. [75]

    arXiv preprint arXiv:2504.01934 , year=

    Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement , author=. arXiv preprint arXiv:2504.01934 , year=

    Pith/arXiv arXiv

  76. [76]

    arXiv preprint arXiv:2309.04669 , year=

    Unified language-vision pretraining in llm with dynamic discrete visual tokenization , author=. arXiv preprint arXiv:2309.04669 , year=

    arXiv

  77. [77]

    Advances in Neural Information Processing Systems , volume=

    Autoregressive image generation without vector quantization , author=. Advances in Neural Information Processing Systems , volume=

  78. [78]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Tokenflow: Unified image tokenizer for multimodal understanding and generation , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  79. [79]

    arXiv preprint arXiv:2402.08268 , year=

    World model on million-length video and language with blockwise ringattention , author=. arXiv preprint arXiv:2402.08268 , year=

    Pith/arXiv arXiv

  80. [80]

    arXiv preprint arXiv:2503.06764 , year=

    Semhitok: A unified image tokenizer via semantic-guided hierarchical codebook for multimodal understanding and generation , author=. arXiv preprint arXiv:2503.06764 , year=

    arXiv

Showing first 80 references.