Pith. sign in

REVIEW 3 major objections 6 minor 14 cited by

LaViDa: A Large Diffusion Language Model for Multimodal Understanding

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LaViDa shows that discrete diffusion language models, which generate text by unmasking tokens in parallel, can serve as the backbone of a competitive vision-language model while preserving controllable infilling and a tunable…

desk verdict Solid diffusion-VLM engineering with a real evaluation leak: LaViDa-Reason's +18% MathVision gain was selected on the test split, and the abstract's speedup numbers don't match the tables. read the letter →

arxiv 2505.16839 v4 pith:2EBPGS53 submitted 2025-05-22 cs.CV

Shufan Li , Konstantinos Kallidromitis , Hritik Bansal , Akash Gokul , Yusuke Kato , Kazuki Kozuka , Jason Kuen , Zhe Lin
show 2 more authors
Kai-Wei Chang Aditya Grover
This is my paper · ORCID
classification cs.CV
keywords diffusionlanguagemodelsvision-languagediscretemultimodalunderstandingtextinfillingparalleldecodingKVcacheinstructionfollowing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to prove that discrete diffusion models—which generate text by starting from a string of mask tokens and unmasking them in parallel—can serve as the language backbone of a vision-language model, not just a language-only curiosity. It introduces LaViDa, built by grafting a SigLIP vision encoder and an MLP projector onto pretrained diffusion language models (LLaDA-8B and Dream-7B), and shows the result is competitive with comparable autoregressive VLMs like LLaVA-1.6 and Open-LLaVA-Next on general, reasoning, OCR, and science benchmarks. The reason a reader should care is that diffusion decoding changes what a VLM can do: generation speed becomes a dial rather than a fixed cost, and outputs can be constrained by bidirectional infilling, which matters for structured extraction and format-adherent generation. Three engineering contributions make this possible: complementary masking so every answer token is used in training, a prefix KV cache so multimodal prompts are computed once, and a timestep-shifting schedule that preserves quality at few diffusion steps.

What carries the argument

The central object is the masked discrete diffusion language model, a non-causal Transformer trained to predict clean tokens from partially masked sequences. Starting from a fully masked output sequence, inference unmasks tokens over $K$ steps, so the number of forward passes is a free parameter. Three mechanisms carry the argument: complementary masking, which creates two complementary masked copies per sample so every answer token contributes to the loss; Prefix-DLM, an attention mask that lets image and prompt tokens attend only to other image and prompt tokens so their key–value pairs are cached once; and timestep shifting, a convex schedule $t'_i = \alpha t_i / (1 + (\alpha-1)t_i)$ with $\alpha = 1/3$ that unmasks more tokens early and is reported to be essential at low $K$.

What would settle it

Take LaViDa-Reason, select its checkpoint on a held-out validation split rather than on MathVision testmini, and compare the resulting MathVision accuracy to the testmini-selected checkpoint; if the reported +18% relative gain collapses to noise, the claimed reasoning advantage from diffusion distillation is not supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a masked discrete diffusion language model can be turned into a competitive vision-language model by attaching a vision encoder and fine-tuning the whole stack with a diffusion objective. LaViDa, built on the pretrained diffusion LMs LLaDA-8B and Dream-7B with a SigLIP vision encoder and an MLP projector, is reported to match or beat similar-sized autoregressive VLMs on MMMU, MathVista, ChartQA, and ScienceQA, and to outperform them on tasks that need bidirectional context: 100% constraint satisfaction on constrained poem completion versus below 50% for AR baselines, and +4.1 CIDEr on COCO captioning with a 1.92x speedup. The central discovery is a new application rather than a new architecture from scratch: diffusion language models, which decode by unmasking tokens in parallel, can carry multimodal instruction following while keeping their native controllability and speed–quality dial.

Load-bearing premise

The whole comparison treats the benchmark harness, the chosen generation lengths, and the checkpoint-selection procedure as a fair common test of both diffusion and autoregressive decoding, and the math-reasoning gain in particular depends on picking the checkpoint that scored best on MathVision testmini.

Editorial extensions

If this is right

  • A diffusion-based VLM can be built by fine-tuning a pretrained discrete diffusion language model with a vision encoder, and at the same data and parameter scale it matches autoregressive VLMs on general, reasoning, and science benchmarks.
  • Generation speed becomes a user-controlled dial: on COCO captioning, running at 50% of the tokens per forward pass is faster than the AR baselines while still producing higher CIDEr, and at 25% it trades quality for a large speedup.
  • Native bidirectional infilling lets the model satisfy hard output constraints, such as fixed line-initial syllables in poems or JSON schemas, without prompt engineering; the paper reports 100% constraint satisfaction where AR baselines stay below 50%.
  • The Prefix-DLM cache makes long multimodal prompts practical by computing image and prompt key–value pairs once, giving up to a 3.9x speedup on captioning with a small quality cost.
  • A short stage-3 distillation on 19.2K chain-of-thought examples improves math reasoning, with the largest relative gain (+18%) on MathVision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if diffusion backbones scale with data and parameters the way autoregressive backbones have, the controllability and parallel-decoding advantages could make diffusion the default for schema-constrained vision-language applications such as structured extraction, form filling, and document editing.
  • Beyond the paper, the finding that a convex unmasking schedule helps text while concave schedules help image diffusion suggests the information profile of language tokens differs from image latents; a direct test would compare per-step token entropy on captioning versus image generation under each schedule.
  • Beyond the paper, the prefix cache idea could be combined with block-wise semi-autoregressive decoding to recover some of the KV-cache benefit while keeping longer bidirectional context, a direction the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces LaViDa, a family of vision-language models (VLMs) built on discrete diffusion language models (LLaDA-8B and Dream-7B). The authors propose three technical contributions: complementary masking for more efficient training, Prefix-DLM (a prefix attention mask that enables KV caching at inference), and a timestep-shifting schedule for improved sampling quality. They evaluate LaViDa on general (MMMU, MME, MMBench, VQAv2), reasoning (MathVista, MathVerse, MathVision), science (ScienceQA, AI2D), and OCR (TextVQA, DocVQA, ChartQA, InfoVQA) benchmarks, reporting competitive performance against similarly sized autoregressive VLMs such as LLaVA-1.6-7B and Open-LLaVA-Next-Llama3-8B. They also demonstrate speed-quality tradeoffs on COCO captioning and controllable text infilling (e.g., constrained poem completion). The abstract and introduction additionally claim a +4.1 CIDEr improvement with a 1.92x speedup over Open-LLaVA-Next-8B on COCO.

Significance. If the claims withstand scrutiny, this is a valuable contribution: it provides the first systematic demonstration that discrete diffusion language models can serve as viable backbones for multimodal instruction following, with practical benefits such as bidirectional context for constrained generation (text infilling, poem completion) and a tunable speed-quality tradeoff. The paper is thorough in its ablations (complementary masking, image resolution, vision encoder, attention-mask variants, schedules) and it evaluates across a broad set of standard benchmarks using a common harness (LMMS-Eval). The proposed techniques—especially Prefix-DLM and timestep shifting—are clearly described and could inform future work on non-autoregressive multimodal models. However, two load-bearing issues—test-set checkpoint selection for LaViDa-Reason and an internally inconsistent abstract number—currently prevent full confidence in the central performance claims.

major comments (3)
  1. [Appendix B.3 and Table 2a] The LaViDa-Reason model was selected by choosing 'the checkpoint that achieves the best performance on MathVision (testmini)' after 5 epochs of fine-tuning. Because the reported +18% relative improvement on MathVision is the maximum over the checkpoints evaluated directly on the test split, this number is likely inflated by selection-on-the-test-set bias. The same testmini set is also used in Table 1 for the base-model comparison, so this is not an independent evaluation. Please select checkpoints on a held-out validation split (or report all epoch-wise results) and recompute the claimed improvements; at minimum, state the selection procedure clearly and discuss the potential inflation.
  2. [Abstract vs. Section 4.5 and Table 3a] The abstract and the introduction claim that LaViDa 'surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x speedup' on COCO captioning, but no operating point in Table 3a matches these numbers. At NFE=50%, Prefix-DLM gives CIDEr 114.8 (only +3.0 over Open-LLaVA-Next) with 1.23s latency versus 1.71s for the baseline, i.e., a 1.39x speedup; at NFE=100%, CIDEr is 117.3 (+5.5) but latency is 1.93s, slower than the baseline. Please correct the abstract or specify the exact configuration (NFE, schedule, generation length, and latency measurement) that yields +4.1 CIDEr and 1.92x speedup. As written, the headline fast-inference advantage is not reproducible from the reported experiments.
  3. [Section 4.3 and Table 6] The comparison in Table 2a between LaViDa (stage-2) and LaViDa-Reason is confounded by generation length: according to Table 6, the stage-2 model is evaluated with L=100, while LaViDa-Reason uses L=1024. The reported gains on MathVista, MathVerse, and MathVision may therefore reflect the larger generation budget rather than the long-CoT distillation itself. To support the claim that the distillation is responsible, please report a control where the stage-2 model is evaluated with L=1024 (even if it produces degenerate output) or otherwise isolate the effect of the additional training.
minor comments (6)
  1. [Abstract] The abstract contains a grammatical error: 'Code and models is available' should be 'Code and models are available'.
  2. [Section 4.1] Typo: 'differet NFE' should be 'different NFE'.
  3. [Section 3.3] The definition of NFE as 'fraction of the number of functional evaluations' is confusing; please state clearly that NFE = K/L, where K is the number of diffusion steps and L is the generation length.
  4. [Table 3a] The speedup factors implied by the latency rows (e.g., 3.9x for Full-DLM vs Prefix-DLM at NFE=100%) are not shown; adding a column with the speedup relative to the uncached baseline would make the result easier to parse.
  5. [Appendix B.3] The sentence 'We observe that the same checkpoint achieved a good performance on the MathVerse and MathVista dataset too' should specify that these evaluations were also on the testmini splits, to avoid ambiguity about whether the checkpoint was selected using those sets.
  6. [Section 5] Typo in the conclusion: 'exitsing AR VLMs' should be 'existing AR VLMs'.

Circularity Check

1 steps flagged · score 6.0 of 10

MathVision testmini checkpoint selection makes the reported +18% reasoning gain a selected maximum rather than an independent prediction; the paper's other central claims remain externally benchmarked.

  1. fitted input called prediction [Appendix B.3, 'Math Reasoning' (data and training setup); Table 2a]
    "we finetune LaViDa on this data for 5 epochs using the identical training setup as stage-2 (e.g., batch size, learning rates) and chose the checkpoint that achieves the best performance on MathVision (testmini)."

    The headline MathVision result for LaViDa-Reason (24.0, +18% relative in Table 2a) is exactly the objective used for checkpoint selection: the chosen checkpoint is, by construction, the argmax over epochs on MathVision testmini. The reported 'most significant gains observed on the most challenging MathVision reasoning dataset' is therefore a selection maximum, not a held-out prediction of reasoning improvement from long-CoT distillation. The same selected checkpoint also determines all MathVision numbers in Table 7, so those comparisons inherit the same selection. This matches the pattern of fitting model selection on the target benchmark and then presenting the resulting score as a measured gain.

full rationale

The paper's central claim—that a diffusion-based VLM can be trained with a standard masked-diffusion objective (Eqs. 1-2) and reach competitive scores on MMMU, MathVista, ChartQA, ScienceQA, and COCO—is not defined in terms of any fitted quantity. These benchmarks are external, evaluation is done with LMMS-Eval, and the reported stage-2 results come from a fixed training run. Complementary masking, Prefix-DLM, and timestep shifting are empirical design choices ablated on held-out tasks; they are not derived from the claims they support. Self-citations (e.g., OmniFlow in Appendix C, MathVista as a benchmark, Bansal et al. as a filtering reference) are contextual or benchmark citations and are not load-bearing for the main derivation. One genuine circular step exists: LaViDa-Reason's MathVision score is produced by choosing the checkpoint with best MathVision (testmini) performance, so the +18% gain in Table 2a is a selected maximum rather than an independent prediction; Table 7 inherits this. Separately, but not circularity, the abstract's '+4.1 CIDEr with 1.92x speedup' has no matching row in Table 3a: at NFE=100% LaViDa is slower (1.93s vs 1.71s) and at NFE=50% it is 1.39x faster with +3.0 CIDEr. This is a support gap, not a circular derivation. Overall score 6 reflects one prediction that reduces by construction, with the paper's main competitive-performance claim remaining externally benchmarked.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on standard masked-diffusion math, an architectural assumption that visual features can be injected into a text-only DLM via an MLP projector, and the empirical validity of an inference-time attention-mask change. No invented entities are introduced.

free parameters (3)
  • timestep shift alpha = 1/3
    Selected by maximizing COCO captioning CIDEr over {1/3, 3, linear, cosine} (Table 3b); used in the speed-quality claims.
  • stage-2 data re-weighting weights = not specified
    Appendix B.1 says 'small adjustments to the weight of each data source' were made; exact values are not disclosed and they influence benchmark results.
  • vision token count after pooling = 980 tokens per image (5 views, 2x2 average pooling)
    Chosen to fit the 4096 and 2048 token context of LLaDA and Dream (Appendix D); OCR performance is directly affected.
assumptions (4)
  • standard math MDLM posterior and training loss (Eq. 6 and Eq. 9 of Appendix A.1) are valid for masked discrete diffusion.
    Taken from prior work [64, 57]; the paper builds on these without reproving them.
  • domain assumption Visual embeddings from SigLIP, after average pooling and MLP projection, are a sufficient conditioning signal for the diffusion LM.
    This mirrors the LLaVA-style architecture; no theoretical guarantee is given, and the paper measures the resulting benchmark performance.
  • ad hoc to paper Using a prefix attention mask at inference time (Prefix-DLM), despite full-attention training, does not materially degrade the learned distribution.
    Section 3.3 and B.4 test this empirically; Table 3a shows a 3-4 CIDEr drop. The claim of 'marginal performance cost' is a judgment call.
  • domain assumption Complementary masking yields a valid estimator of the diffusion loss.
    Section 3.2 constructs two disjoint corrupted views per sample. This changes the loss estimator; the paper validates it only by downstream benchmark scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LaViDa: A Large Diffusion Language Model for Multimodal Understanding." pith.science (2026). https://pith.science/paper/2EBPGS53

@misc{pith2026250516839,
  author       = {Pith},
  title        = {Pith review of: LaViDa: A Large Diffusion Language Model for Multimodal Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2EBPGS53}},
  note         = {Machine review of arXiv:2505.16839}
}
read the original abstract

Modern Vision-Language Models (VLMs) can solve a wide range of tasks requiring visual reasoning. In real-world scenarios, desirable properties for VLMs include fast inference and controllable generation (e.g., constraining outputs to adhere to a desired format). However, existing autoregressive (AR) VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs) offer a promising alternative, enabling parallel decoding for faster inference and bidirectional context for controllable generation through text-infilling. While effective in language-only settings, DMs' potential for multimodal tasks is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build LaViDa by equipping DMs with a vision encoder and jointly fine-tune the combined parts for multimodal instruction following. To address challenges encountered, LaViDa incorporates novel techniques such as complementary masking for effective training, prefix KV cache for efficient inference, and timestep shifting for high-quality sampling. Experiments show that LaViDa achieves competitive or superior performance to AR VLMs on multi-modal benchmarks such as MMMU, while offering unique advantages of DMs, including flexible speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x speedup. On bidirectional tasks, it achieves +59% improvement on Constrained Poem Completion. These results demonstrate LaViDa as a strong alternative to AR VLMs. Code and models will be released in the camera-ready version.

Figures

Figures reproduced from arXiv: 2505.16839 by the authors.

Figure 2
Figure 2. Overall design of LaViDa. LaViDa’s architecture consists of a vision encoder, a diffusion language model, and an MLP vision projector. The bottom half of the figure illustrates the image encoding process, while the top half depicts the diffusion language modeling process. These two pipelines are described in detail in Sec. 3.1. 3 Method 3.1 Model Architecture LaViDa’s model architecture follows a similar design to c… view at source ↗
Figure 3
Figure 3. Technical Details of LaViDa. (a) We propose Complementary Masking to ensure loss is calculated over all tokens in the data for training efficiency. (b) We propose Prefix-DLM attention mask that enables KV caching. We visualize the attention mask of image tokens (Img1-2), prompt tokens (Pmt1-2), and text and mask tokens in the noise answer Xt (Ans1, [M], Ans3). Rows represent queries, while columns are keys. Colored … view at source ↗
Figure 4
Figure 4. We showcase the advantages of LaViDa over AR VLMS in terms of controllability and speed. (a) Qualitative comparison on constrained poem generation between LaViDa /LaViDa-FIM and AR models. LaViDa variants successfully satisfy line-level constraints and adapt token length per line, unlike AR baselines. (b) Speed–quality tradeoff for image captioning on COCO 2017. By adjusting the number of discretization steps (K), L… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Visualization of Schedules (a) We visualize discretization results of the same continuous schedule (dashed line) under different numbers of sampling steps K ∈ {16, 24, 32} at L = 32. (b) We visualize various choices of continuous schedules: Shift(α = 3−1 ), Shift(α = 3…
Figure 6
Figure 6. Figure 6: Additional Qualitative Results for Text Infilling. We showcase several useful applications of text-infilling capabilities. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Visualization of Different Choices of Attention Mask at Inference Time. I represents the image embeddings, P represents the prompt tokens, and Xt represents the partially masked answer tokens. Each row represents a query and each column represents a key. Colored region…
Figure 8
Figure 8. Figure 8: Visualization of Training Strategies for Prefix-DLM. Given each triplet of image I, prompt P and answer X, we create two versions of partially masked answer Xt and XC t with complementary masking.(Left) By default, we construct two sequence and apply full attention mas…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Grounding

    cs.AI 2026-07 conditional novelty 6.0 of 10

    ST-Veto improves reasoning in diffusion MLLMs by vetoing temporally unstable tokens and tokens with weak image grounding, swapping in safer near-boundary candidates.

  2. Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A diffusion multimodal LLM's early-layer MLP sparsity at denoising step 0 pinpoints the semantic boundary, enabling one-shot suffix truncation that raises throughput up to ~31x.

  3. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

    cs.CL 2026-07 accept novelty 6.0 of 10

    Joint AR–diffusion training yields one tri-mode LM that switches AR, diffusion, and self-speculation, beating open AR/diffusion models on accuracy and tokens-per-forward.

  4. Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Masked discrete diffusion with token editing and grouped cross-entropy reaches strong text-to-image generation scores in an 8B decoder-only model, reporting GenEval 0.90, DPG 86.9, HPSv3 10.76.

  5. High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models

    cs.CV 2025-12 unverdicted novelty 6.0 of 10

    High-entropy tokens act as concentrated multimodal failure points in VLMs, enabling sparse Entropy-Guided Attacks that achieve 93-95% success and 30-38% harmful rates with cross-model transfer.

  6. Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Sparse-LaViDa speeds up masked diffusion LLMs by truncating masked tokens, using register tokens and a step-causal attention mask, reaching ~2-2.8x speedup over LaViDa-O with comparable generation quality.

  7. A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Visual token pruning causes severe loss in discrete diffusion MLLMs; only from-scratch models on long-answer tasks recover via late denoising, so redundancy is recoverability, not dispensability.

  8. Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Lavida-O introduces an elastic mixture-of-transformers architecture that brings high-resolution text-to-image generation, object grounding, and image editing into a single masked diffusion model, using planning and se...

  9. Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphs

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Spiffy speeds up diffusion LLM inference up to about 3x (and up to 7.9x with parallel decoding) by verifying multiple candidate unmasked states in one batched model call, while preserving greedy output.

  10. Inpainting-Guided Policy Optimization for Diffusion Large Language Models

    cs.LG 2025-09 reject novelty 6.0 of 10

    IGPO injects partial ground-truth reasoning hints into masked diffusion LLM generation during RL sampling, restoring non-zero advantages and improving math benchmarks, but as written the RL training set overlaps the e...

  11. LLaDA-VLA: Vision Language Diffusion Action Models

    cs.RO 2025-09 conditional novelty 6.0 of 10

    LLaDA-VLA applies a masked diffusion vision-language model to robot control with localized action-token classification and hierarchical decoding, achieving SOTA success rates on SimplerEnv, CALVIN, and real-robot tasks.

  12. DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 7B masked-diffusion code model plus complementary-mask GRPO (coupled-GRPO) improves benchmark scores and shifts decoding away from strict left-to-right order.

  13. Mask-Aware Policy Gradients for Diffusion Language Models

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Optimizing both token predictions and unmasking-position probabilities in MDLM policy gradients improves RL fine-tuning, reaching 87.1% GSM8K and 53.4% MBPP.

  14. A Survey on Diffusion Language Models

    cs.CL 2025-08 unverdicted novelty 3.0 of 10

    A comprehensive survey of diffusion language models covering taxonomy, training and inference techniques, and comparisons with autoregressive models.

Reference graph

Works this paper leans on

90 extracted references · 10 canonical work pages · cited by 14 Pith papers

  1. [1]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [2]

    Claude 3.5 family

    Anthropic. Claude 3.5 family. https://www.anthropic.com/claude/sonnet, 2024. Ac- cessed: 2025-05-15

    2024

  3. [3]

    Block diffusion: Interpolating between autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573, 2025

    Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and V olodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573, 2025

    arXiv 2025

  4. [4]

    Structured denoising diffusion models in discrete state-spaces.Advances in neural information processing systems, 34:17981–17993, 2021

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces.Advances in neural information processing systems, 34:17981–17993, 2021

    2021

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  6. [6]

    Smaller, weaker, yet better: Training llm reasoners via compute-optimal sampling.arXiv preprint arXiv:2408.16737, 2024

    Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training llm reasoners via compute-optimal sampling.arXiv preprint arXiv:2408.16737, 2024

    arXiv 2024

  7. [7]

    Efficient training of language models to fill in the middle.arXiv preprint arXiv:2207.14255, 2022

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle.arXiv preprint arXiv:2207.14255, 2022

    arXiv 2022

  8. [8]

    Scheduled sampling for sequence prediction with recurrent neural networks.Advances in neural information processing systems, 28, 2015

    Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks.Advances in neural information processing systems, 28, 2015

    2015

Show all 90 references
  1. [9]

    Level up your tutorials: Vlms for game tutorials quality assessment

    Daniele Rege Cambrin, Gabriele Scaffidi Militone, Luca Colomba, Giovanni Malnati, Daniele Apiletti, and Paolo Garza. Level up your tutorials: Vlms for game tutorials quality assessment. arXiv preprint arXiv:2408.08396, 2024

    2024 arXiv

  2. [10]

    Muse: Text-to-image generation via masked generative transformers.arXiv preprint arXiv:2301.00704, 2023

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers.arXiv preprint arXiv:2301.00704, 2023

    2023 arXiv

  3. [11]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315–11325, 2022

    2022

  4. [12]

    Allava: Harnessing gpt4v- synthesized data for a lite vision-language model, 2024

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v- synthesized data for a lite vision-language model, 2024. 10

    2024

  5. [13]

    Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning

    Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint arXiv:2105.14517, 2021

    2021 arXiv

  6. [14]

    Sharegpt4v: Improving large multi-modal models with better captions.arXiv preprint arXiv:2311.12793, 2023

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions.arXiv preprint arXiv:2311.12793, 2023

    2023 arXiv

  7. [15]

    Open-llava-next: An open-source implementation of llava-next series for facilitating the large multi-modal model community

    Lin Chen and Long Xing. Open-llava-next: An open-source implementation of llava-next series for facilitating the large multi-modal model community. https://github.com/ xiaoachen98/Open-LLaVA-NeXT, 2024

    2024

  8. [16]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101, 2024

    2024

  9. [17]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer visi...

    2024

  10. [18]

    Open- vlthinker: An early exploration to complex vision-language reasoning via iterative self- improvement.arXiv preprint arXiv:2503.17352, 2025

    Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Open- vlthinker: An early exploration to complex vision-language reasoning via iterative self- improvement.arXiv preprint arXiv:2503.17352, 2025

    2025 arXiv

  11. [19]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langua...

    2019

  12. [20]

    Flex attention: A pro- gramming model for generating optimized attention kernels.arXiv preprint arXiv:2412.05496, 2024

    Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A pro- gramming model for generating optimized attention kernels.arXiv preprint arXiv:2412.05496, 2024

    2024 arXiv

  13. [21]

    Scaling rectified flow trans- formers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. InForty-first international conference on machine...

    2024

  14. [22]

    Incoder: A generative model for code infilling and synthesis.arXiv preprint arXiv:2204.05999, 2022

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis.arXiv preprint arXiv:2204.05999, 2022

    2022 arXiv

  15. [23]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  16. [24]

    Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017

    2017

  17. [25]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  18. [26]

    Llava-uhd: an lmm perceiving any aspect ratio and high- resolution images

    Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high- resolution images. InEuropean Conference on Computer Vision, pages 390–406. Springer, 2024

    2024

  19. [27]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022. 11

    2022

  20. [28]

    Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    2024 arXiv

  21. [29]

    Unified discrete diffusion for simultaneous vision-language generation.arXiv, 2022

    Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang, Zuopeng Yang, Dacheng Tao, and Ponnuthurai N Suganthan. Unified discrete diffusion for simultaneous vision-language generation.arXiv, 2022

    2022

  22. [30]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  23. [31]

    Dvqa: Understanding data visualizations via question answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656, 2018

    2018

  24. [32]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. S...

    2016

  25. [33]

    Ocr-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. InEuropean Conference on Computer Vision (ECCV), 2022

    2022

  26. [34]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  27. [35]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123:32–73, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer v...

    2017

  28. [36]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  29. [37]

    Multimodal foundation models: From specialists to general-purpose assistants

    Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends® in Computer Graphics and Vision, 16(1-2):1–214, 2024

    2024

  30. [38]

    Omniflow: Any-to-any generation with multi-modal rectified flows

    Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Zichun Liao, Yusuke Kato, Kazuki Kozuka, and Aditya Grover. Omniflow: Any-to-any generation with multi-modal rectified flows. arXiv preprint arXiv:2412.01169, 2024

    2024 arXiv

  31. [39]

    Diffusion-lm improves controllable text generation.Advances in neural information processing systems, 35:4328–4343, 2022

    Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation.Advances in neural information processing systems, 35:4328–4343, 2022

    2022

  32. [40]

    Open-ended long text generation via masked language modeling

    Xiaobo Liang, Zecheng Tang, Juntao Li, and Min Zhang. Open-ended long text generation via masked language modeling. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 223–241, 2023

    2023

  33. [41]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26689–26699, 2024

    2024

  34. [42]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings,...

    2014

  35. [43]

    Llava v1.6 vicuna-7b

    Haotian Liu. Llava v1.6 vicuna-7b. https://huggingface.co/liuhaotian/llava-v1. 6-vicuna-7b, 2023. Accessed: 2025-05-14

    2023

  36. [44]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  37. [45]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  38. [46]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  39. [47]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

    2024

  40. [48]

    Nvila: Efficient frontier visual language models

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04468, 2024

    2024 arXiv

  41. [49]

    Discrete diffusion modeling by estimating the ratios of the data distribution.arXiv preprint arXiv:2310.16834, 2023

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution.arXiv preprint arXiv:2310.16834, 2023

    2023 arXiv

  42. [50]

    Latent diffusion for language generation.Advances in Neural Information Processing Systems, 36:56998–57025, 2023

    Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q Weinberger. Latent diffusion for language generation.Advances in Neural Information Processing Systems, 36:56998–57025, 2023

    2023

  43. [51]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    arXiv

  44. [52]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. InThe 36th Conference on Neural Information Process- ing Systems...

    2022

  45. [53]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, May

    2022

  46. [54]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographicvqa. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022

    2022

  47. [55]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021

    2021

  48. [56]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. InICDAR, 2019

    2019

  49. [57]

    Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    2025 arXiv

  50. [58]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    OpenAI. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  51. [59]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023. 13

    2023 arXiv

  52. [60]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  53. [61]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  54. [62]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  55. [63]

    Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  56. [64]

    Simple and effective masked diffusion language models.Advances in Neural Information Processing Systems, 37:130136–130184, 2024

    Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and V olodymyr Kuleshov. Simple and effective masked diffusion language models.Advances in Neural Information Processing Systems, 37:130136–130184, 2024

    2024

  57. [65]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019

    2019

  58. [66]

    Unified multimodal discrete diffusion.arXiv preprint arXiv:2503.20853, 2025

    Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragkiadaki. Unified multimodal discrete diffusion.arXiv preprint arXiv:2503.20853, 2025

    2025 arXiv

  59. [67]

    Improved artgan for conditional synthesis of natural image and artwork.IEEE Transactions on Image Processing, 28(1):394–409, 2019

    Wei Ren Tan, Chee Seng Chan, Hernan Aguirre, and Kiyoshi Tanaka. Improved artgan for conditional synthesis of natural image and artwork.IEEE Transactions on Image Processing, 28(1):394–409, 2019

    2019

  60. [68]

    Any-to-any generation via composable diffusion.Advances in Neural Information Processing Systems, 36:16083–16099, 2023

    Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion.Advances in Neural Information Processing Systems, 36:16083–16099, 2023

    2023

  61. [69]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  62. [70]

    Open Thoughts

    OpenThoughts Team. Open Thoughts. https://open-thoughts.ai, January 2025

    2025

  63. [71]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  64. [72]

    Q-vlm: Post-training quantization for large vision-language models.arXiv preprint arXiv:2410.08119, 2024

    Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu. Q-vlm: Post-training quantization for large vision-language models.arXiv preprint arXiv:2410.08119, 2024

    2024 arXiv

  65. [73]

    Vl- rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning

    Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vl- rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837, 2025

    2025 arXiv

  66. [74]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

    2024

  67. [75]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 14

    2024 arXiv

  68. [76]

    Demystifying clip data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309.16671, 2023

    2023 arXiv

  69. [77]

    Pvc: Progressive visual token compression for unified image and video processing in large vision-language models.arXiv preprint arXiv:2412.09613, 2024

    Chenyu Yang, Xuan Dong, Xizhou Zhu, Weijie Su, Jiahao Wang, Hao Tian, Zhe Chen, Wenhai Wang, Lewei Lu, and Jifeng Dai. Pvc: Progressive visual token compression for unified image and video processing in large vision-language models.arXiv preprint arXiv:2412.09613, 2024

    2024 arXiv

  70. [78]

    Multimodal large diffusion language models.arXiv preprint arXiv:2505.15809, 2025

    Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Multimodal large diffusion language models.arXiv preprint arXiv:2505.15809, 2025

    2025 arXiv

  71. [79]

    Dream 7b, 2025

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025

    2025

  72. [80]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  73. [81]

    Star: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022

    2022

  74. [82]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023

    2023

  75. [83]

    Lmms-eval: Reality check on the evaluation of large multimodal models, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024

    2024

  76. [84]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?arXiv preprint arXiv:2403.14624, 2024

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?arXiv preprint arXiv:2403.14624, 2024

    2024 arXiv

  77. [85]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  78. [86]

    Masked audio generation using a single non-autoregressive transformer.arXiv preprint arXiv:2401.04577, 2024

    Alon Ziv, Itai Gat, Gael Le Lan, Tal Remez, Felix Kreuk, Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. Masked audio generation using a single non-autoregressive transformer.arXiv preprint arXiv:2401.04577, 2024. 15 A Additional Technical Details A.1 Formulat...

    2024 arXiv

  79. [88]

    MDLM [64] showed that the posterior of the reversal processp(X s|Xt, X0)can be simplified into the following p(X i s|X i t , Xi

    =Cat(X i t ; (1−t)X i 0 +tM)(5) where Cat(.) denotes a categorical distribution and M,X i 0,X i s are probability vectors. MDLM [64] showed that the posterior of the reversal processp(X s|Xt, X0)can be simplified into the following p(X i s|X i t , Xi

  80. [89]

    In this process, we also assume pθ(Xtk−1 |Xtk ) factorize intoQL i=1 pθ(X i tk−1 |Xtk )following previous works [57, 64, 49]

    = Cat(X i s;X i t),ifX i s ̸=M Cat(X i t ; t−s t Xi 0 + s t M),ifX i s =M (6) In practice, we use the categorical distribution induced by the neural network’s prediction pθ(X i 0|Xt) in place ofX i 0 the sample from the reverse process, which gives the following parametrizatio...

  81. [90]

    object_class

    Overall, the conclusion on CoT math reasoning task is similar to that on image captioning task, with the convex schedule performing the best across different choices of sampling steps. 1We will release this data in the camera-ready version. 19 Classify the Image { "object_clas...

  82. [2022]

    Association for Computational Linguistics

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.