Pith. sign in

REVIEW 4 major objections 4 minor 16 cited by

Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces Dimple, the first discrete diffusion multimodal large language model, and shows it outperforms autoregressive LLaVA-NEXT by 3.9% on a 13-benchmark average while decoding in about one-third the iterations.

desk verdict First real DMLLM, but the headline margin over LLaVA-NEXT is confounded by a stronger base LLM and no AR-native control; the training recipe and structure priors are the genuine contributions. read the letter →

arxiv 2505.16990 v2 pith:Z6426YSS submitted 2025-05-22 cs.CV

Runpeng Yu , Xinyin Ma , Xinchao Wang This is my paper · ORCID
classification cs.CV
keywords discretediffusionlanguagemodelmultimodallargeautoregressive-then-diffusiontrainingconfidentdecodingstructurepriorsparallelvision-languagebenchmarkslengthbias
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dimple is the first multimodal large language model whose text decoder is a discrete diffusion model rather than a next-token autoregressive generator. The paper argues that a purely diffusion-trained vision-language model is unstable, underuses the training signal, and suffers from length bias, so it trains Dimple in two stages: first an autoregressive multimodal alignment and instruction-tuning stage, then a diffusion stage that restores parallel decoding. Trained on the same data and pipeline as LLaVA-NEXT, Dimple-7B scores 62.4% average on 13 vision-language benchmarks, 3.9 points higher than LLaVA-NEXT. The same architecture enables confident decoding, which cuts generation iterations to about one-third of response length, plus structure priors that fix tokens in the output to control format and length. The paper's central claim is that discrete diffusion is now a viable alternative backbone for multimodal instruction-following, not just a speed-up trick.

What carries the argument

The central object is the absorbing-state discrete diffusion process: a sequence is initialized as [MASK] tokens and progressively unmasked by a model trained with a reweighted cross-entropy loss on masked positions. The load-bearing mechanism is the two-phase training paradigm: Phase I runs the same autoregressive alignment and instruction tuning as LLaVA-NEXT with causal attention; Phase II switches back to full bidirectional attention and trains with the diffusion loss on the same instruction data. This combination lets the model keep the knowledge acquisition of next-token prediction while recovering parallel decoding. A second mechanism, confident decoding, replaces the fixed number of tokens decoded per MaskGIT step with a confidence threshold: all positions whose maximum softmax probability clears the threshold are decoded at once, with a fallback to one highest-confidence token when none do.

What would settle it

Run the same evaluation protocol with response lengths 4, 8, and 16 on LLaVA-NEXT, take its best score per benchmark, apply the same rescaling to MME and OCRBench, and recompute the 13-benchmark average; if the gap shrinks or reverses, the parity claim is not robust to the selection procedure.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that an absorbing-state discrete diffusion language model can be taught to follow visual instructions as well as an autoregressive MLLM if the training is split into an autoregressive phase and a diffusion phase. The resulting model, Dimple-7B, uses Dream as its diffusion backbone and Qwen2.5-VL's vision encoder, is trained on LLaVA-NEXT's data, and reaches a 62.4% average across 13 benchmarks, exceeding LLaVA-NEXT by 3.9% after rescaling MME and OCRBench. The paper also reports that pure-diffusion training produces severe length bias—accuracy on ChartQA drops from 42.7% to 8.6% as response length grows from 8 to 32—while the hybrid model stays stable. It further claims that confident decoding dynamically updates all positions whose confidence exceeds a threshold, so generation finishes in roughly response-length/3 iterations instead of response-length iterations, and that prefilling adds a 1.5x to 7x speedup with a 0.8% average accuracy drop.

Load-bearing premise

The headline comparison takes Dimple's best result across response lengths 4, 8, and 16 and rescales two benchmarks before averaging, while most baseline numbers come from official reports without the same best-of-N selection; if baselines were tuned and rescored the same way, the reported 3.9% gap could shrink or reverse.

Editorial extensions

If this is right

  • Other multimodal instruction-following models can be built on discrete diffusion backbones without giving up benchmark performance.
  • Generation cost no longer scales one forward pass per token: the model reports roughly one iteration per three response tokens via Confident Decoding.
  • Prefilling recovers most of the speed advantage of autoregressive KV-cache reuse in diffusion decoding, with an average accuracy cost of 0.8%.
  • The autoregressive-then-diffusion schedule removes the length-bias failure mode where pure diffusion accuracy collapses from 42.7% to 8.6% as response length grows.
  • Structure priors let a user fix tokens (e.g., an opening like 'The answer is' or a closing bracket) before generation, enabling controlled JSON-style output and forced stopping positions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 3.9% gap is reported under a best-of-three response-length protocol for Dimple but official-report baselines; re-running baselines under the same protocol would test how much of the margin is architectural versus evaluational.
  • If the two-phase recipe transfers to larger diffusion backbones and more data, the parity result suggests diffusion MLLMs could scale competitively while keeping parallel decoding.
  • Structure priors amount to a native constrained-decoding interface; combining them with schema or grammar constraints could make structured outputs like JSON reliable without external validators.
  • Confident decoding's threshold could be tuned per task or learned rather than fixed, which is a natural next experiment not explored in the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Dimple, a 7B discrete-diffusion multimodal large language model (DMLLM) trained with a two-phase 'autoregressive-then-diffusion' recipe on the LLaVA-NEXT datasets and evaluation pipeline. The central claim is that Dimple-7B outperforms LLaVA-NEXT by 3.9% on a 13-benchmark average, showing that a discrete-diffusion MLLM can match autoregressive MLLMs under a similar training budget. The paper also introduces confident decoding, re-implements prefilling for diffusion decoding, and demonstrates structure-prior control over response format and length.

Significance. If the performance claim holds, the paper opens a credible new direction: a diffusion-based MLLM that reaches autoregressive-level accuracy on standard benchmarks while offering parallel decoding and explicit output control. The authors provide a concrete model, code, model weights, and a detailed training configuration, and their ablation study usefully documents the length-bias problem in pure diffusion training and the benefit of the hybrid AR-then-diffusion recipe. The central comparison, however, is currently under-controlled: the reported margin over LLaVA-NEXT could be explained by the stronger base LLM and by the per-benchmark best-of-three response-length selection, so the paper's headline conclusion needs additional experiments before it can be accepted as stated.

major comments (4)
  1. [Sec. 4.1, Table 1] The comparison to LLaVA-NEXT does not isolate the generation mechanism because Dimple is built on Dream (Qwen2.5) while LLaVA-NEXT uses Vicuna-1.5. The authors explicitly acknowledge this in Sec. 4.1 ('Dimple benefits from a stronger language model compared to LLaVA-NEXT and Eagle'). Since the 3.9% margin is the paper's central evidence that DMLLM can match autoregressive MLLMs, the manuscript needs an autoregressive control with the same Qwen2.5 backbone, trained on the same LLaVA-NEXT data, and evaluated with its native AR decoding; without such a control, the margin cannot be attributed to the diffusion paradigm.
  2. [Sec. S1.3, Table 1] The reported Dimple numbers are selected as the best among response lengths 4, 8, and 16 for each benchmark, while the baseline numbers mostly come from official reports or prior papers without the same best-of-N selection. This asymmetric protocol biases the 3.9% average upward; for example, a baseline evaluated at a single fixed length has no opportunity to benefit from length tuning. The authors should report results at a fixed response length (e.g., 8) for all models, or report the per-length results so the reader can assess the sensitivity.
  3. [Footnote 1, Sec. 4.1] The statement that 'metrics on MME and OCRBench are rescaled when calculating the average score' leaves the exact rescaling unspecified, making the headline 62.4% versus 58.5% average non-reproducible. The authors should give the precise transformation (e.g., min-max or percentage-of-max scaling) and show the per-benchmark normalized scores.
  4. [Sec. 4.2, Table 2] The ablation in Table 2 does not provide the missing autoregressive control: strategies C and D are autoregressively trained, but the text states that 'all models were evaluated using the diffusion generation pipeline.' Consequently, no model in the table is an autoregressive model evaluated with its native decoding, so the table cannot separate the contribution of the diffusion phase from the contribution of the additional training data/epochs or the stronger backbone. The authors should evaluate Strategy D (or an equivalent AR-only model) with its native AR decoding and compare that against Dimple's diffusion decoding.
minor comments (4)
  1. [Abstract and Sec. S1.3] The abstract states that with confident decoding 'the number of iterations needed by Dimple is even only response length/3', but the benchmark evaluation protocol in Sec. S1.3 sets the number of generation steps equal to the response length, and the confident-decoding iteration counts are only demonstrated in the anecdotal examples of Sec. 5; a systematic measurement of iteration counts across benchmarks would substantiate this claim.
  2. [Sec. 2.2] The text says 'with caucal attention'; this should be 'causal attention'.
  3. [Sec. 3.2] The phrase 'To mitigate theses inefficiencies' contains a typo; it should be 'these inefficiencies'.
  4. [Sec. 5, Tables 4-6] The generation-history visualizations are difficult to read in the printed rendering because the per-token iteration indices are dense and the color legend is not reproduced in a grayscale-friendly form; consider using a cleaner layout or a supplementary figure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 3.9% claim is an external benchmark comparison, not a construction-defined result.

full rationale

The derivation chain in Dimple is empirical rather than self-referential. The model is constructed from externally released components (Dream [39], the Qwen2.5-VL vision encoder, and the LLaVA-NEXT training recipe) and then evaluated on 13 external benchmarks; the central claim of a 3.9% improvement over LLaVA-NEXT is a measured comparison, not an equation that reduces to its own inputs. The hybrid autoregressive-then-diffusion training strategy is supported by the ablations in Sec. 4.2 (Table 2), which compare six instruction-tuning strategies, and the length-bias analysis in Figs. 1b and 1c compares diffusion-only and hybrid models on ChartQA and AI2D; these are controlled experiments rather than definitions. The only self-citation involving a Dimple coauthor is MMVet [40], but MMVet is a fixed external evaluation benchmark, not a training target or a fitted quantity, so it is not load-bearing. The response-length selection in S1.3 (best of 4, 8, and 16) and the rescaling of MME and OCRBench are evaluation-protocol choices that could inflate the headline average, and the Qwen2.5-versus-Vicuna backbone difference confounds the comparison to LLaVA-NEXT; however, these are validity or attribution concerns, not circular reduction by construction. No fitted parameter is renamed as a prediction, and no load-bearing conclusion is imported solely from a self-citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central claims rest on the diffusion framework and metrics inherited from prior work, not on new postulated entities. The main 'cost' items are the per-dataset response-length selection and the unspecified decoding threshold, which are fitted or selected rather than derived.

free parameters (4)
  • Response length per benchmark = 4, 8, or 16, best selected per dataset (exact choices not reported)
    Response length is described as the most influential evaluation parameter (Sec. S1.3); Dimple's reported scores are the best of three lengths per dataset, while baselines use official single settings.
  • Confident decoding threshold gamma = not reported
    Gamma determines how many tokens are decoded per step in confident decoding (Alg. 1); the efficiency claim depends on it, but its value is never given.
  • Autoregressive tuning learning rate = 5e-6 (ablation also uses 2e-5)
    Chosen by hand; Table 2 shows performance varies with this learning rate, and it is part of the hybrid recipe.
  • Diffusion tuning learning rate = 5e-7 (ablation uses 5e-6)
    Chosen by hand; Table 2 shows lower diffusion-tune LR improves the final model, so the claim of hybrid-training superiority depends on this choice.
assumptions (4)
  • standard math The absorbing-state discrete diffusion loss (eq. 3) is a valid generative objective and can be optimized by a transformer with full attention.
    The paper builds on Dream and the MDLM-style masked diffusion framework (refs 25, 33, 39); eq. 3 is taken from prior work, not derived here.
  • domain assumption A model fine-tuned autoregressively can later be fine-tuned into a diffusion model without losing multimodal alignment.
    Invoked in Sec. 3.2 with citations [10, 39]; the phase transition from causal to full attention assumes the AR-then-diffusion transfer works for vision-language data.
  • domain assumption The LLaVA-NEXT dataset and training recipe transfer to a diffusion backbone.
    Sec. 3.3 reuses LLaVA-NEXT's 1.3M samples, treating them as sufficient for both alignment and diffusion instruction tuning; no analysis of dataset adequacy is given.
  • domain assumption The selected benchmarks measure capabilities relevant to MLLM performance, and evaluation via lmms-eval is comparable across models.
    Sec. 4.1 and S1.3 mix numbers from official reports and lmms-eval; the comparison assumes these are not materially affected by protocol differences.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding." pith.science (2026). https://pith.science/paper/Z6426YSS

@misc{pith2026250516990,
  author       = {Pith},
  title        = {Pith review of: Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z6426YSS}},
  note         = {Machine review of arXiv:2505.16990}
}
abstract

In this work, we propose Dimple, the first Discrete Diffusion Multimodal Large Language Model (DMLLM). We observe that training with a purely discrete diffusion approach leads to significant training instability, suboptimal performance, and severe length bias issues. To address these challenges, we design a novel training paradigm that combines an initial autoregressive phase with a subsequent diffusion phase. This approach yields the Dimple-7B model, trained on the same dataset and using a similar training pipeline as LLaVA-NEXT. Dimple-7B ultimately surpasses LLaVA-NEXT in performance by 3.9%, demonstrating that DMLLM can achieve performance comparable to that of autoregressive models. To improve inference efficiency, we propose a decoding strategy termed confident decoding, which dynamically adjusts the number of tokens generated at each step, significantly reducing the number of generation iterations. In autoregressive models, the number of forward iterations during generation equals the response length. With confident decoding, however, the number of iterations needed by Dimple is even only $\frac{\text{response length}}{3}$. We also re-implement the prefilling technique in autoregressive models and demonstrate that it does not significantly impact performance on most benchmark evaluations, while offering a speedup of 1.5x to 7x. Additionally, we explore Dimple's capability to precisely control its response using structure priors. These priors enable structured responses in a manner distinct from instruction-based or chain-of-thought prompting, and allow fine-grained control over response format and length, which is difficult to achieve in autoregressive models. Overall, this work validates the feasibility and advantages of DMLLM and enhances its inference efficiency and controllability. Code and models are available at https://github.com/yu-rp/Dimple.

Figures

Figures reproduced from arXiv: 2505.16990 by the authors.

Figure 1
Figure 1. Comparison of Captioning Capabilities After Alignment (Fig. 1a), and the Length Bias [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. An example of Length Bias Phenomenon Collected from ChartQA. The response is [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DF3DV-1K: A Large-Scale Dataset and Benchmark for Distractor-Free Novel View Synthesis

    cs.CV 2026-04 unverdicted novelty 8.0 of 10

    DF3DV-1K supplies 1,048 scenes with clean and cluttered image pairs plus a challenging 41-scene subset to benchmark and improve distractor-free radiance field methods.

  2. CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Credit

    cs.CL 2025-10 unverdicted novelty 7.0 of 10

    CreditDecoding accelerates parallel decoding in diffusion LLMs by fusing accumulated Trace Credit with current logits to accept early-correct tokens sooner, yielding up to 5.48x speedup and accuracy gains.

  3. Rethinking the Generation Order of Block Diffusion Language Models

    cs.CL 2026-07 accept novelty 6.0 of 10

    BDLMs are training-biased toward left-to-right contexts, and PARD exploits that bias with leftmost-prefix parallel unmasking to improve quality–speed trade-offs over existing parallel DLM samplers.

  4. Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A diffusion multimodal LLM's early-layer MLP sparsity at denoising step 0 pinpoints the semantic boundary, enabling one-shot suffix truncation that raises throughput up to ~31x.

  5. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

    cs.CL 2026-07 accept novelty 6.0 of 10

    Joint AR–diffusion training yields one tri-mode LM that switches AR, diffusion, and self-speculation, beating open AR/diffusion models on accuracy and tokens-per-forward.

  6. Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Diffusion-LLM inference can be accelerated with one measured signal — representational drift — that decides which cached positions to refresh and which tokens to commit early, reaching 3.67 tokens per forward pass.

  7. Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion

    cs.CV 2026-03 conditional novelty 6.0 of 10

    A unified mask-based discrete diffusion model jointly models text, speech, and image tokens and matches or exceeds several any-to-any and specialist multimodal baselines.

  8. DODO: Discrete OCR Diffusion Models

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Block-based discrete diffusion can transcribe documents in parallel, roughly matching autoregressive OCR accuracy while cutting inference time by up to about 3x in a lower-accuracy fast variant.

  9. Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Sparse-LaViDa speeds up masked diffusion LLMs by truncating masked tokens, using register tokens and a step-causal attention mask, reaching ~2-2.8x speedup over LaViDa-O with comparable generation quality.

  10. A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Visual token pruning causes severe loss in discrete diffusion MLLMs; only from-scratch models on long-answer tasks recover via late denoising, so redundancy is recoverability, not dispensability.

  11. Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Lavida-O introduces an elastic mixture-of-transformers architecture that brings high-resolution text-to-image generation, object grounding, and image editing into a single masked diffusion model, using planning and se...

  12. Set Block Decoding is a Language Model Inference Accelerator

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Fine-tuning an autoregressive LLM with an extra masked-token loss lets it decode several non-consecutive future tokens per forward pass, giving 3-5x fewer passes at similar accuracy.

  13. DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 7B masked-diffusion code model plus complementary-mask GRPO (coupled-GRPO) improves benchmark scores and shifts decoding away from strict left-to-right order.

  14. TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decoding

    cs.CL 2026-07 conditional novelty 5.5 of 10

    Trajectory-aware commit gating (TILG + History Gate + capped extra promotion) improves or preserves DLLM accuracy while reducing steps and raising tokens-per-forward without retraining.

  15. Adaptive Multi-Step Lookahead Decoding for Diffusion Language Models

    cs.CL 2026-07 conditional novelty 5.0 of 10

    An adaptive lookahead decoder that gates rollout depth by candidate-score variance and re-triggers exploration from informative branches improves accuracy per decoding step over one-step lookahead on masked diffusion ...

  16. A Survey on Diffusion Language Models

    cs.CL 2025-08 unverdicted novelty 3.0 of 10

    A comprehensive survey of diffusion language models covering taxonomy, training and inference techniques, and comparisons with autoregressive models.

Reference graph

Works this paper leans on

44 extracted references · 22 canonical work pages · cited by 16 Pith papers

  1. [1]

    Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg

    Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces, 2023

    2023

  2. [2]

    Leo: Boosting mixture of vision encoders for multimodal large language models, 2025

    Mozhgan Nasr Azadani, James Riddell, Sean Sedwards, and Krzysztof Czarnecki. Leo: Boosting mixture of vision encoders for multimodal large language models, 2025

    work page 2025

  3. [3]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  4. [4]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025

    2025

  5. [5]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer, 2022

    work page 2022

  6. [6]

    Eagle 2.5: Boosting long-context post-training for frontier vision-language models, 2025

    Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, Tyler Poon, Max Ehrlich, Tuomas Rintamaki, Tyler Poon, Tong Lu, Limin Wang, Bryan Catanzaro, Jan Kautz, Andrew Tao, Zhiding Yu, and Guilin Liu. Eagle 2.5: Boosting long-context post-training for frontier vision-language models, 2025

    work page 2025

  7. [7]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu...

    2024

  8. [8]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, 2024

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, 2024

    2024

Show all 44 references
  1. [9]

    MME: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306.13394, 2023

    2023 arXiv

  2. [10]

    Scaling diffusion language models via adaptation from autoregressive models, 2025

    Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models, 2025

    2025

  3. [11]

    Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control, 2023

    Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control, 2023

    2023

  4. [12]

    Argmax flows and multinomial diffusion: Learning categorical distributions, 2021

    Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions, 2021

    2021

  5. [13]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  6. [14]

    A diagram is worth a dozen images, 2016

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images, 2016

    2016

  7. [15]

    Llava-next: Stronger llms supercharge multimodal capabilities in the wild, 2024

    Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capabilities in the wild, 2024

    2024

  8. [16]

    Lmms-eval: Accelerating the development of large multimoal models, 2024

    Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, 2024

    2024

  9. [17]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. 19

    2023

  10. [18]

    Diffusion models for non-autoregressive text generation: A survey, 2023

    Yifan Li, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. Diffusion models for non-autoregressive text generation: A survey, 2023

    2023

  11. [19]

    Alvarez, Bryan Catanzaro, Jan Kautz, Andrew Tao, Guilin Liu, and Zhiding Yu

    Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, Nadine Chang, Karan Sapra, Amala Sanjay Deshmukh, Tuomas Rintamaki, Matthieu Le, Ilia Karmanov, Lukas V oegtle, Philipp Fischer, De-An Huang, Timo ...

    2025

  12. [20]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  13. [21]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Conference on Neural Information Processing Systems (NeurlPS) , 2023

    2023

  14. [22]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  15. [23]

    Mmbench: Is your multi-modal model an all-around player? CoRR, abs/2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? CoRR, abs/2307.06281, 2023

    2023 arXiv

  16. [24]

    Ocrbench: on the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences , 67(12), 2024

    2024

  17. [25]

    Discrete diffusion modeling by estimating the ratios of the data distribution, 2024

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution, 2024

    2024

  18. [26]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  19. [27]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representat...

    2024

  20. [28]

    Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training, 2025

    Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training, 2025

    2025

  21. [29]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022

    2022

  22. [30]

    Fnu Mohbat and Mohammed J. Zaki. Llava-chef: A multi-modal generative model for food recipes, 2024

    2024

  23. [31]

    Large language diffusion models, 2025

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models, 2025

    2025

  24. [33]

    Simple and effective masked diffusion language models, 2024

    Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and V olodymyr Kuleshov. Simple and effective masked diffusion language models, 2024

    2024

  25. [34]

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data, 2025

    2025

  26. [35]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), pages 8317–8326, 2019

    2019

  27. [36]

    Self-conditioned embedding diffusion for text generation, 2022

    Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, and Rémi Leblond. Self-conditioned embedding diffusion for text generation, 2022

    2022

  28. [37]

    Clip-diffusion-lm: Apply diffusion model on image captioning, 2022

    Shitong Xu. Clip-diffusion-lm: Apply diffusion model on image captioning, 2022

    2022

  29. [38]

    Beyond autoregression: Discrete diffusion for complex reasoning and planning, 2025

    Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning, 2025. 20

    2025

  30. [39]

    Dream 7b, 2025

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025

    2025

  31. [40]

    Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023

    2023

  32. [41]

    Seqdiffuseq: Text diffusion with encoder-decoder transformers, 2023

    Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. Seqdiffuseq: Text diffusion with encoder-decoder transformers, 2023

    2023

  33. [42]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2023 arXiv

  34. [43]

    Joint vision-language social bias removal for clip, 2024

    Haoyu Zhang, Yangyang Guo, and Mohan Kankanhalli. Joint vision-language social bias removal for clip, 2024

    2024

  35. [44]

    Lmms-eval: Reality check on the evaluation of large multimodal models, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024

    2024

  36. [45]

    A reparameterized discrete diffusion model for text generation, 2024

    Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation, 2024. 21

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.