Pith. sign in

REVIEW 3 major objections 6 minor 5 cited by

Sparse-LaViDa claims that masked discrete diffusion models can be accelerated about 2x by truncating masked tokens at each sampling step, using 64 register tokens to stand in for dropped positions and a step-causal attention mask to keep tr

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 16:16 UTC pith:EXSBQNCL

load-bearing objection Solid efficiency work on masked diffusion acceleration, but the 'equivalent parameterization' claim is undercut by the authors' own no-training ablation. the 3 major comments →

arxiv 2512.14008 v2 pith:EXSBQNCL submitted 2025-12-16 cs.CV

Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models

Shufan Li , Jiuxiang Gu , Kangning Liu , Zhe Lin , Zijun Wei , Aditya Grover , Jason Kuen This is my paper
classification cs.CV
keywords masked discrete diffusion modelssparse parameterizationregister tokensstep-causal attention maskKV cachetext-to-image generationimage editingmultimodal understanding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that the main computational waste of masked discrete diffusion models—reprocessing the same masked tokens at every sampling step—can be removed without losing the benefits that make these models attractive. The proposed Sparse-LaViDa keeps only the prompt, the tokens already decoded, and the small subset of masked tokens being decoded at the current step; 64 register tokens stand in for all truncated masked positions, and a step-causal attention mask lets training mimic the truncated, KV-cached inference procedure. On top of an existing unified multimodal diffusion model, this yields roughly 2x lower end-to-end latency on text-to-image generation, image editing, and visual math reasoning while matching or slightly improving quality metrics. If the central premise holds—that masked tokens carry no information beyond marking their positions—then this is a general way to make MDM sampling sparse, parallel, and bidirectional at once. The paper itself notes that the speedup is confined to long generations and that short-answer vision-language tasks see little benefit.

Core claim

The paper claims that masked discrete diffusion sampling can be made sparse without changing the underlying diffusion objective. Since a masked token only signals 'this position is not yet revealed,' a partially masked sequence can be written as the clean tokens with their positional embeddings plus the total sequence length, which implicitly fixes all masked locations. Truncating masked tokens outright hurts image quality, so Sparse-LaViDa appends 64 register tokens—special vocabulary entries at fixed positions after the sequence—as compact learned surrogates for the removed masked tokens. The model is fine-tuned with a step-causal attention mask that mirrors the KV-cached inference procedu

What carries the argument

Three components carry the argument. (1) Sparse parameterization: at each sampling step only the prompt, cached decoded tokens, and the subset of masked tokens to decode are fed to the network; other masked positions are dropped. (2) Register tokens: 64 special tokens with distinct positional embeddings appended after the sequence serve as compressed representations of the truncated masked tokens; they attend to everything, while decode tokens and other registers attend to them. (3) Step-causal attention mask: during training the sequence is partitioned into prompt/clean/masked blocks with block-wise attention rules that simulate arbitrary KV-cache sampling paths in a single pass, closing th

Load-bearing premise

The load-bearing premise, stated in Sec. 3.1, is that masked tokens carry no information beyond marking a position as masked, so 64 fixed register tokens can stand in for any number of truncated masked tokens without losing needed context; the paper's own ablations show this premise only holds after fine-tuning (without it, GenEval collapses from 0.77 to 0.24) and that understanding quality still drops on MME (488 to 450) after fine-tuning.

What would settle it

The paper's Table 8 nearly falsifies the central premise at initialization: running the sparse pipeline on the pretrained dense base model without fine-tuning collapses GenEval from 0.77 to 0.24, so 'masked tokens carry no substantive information' is only true after step-causal fine-tuning re-learns the representation. A decisive experiment would be to take the fine-tuned Sparse-LaViDa and, on a fixed prompt set, swap the 64 register tokens for the actual truncated masked tokens at a mid-sampling step; if outputs change measurably (e.g., GenEval or DPG shifts beyond noise), then registers are

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Latency for text-to-image generation drops from 21.27s to 10.86s (1.95x), image editing from 63.98s to 22.55s (2.83x), and MathVista reasoning from 10.41s to 3.72s (2.80x) on a single A100.
  • Generation quality is preserved or slightly improved: GenEval 0.78 vs 0.77, DPG 82.4 vs 81.8, and ImgEdit 3.79 vs 3.71.
  • Because decoding order is no longer constrained to left-to-right, the model keeps bidirectional context and can still perform inpainting/outpainting, parallel grounding, and constrained captioning, unlike block-causal diffusion approaches.
  • The method is a generalization of the standard MDM: if no masked tokens are truncated, Sparse-LaViDa reduces to the dense parameterization.
  • Speedup applies mainly to long generations such as 1024-token reasoning or 4096-token images; short-answer tasks gain little because their output fits in a single block.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper only fine-tunes an existing dense model, but if the sparse parameterization holds when trained from scratch, it would likely also reduce pretraining cost, not just inference cost.
  • Register tokens can be read as a learned summary of the context that truncated masked tokens would have supplied; a natural extension is to let the number or placement of registers adapt per step or per task rather than fixing 64 at the end of the sequence.
  • Because decoding order is arbitrary, the approach could combine with confidence-based dynamic unmasking for text generation, potentially removing the block-size constraint of semi-autoregressive sampling.
  • A testable extension is to apply the same truncation idea to video or higher-resolution images, where per-frame masked-token overhead is even larger.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. Masked diffusion models (MDMs) must process all masked tokens at every sampling step, limiting inference speed. Sparse-LaViDa addresses this by truncating masked tokens, introducing 64 register tokens as compressed placeholders, and training with a step-causal attention mask that enables KV caching while preserving bidirectional context and arbitrary unmasking order. Built on the 10.4B LaViDa-O, the method reports end-to-end speedups of 1.95x on text-to-image generation (21.27s→10.86s), 2.83x on image editing (63.98s→22.55s), and 2.80x on MathVista (10.41s→3.72s), with maintained/slightly improved generation quality on GenEval, DPG, MJHQ-30k, and ImgEdit. The authors also report competitive results on image understanding, object grounding, and bidirectional-context tasks. The method requires supervised fine-tuning (100k steps, 64 H100 GPUs) and is positioned as a post-training acceleration of LaViDa-O.

Significance. If the reported results hold, the paper makes a practically significant contribution: it demonstrates that MDM sampling can be accelerated by truncating masked tokens without imposing a left-to-right block-causal order, thus retaining the bidirectional-context advantages that are essential for image generation, editing, and infilling. The speedups are large (1.95–2.84x), measured end-to-end, and the generation-quality evidence is reasonably strong: multiple benchmarks, multiple metrics, and a same-data comparison (LaViDa-O* FID 8.11 vs. Sparse-LaViDa 7.63) support the empirical claim. The ablations in Tables 6–8 are well designed and isolate the contributions of caching, truncation, registers, and the step-causal mask. The main weaknesses are overstated equivalence and unaddressed understanding-quality regressions, which are fixable in revision.

major comments (3)
  1. [Sec. 3.1, Sec. 2.2, Table 8] The paper's central premise is that masked tokens 'carry no substantive information beyond indicating that a position was masked,' and that Sparse-LaViDa is 'an alternative parameterization' that preserves MDM properties 'without any quality compromise or training–inference gap.' The 'No Training' row in Table 8 contradicts this: applying the sparse pipeline to pretrained LaViDa-O without SFT collapses GenEval from 0.77 to 0.24 and DPG from 81.8 to 47.9. An information-equivalent reparameterization would not require fine-tuning to avoid catastrophic degradation. The recovery after SFT shows learned adaptation, not equivalence. Sec. 7.2 only proves reduction to the dense case when C=B, not equivalence for C⊂B. Please provide a formal equivalence condition or reframe the method as a trained sparse approximation.
  2. [Table 5, Sec. 4.4/5] The claim of maintaining quality 'without any quality compromise' is not supported by the understanding benchmarks. Relative to LaViDa-O, Sparse-LaViDa drops on MME (488→450), MMMU (45.1→43.6), and MMB (76.4→75.0), while improving ChartQA, DocVQA, and MathVerse. The text calls these results 'competitive' but does not discuss the systematic drop on three general-purpose suites. Since Sec. 4.4 and the Limitations acknowledge that speedups are minimal for short QA outputs, this regression is not offset by a latency gain in those settings. Please provide error bars/significance, explain the likely cause, or restrict the 'no quality compromise' claim to the text-to-image and editing tasks for which the data are positive.
  3. [Sec. 3.3, Algorithm 1] The step-causal mask is stated to 'faithfully match' the truncated sampling procedure, but the match is not demonstrated. During inference, the partition C_k is determined by confidence-based or stratified-random unmasking that depends on the model's current predictions (Sec. 3.2); in training, clean and masked tokens are randomly assigned to M and N blocks (Sec. 3.3). These are different stochastic processes, and the hyperparameters M,N are not justified or ablated. The 'No Step-Causal Attention Mask' and 'No Training' rows in Table 8 show that both components are needed to recover quality, but they do not establish that the random block assignment covers the inference-time distribution. A distributional argument or an ablation over M,N and masking schedules would substantiate the faithfulness claim.
minor comments (6)
  1. [Abstract/Intro] The abstract reports 'up to a 2× speedup' while Tables 3 and 4 show 2.83× and 2.80×; use consistent speedup numbers.
  2. [Throughout] Typos: 'Quantative' (Table 4 caption), 'key deigns' (Table 6 caption), 'beahvior' (Sec. 7.2), 'Future works' (Sec. 5).
  3. [Sec. 8.1] The data-pipeline section cites [20] for UniWorld-V1; the reference list suggests [34] is the correct citation. Please verify.
  4. [Tables 1/3/4] Please report exact inference configurations (number of diffusion steps, block size S, truncation criterion, batch size, GPU details) so the latency comparisons can be reproduced.
  5. [Table 5] The understanding benchmark table does not report latency; adding a latency column or an explicit 'no speedup' note would make the trade-off transparent.
  6. [Reproducibility] No code or model release is mentioned; making the inference code available would materially aid reproducibility of a systems-oriented paper.

Circularity Check

0 steps flagged

No significant circularity: the speedup is measured latency, quality is evaluated on external benchmarks, and no fitted quantity is renamed as a prediction.

full rationale

The paper's load-bearing claims do not reduce to their own inputs. The speedup (Sec. 4.2-4.4, Tables 1, 3, 4) is directly measured end-to-end latency against a fixed baseline, not a number predicted from the sparse parameterization. The quality claims are checked on external benchmarks (GenEval, DPG, MJHQ, ImgEdit, MME, MathVista, etc.) and against external models, so no fitted parameter is passed off as a prediction. The register count (64) and the step-causal mask are selected and validated by ablations (Tables 7-8), not derived from a self-cited uniqueness theorem or borrowed ansatz. Citations to LaViDa-O [32] supply the base model and a baseline, but the sparse representation and its training are the paper's own construction. The paper even reports evidence against its ideal equivalence claim—Table 8 shows GenEval collapsing from 0.77 to 0.24 without fine-tuning, and Table 5 shows understanding drops (MME 488->450)—which is a limitation of the method's assumptions, not a circular derivation. Section 9 explicitly acknowledges that speedups do not apply to short QA/grounding tasks. Thus there is no step where an output is equivalent by definition to an input; at most there is self-benchmarking against the authors' own baseline, which is not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 1 invented entities

The central claim rests on the assumption that masked tokens are compressible and that registers recover lost capacity. The ledger shows one fitted hyperparameter (64 registers), two unreported/choice hyperparameters, and the step-causal training assumption, which is empirically validated only through ablations.

free parameters (3)
  • register token count (m) = 64
    Chosen based on ablations in Table 7 (0/1/32/64); affects FID and DPG. No theoretical principle fixes the count.
  • clean/masked block counts (M, N) in step-causal mask = not reported
    Algorithm 1 and Sec. 3.3 require M,N as inputs; they control how many cache steps and masked blocks are simulated per training step. The paper never states the values used, despite their effect on training–inference fidelity.
  • text generation block size S = 32
    Used for semi-AR sampling in MathVista (Sec 4.4); no ablation justifying the choice for the speed/quality tradeoff.
axioms (5)
  • standard math pθ(X0|Xt) factorizes as ∏_i pθ(X_i0|Xt)
    Standard MDM independence assumption from prior work (Sahoo et al.); used to justify predicting only a subset of positions (Sec 3.1).
  • ad hoc to paper Masked tokens carry no substantive information beyond indicating a position is masked, so they can be compressed into register tokens.
    Core of the truncation idea (Sec 3.1); not proven, and the need for 64 registers and SFT suggests capacity loss.
  • ad hoc to paper Random block assignment during training covers the inference-time unmasking paths.
    Sec 3.3 assigns clean/masked tokens random blocks; no proof that this approximates arbitrary confidence-based or pre-generated decode orders.
  • domain assumption Pretrained LaViDa-O weights provide a sufficient initialization for SFT to the sparse parameterization.
    All results use SFT from LaViDa-O; the 'No Training' ablation (Table 8) shows the sparse inference pipeline fails without fine-tuning, so the method's success depends on this base model.
  • domain assumption Latency measurements are representative of end-to-end inference under identical hardware/settings.
    Tables report single A100 latencies for base and Sparse, but no code/scripts are provided; speedup could be implementation-sensitive.
invented entities (1)
  • register token ([reg]) with 64 positional instantiations no independent evidence
    purpose: Compressed representation of truncated masked tokens during sparse inference; attends to all tokens but is mainly attended to by current decode tokens and other registers.
    No external falsifiable prediction; only internal ablations (Table 7) show it helps. The paper provides no theoretical reason why 64 position-fixed tokens suffice for arbitrary truncation patterns.

pith-pipeline@v1.3.0-alltime-deepseek · 22083 in / 15695 out tokens · 124081 ms · 2026-08-03T16:16:00.991109+00:00 · methodology

0 comments
read the original abstract

Masked Discrete Diffusion Models (MDMs) have achieved strong performance across a wide range of multimodal tasks, including image understanding, generation, and editing. However, their inference speed remains suboptimal due to the need to repeatedly process redundant masked tokens at every sampling step. In this work, we propose Sparse-LaViDa, a novel modeling framework that dynamically truncates unnecessary masked tokens at each inference step to accelerate MDM sampling. To preserve generation quality, we introduce specialized register tokens that serve as compact representations for the truncated tokens. Furthermore, to ensure consistency between training and inference, we design a specialized attention mask that faithfully matches the truncated sampling procedure during training. Built upon the state-of-the-art unified MDM LaViDa-O, Sparse-LaViDa achieves up to a 2x speedup across diverse tasks including text-to-image generation, image editing, and mathematical reasoning, while maintaining generation quality.

Figures

Figures reproduced from arXiv: 2512.14008 by Aditya Grover, Jason Kuen, Jiuxiang Gu, Kangning Liu, Shufan Li, Zhe Lin, Zijun Wei.

Figure 1
Figure 1. Figure 1: We propose Sparse-LaViDa, a novel modeling tech￾nique for unified multimodal masked discrete diffusion models. Sparse-LaViDa achieves substantial speedup across a wide range of tasks, including text-to-image generation, image editing, and visual math reasoning, compared with the baseline LaViDa-O. with continuous diffusion models to handle understanding and generation tasks respectively. More recently, Mas… view at source ↗
Figure 2
Figure 2. Figure 2: Overall design of Sparse-LaViDa. Left: Vanilla MDMs materialize all masked tokens and support arbitrary-order decoding (top-down). Unlike AR models, they have bidirectional context and naturally support tasks such as image generation and inpainting. Middle: Block Diffusion truncates redundant masked tokens from the right but imposes a left-to-right generation order using a block￾causal attention mask, losi… view at source ↗
Figure 3
Figure 3. Figure 3: Inference pipeline of Sparse-LaViDa. (a) At a specific decoding step (top-down), the input of Sparse-LaViDa consists of four types of tokens: (1) previously decoded tokens stored in the KV cache; (2) newly decoded tokens from the previous step, which will be added to the cache at this step; (3) masked tokens to be decoded at the current step; and (4) register tokens. (b) During sampling, we apply a special… view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of Sparse Representation of Masked Se￾quence. Instead of materializing all masked tokens, a partially masked sequence can be uniquely represented by non-mask to￾kens, their locations, and the number of total tokens in the original sequence. tokens. However, similar to the previous category, they are mostly designed for language modeling and assume a left￾to-right semi-AR decoding order. Furthe… view at source ↗
Figure 5
Figure 5. Figure 5: Step-Causal Mask. We employ a step-causal attention mask during training to match the inference behavior of Sparse￾LaViDa. Consider a sequence containing prompt tokens P0–P3 and answer tokens X0–X15, where some tokens are clean and others are masked (color-coded in blue and gray). During infer￾ence, prompt tokens and clean tokens are sequentially added to the KV cache. To simulate this behavior during trai… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative results. Unlike semi-AR approaches like Block Diffusion, Sparse-LaViDa supports tasks requiring bidirec￾tional context, such as inpainting/outpainting, parallel grounding, and constrained captioning. In text generation examples, colored regions denote masked tokens initialized for infilling [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Sparse-Representation of Partially Masked Sequence. Given a partially masked sequence, we can partition its tokens into two subsets A and B, where A consists of clean tokens and B consists of masked tokens. The standard MDM (Left) need to pass all |A| + |B| tokens. By contrast, Sparse-LaViDa (Middle and Right) only need to pass |A| + |C| + m tokens, where C is a subset of B and m is the number of register … view at source ↗
Figure 8
Figure 8. Figure 8: Connection between the Diffusion Sampling Process and the Step-Causal-Attention-Mask. (Left) Given prompt tokens P0...P2 and response tokens X0...X5, we partition the response tokens with C1...C4 and assign prompt tokens to C0. Both 0-1-2-3-4 and 0-1-2-4-3 are viable sampling order of tokens in a 4-step diffusion process. We use M to represent mask tokens and R represent registers. (Right), for the two sam… view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative Comparisons of Sparse-LaViDa and LaViDa-O baseline. We show qualitative results of text-to-image generation and image editing results of Sparse-LaViDa and LaViDa-O baseline.Sparse-LaViDa achieves a speedup of 1.95× on text-to-image gener￾ation and a speedup of 2.83× on image editing, while maintaining comparable visual quality 6 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding

    cs.LG 2026-07 unverdicted novelty 7.0

    Set diffusion factorizes likelihood over arbitrary token sets and uses a set-causal diffusion architecture to support KV caching and any-order decoding, yielding improved speed-quality tradeoffs versus prior diffusion LMs.

  2. Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation

    cs.AI 2026-07 conditional novelty 6.0

    A diffusion multimodal LLM's early-layer MLP sparsity at denoising step 0 pinpoints the semantic boundary, enabling one-shot suffix truncation that raises throughput up to ~31x.

  3. Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis

    cs.CV 2026-06 unverdicted novelty 6.0

    A masked discrete diffusion model adds token editing at inference and grouped cross-entropy training to reach 0.90 GenEval, 86.9 DPG, and 10.76 HPSv3 scores.

  4. Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis

    cs.CV 2026-06 conditional novelty 6.0

    Masked discrete diffusion with token editing and grouped cross-entropy reaches strong text-to-image generation scores in an 8B decoder-only model, reporting GenEval 0.90, DPG 86.9, HPSv3 10.76.

  5. Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Models

    cs.LG 2026-05 unverdicted novelty 6.0

    VRCD prioritizes visually complementary positions during parallel decoding in dMLLMs by measuring attention overlap with the new Visual Redundancy Index, yielding accuracy gains over confidence-based baselines on M^3C...

Reference graph

Works this paper leans on

82 extracted references · 37 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Block diffusion: Interpolating be- tween autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573, 2025

    Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and V olodymyr Kuleshov. Block diffusion: Interpolating be- tween autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573, 2025. 2, 3

    Pith/arXiv arXiv 2025

  2. [2]

    Structured denoising dif- fusion models in discrete state-spaces.Advances in neural information processing systems, 34:17981–17993, 2021

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tar- low, and Rianne Van Den Berg. Structured denoising dif- fusion models in discrete state-spaces.Advances in neural information processing systems, 34:17981–17993, 2021. 3

    2021

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 5

    Pith/arXiv arXiv 2025

  4. [4]

    Halton scheduler for masked gen- erative image transformer.arXiv preprint arXiv:2503.17076,

    Victor Besnier, Mickael Chen, David Hurych, Eduardo Valle, and Matthieu Cord. Halton scheduler for masked gen- erative image transformer.arXiv preprint arXiv:2503.17076,

    Pith/arXiv arXiv

  5. [5]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 18392–18402, 2023. 7

    2023

  6. [6]

    Coyo-700m: Image-text pair dataset.https : / / github

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset.https : / / github . com / kakaobrain/coyo-dataset, 2022. 6, 4

    2022

  7. [7]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 11315–11325, 2022. 2

    2022

  8. [8]

    Muse: Text-to-image generation via masked generative transform- ers.arXiv preprint arXiv:2301.00704, 2023

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Mur- phy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transform- ers.arXiv preprint arXiv:2301.00704, 2023. 2

    Pith/arXiv arXiv 2023

  9. [9]

    Sharegpt-4o-image: Aligning multimodal mod- els with gpt-4o-level image generation.arXiv preprint arXiv:2506.18095, 2025

    Junying Chen, Zhenyang Cai, Pengcheng Chen, Shunian Chen, Ke Ji, Xidong Wang, Yunjin Yang, and Benyou Wang. Sharegpt-4o-image: Aligning multimodal mod- els with gpt-4o-level image generation.arXiv preprint arXiv:2506.18095, 2025. 6, 4

    Pith/arXiv arXiv 2025

  10. [10]

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Sil- vio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025. 6, 4

    Pith/arXiv arXiv 2025

  11. [11]

    Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation.arXiv preprint arXiv:2510.06303, 2025

    Shuang Cheng, Yihan Bian, Dawei Liu, Yuhua Jiang, Yihao Liu, Linfeng Zhang, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, et al. Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation.arXiv preprint arXiv:2510.06303, 2025. 3

    arXiv 2025

  12. [12]

    Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025. 1, 6, 7

    Pith/arXiv arXiv 2025

  13. [13]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. InProceedings of the 2019 conference of the North American chapter of the asso- ciation for computational linguistics: human language tech- nologies, volume 1 (long and short papers), pages 4171– 4186, 2019. 2

    2019

  14. [14]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 2

    2021

  15. [15]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. InForty-first international conference on machine learning,

  16. [16]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023. 6, 7

    Pith/arXiv arXiv 2023

  17. [17]

    Geneval: An object-focused framework for evaluating text- to-image alignment.Advances in Neural Information Pro- cessing Systems, 36:52132–52152, 2023

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text- to-image alignment.Advances in Neural Information Pro- cessing Systems, 36:52132–52152, 2023. 6

    2023

  18. [18]

    Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale.arXiv preprint arXiv:2412.05237,

    Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xi- ang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale.arXiv preprint arXiv:2412.05237,

    Pith/arXiv arXiv

  19. [19]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000– 16009, 2022. 2

    2022

  20. [20]

    Unified discrete diffusion for si- multaneous vision-language generation.arXiv, 2022

    Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang, Zuopeng Yang, Dacheng Tao, and Ponnuthurai N Suganthan. Unified discrete diffusion for si- multaneous vision-language generation.arXiv, 2022. 4

    2022

  21. [21]

    Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 5(7): 16, 2024

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu Ella. Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 5(7): 16, 2024. 6

    Pith/arXiv arXiv 2024

  22. [22]

    Visualwebinstruct: Scaling up mul- timodal instruction data through web search.arXiv preprint arXiv:2503.10582, 2025

    Yiming Jia, Jiachen Li, Xiang Yue, Bo Li, Ping Nie, Kai Zou, and Wenhu Chen. Visualwebinstruct: Scaling up mul- timodal instruction data through web search.arXiv preprint arXiv:2503.10582, 2025. 6, 4

    Pith/arXiv arXiv 2025

  23. [23]

    Referitgame: Referring to objects in pho- 9 tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- 9 tographs of natural scenes. InProceedings of the 2014 con- ference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014. 4

    2014

  24. [24]

    Mer- cury: Ultra-fast language models based on diffusion.arXiv preprint arXiv:2506.17298, 2025

    Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Ya- nis Miraoui, Akash Palrecha, Stefano Ermon, et al. Mer- cury: Ultra-fast language models based on diffusion.arXiv preprint arXiv:2506.17298, 2025. 3

    Pith/arXiv arXiv 2025

  25. [25]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InProceedings of the IEEE/CVF international confer- ence on computer vision, pages 4015–4026, 2023. 6, 4

    2023

  26. [26]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation.Ad- vances in neural information processing systems, 36:36652– 36663, 2023

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Ma- tiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation.Ad- vances in neural information processing systems, 36:36652– 36663, 2023. 7

    2023

  27. [27]

    Flux.https://github.com/ black-forest-labs/flux, 2024

    Black Forest Labs. Flux.https://github.com/ black-forest-labs/flux, 2024. 6

    2024

  28. [28]

    Flux.1 kontext: Flow matching for in-context image generation and editing in latent space,

    Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dock- horn, Jack English, Zion English, Patrick Esser, Sumith Ku- lal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas M¨uller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. Flux.1 kontext: Flow matching for in-context i...

  29. [29]

    Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing

    Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing. InAdvances in Neural Information Processing Systems, pages 30146–30166. Curran Associates, Inc., 2023. 4

    2023

  30. [30]

    Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image genera- tion, 2024

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Lin- miao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image genera- tion, 2024. 6

    2024

  31. [31]

    Instruc- tany2pix: Flexible visual editing via multimodal instruction following.arXiv preprint arXiv:2312.06738, 2023

    Shufan Li, Harkanwar Singh, and Aditya Grover. Instruc- tany2pix: Flexible visual editing via multimodal instruction following.arXiv preprint arXiv:2312.06738, 2023. 7

    Pith/arXiv arXiv 2023

  32. [32]

    Lavida-o: Elastic masked diffusion models for unified multimodal understanding and generation.arXiv preprint arXiv:2509.19244, 2025

    Shufan Li, Jiuxiang Gu, Kangning Liu, Zhe Lin, Zijun Wei, Aditya Grover, and Jason Kuen. Lavida-o: Elastic masked diffusion models for unified multimodal understanding and generation.arXiv preprint arXiv:2509.19244, 2025. 1, 2, 3, 4, 6, 7, 8

    Pith/arXiv arXiv 2025

  33. [33]

    Lavida: A large diffu- sion language model for multimodal understanding.arXiv preprint arXiv:2505.16839, 2025

    Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai-Wei Chang, and Aditya Grover. Lavida: A large diffu- sion language model for multimodal understanding.arXiv preprint arXiv:2505.16839, 2025. 1, 3

    Pith/arXiv arXiv 2025

  34. [34]

    Uniworld: High-resolution semantic en- coders for unified visual understanding and generation.arXiv preprint arXiv:2506.03147, 2025

    Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, et al. Uniworld: High-resolution semantic en- coders for unified visual understanding and generation.arXiv preprint arXiv:2506.03147, 2025. 7

    Pith/arXiv arXiv 2025

  35. [35]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuro- pean conference on computer vision, pages 38–55. Springer,

  36. [36]

    Step1x-edit: A practical framework for general image editing.arXiv preprint arXiv:2504.17761, 2025

    Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chun- rui Han, et al. Step1x-edit: A practical framework for general image editing.arXiv preprint arXiv:2504.17761, 2025. 7

    Pith/arXiv arXiv 2025

  37. [37]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vi- sion, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vi- sion, pages 216–233. Springer, 2024. 6

    2024

  38. [38]

    Discrete diffusion modeling by estimating the ratios of the data distri- bution.arXiv preprint arXiv:2310.16834, 2023

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distri- bution.arXiv preprint arXiv:2310.16834, 2023. 3, 1

    Pith/arXiv arXiv 2023

  39. [39]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 6

    Pith/arXiv arXiv 2023

  40. [40]

    Unitok: a unified tokenizer for visual generation and understanding

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Ze- huan Yuan, BINGYUE PENG, and XIAOJUAN QI. Unitok: a unified tokenizer for visual generation and understanding. InThe Thirty-ninth Annual Conference on Neural Informa- tion Processing Systems, 2025. 1

    2025

  41. [41]

    dkv-cache: The cache for diffusion language models.arXiv preprint arXiv:2505.15781, 2025

    Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models.arXiv preprint arXiv:2505.15781, 2025. 3

    Pith/arXiv arXiv 2025

  42. [42]

    Hpsv3: Towards wide-spectrum human preference score

    Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15086–15095, 2025. 7

    2025

  43. [43]

    ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Ena- mul Hoque. ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning. InFind- ings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, 2022. Association for Computational Linguistics. 6, 7

    2022

  44. [44]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 6, 7

    2021

  45. [45]

    Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025. 3, 1

    Pith/arXiv arXiv 2025

  46. [46]

    Dall·e 3.https://openai.com/index/ dall-e-3/, 2023

    OpenAI. Dall·e 3.https://openai.com/index/ dall-e-3/, 2023. 6

    2023

  47. [47]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    OpenAI. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 7

    Pith/arXiv arXiv 2024

  48. [48]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023. 6

    Pith/arXiv arXiv 2023

  49. [49]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, 10 Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021. 4

    2021

  50. [50]

    Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S. Khan. Glamm: Pixel grounding large multimodal model.The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 4

    2024

  51. [51]

    Simple and effective masked dif- fusion language models.Advances in Neural Information Processing Systems, 37:130136–130184, 2024

    Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and V olodymyr Kuleshov. Simple and effective masked dif- fusion language models.Advances in Neural Information Processing Systems, 37:130136–130184, 2024. 3, 1

    2024

  52. [52]

    Laion-aesthetics.https : / / laion.ai/blog/laion- aesthetics/, 2022

    Christoph Schuhmann. Laion-aesthetics.https : / / laion.ai/blog/laion- aesthetics/, 2022. Ac- cessed: 2024 - 03 - 06. 4

    2022

  53. [53]

    Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in neural in- formation processing systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in neural in- formation processing systems, 35:25278–25294, 2022. 6, 4

    2022

  54. [54]

    Muddit: Liberating gener- ation beyond text-to-image with a unified discrete diffusion model.arXiv preprint arXiv:2505.23606, 2025

    Qingyu Shi, Jinbin Bai, Zhuoran Zhao, Wenhao Chai, Kaidong Yu, Jianzong Wu, Shuangyong Song, Yunhai Tong, Xiangtai Li, Xuelong Li, et al. Muddit: Liberating gener- ation beyond text-to-image with a unified discrete diffusion model.arXiv preprint arXiv:2505.23606, 2025. 1, 3

    Pith/arXiv arXiv 2025

  55. [55]

    Sparse-dllm: Accelerating diffusion llms with dynamic cache eviction.arXiv preprint arXiv:2508.02558, 2025

    Yuerong Song, Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. Sparse-dllm: Accelerating diffusion llms with dynamic cache eviction.arXiv preprint arXiv:2508.02558, 2025. 3

    arXiv 2025

  56. [56]

    Journeydb: A benchmark for generative im- age understanding.Advances in neural information process- ing systems, 36:49659–49678, 2023

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative im- age understanding.Advances in neural information process- ing systems, 36:49659–49678, 2023. 6, 4

    2023

  57. [57]

    Diffusion llms can do faster-than-ar inference via discrete diffusion forcing.arXiv preprint arXiv:2508.09192, 2025

    Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. Diffusion llms can do faster-than-ar inference via discrete diffusion forcing.arXiv preprint arXiv:2508.09192, 2025. 3, 6

    Pith/arXiv arXiv 2025

  58. [58]

    Segllm: Multi-round reasoning segmentation with large language models

    XuDong Wang, Shaolun Zhang, Shufan Li, Kehan Li, Kon- stantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Segllm: Multi-round reasoning segmentation with large language models. InThe Thirteenth International Conference on Learning Representations, 2025. 5

    2025

  59. [59]

    Gpt-image-edit- 1.5 m: A million-scale, gpt-generated image dataset.arXiv preprint arXiv:2507.21033, 2025

    Yuhan Wang, Siwei Yang, Bingchen Zhao, Letian Zhang, Qing Liu, Yuyin Zhou, and Cihang Xie. Gpt-image-edit- 1.5 m: A million-scale, gpt-generated image dataset.arXiv preprint arXiv:2507.21033, 2025. 6, 7, 4

    Pith/arXiv arXiv 2025

  60. [60]

    Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618, 2025

    Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618, 2025. 3, 7

    Pith/arXiv arXiv 2025

  61. [61]

    Omnigen2: Exploration to advanced multimodal generation.arXiv preprint arXiv:2506.18871, 2025

    Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjie Zhou, et al. Omnigen2: Exploration to advanced multimodal generation.arXiv preprint arXiv:2506.18871, 2025. 7

    Pith/arXiv arXiv 2025

  62. [62]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341,

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341,

    Pith/arXiv arXiv

  63. [63]

    VILA-u: a unified foun- dation model integrating visual understanding and genera- tion

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, and Yao Lu. VILA-u: a unified foun- dation model integrating visual understanding and genera- tion. InThe Thirteenth International Conference on Learn- ing Representations, 2025. 1

    2025

  64. [64]

    Omnigen: Unified image genera- tion

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xin- grun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image genera- tion. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 13294–13304, 2025. 7

    2025

  65. [65]

    Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024. 6

    Pith/arXiv arXiv 2024

  66. [66]

    Multimodal large diffusion language models.arXiv preprint arXiv:2505.15809, 2025

    Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Multimodal large diffusion language models.arXiv preprint arXiv:2505.15809, 2025. 1, 3, 6

    Pith/arXiv arXiv 2025

  67. [67]

    Dream 7b,

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b,

  68. [68]

    Imgedit: A uni- fied image editing dataset and benchmark.arXiv preprint arXiv:2505.20275, 2025

    Yang Ye, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, and Li Yuan. Imgedit: A uni- fied image editing dataset and benchmark.arXiv preprint arXiv:2505.20275, 2025. 6, 7

    Pith/arXiv arXiv 2025

  69. [69]

    Llada-v: Large language diffusion models with visual instruction tuning

    Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li. Llada-v: Large language diffusion models with visual instruction tuning. arXiv preprint arXiv:2505.16933, 2025. 1

    Pith/arXiv arXiv 2025

  70. [70]

    Anyedit: Mastering unified high-quality image editing for any idea

    Qifan Yu, Wei Chow, Zhongqi Yue, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Anyedit: Mastering unified high-quality image editing for any idea. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 26125–26135, 2025. 7

    2025

  71. [71]

    Dimple: Dis- crete diffusion multimodal large language model with par- allel decoding.arXiv preprint arXiv:2505.16990, 2025

    Runpeng Yu, Xinyin Ma, and Xinchao Wang. Dimple: Dis- crete diffusion multimodal large language model with par- allel decoding.arXiv preprint arXiv:2505.16990, 2025. 1, 3

    Pith/arXiv arXiv 2025

  72. [72]

    Mmmu: A massive multi-discipline multimodal understand- ing and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Ren- liang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understand- ing and reasoning benchmark for...

    2024

  73. [73]

    Magicbrush: A manually annotated dataset for instruction- 11 guided image editing.Advances in Neural Information Pro- cessing Systems, 36:31428–31449, 2023

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction- 11 guided image editing.Advances in Neural Information Pro- cessing Systems, 36:31428–31449, 2023. 7

    2023

  74. [74]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?arXiv preprint arXiv:2403.14624, 2024

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?arXiv preprint arXiv:2403.14624, 2024. 6, 7

    Pith/arXiv arXiv 2024

  75. [75]

    Ultraedit: Instruction-based fine-grained image editing at scale.Advances in Neural Information Pro- cessing Systems, 37:3058–3093, 2024

    Haozhe Zhao, Xiaojian Shawn Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-based fine-grained image editing at scale.Advances in Neural Information Pro- cessing Systems, 37:3058–3093, 2024. 7

    2024

  76. [76]

    Transfusion: Pre- dict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024. 1

    Pith/arXiv arXiv 2024

  77. [77]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 5 12 Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models Supplementary Material

    Pith/arXiv arXiv 2025

  78. [78]

    Additional Technical Details 7.1. Formulation of Discrete Diffusion Models In this section, we include an overview of the standard formulation of Masked Diffusion Models (MDMs) that are widely adopted by literature [32, 33, 38, 51, 69]. Given a sequenceX 0 consisting of discrete tokens[X 1 0 , X2 0 , . . . , XL 0 ], whereLis the sequence length, the forwa...

  79. [79]

    =Cat(X i t ; (1−t)Xi 0 +tM).(3) Prior works [51] show that the posterior of the reverse processp(X s|Xt, X0)has the following form: p(X i s|X i t , Xi

  80. [80]

    [reg]” that is similar to mask token “[M]

    = ( Cat(X i s;X i t),ifX i s ̸= [M] Cat(X i s; t−s t Xi 0 + s t M),ifX i s = [M]. (4) At inference,X 0 is not known, so we replaceXi 0 with the neural network predictionpθ(X i 0|Xt), which gives the following emprical sampling process: pθ(X i s|Xt) = ( Cat(X i s;X i t),ifX i s ̸= [M] Cat(X i s; t−s t pθ(X i 0|Xt) +s t M),ifX i s = [M]. (5) Sampling proces...

Showing first 80 references.