Pith. sign in

REVIEW 3 major objections 5 minor 52 references

LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformer

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LaTtE-Flow claims that splitting transformer layers into timestep experts reduces flow-matching inference complexity from $O(L\times T')$ to $O(M\times T')$, delivering ~6x faster unified image generation than Janus Pro at competitive…

desk verdict Clever efficiency idea with real ablations, but the central group-input ambiguity must be resolved before the speedup claim can be trusted. read the letter →

arxiv 2506.06952 v2 pith:64DOGDEF submitted 2025-06-08 cs.CV

Ying Shen , Zhiyang Xu , Jiuhai Chen , Shizhe Diao , Jiaxin Zhang , Yuguang Yao , Joy Rimchala , Ismini Lourentzou
show 1 more author
Lifu Huang
This is my paper · ORCID
classification cs.CV
keywords unifiedmultimodalmodelflowmatchinglayerwisetimestepexpertstimestep-conditionedresidualattentionefficientimagegenerationvision-languageinferenceaccelerationtransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single multimodal model can handle both image understanding and image generation without paying the usual sampling-time cost. The proposed architecture, LaTtE-Flow, partitions the transformer backbone into groups of layers and assigns each group a distinct slice of the flow-matching timestep axis, so that at each denoising step only one group executes. The authors claim this changes inference complexity from $O(L\times T')$ to $O(M\times T')$ with $M=L/K$, and they report a best ImageNet-256 FID of 5.79 with roughly 6x faster generation than the unified-model baseline Janus Pro and 48x faster than Show-o. A second mechanism, timestep-conditioned residual attention, reuses earlier attention maps under per-head timestep-dependent gating and is shown to improve convergence speed and FID by about 2.5 points. If these claims hold, fast unified vision-language models for real-time and resource-constrained settings become practical without sacrificing either mode's quality.

What carries the argument

The load-bearing object is the Layerwise Timestep Expert: a partition of the transformer's $L$ layers into $K=4$ consecutive groups of $M=7$ layers, where group $k$ is trained exclusively on timestep interval $[t_k, t_{k+1}]$ with 100-step overlaps during training and is the only group activated during inference on that interval. The companion mechanism is Timestep-Conditioned Residual Attention, which augments the self-attention matrix at layer $l+1$ as $A_{l+1} + g(t)\odot A_l$, with per-head gate $g(t)=\tanh(h_t W_t)$ computed from the timestep embedding; it applies only to the noisy-image self-attention, not to multimodal context attention. The multimodal context hidden states are precomputed once and cached for all timesteps, which is what allows the per-timestep forward pass to start from the cached context rather than from the full backbone.

What would settle it

Instrument the released model during ImageNet sampling and compare the hidden state entering the first layer of each expert group with the hidden state the full 28-layer network would produce at that same point; if they diverge (or if the code silently runs the preceding group to produce that input), the claimed $O(M\times T')$ complexity does not hold as stated. A simpler check is to measure total FLOPs per generated image and see whether it scales with $M=7$ or with the full $L=28$ layers.

Watch

Extended reading notes

Core claim

The paper's central claim is that the flow-matching generative process can be distributed across layerwise timestep experts: the $L=28$ transformer layers are split into $K=4$ non-overlapping groups of $M=7$ consecutive layers, each group trained to predict the velocity field only on its assigned timestep interval, and at inference time only the group for the current timestep is activated. This is stated to reduce inference-time complexity from $O(L\times T')$ to $O(M\times T')$ and, combined with the timestep-conditioned residual attention defined by $\tilde{A}_{l+1}=A_{l+1}+\tanh(h_t W_t)\odot A_l$, to yield an FID of 5.79 on ImageNet-256 with 40 sampling steps and 500M activated parameters per step. The paper also reports that removing the residual attention mechanism raises FID to 8.26, and that both the Couple and Blend variants outperform their vanilla counterparts while running 3–4x faster.

Load-bearing premise

The speedup assumes that each layer group can begin denoising directly from the noisy image latent plus the cached multimodal context, without needing the output of the layer group that would precede it in a full forward pass, so that the first layer of every group has a valid input at inference time.

Editorial extensions

If this is right

  • Using 40 sampling steps and CFG=5 on a single L40 GPU, LaTtE-Flow Couple generates a 256×256 ImageNet image in about 0.052 seconds, roughly 6x faster than Janus Pro and 48x faster than Show-o, while activating only 500M parameters per step.
  • The timestep-expert split accelerates training as well: both LaTtE-Flow variants reach competitive FID in far fewer steps than their vanilla counterparts, consistent with the argument that separating distant timesteps reduces conflicting optimization gradients.
  • Timestep-conditioned residual attention is a load-bearing component: removing it degrades FID from 5.79 to 8.26 and Inception Score from 213.1 to 157.0, at no extra inference cost.
  • The group size controls the quality–speed trade-off: $M=7$ and $M=14$ both beat the vanilla 28-layer baseline on FID at the same step count, with smaller groups being faster and larger groups higher quality.
  • Because the Couple variant freezes the pretrained VLM and adds a parallel generative path, it inherits strong understanding scores (e.g., MMBench 74.9, MMMU 41.1, TextVQA 79.7) without additional understanding finetuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the per-group forward pass truly starts from the cached multimodal context and the noisy latent alone, the same layerwise timestep-expert recipe should transfer to other flow or diffusion backbones (e.g., text-to-image or video DiTs), with the same complexity reduction; the main obstacle would be contexts that change during sampling, which invalidate the cache.
  • The observed gating patterns—stable within a head, distinct across heads—suggest the model learns reusable attention templates; one testable extension is whether these gate patterns transfer to new datasets or resolutions without retraining.
  • The reported 6x speedup assumes static conditioning; in interactive use where the text prompt is edited mid-sampling, the cached context would have to be recomputed, which would shrink the speed gap to baselines.
  • A FLOPs-level measurement of the released implementation would directly test the complexity claim: if the input to the first layer of each group still requires the preceding group's output, effective complexity approaches $O(L\times T')$, and the wall-clock gains come from other engineering choices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LaTtE-Flow, a unified multimodal architecture built on Qwen2-VL-2B that adds flow-matching-based image generation. Its two main contributions are (i) Layerwise Timestep Experts, which partition the transformer layers into K groups assigned to disjoint timestep intervals and activate only one group per sampling step, and (ii) Timestep-Conditioned Residual Attention, which reuses previous-layer attention maps under timestep-dependent per-head gating. Two instantiations, Couple and Blend, are evaluated on ImageNet 256 generation (FID 5.79 for Couple) and on several multimodal understanding benchmarks, with reported inference speedups of 6x over Janus Pro and 3-4x over matched vanilla baselines. The central claim is that inference complexity drops from O(L*T') to O(M*T') when only M=L/K layers run per timestep.

Significance. If the efficiency claim holds, the layerwise timestep-expert idea is a useful and simple contribution: it decouples sampling cost from total model depth and is compatible with pretrained VLMs. The paper's strengths are the matched vanilla baselines, the clean ablations on group size and residual attention, and the released code/checkpoints. The main result, however, rests on an architectural semantics that is not fully specified: the O(M*T') complexity reduction is only valid if each expert group can be evaluated without executing earlier layers, and the paper does not establish that. The generation-quality evidence is also limited to a single dataset and resolution, which tempers the generality of the claims.

major comments (3)
  1. [Section 4.2, Eq. (3), Appendix B] The central efficiency claim, O(L*T') -> O(M*T'), requires that each expert layer group can be evaluated in isolation. The paper never specifies how the input to the first layer of a group is obtained. Eq. (3) states that the group receives x_t and the multimodal context m_l, with m_l 'derived from the preceding layer l-1'; for groups 2-4, layer l-1 lies in the preceding group. At inference, the text says only that 'the associated expert layer group G_{l,l+M}^k is activated to perform a forward pass from layer l to layer M', but it does not say whether the first layer of the group receives (a) a fresh embedding of x_t, (b) a cached hidden state from an earlier timestep, or (c) the output of layers 1..l-1 at the current timestep. Under reading (c), the per-step cost is cumulative (average L/2 layers per step) and the asymptotic saving disappears. Under reading (a), each group is effectively a standalone M-layer denoiser, which contradicts the 'consecutive layers' description and would require per-group input projections or output heads that are not described. This ambiguity is load-bearing: the 6x speedup and the complexity reduction depend on which semantics is implemented. The authors must clarify the data flow and, if the groups are sequential, correct the complexity claim and the reported speedup.
  2. [Table 1, Section 6.1, Abstract] The abstract and Section 6.1 claim 'competitive image generation quality', but the only generation experiments are class-conditioned generation on ImageNet at 256x256. There are no experiments with text prompts, other resolutions, other datasets, or zero-shot generation. Since the model is built on a pretrained VLM and trained on ImageNet only, the generality of the quality claim is not established. The authors should either add experiments outside ImageNet 256 or substantially soften the claim.
  3. [Section 4.2, Eq. (3), Section 5 (Timestep Distribution)] The definition of a layer group is inconsistent. Eq. (3) defines G_{l,l+M}^k = {l, l+1, ..., l+M} with M=L/K, which contains M+1 layers rather than M layers. The setup text says K=4 groups of M=7 consecutive layers for L=28, which is consistent with groups {1..7}, {8..14}, {15..21}, {22..28} only if the set notation is corrected to {l,...,l+M-1}. Please fix the notation and ensure that all complexity statements use the correct group cardinality.
minor comments (5)
  1. [Section 4.2, Inference paragraph] The phrase 'perform a forward pass from layer l to layer M' is confusing; it should be 'from layer l to layer l+M-1' (or 'from layer l to layer l+M' if inclusive notation is intended).
  2. [Appendix B, Timestep Distribution] The text says a 100-step overlap is introduced between adjacent intervals, but the listed intervals [1000,700], [700,450], [450,200], [200,0] share only endpoints and have no overlap. Please clarify the actual overlap schedule.
  3. [Table 1] The caption says inference times are measured on a single L40 GPU with batch size 50, but it is unclear whether 'Time (s / img)' is per image at that batch size or per image in a batch. Please state the batching convention used for all rows.
  4. [Figure 5] The legend labels 'Group 4', 'Group 7', 'Group 14' are ambiguous: they could mean group size M or number of groups. Please label them explicitly as group size M.
  5. [Section 4.3, Eq. (5)] Please specify whether A_l is the post-softmax attention matrix and describe how the per-head gating vector g(t) is broadcast over the spatial dimensions of A_l.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the central efficiency claim is architectural bookkeeping and the quality claims are empirical measurements; the few self-citations are incidental.

full rationale

The paper's claimed complexity reduction, from O(L x T') to O(M x T'), is a direct consequence of the architecture's definition: only M = L/K layers are activated at each sampling step. This is an accounting identity of the proposed design, not a fitted prediction or a first-principles result whose output is equivalent to its input. The generative-quality claims (FID 5.79, 6x faster inference) are empirical results measured against external baselines on ImageNet, not quantities reconstructed from the model's own parameters. Equation (3) is the standard conditional flow-matching loss restricted to a layer group; it does not encode the reported FID or timing numbers. The self-citations that appear, e.g. references [4] and [47], are contextual examples in the related-work survey rather than load-bearing support for the architecture's efficiency or quality, and no author-imported uniqueness theorem is used. The under-specification of how the first layer of a timestep-expert group receives its input (Eq. 3 conditions group k on m_l from layer l-1, while Section 4.2 says the forward pass starts at layer l) is an implementation ambiguity that could affect the validity of the asymptotic claim, but it is a correctness or completeness issue, not a circularity: resolving it would not make any predicted quantity equal to a fitted input by construction. No quoted reduction of the kind required by the circularity rules is present, so the appropriate finding is no significant circularity.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The central claims rest on standard flow matching theory, the quality of pretrained components (Qwen2-VL, DC-AE), and a set of hand-chosen hyperparameters. No new physical or conceptual entities are introduced; the architectural modules are standard trainable components.

free parameters (8)
  • number of layer groups K = 4
    Chosen by hand; not fitted. Ablation explores K=2,4,7 (M=14,7,4).
  • group size M (layers per expert) = 7
    Selected based on trade-off between quality and efficiency (Figure 5).
  • timestep overlap between groups = 100 steps
    Introduced to smooth boundary transitions; hand-chosen, not optimized.
  • training steps = 240K
    Limited by compute; authors note more steps could improve performance.
  • learning rate = 5e-4
    Standard choice, not tuned per model.
  • global batch size = 2048
    Fixed for all variants.
  • CFG scale at evaluation = 5
    Selected based on ablation study (Figure 6).
  • number of sampling steps = 40
    Selected based on ablation; more steps give marginal gains.
assumptions (4)
  • standard math Flow matching objective with linear interpolation paths is a valid generative modeling framework (Liu et al., Lipman et al.).
    Used as the training objective; accepted in the literature.
  • domain assumption DC-AE autoencoder provides a 32x downsampled latent space that preserves enough information for high-quality generation.
    Relies on the DC-AE paper for the autoencoder's quality; no verification in this paper.
  • domain assumption Qwen2-VL-2B-Instruct provides strong multimodal understanding capabilities that can be transferred to generation via a parallel branch.
    The understanding performance is inherited from this pretrained model, not trained jointly.
  • domain assumption ImageNet 256x256 is a sufficient benchmark to validate unified image understanding and generation.
    Evaluation is limited to this dataset; no text-to-image or other benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformer." pith.science (2026). https://pith.science/paper/64DOGDEF

@misc{pith2026250606952,
  author       = {Pith},
  title        = {Pith review of: LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/64DOGDEF}},
  note         = {Machine review of arXiv:2506.06952}
}
read the original abstract

Recent advances in multimodal foundation models unifying image understanding and generation have opened exciting avenues for tackling a wide range of vision-language tasks within a single framework. Despite progress, existing unified models typically require extensive pretraining and struggle to achieve the same level of performance compared to models dedicated to each task. Additionally, many of these models suffer from slow image generation speeds, limiting their practical deployment in real-time or resource-constrained settings. In this work, we propose Layerwise Timestep-Expert Flow-based Transformer (LaTtE-Flow), a novel and efficient architecture that unifies image understanding and generation within a single multimodal model. LaTtE-Flow builds upon powerful pretrained Vision-Language Models (VLMs) to inherit strong multimodal understanding capabilities, and extends them with a novel Layerwise Timestep Experts flow-based architecture for efficient image generation. LaTtE-Flow distributes the flow-matching process across specialized groups of Transformer layers, each responsible for a distinct subset of timesteps. This design significantly improves sampling efficiency by activating only a small subset of layers at each sampling timestep. To further enhance performance, we propose a Timestep-Conditioned Residual Attention mechanism for efficient information reuse across layers. Experiments demonstrate that LaTtE-Flow achieves strong performance on multimodal understanding tasks, while achieving competitive image generation quality with around 6x faster inference speed compared to recent unified multimodal models.

Figures

Figures reproduced from arXiv: 2506.06952 by the authors.

Figure 1
Figure 1. Comparison of the flow-matching process between standard diffusion / flow-matching models and our proposed LaTtE-Flow. Unlike diffusion / flow-matching based models, which invoke the entire model at each sampling timestep, LaTtE-Flow activates only a subset of layers at each step, improving efficiency. integrating vision and language generation within a single framework. The second category leverages diffusion-based… view at source ↗
Figure 2
Figure 2. LaTtE-Flow overall architecture. Furthermore, we introduce two core architectural innovations applicable to both variants to en￾hance image generation efficiency and quality: (1) Layerwise Timestep Experts (Section 4.2), which partition the model into timestep-specialized modules to reduce sampling complexity, and (2) Timestep-Conditioned Residual Attention (Section 4.3), which injects timestep-aware residual attent… view at source ↗
Figure 3
Figure 3. Timestep-conditioned residual attention To facilitate information reuse across transformer layers and improve both training efficiency and generative performance, we propose Timestep-Conditioned Residual Attention, a novel mechanism that introduces adaptive residual connections be￾tween successive image attention layers based on the current timestep. The goal is to enable later layers to reuse and refine the attenti… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Training dynamics of LaTtE￾Flow vs. baselines. FID on ImageNet 50K. We observe that both LaTtE-Flow Blend and LaTtE￾Flow Couple exhibit a significantly faster convergence rate during training, reaching competitive image gen￾eration performance (lower FID) in fewer trai…
Figure 5
Figure 5. Figure 5: Effect of group size in LaTtE-Flow Couple. Impact of Varying Group Size. We also investigate how the timestep-expert group size M affects the trade-off between generation quality and inference efficiency. Specifically, we train LaTtE-Flow Couple with group sizes M ∈ {4…
Figure 6
Figure 6. Figure 6: Impact of # sampling steps and CFG strength on Inception Score and FID. (a) Timestep‐varying sequential similarity across adjacent transformer layers (b) Timestep-conditioned residual attention gate for head 5 across transformer layers [PITH_FULL_IMAGE:figures/full_fi…
Figure 7
Figure 7. Figure 7: Timestep-conditioned residual attention analysis. (a) Visualization of attention behavior in Vanilla Couple and (b) learned residual gating patterns in LaTtE-Flow Couple. Blend. Specifically, we first investigate how attention patterns evolve across transformer layers …
Figure 8
Figure 8. Figure 8: illustrates the architecture of the LaTtE-Flow Attention module. Our framework applies 3D Rotary Positional Embeddings (RoPE) [35] from the pretrained VLM to multimodal hidden states and uses a new 2D Rotary Positional Embeddings to the generative image tokens. We adop…
Figure 9
Figure 9. Figure 9: Generated 256×256 samples by LaTtE-Flow Couple trained on ImageNet. validation split. Following previous convention [27], we compute Precision and Recall using 1,000 generated images. All scores are calculated using standard implementations from torch-fidelity 2 . C Qu…
Figure 10
Figure 10. Figure 10: Visualization of attention in Baseline Blend and LaTtE-Flow Blend. (a) Sequential similarity between adjacent layers increases over timesteps, particularly in early layers. (b) Residual attention gating in LaTtE-Flow Blend (head 11) shows relatively consistent gating …
Figure 11
Figure 11. Figure 11: Timestep-conditioned residual attention gates across transformer layer in LaTtE￾Flow Couple. White regions indicate positions without gating values since residual attention is applied only within predefined layer groups. Notably, different heads exhibit distinct gatin…
Figure 12
Figure 12. Figure 12: Timestep-conditioned residual attention gates across transformer layer in LaTtE￾Flow Blend. White regions indicate positions without gating values since residual attention is applied only within predefined layer groups. Notably, different heads exhibit distinct gating…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 31 canonical work pages

  1. [1]

    Building normalizing flows with stochastic interpolants

    Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  2. [2]

    ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers.arXiv preprint arXiv:2211.01324, 2022

    Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers.arXiv preprint arXiv:2211.01324, 2022

    arXiv 2022

  3. [3]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    work page 2022

  4. [4]

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025. URL https://arxiv.org/abs/2505.09568

    arXiv 2025

  5. [5]

    Deep compression autoencoder for efficient high-resolution diffusion models

    Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models. In International Conference on Learning Representations (ICLR), 2025

    work page 2025

  6. [6]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    arXiv 2025

  7. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009

    work page 2009

  8. [8]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

Show all 52 references
  1. [9]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021

    2021

  2. [10]

    Remix-dit: Mixing diffusion transformers for multi-expert denoising

    Gongfan Fang, Xinyin Ma, and Xinchao Wang. Remix-dit: Mixing diffusion transformers for multi-expert denoising. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  3. [11]

    MME: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  4. [12]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    2024 arXiv

  5. [13]

    Efficient diffusion training via min-snr weighting strategy

    Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. InInternational Conference on Computer Vision (ICCV), 2023

    2023

  6. [14]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. InNeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications (NeurIPS), 2022. 10

    2021

  7. [15]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  8. [16]

    Cascaded diffusion models for high fidelity image generation

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. InJournal of Machine Learning Research (JMLR), 2022

    2022

  9. [17]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  10. [18]

    Multi-architecture multi-expert diffusion models

    Yunsung Lee, JinYoung Kim, Hyojun Go, Myeongho Jeong, Shinhyeok Oh, and Seungtaek Choi. Multi-architecture multi-expert diffusion models. InAAAI Conference on Artificial Intelligence (AAAI), 2024

    2024

  11. [19]

    Seed-bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  12. [20]

    Mage: Masked generative encoder to unify representation learning and image synthesis

    Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  13. [21]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  14. [22]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  15. [23]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  16. [24]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sat...

    2024

  17. [25]

    Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. InIEEE Conference on Computer Vision ...

    2025

  18. [26]

    Transfer between modalities with metaqueries.CoRR, abs/2504.06256, 2025

    Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Ji Hou, and Saining Xie. Transfer between modalities with metaqueries.CoRR, abs/2504.06256, 2025. doi: 10.48550/ARXIV . 2504.06...

    2025 doi arXiv

  19. [27]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InInternational Conference on Computer Vision (ICCV), 2023

    2023

  20. [28]

    Generating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. InAdvances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  21. [29]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  22. [30]

    Efficient diffusion models: A survey.arXiv preprint arXiv:2502.06805, 2025

    Hui Shen, Jingxuan Zhang, Boning Xiong, Rui Hu, Shoufa Chen, Zhongwei Wan, Xin Wang, Yu Zhang, Zixuan Gong, Guangyin Bao, et al. Efficient diffusion models: A survey.arXiv preprint arXiv:2502.06805, 2025. 11

    2025 arXiv

  23. [31]

    Diffmoe: Dynamic token selection for scalable diffusion transformers.arXiv preprint arXiv:2503.14487, 2025

    Minglei Shi, Ziyang Yuan, Haotian Yang, Xintao Wang, Mingwu Zheng, Xin Tao, Wenliang Zhao, Wenzhao Zheng, Jie Zhou, Jiwen Lu, et al. Diffmoe: Dynamic token selection for scalable diffusion transformers.arXiv preprint arXiv:2503.14487, 2025

    2025 arXiv

  24. [32]

    Llamafusion: Adapting pretrained language models for multimodal generation

    Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Llamafusion: Adapting pretrained language models for multimodal generation. arXiv preprint arXiv:2412.15188, 2024

    2024 arXiv

  25. [33]

    Towards VQA models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  26. [34]

    Generative modeling by estimating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. InAdvances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  27. [35]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 2024

    2024

  28. [36]

    Ec-dit: Scaling diffusion transformers with adaptive expert-choice routing

    Haotian Sun, Tao Lei, Bowen Zhang, Yanghao Li, Haoshuo Huang, Ruoming Pang, Bo Dai, and Nan Du. Ec-dit: Scaling diffusion transformers with adaptive expert-choice routing. In International Conference on Learning Representations (ICLR), 2025

    2025

  29. [37]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  30. [38]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  31. [39]

    Metamorph: Multimodal under- standing and generation via instruction tuning.arXiv preprint arXiv:2412.14164, 2024

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal under- standing and generation via instruction tuning.arXiv preprint arXiv:2412.14164, 2024

    2024 arXiv

  32. [40]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  33. [41]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  34. [42]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  35. [43]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. InIEEE Conference on Computer Vision and Pattern Recognition (...

    2025

  36. [44]

    Vila-u: a unified foundation model integrating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  37. [45]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. InInternational Conference on Learning Represe...

    2025

  38. [46]

    Autoregressive models in vision: A survey

    Jing Xiong, Gongye Liu, Lun Huang, Chengyue Wu, Taiqiang Wu, Yao Mu, Yuan Yao, Hui Shen, Zhongwei Wan, Jinfa Huang, et al. Autoregressive models in vision: A survey. In Transactions on Machine Learning Research (TMLR), 2025. 12

    2025

  39. [47]

    Modality-specialized synergizers for interleaved vision-language generalists

    Zhiyang Xu, Minqian Liu, Ying Shen, Joy Rimchala, Jiaxin Zhang, Qifan Wang, Yu Cheng, and Lifu Huang. Modality-specialized synergizers for interleaved vision-language generalists. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, Apr...

    2025

  40. [48]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. InInternational Conference on Learning Representations (ICLR), 2022

    2022

  41. [49]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International Conference on Machine Learning (ICML), 2024

    2024

  42. [50]

    MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI

    Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A m...

    2024

  43. [51]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. InInternational Conference on Learning Representati...

    2025

  44. [52]

    Timestep master: Asymmetri- cal mixture of timestep lora experts for versatile and efficient diffusion models in vision.arXiv preprint arXiv:2503.07416, 2025

    Shaobin Zhuang, Yiwei Guo, Yanbo Ding, Kunchang Li, Xinyuan Chen, Yaohui Wang, Fangyikang Wang, Ying Zhang, Chen Li, and Yali Wang. Timestep master: Asymmetri- cal mixture of timestep lora experts for versatile and efficient diffusion models in vision.arXiv preprint arXiv:2503...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.