Pith. sign in

REVIEW 4 major objections 4 minor 6 cited by

Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that ordering visual tokens by diffusion timesteps instead of spatial position gives multimodal LLMs a recursive visual language that makes unified comprehension and generation work.

desk verdict Novel diffusion-timestep tokenization with strong but under-verified results; the recursivity claim needs a decisive ordering ablation. read the letter →

arxiv 2504.14666 v1 pith:CIC6WCA2 submitted 2025-04-20 cs.CV

Kaihang Pan , Wang Lin , Zhongqi Yue , Tenglong Ao , Liyu Jia , Wei Zhao , Juncheng Li , Siliang Tang
show 1 more author
Hanwang Zhang
This is my paper · ORCID
classification cs.CV
keywords discretediffusiontimesteptokensrecursivevisualmultimodallargelanguagemodelsunifiedcomprehensionandgenerationvectorquantizationimageeditingtext-to-imagenext-tokenprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that spatial visual tokens—image patches read in spatial order—lack the recursive structure of language, so an LLM trained on them is memorizing an impossible language rather than learning to see. It replaces that ordering with one inherited from the diffusion process: as noise is added to an image in increasing timesteps, an encoder emits one new discrete token per timestep that compensates for the attributes lost so far. These Discrete Diffusion Timestep (DDT) tokens form an expanding prefix, so the diffusion decoder can reconstruct the original image from the noised image plus the first $t$ tokens. The authors build an 8-billion-parameter multimodal LLM around this tokenizer, train it with next-token prediction on 200M image-text pairs, and report that it surpasses other generalist MLLMs on text-to-image generation, zero-shot image editing, and several visual comprehension benchmarks even though the tokenizer was trained only on ImageNet.

What carries the argument

The load-bearing object is the DDT tokenizer: a transformer encoder with $T=480$ learnable query tokens, a vector-quantization codebook of 65,536 entries, and an MMDiT diffusion decoder. During training, the noised image is sampled by Rectified Flow interpolation, $x_t = t\epsilon + (1-t)x_0$; the decoder receives $x_t$, the timestep $t$, and only the first $t$ quantized tokens, and is trained to return $x_0$. This expanding-prefix reconstruction loss is what forces each new token to carry the information removed by the extra noise. The paper also uses an order-perturbation experiment, borrowed from the impossible-language test, to show that DDT sequences lose predictive performance when shuffled, while spatial token sequences are nearly order-insensitive.

What would settle it

Take a complete DDT token sequence and decode it after shuffling the token order, then compare reconstruction quality with the ordered case: if the recursion claim holds, shuffled tokens should reconstruct the image substantially worse, whereas if the tokens just encode the image redundantly, a shuffled but complete set would still reconstruct well. A second check is to train the tokenizer with a reversed timestep order under the same loss and see whether the editing and generation advantages disappear.

Watch

Extended reading notes

Core claim

The central claim is that images can be tokenized into a discrete, recursive visual language by aligning token order with diffusion timesteps. For a clean image $x_0$, the encoder $f$ produces a sequence $(V_1,\ldots,V_T)$; at timestep $t$, the first $t$ tokens together with the noised image $x_t = t\epsilon + (1-t)x_0$ are enough for the diffusion decoder to reconstruct $x_0$. Because each appended token accounts for the attributes destroyed by the additional noise, the sequence satisfies $f_t(x_0) = (f_{t-1}(x_0), V_t)$, which gives the token stream a language-like recursive structure. The paper supports this with three kinds of evidence: autoregressive training curves that degrade when DDT token order is perturbed, as human language does; counterfactual interpolation in which swapping a subset of tokens changes only the corresponding attributes instead of cutting and mixing image regions; and progressive decoding in which the earliest tokens fix fine details while later tokens build contours and color. With this tokenizer, the MLLM learns to translate between text and DDT tokens under a single next-token prediction objective, and the same diffusion decoder serves both generation and comprehension.

Load-bearing premise

The load-bearing premise is that a single fixed ordering of tokens along diffusion timesteps induces a consistent, semantically meaningful attribute hierarchy across all images, so that the first $t$ tokens together with the noised image suffice to reconstruct the original image.

Editorial extensions

If this is right

  • An MLLM can be built without a pretrained CLIP-style vision encoder and still match or beat comprehension-specialized models on captioning, VQA, and MLLM benchmarks.
  • Zero-shot instruction-based image editing improves because editing becomes a matter of changing the specific tokens that carry the modified attributes; DDT-LLaMA outperforms both generalist and specialist editors on EVR, MA5K, and MagicBrush.
  • Counterfactual interpolation with DDT tokens changes only the substituted attributes, whereas spatial tokenizers behave like CutMix, producing a patch-wise blend rather than a semantic one.
  • Scaling the LLM backbone from 2B to 8B parameters and increasing training compute improves visual quality, consistent with scaling laws for the DDT-based formulation.
  • The paper's stated limitation is that tokenizer reconstruction quality, not token ordering, is the bottleneck: a tokenizer trained only on ImageNet at 256x256 restricts aesthetic quality for open-domain generation, so scaling tokenizer data is the clear next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Any signal with a natural degradation process—video, audio, or 3D scenes under corruption—could be tokenized along its own timestep ordering; the design principle of recovering lost attributes with expanding discrete prefixes is not image-specific.
  • The learned attribute hierarchy is encouraged by the reconstruction loss rather than enforced by a structural constraint, so a direct test of the visual-language claim would measure how well the ordering transfers to out-of-domain images, where the paper reports strong results but does not quantify attribute disentanglement.
  • A testable extension is to train the tokenizer with reversed or shuffled timestep order under the same loss: if the advantages in editing and generation persist, recursion is not the active ingredient; if they vanish, the diffusion-timestep ordering is doing the work.
  • Because editing quality improves when the backbone grows from 2B to 8B, the DDT formulation may offer a direct path to attribute-level controllable generation at scale, since token-level interventions become semantically meaningful.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Discrete Diffusion Timestep (DDT) tokens: an image tokenizer that encodes an image into a sequence of discrete codes, one per diffusion timestep, trained so that the first t tokens together with the noised image x_t reconstruct x_0 (Eq. 1). The authors argue that this yields a recursive visual language whose tokens progressively compensate for attribute loss as the timestep increases, in contrast to spatial tokens that form an 'impossible language' for LLMs. They integrate these tokens into an 8B LLaMA-3-based MLLM trained on 200M image-text pairs and report improved results over other MLLMs on text-to-image generation (GenEval, T2I-CompBench, DrawBench), instruction-based image editing (EVR, MA5K, MagicBrush), and vision-language comprehension benchmarks, despite a tokenizer trained only on ImageNet at 256x256. Additional analyses cover class-conditional ImageNet generation, counterfactual interpolation, progressive decoding with token subsets, a perplexity comparison, an A/B editing test with a 2B Gemma backbone, and preliminary scaling observations.

Significance. If the central claim holds, the paper makes a useful conceptual contribution: replacing spatial token order with a diffusion-timestep order that is claimed to be recursive and semantically structured, while keeping a single next-token-prediction objective for both text and image. The breadth of benchmarks is a strength, and the reported results are generally strong for an MLLM generalist. The paper also presents a controlled A/B comparison (DDT-Gemma vs. MoVQ-Gemma) and an ImageNet class-conditional FID comparison, which are helpful evidence beyond the headline tables. However, the load-bearing property—that the learned token order is genuinely recursive and semantically consistent across images—is currently supported mainly by qualitative demonstrations and by an order-perturbation experiment on the LLM, not by a direct ablation of the tokenizer's ordering. The single-run point estimates in Tables 1-3 and the use of previously published baseline numbers also weaken the confidence in the claimed superiority. The idea is timely and potentially impactful, but the evidence as presented is not yet decisive.

major comments (4)
  1. [Section 3, Eq. (1); Section 5.4.2] The recursion claim f_{t+1}(x0) = (f_t(x0), V_{t+1}) is not entailed by the training objective in Eq. (1). The loss only requires that the decoder map (x_t, t, V_1, ..., V_t) to x0; since x_t itself preserves most information for small t, the objective can be satisfied by late tokens that are redundant or that encode image-specific statistics rather than the incremental attribute loss between t and t+1. The counterfactual interpolation and progressive-decoding figures are consistent with the claimed hierarchy, but they are also consistent with the decoder exploiting the known noise level t and a few coarse tokens. To make the central claim load-bearing, the authors should provide a decisive ordering ablation: train the tokenizer with (i) the proposed order, (ii) the reversed order, and (iii) a fixed random permutation of timestep indices, holding architecture and data fixed, and then compare reconstruction PSNR and the downstream editing A/B test. If ordering has little effect, the 'recursive visual language' interpretation is a training artifact; if it has a large effect, the size and direction of that effect should be quantified.
  2. [Tables 1-3 in Sections 5.1-5.3] The headline comparisons are based on single-run point estimates without error bars, confidence intervals, or significance tests, and many baseline numbers are taken from published papers rather than rerun in an identical setup. This matters because several headline margins are small, for example GenEval Overall 0.66 vs. Transfusion 0.63 and DrawBench Clip-T 0.267 tied with LaVIT and Lumina-mGPT. The authors should either provide multiple runs (at least for a smaller controlled setting) or state clearly that the differences are not statistically assessed; without this, the claim of systematic superiority over other MLLMs is not fully supported.
  3. [Section 5.4.2 and Figure 1] The 'impossible language' argument and the recursion interpretation rely on Figure 1, which shows LLM training loss under different perturbation types for DDT tokens and spatial tokens. The figure lacks experimental detail such as the number of seeds, architecture, training length, and the magnitude of the perturbation; more importantly, order sensitivity of the LLM does not establish that the token order corresponds to a semantically meaningful attribute hierarchy. An LLM can be sensitive to arbitrary positional statistics without the tokens encoding recursive, disentangled attributes. The authors should report quantitative order-sensitivity measures (e.g., final loss gaps averaged over seeds) and connect them to the downstream editing and counterfactual results.
  4. [Section 5.4.4 and Figure 9(b)] The A/B test on image editing is potentially the strongest evidence for the claimed advantage, but the description is incomplete. The paper does not report the number of human raters, the exact instructions, whether ties were allowed, how the test cases were selected, or the total number of cases. The text states that DDT-Gemma outperforms MoVQ-Gemma in 65 editing cases and that MoVQ-Gemma surpasses DDT-Gemma in only 10, but without a total and a protocol the reader cannot judge significance or bias. This should be documented, together with inter-annotator agreement if multiple raters were used.
minor comments (4)
  1. [Appendix A.1 and B.1] There are typos: 'two weaks' should read 'two weeks' (appears twice), and the reference 'Coyo [ ? ]' in Appendix B.1 is incomplete.
  2. [Section 5.1 and 5.4.2] Main-text figure references are inconsistent with the numbering: Section 5.1 refers to 'Figure 10' for qualitative T2I examples, and Section 5.4.2 refers to 'Figure 16' and 'Figure 17' for counterfactual interpolation and subset decoding, while the corresponding main-paper figures are numbered 3, 6, and 8.
  3. [Appendix B.2] In the GenEval subtask list, 'single object (TwoObj)' should be 'single object (SingObj) and two objects (TwoObj)'.
  4. [Figure 12 captions] The captions contain article errors: 'an blue cell phone' and 'an purple suitcase' should be 'a blue cell phone' and 'a purple suitcase'.

Circularity Check

1 steps flagged · score 4.0 of 10

The recursive-token claim is largely built into Eq. (1), while the downstream MLLM results rest on external benchmarks and are not circular.

  1. self definitional [Section 3, Training (Eq. 1); Section 5.4.2, DDT Tokens are Recursive]
    "Hence we use an expanding set of tokens (V1,..., Vt) as the input to the decoder d, and train everything end-to-end with the reconstruction loss: L = E t,x0,ϵ [ || d (tϵ + (1−t)x0,t, (V1,..., Vt))− x0||2 ] ... As shown in Figure 17, with the number of tokens increasing, the image’s attributes are progressively recovered. ... This finding further demonstrates that DDT tokens are recursive, and DDT-LLaMA effectively disentangles visual attributes."

    Eq. (1) is exactly an expanding-prefix reconstruction objective: at every timestep t the decoder receives (V1,...,Vt) and the noised image xt and must reconstruct x0. Therefore the Section 5.4.2 observation that decoding with the first t tokens progressively recovers the image is a direct consequence of the training loss, not an independent discovery of recursivity. The abstract's phrase 'tokens recursively compensate for the progressive attribute loss' restates this same training objective. The additional claim of a semantically meaningful attribute hierarchy (fine details first, coarse attributes later) is not forced by Eq. (1), but the load-bearing recursive-prefix property is true by construction, making the evidence partly self-definitional.

full rationale

The central empirical result of the paper—DDT-LLaMA outperforming other MLLM generalists on GenEval, image editing, and VQA/captioning—is evaluated on external benchmarks and does not reduce to the tokenizer loss. The A/B test against MoVQ-Gemma and the perplexity comparison also provide independent, non-circular evidence. However, the paper's structural claim that DDT tokens are 'recursive' is largely built into the method: Section 3 defines ft+1(x0) = (ft(x0), Vt+1) and Eq. (1) trains the decoder to reconstruct x0 from every prefix (V1,...,Vt) at every timestep. Consequently, Section 5.4.2's progressive-decoding demonstration that an expanding subset of tokens reconstructs the image is a consequence of the training objective rather than an independent confirmation. The counterfactual interpolation experiment offers some non-forced evidence for attribute disentanglement, and the decoder's dependence on the noise level t could explain progressive reconstruction without a semantically ordered token hierarchy, so the circularity is partial rather than total. No load-bearing self-citation or imported uniqueness theorem was found; the cited 'impossible language' evaluation [34] is re-run by the authors on their own tokens and spatial baselines, so it does not create a circularity chain.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The architecture introduces a new token representation but no new natural-kind entity. The main assumptions concern the validity of the "impossible language" criterion and the existence of a consistent attribute hierarchy along diffusion timesteps, neither of which is proven independently of the training objective.

free parameters (5)
  • Number of timestep tokens T = 480
    Hand-chosen sequence length; determines the recursion depth of the visual token sequence and the vocabulary context length for the MLLM.
  • Visual codebook size |C| = 65,536
    Hand-chosen vocabulary size; added directly to the LLaMA token vocabulary and used by the shared prediction head.
  • VQ projection dimension m = 16
    Adopted from [84] to improve codebook usage; affects the granularity of the quantized representation.
  • Inference sampling hyperparameters = topk=4096, topp=0.9, CFG scale=8.0
    Hand-chosen sampling settings for visual token prediction; directly affect text-to-image generation quality reported in the tables.
  • Pretraining data recipe ratios = 60% long caption, 40% short caption, 10% caption dropout, 10% text data
    Hand-set data mixture; influences alignment between text and visual tokens during pretraining, though it is not a fitted scientific constant.
assumptions (3)
  • domain assumption The order-perturbation test from Kallini et al. [34] is a valid diagnostic for whether a token sequence is a learnable "language" for an LLM.
    Section 1 uses Figure 1 loss curves to infer that spatial tokens form an "impossible language"; this transfers an external criterion without validating it on the exact tokenizer and LLM setting.
  • domain assumption There exists a single consistent attribute hierarchy ordered by diffusion timestep that the encoder can approximate.
    Section 3 trains the encoder and decoder jointly with expanding prefix tokens; reconstruction performance alone does not prove the hierarchy is semantically meaningful rather than a decoder compensation artifact.
  • standard math Rectified flow interpolation x_t = t*epsilon + (1-t)*x0 is a valid noising schedule for learning the token-to-attribute mapping.
    Section 3, Eq. (1), adopts the Rectified Flow schedule from [54], which is standard generative modeling practice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens." pith.science (2026). https://pith.science/paper/CIC6WCA2

@misc{pith2026250414666,
  author       = {Pith},
  title        = {Pith review of: Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CIC6WCA2}},
  note         = {Machine review of arXiv:2504.14666}
}
read the original abstract

Recent endeavors in Multimodal Large Language Models (MLLMs) aim to unify visual comprehension and generation by combining LLM and diffusion models, the state-of-the-art in each task, respectively. Existing approaches rely on spatial visual tokens, where image patches are encoded and arranged according to a spatial order (e.g., raster scan). However, we show that spatial tokens lack the recursive structure inherent to languages, hence form an impossible language for LLM to master. In this paper, we build a proper visual language by leveraging diffusion timesteps to learn discrete, recursive visual tokens. Our proposed tokens recursively compensate for the progressive attribute loss in noisy images as timesteps increase, enabling the diffusion model to reconstruct the original image at any timestep. This approach allows us to effectively integrate the strengths of LLMs in autoregressive reasoning and diffusion models in precise image generation, achieving seamless multimodal comprehension and generation within a unified framework. Extensive experiments show that we achieve superior performance for multimodal comprehension and generation simultaneously compared with other MLLMs. Project Page: https://DDT-LLaMA.github.io/.

Figures

Figures reproduced from arXiv: 2504.14666 by the authors.

Figure 1
Figure 1. Auto-regressive training curves of diffusion timestep [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of our methods. (a): The architecture of diffusion timestep tokenizer encodes an image to a recursive sequence of discrete tokens. (b): An MLLM architecture that unifies comprehension and generation based on next token prediction. transformer-based model with learnable query tokens as our encoder. The input to the encoder is a noise-free im￾age (patchified and flattened into image tokens similar to ViT … view at source ↗
Figure 3
Figure 3. Qualitative results of DDT-LLaMA text-to-image generation. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison with EMU3 on T2I generation. DDT-LLaMA better responses to prompts related to counting or position. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison on image editing. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Results of Counterfactual Interpolation with DDT tokens [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Scaling MLLM size (2B, 8B) and training compute (50%, 75%, 100% of total tokens) improves visual quality. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: (a) Perplexity of MLLM in text generation during image [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: More qualitative results of DDT-LLaMA text-to-image generation. ( [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: More qualitative comparison with EMU3 on T2I generation (PART-1). DDT-LLaMA can better respond to prompts related to [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: More qualitative comparison with EMU3 on T2I generation (PART-2). DDT-LLaMA can better respond to prompts related to [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Qualitative comparison of MOVQ-Gemma(2B), DDT-Gemma(2B), and DDT-LLaMA(8B) in the image editing task. In most [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: (More examples of text-to-image generation with different MLLM size (2B, 8B) and training compute (50%, 75%, 100% of [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Qualitative results of DDT-LLaMA visual comprehen [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 17
Figure 17. Figure 17: More results of decoding images with an expanding [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 16
Figure 16. Figure 16: More results of counterfactual interpolation with DDT [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RL

    cs.CV 2025-06 conditional novelty 7.0 of 10

    FocusDiff improves autoregressive text-to-image generation by training on paired similar prompts with a modified GRPO objective, achieving state-of-the-art alignment on PairComp and gains on GenEval and T2I-CompBench.

  2. D-AR: Diffusion via Autoregressive Models

    cs.CV 2025-05 conditional novelty 7.0 of 10

    D-AR recasts pixel-space diffusion as vanilla autoregressive next-token prediction using a diffusion-ordered discrete tokenizer, reaching 2.09 FID on ImageNet 256x256 with a 775M Llama backbone.

  3. Towards Meta-Cognitive Knowledge Editing for Multimodal LLMs

    cs.AI 2025-09 conditional novelty 6.0 of 10

    CogEdit and MIND shift multimodal knowledge editing toward evaluating and enabling meta-cognitive skills: self-awareness, boundary monitoring, and noise robustness.

  4. What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A self-generating graph benchmark produces 36k GUI agent tasks with controllable complexity and ten capability scores, and fine-tuning on its trajectories gives small gains on AndroidControl and OmniAct.

  5. FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A 1.5B unified multimodal model trained with discrete flow matching and metric-induced probability paths matches autoregressive baselines of similar size on generation and understanding benchmarks.

  6. KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A new benchmark, KRIS-Bench, evaluates image editing models on knowledge-grounded reasoning across factual, conceptual, and procedural tasks, and finds large performance gaps in current models.

Reference graph

Works this paper leans on

101 extracted references · 36 canonical work pages · cited by 6 Pith papers

  1. [1]

    Nocaps: Novel object captioning at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pages 8948–8957,

  2. [2]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425–2433, 2015. 6

    2015

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 7, 17

    arXiv 2023

  4. [4]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 4, 16

    2023

  5. [5]

    Vizwiz: nearly real-time answers to visual questions

    Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Lit- tle, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceed- ings of the 23nd annual ACM symposium on User interface software and technology, pages 333–342, 2010. 19

    2010

  6. [6]

    In- structpix2pix: Learning to follow image editing instruc- tions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instruc- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18392– 18402, 2023. 4, 6, 16, 17

    2023

  7. [7]

    Coyo-700m: Image-text pair dataset

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https : / / github . com / kakaobrain/coyo-dataset, 2022. 2, 4

    2022

  8. [8]

    Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Juny- ing Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jian- quan Li, Xiang Wan, and Benyou Wang. Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024. 16

    arXiv 2024

Show all 101 references
  1. [9]

    Pixart- α: Fast train- ing of diffusion transformer for photorealistic text-to-image synthesis, 2023

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- α: Fast train- ing of diffusion transformer for photorealistic text-to-image synthesis, 2023. 4, 16

    2023

  2. [10]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 7, 15, 16, 17

    2023 arXiv

  3. [11]

    Janus- pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus- pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811,

    arXiv

  4. [12]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1–113, 2023. 15

    2023

  5. [13]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  6. [14]

    Imagenet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2, 15, 21

    2009

  7. [15]

    Dreamllm: Synergistic multimodal com- prehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal com- prehension and creation. arXiv preprint arXiv:2309.11499,

    arXiv

  8. [16]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3

    2010 arXiv

  9. [17]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  10. [18]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 1, 7, 21

    2021

  11. [19]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machin...

  12. [20]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. 6, 19

    2024

  13. [21]

    Guiding instruction-based im- 9 age editing via multimodal large language models

    Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guiding instruction-based im- 9 age editing via multimodal large language models. arXiv preprint arXiv:2309.17102, 2023. 4, 5, 6, 17, 18

    2023 arXiv

  14. [22]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, An- ish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. 15

    2020 arXiv

  15. [23]

    Making llama see and draw with seed tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218 ,

    arXiv

  16. [24]

    Seed-x: Multimodal models with unified multi-granularity compre- hension and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity compre- hension and generation. arXiv preprint arXiv:2404.14396,

    arXiv

  17. [25]

    Geneval: An object-focused framework for evaluating text- to-image alignment

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text- to-image alignment. Advances in Neural Information Pro- cessing Systems, 36, 2024. 4, 17, 18

    2024

  18. [26]

    Making the v in vqa matter: El- evating the role of image understanding in visual ques- tion answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: El- evating the role of image understanding in visual ques- tion answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913,

  19. [27]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  20. [28]

    Mea- suring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding, 2021. 21

    2021

  21. [29]

    Clipscore: A reference-free evalu- ation metric for image captioning, 2022

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evalu- ation metric for image captioning, 2022. 18

    2022

  22. [30]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xi- hui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. arXiv preprint arXiv:2307.06350, 2023. 4, 17, 18

    2023 arXiv

  23. [31]

    Vbench: Comprehensive benchmark suite for video generative mod- els, 2023

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video generative mod- els,...

    2023

  24. [32]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 6, 19

    2019

  25. [33]

    Unified language-vision pre- training in llm with dynamic discrete visual tokenization

    Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jian- chao Tan, Yadong Mu, et al. Unified language-vision pre- training in llm with dynamic discrete visual tokenization. In International Conference on Learning Representations ,

  26. [34]

    Mission: Impossible language models

    Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. Mission: Impossible language models. arXiv preprint arXiv:2401.06416, 2024. 1

    2024 arXiv

  27. [35]

    Gen- erating images with multimodal language models

    Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Gen- erating images with multimodal language models. Ad- vances in Neural Information Processing Systems , 36,

  28. [36]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11523–11532, 2022. 1, 7

    2022

  29. [37]

    Seed-bench: Benchmarking multi- modal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking multi- modal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 19

    2023 arXiv

  30. [38]

    Seed-bench: Bench- marking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Bench- marking multimodal large language models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024. 6

    2024

  31. [39]

    Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation, 2022

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation, 2022. 18

    2022

  32. [40]

    Fine-tuning multimodal llms to follow zero-shot demonstrative instructions

    Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Han- wang Zhang, and Yueting Zhuang. Fine-tuning multimodal llms to follow zero-shot demonstrative instructions. arXiv preprint arXiv:2308.04152, 2023. 21

    2023 arXiv

  33. [41]

    Variational cross-graph reasoning and adaptive structured semantics learning for compositional temporal grounding

    Juncheng Li, Siliang Tang, Linchao Zhu, Wenqiao Zhang, Yi Yang, Tat-Seng Chua, Fei Wu, and Yueting Zhuang. Variational cross-graph reasoning and adaptive structured semantics learning for compositional temporal grounding. IEEE Transactions on Pattern Analysis and Machine Intel...

    2023

  34. [42]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 6

    2023 arXiv

  35. [43]

    Evaluating object hallucination in large vision-language models, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023. 19

    2023

  36. [44]

    Vila: On pre-training for visual language models, 2023

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023. 7, 17

    2023

  37. [45]

    Action imitation in common action space for customized action image synthesis

    Wang Lin, Jingyuan Chen, Jiaxin Shi, Zirun Guo, Yichen Zhu, Zehan Wang, Tao Jin, Zhou Zhao, Fei Wu, Y AN Shuicheng, et al. Action imitation in common action space for customized action image synthesis. InThe Thirty-eighth Annual Conference on Neural Information Processing Sys-...

  38. [46]

    Tavt: Towards transferable audio-visual text generation

    Wang Lin, Tao Jin, Wenwen Pan, Linjun Li, Xize Cheng, Ye Wang, and Zhou Zhao. Tavt: Towards transferable audio-visual text generation. In Proceedings of the 61st An- 10 nual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 14983–14999, 2023

    2023

  39. [47]

    Exploring group video captioning with efficient relational approximation

    Wang Lin, Tao Jin, Ye Wang, Wenwen Pan, Linjun Li, Xize Cheng, and Zhou Zhao. Exploring group video captioning with efficient relational approximation. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 15281–15290, 2023

    2023

  40. [48]

    Non-confusing generation of cus- tomized concepts in diffusion models

    Wang Lin, Jingyuan Chen, Jiaxin Shi, Yichen Zhu, Chen Liang, Junzhong Miao, Tao Jin, Zhou Zhao, Fei Wu, Shuicheng Yan, et al. Non-confusing generation of cus- tomized concepts in diffusion models. arXiv preprint arXiv:2405.06914, 2024. 21

    2024 arXiv

  41. [49]

    Lumina-mgpt: Il- luminate flexible photorealistic text-to-image generation with multimodal generative pretraining

    Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-mgpt: Il- luminate flexible photorealistic text-to-image generation with multimodal generative pretraining. arXiv preprint arXiv:2408.02657, 2024. 2, 4, 16

    2024 arXiv

  42. [50]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 7, 17

    2024

  43. [51]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 7, 17

    2024

  44. [52]

    World model on million-length video and language with ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv e-prints, pages arXiv–2402, 2024. 14

    2024

  45. [53]

    Llava-plus: Learning to use tools for creating multi- modal agents

    Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multi- modal agents. arXiv preprint arXiv:2311.05437, 2023. 1, 7

    2023 arXiv

  46. [54]

    Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022. 3

    2022 arXiv

  47. [55]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 6, 19

    2019

  48. [56]

    OpenAI. Chatgpt. https://chat.openai.com ,

  49. [57]

    Self-supervised meta- prompt learning with meta-gradient regularization for few- shot generalization

    Kaihang Pan, Juncheng Li, Hongye Song, Jun Lin, Xi- aozhong Liu, and Siliang Tang. Self-supervised meta- prompt learning with meta-gradient regularization for few- shot generalization. arXiv preprint arXiv:2303.12314 ,

    arXiv

  50. [58]

    Towards unified multimodal editing with enhanced knowledge collaboration

    Kaihang Pan, Zhaoyu Fan, Juncheng Li, Qifan Yu, Hao Fei, Siliang Tang, Richang Hong, Hanwang Zhang, and Qianru Sun. Towards unified multimodal editing with enhanced knowledge collaboration. Advances in Neural Information Processing Systems, 37:110290–110314, 2024. 21

    2024

  51. [59]

    I3: I ntent-i ntrospective retrieval conditioned on i nstructions

    Kaihang Pan, Juncheng Li, Wenjie Wang, Hao Fei, Hongye Song, Wei Ji, Jun Lin, Xiaozhong Liu, Tat-Seng Chua, and Siliang Tang. I3: I ntent-i ntrospective retrieval conditioned on i nstructions. In Proceedings of the 47th International ACM SIGIR Conference on Research and Develo...

    2024

  52. [60]

    Auto-encoding morph-tokens for multimodal llm

    Kaihang Pan, Siliang Tang, Juncheng Li, Zhaoyu Fan, Wei Chow, Shuicheng Yan, Tat-Seng Chua, Yueting Zhuang, and Hanwang Zhang. Auto-encoding morph-tokens for multimodal llm. arXiv preprint arXiv:2405.01926 , 2024. 2

    2024 arXiv

  53. [61]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In Pro- ceedings of the IEEE international conference on computer ...

    2015

  54. [62]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 2, 4, 16

    2023 arXiv

  55. [63]

    Step: Enhancing video-llms’ compositional reasoning by spatio-temporal graph-guided self-training

    Haiyi Qiu, Minghe Gao, Long Qian, Kaihang Pan, Qi- fan Yu, Juncheng Li, Wenjie Wang, Siliang Tang, Yueting Zhuang, and Tat-Seng Chua. Step: Enhancing video-llms’ compositional reasoning by spatio-temporal graph-guided self-training. arXiv preprint arXiv:2412.00161, 2024. 21

    2024 arXiv

  56. [64]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International conference on machine learning...

    2021

  57. [65]

    Hierarchical text-conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022. 4, 16

    2022 arXiv

  58. [66]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 4, 16

    2022

  59. [67]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Sali- mans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural in- forma...

    2022

  60. [68]

    Laion-5b: An open large-scale dataset for train- ing next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for train- ing next generation image-text models. Advances in Neural Inf...

    2022

  61. [69]

    Learning by planning: Language-guided global image editing

    Jing Shi, Ning Xu, Yihang Xu, Trung Bui, Franck Der- noncourt, and Chenliang Xu. Learning by planning: Language-guided global image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13590–13599, 2021. 5, 18 11

    2021

  62. [70]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 2

    2023 arXiv

  63. [71]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pa...

    2024

  64. [72]

    Expressing visual relationships via language

    Hao Tan, Franck Dernoncourt, Zhe Lin, Trung Bui, and Mohit Bansal. Expressing visual relationships via language. arXiv preprint arXiv:1906.07689, 2019. 5, 18

    1906 arXiv

  65. [73]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 ,

    arXiv

  66. [74]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, L ´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. 8, 20

    2024 arXiv

  67. [75]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 1, 2

    2023 arXiv

  68. [76]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. 3

    2017

  69. [77]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 1, 2, 4, 7, 14, 16, 17

    2024 arXiv

  70. [78]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation, 2024

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Conghui He, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation, 2024. 21

    2024

  71. [79]

    Mmlu-pro: A more robust and challenging multi- task language understanding benchmark, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi- task language under...

    2024

  72. [80]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024. 2

    arXiv

  73. [81]

    Vila-u: a unified foundation model inte- grating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model inte- grating visual understanding and generation. arXiv preprint arXiv:2409.04429, 2024. 2, 4, 7, 16, 17

    2024 arXiv

  74. [82]

    Show-o: One single transformer to unify multimodal under- standing and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal under- standing and generation. arXiv preprint arXiv:2408.12528,

    arXiv

  75. [83]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2024

  76. [84]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021. 1, 3, 14

    2021 arXiv

  77. [85]

    Visually-prompted language model for fine-grained scene graph generation in an open world

    Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang, Wei Ji, and Yueting Zhuang. Visually-prompted language model for fine-grained scene graph generation in an open world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 21560–21571, 2023. 21

    2023

  78. [86]

    Anyedit: Mastering unified high-quality image editing for any idea

    Qifan Yu, Wei Chow, Zhongqi Yue, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Anyedit: Mastering unified high-quality image editing for any idea. arXiv preprint arXiv:2411.15738, 2024

    2024 arXiv

  79. [87]

    Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data

    Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wen- tao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  80. [88]

    Cutmix: Regu- larization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu- larization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032, 2019. 7

    2019

  81. [89]

    Soundstream: An end- to-end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end- to-end neural audio codec. IEEE/ACM Transactions on Au- dio, Speech, and Language Processing, 30:495–507, 2021. 3

    2021

  82. [90]

    Magicbrush: A manually annotated dataset for instruction-guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural In- formation Processing Systems, 36:31428–31449, 2023. 5, 18

    2023

  83. [91]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 5, 18

    2018

  84. [92]

    Hive: Harnessing human feed- back for instructional visual editing

    Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. Hive: Harnessing human feed- back for instructional visual editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  85. [93]

    Ultraedit: Instruction-based fine-grained image editing at scale

    Haozhe Zhao, Xiaojian Ma, Liang Chen, Shuzheng Si, Ru- jie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-based fine-grained image editing at scale. arXiv preprint arXiv:2407.05282 ,

    arXiv

  86. [94]

    Movq: Modulating quantized vectors for high- fidelity image generation

    Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high- fidelity image generation. Advances in Neural Information Processing Systems, 35:23412–23425, 2022. 1, 7, 21

    2022

  87. [95]

    Transfusion: Pre- dict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024. 1, 2, 4

    2024 arXiv

  88. [96]

    Simple multi-dataset detection, 2022

    Xingyi Zhou, Vladlen Koltun, and Philipp Kr ¨ahenb¨uhl. Simple multi-dataset detection, 2022. 18

    2022

  89. [97]

    Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%

    Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%. arXiv preprint arXiv:2406.11837, 2024. 1, 7

    2024 arXiv

  90. [98]

    Rosa: A robust self-aligned framework for node-node graph con- trastive learning

    Yun Zhu, Jianhao Guo, Fei Wu, and Siliang Tang. Rosa: A robust self-aligned framework for node-node graph con- trastive learning. arXiv preprint arXiv:2204.13846, 2022. 21

    2022 arXiv

  91. [99]

    Graphclip: Enhancing transferability in graph foun- dation models for text-attributed graphs

    Yun Zhu, Haizhou Shi, Xiaotang Wang, Yongchao Liu, Yaoke Wang, Boci Peng, Chuntao Hong, and Siliang Tang. Graphclip: Enhancing transferability in graph foun- dation models for text-attributed graphs. arXiv preprint arXiv:2410.10329, 2024. 21

    arXiv

  92. [100]

    MLLM-pt” de- notes the pretraining of DDT-LLaMA, “MLLM-ft

    Xianwei Zhuang, Yuxin Xie, Yufan Deng, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt: Uni- fied understanding and generation in a visual autoregres- sive multimodal large language model. arXiv preprint arXiv:2501.12327, 2025. 2 13 Generative Multimodal Pretraini...

    2025 arXiv

  93. [2024]

    1, 2, 4, 6, 7, 16, 17

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.