Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A tiny English-trained adapter can make frozen diffusion models generate images from prompts in more than 110 languages.

desk verdict A practical, cheap multilingual adapter worth taking seriously, but the headline parity claim is carried by a self-referential metric and the independent Laion CLIP scores tell a more modest story. read the letter →

arxiv 2412.01271 v2 pith:W3K37USH submitted 2024-12-02 cs.CL cs.AI

Sen Xing , Muyan Zhong , Zeqiang Lai , Liangchen Li , Jiawen Liu , Yaohui Wang , Jifeng Dai , Wenhai Wang This is my paper · ORCID
classification cs.CLcs.AI
keywords multilingualtext-to-imagegenerationdiffusionmodelslanguageadapterzero-shotcross-lingualtransferimage-centeredalignmentEnglish-onlytrainingfrozentextencoderCLIPsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that native multilingual text-to-image generation does not require multilingual training data. The authors argue that when the text encoder has already been aligned across languages by contrastive training on noisy, web-scale image-text pairs, a single lightweight adapter trained only on English image-text pairs can map prompts in more than 110 languages into a frozen image generator's conditioning space. They report CLIP similarity scores of 39.57 for English and 39.61 on average for other languages, with adapter training for Stable Diffusion-class models costing about 12 hours on eight A100 GPUs using 17 million English samples. If true, multilingual capability becomes a cheap bolt-on for existing diffusion models rather than a data-intensive retraining problem.

What carries the argument

The load-bearing object is the image-centered aligned multilingual text encoder, here InternVL-LLaMA, whose contrastive pretraining makes the same meaning expressed in different languages land on nearby vectors. The adapter is a small trainable network of fewer than 20 million parameters, implemented as a one-layer encoder-decoder transformer with learnable queries for Stable Diffusion models, two transformers with an attention pooling layer for SDXL, and a simple MLP for PixArt-α. Its job is to project the frozen encoder's embeddings into the frozen diffusion decoder's conditioning space, so the only learning signal required is English text-to-image pairs. The paper's structural claim is that the encoder's cross-lingual alignment, not the adapter's capacity, is what supplies the multilingual generalization.

What would settle it

Take a set of low-resource languages with non-Latin scripts that are rare in the encoder's pretraining data, translate COCO2014 prompts into them, and measure MuLan's CLIP similarity; if scores collapse toward the as-is baseline or fall below a translation-based pipeline, then the English-trained adapter has not actually transferred to those languages.

Watch

Extended reading notes

Core claim

The central claim is that image-centered multilingual alignment inside the text encoder is what lets an English-only adapter transfer across languages. MuLan freezes both the multilingual text encoder (InternVL-LLaMA, a decoder-only language model trained with next-token prediction and contrastive image-text learning on noisy web data) and the diffusion decoder, and trains only a small language adapter that re-projects text embeddings into the decoder's conditioning space. Because semantically equivalent prompts in different languages already sit close together in the encoder's vector space, the adapter learned from English prompts applies to every language the encoder understands. The paper reports generation quality on 12 mainstream benchmark languages and 85 translated COCO languages that matches or exceeds translation-based pipelines and dedicated multilingual models, at a small fraction of the training cost.

Load-bearing premise

The whole multilingual transfer rests on the premise that the text encoder's internal representations of the same prompt are already nearly identical across languages, a property the paper verifies for only 12 mainstream languages and presumes for the rest of the 110+.

Editorial extensions

If this is right

  • Multilingual text-to-image generation can be added to any compatible frozen diffusion model by training only a sub-20M-parameter adapter on English data.
  • Training cost drops to about 12 hours on eight A100 GPUs for SD-class models and two days for SDXL/PixArt-α, versus thousands of GPU-days for retraining-based multilingual models.
  • Performance on 12 mainstream languages is comparable to translation-based pipelines and exceeds dedicated Chinese and Japanese models on the XM3600 benchmark.
  • Because the base diffusion model is untouched, existing community tools such as LoRA, LCM, ControlNet, and IP-Adapter keep working with multilingual prompts.
  • The 110+ language claim is demonstrated via CLIP scores on translated COCO2014 for 85 languages plus XM3600 for 12, with low-resource languages showing the largest gains over AltDiffusion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method's ceiling is set by the frozen encoder's language coverage: languages the encoder has not seen enough of during contrastive pretraining would receive little or no transfer, so the natural stress test is a set of true low-resource languages with non-Latin scripts.
  • If the claim generalizes, it suggests multilingual generation is mostly a representation-alignment problem, and the same recipe could be applied to other frozen conditional generators such as video, 3D, or audio models.
  • A testable extension is to pair the same adapter design with a different image-centered aligned encoder; if performance holds, the alignment property rather than the specific encoder is the causal factor.
  • The parity in CLIP scores may be partly inherited from the evaluator, since both the generation and the scoring use the same multilingual encoder family; an independent human preference or multilingual VQA evaluation would be a stronger confirmation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. MuLan is a multilingual text-to-image adaptation method. The authors train a lightweight adapter (<20M parameters) on English-only LAION image-text pairs while freezing a multilingual text encoder (InternVL-LLaMA) and the diffusion backbone (SD1.5, SD2.1, SDXL, or PixArt-α). The main claim is that because InternVL-LLaMA was contrastively aligned to images on noisy multilingual web data, the adapter trained in English transfers zero-shot to 110+ languages, with CLIP similarity scores of 39.57 (English) vs 39.61 (other languages), at a training cost of roughly 12 hours on 8 A100 GPUs for SD 1.5/2.1. The paper reports comparisons with translation-based baselines, AltDiffusion, GlueGen, language-specific models, and community tools, plus ablations on dataset size and alignment method.

Significance. If the 110-language parity claim were independently confirmed, MuLan would be a valuable practical contribution: it would let a frozen SD-family model be turned into a multilingual generator at a fraction of the cost of training a multilingual model from scratch, and its plug-and-play compatibility with LoRA, ControlNet, LCM, and IP-Adapter is genuinely useful. The paper also makes a substantive comparison of image-centered vs language-centered alignment and provides a clean data-efficiency ablation (Table 7). The central weakness is that the headline result is produced with the same text-encoder family used inside the model, and the independent Laion CLIP scores in Table 5 do not reproduce the parity; fine-grained VQAScore is only reported for 12 languages. The contribution is therefore plausible but currently not established at the scope claimed.

major comments (3)
  1. [§4.1, Tables 3 and 8, abstract] The headline parity (39.57 English vs 39.61 other languages) is computed with InternVL-LLaMA, the same model family that serves as MuLan's frozen text encoder. Because the adapter is trained to map InternVL text embeddings into the image decoder's conditioning space, InternVL-based CLIP scores can be inflated by this internal alignment and are not an independent measure of image-text alignment. This is not merely a hypothetical concern: the independently computed Laion CLIP scores in Table 5 show Mulan-SD15 at 23.0, essentially tied with AltDiffusion (23.1) and only 0.7 above the SD15(Google) translation baseline (22.3), while Mulan-PixArt (24.2) is below the PixArt(Google) translation baseline (24.3) and below its own English-only score (24.4). In addition, the exact values 39.57/39.61 do not appear in any table, so the computation is not reproducible. Please either (a) report independent multilingual CLIP or VQAScore over the full 85/110-language set and show the parity, or (b) restrict the parity claim to XM12 and state clearly which claims depend on the InternVL metric.
  2. [Table 8] The per-language CLIP scores reported by the authors themselves contradict a strong reading of 'comparable generation capabilities in over 110 languages.' For example, on the COCO2014 validation set, the same InternVL-MuLan-SD15 model scores 38.85 for Chinese, 38.05 for Japanese, and 38.11 for Russian, but 25.82 for Khmer, 27.56 for Irish, 27.95 for Scottish Gaelic, 28.07 for Telugu, and 30.74 for Pashto. A range of roughly 13 points between high- and low-resource languages is not 'comparable' parity; it is substantial degradation. The abstract and conclusion report only an average over languages, which hides this spread. Please report the distribution (or per-language values) for all claimed languages and either set an explicit tolerance for 'comparable' or amend the claim to 'comparable on average, with large low-resource degradation.'
  3. [§3.2 and Appendix A.2] The method's core premise is that InternVL-LLaMA 'maintains a consistent vector space across languages,' and the paper's only direct evidence for this is the t-SNE visualization and the XM12 results. This evidence is thin and, for low-resource languages, Table 8 raises doubts about whether the premise holds. Moreover, the t-SNE description is internally inconsistent: the text says 20 captions were translated into 8 languages (160 inputs), while the Figure 5 caption says 9 prompts in 20 languages. The authors should provide a quantitative alignment check (e.g., cross-lingual text-image retrieval or nearest-neighbor agreement on a broad language sample) and use it to delineate the set of languages for which the English-trained adapter can be expected to transfer.
minor comments (5)
  1. [§A.1 vs §4.1] The data-efficiency ablation says it was run 'without employing any training tricks,' while Section 4.1 lists 10% text-condition dropout and min-SNR weighting; clarify which setting applies to Table 7.
  2. [Table 3] Several GlueGen cells contain '%' placeholders rather than numeric values; please replace them with the actual scores or explain the omission.
  3. [Abstract, §4.1, §5] The language counts should be reconciled: the abstract claims 'over 110 languages,' Table 8 contains 85 COCO languages, and Section 5 says '110 different languages'; please define the exact evaluation set and avoid double counting with XM12.
  4. [§4.2 references] The citations for the translation-based comparison and multilingual-T2I competitors (Sun et al., 2024; Yan et al., 2024) point to papers on 3D shape generation and avatar video, which appear unrelated; please replace them with correct references.
  5. [§4.1] The aesthetic-score threshold of 5.8 used for LAION filtering is a free parameter; please report sensitivity to it or justify the choice.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline cross-lingual parity is scored by MuLan's own frozen text encoder; independent Laion CLIP does not reproduce it.

  1. other [Section 4.1 (Implementation details; Evaluation Metrics) and Section 4.2 (Alternative CLIP for Image-Text Similarity); abstract/Table 3/Table 8/Figure 3]
    "We use InternVL-LLaMA (Chen et al., 2023b) as our text encoder... Regarding evaluation metrics, we employed FID and CLIP Score (SIM) calculated by InternVL-LLaMA (Chen et al., 2023b)."

    InternVL-LLaMA is not an external judge: it is the frozen multilingual text encoder whose embeddings the MuLan adapter is trained to project into the image decoder's conditioning space (Section 3.2, Eq. 4). The abstract's headline parity (39.57 vs. 39.61) and the 110-language results in Figure 3 and Table 8 all use InternVL-LLaMA as the CLIP scorer. Thus the score measures whether a generated image is close, in InternVL's own embedding space, to a prompt embedding that directly conditioned generation. Since InternVL's image-centered multilingual alignment is the paper's stated premise (cited to the authors' own Chen et al. 2023b), cross-lingual score parity largely restates that alignment rather than independently confirming generation quality.

full rationale

The paper's central quantitative claim is the English-vs-other parity (39.57 vs. 39.61) and the 110+ language generalization. That claim is evaluated almost entirely with InternVL-LLaMA CLIP scores, and InternVL-LLaMA is the same model used as MuLan's frozen text encoder. Because the MuLan adapter is trained to re-project InternVL-LLaMA embeddings into the frozen image decoder's conditioning space, and because InternVL-LLaMA's multilingual alignment is the premise of the method, scoring generated images with the same encoder makes the parity partially true by construction. The independent Laion CLIP results (Table 5) do not show the same margin or parity, so the headline result is not externally confirmed. Other reported metrics (FID, VQAScore with GPT-4o, aesthetic scores) are external and provide partial, non-circular support; therefore the paper is not wholly circular, but the headline generalization claim is weakened by the self-referential evaluation.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method rests on properties of a pretrained encoder (InternVL-LLaMA) rather than a mathematical derivation. The key assumptions are the encoder's language-agnostic embedding space, the validity of the CLIP-based metric, and the accuracy of machine-translated evaluation data. No new physical or conceptual entities are introduced.

free parameters (2)
  • LAION aesthetic score threshold = 5.8
    Used to select the training subset of LAION-EN (Section 4.1). The threshold is chosen without stated justification and controls the data distribution the adapter is trained on, so it influences all downstream results.
  • Training steps per backbone = 50k (SD1.5), 100k+1k (SDXL), 118k (PixArt)
    Training durations are set manually in Section 4.1 and may affect convergence and final CLIP scores, but they are not central to the method's logic.
assumptions (3)
  • domain assumption InternVL-LLaMA's embeddings for semantically equivalent prompts in different languages lie close together in a shared vector space, so a single English-trained adapter can map all languages to the decoder's conditioning space.
    Invoked in Section 3.2 ('maintaining a consistent vector space across languages') and is the mechanism by which training only on English transfers to 110+ languages. It is demonstrated only for 8 languages in the t-SNE analysis (Appendix A.2).
  • domain assumption CLIP similarity computed by InternVL-LLaMA is a valid measure of text-image alignment quality for multilingual prompts.
    Used in Section 4.1 as the primary evaluation metric; the paper argues CLIP-ViT is English-biased and InternVL is better, but this substitutes one learned model for another without a human-alignment calibration.
  • domain assumption Google Translate outputs for COCO2014 captions into 85 languages are accurate enough for evaluation.
    Section 4.1: 'We translated the prompts into 85 languages using Google Translate for these datasets.' No human verification is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost." pith.science (2026). https://pith.science/paper/W3K37USH

@misc{pith2026241201271,
  author       = {Pith},
  title        = {Pith review of: MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W3K37USH}},
  note         = {Machine review of arXiv:2412.01271}
}
read the original abstract

In this work, we explore a cost-effective framework for multilingual image generation. We find that, unlike models tuned on high-quality images with multilingual annotations, leveraging text encoders pre-trained on widely available, noisy Internet image-text pairs significantly enhances data efficiency in text-to-image (T2I) generation across multiple languages.Based on this insight, we introduce MuLan, Multi-Language adapter, a lightweight language adapter with fewer than 20M parameters, trained alongside a frozen text encoder and image diffusion model. Compared to previous multilingual T2I models, this framework offers: (1) Cost efficiency. Using readily accessible English data and off-the-shelf multilingual text encoders minimizes the training cost; (2) High performance. Achieving comparable generation capabilities in over 110 languages with CLIP similarity scores nearly matching those in English (39.57 for English vs. 39.61 for other languages); and (3) Broad applicability. Seamlessly integrating with compatible community tools like LoRA, LCM, ControlNet, and IP-Adapter, expanding its potential use cases.

Figures

Figures reproduced from arXiv: 2412.01271 by the authors.

Figure 1
Figure 1. Images generated by MuLan with different backbones, such as Dreamshaper 8, Dreamshaper XL Lightning, and Pixart-Alpha, using a variety of languages or mixed-language prompts. lingual semantic alignment in image generation from the perspective of language and image-text alignment. We also reveal that text encoders trained on large-scale multilingual image-text datasets with noisy data demonstrate remarkable data effi… view at source ↗
Figure 2
Figure 2. Overview of of MuLan. We use a language model as the Multilingual Text Encoder, which has undergone Multilingual Semantic Alignment stage. We only train the Language Adapter, while all other modules are frozen. translation data to align other languages’ vector spaces with the well-supported English vector space. By conducting distillation training with translation data alone, this can be achieved: we designate the S… view at source ↗
Figure 3
Figure 3. Comparison of CLIP Score of InternVL-MuLan￾SD15 and AltDiffusion across hundreds of languages on COCO2014 val. Our model achieved performance comparable to AltDiffusion in mainstream languages while substantially surpass￾ing AltDiffusion in less common languages. Multilingual T2I Comparison. We integrated InternVL￾LLaMA (Chen et al., 2023b) into the adapter’s model, which we call InternVL-MuLan. Specifically, for ea… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Examples of MuLan integrates with community tools. MuLan seamlessly integrates with MVDream (Shi et al., 2023) for 3D model generation and is fully compatible with community tools like LoRA (Hu et al., 2021), ControlNet (Zhang et al., 2023), and LCM (Luo et al., 2023).…
Figure 5
Figure 5. Figure 5: t-SNE analysis on embeddings of 9 prompts in 20 languages produced by 5 text encoders. (a) XLM-RoBERTa-Large (Con￾neau, 2019) (b) CCMatrix pre-trained XLM-RoBERTa-Large (Conneau, 2019) in [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: IP-Adapter Results. Our model enables multilingual style transfer by integrating with the IP-Adapter (Ye et al., 2023b). 16 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Lora Results. Our model can naturally support multilingual input when using LoRA (Hu et al., 2021). Depth-ControlNet Prompt: 스파이더맨 Translation: spiderman Prompt: 蜘蛛侠 Translation: spiderman Prompt: スパイダーマン Translation: spiderman OpenPose-ControlNet Prompt: Un mannequin …
Figure 9
Figure 9. Figure 9: ControlNet Results. Our model can utilize existing ControlNet (Zhang et al., 2023) models, enabling multilingual image generation with conditional inputs such as depth maps and pose images. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. IMAGINE-E: Image Generation Intelligence Evaluation of State-of-the-art Text-to-Image Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A new evaluation suite finds that CLIPScore, HPSv2, and Aesthetic Score misjudge challenging text-to-image outputs, while GPT-4o and human ratings favor FLUX.1 and Ideogram2.0.

Reference graph

Works this paper leans on

52 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Kandinsky 3.0 technical report

    Arkhipkin, V., Filatov, A., Vasilev, V., Maltseva, A., Azizov, S., Pavlov, I., Agafonova, J., Kuznetsov, A., and Dimitrov, D. Kandinsky 3.0 technical report. arXiv preprint arXiv:2312.03511, 2023

    arXiv 2023

  3. [3]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  4. [4]

    Cross-lingual and multilingual clip

    Carlsson, F., Eisen, P., Rekathati, F., and Sahlgren, M. Cross-lingual and multilingual clip. In Proceedings of the Language Resources and Evaluation Conference, pp.\ 6848--6854, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthology.org/2022.lrec-1.739

    work page 2022

  5. [5]

    Pixart- : Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023 a

    Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., and Li, Z. Pixart- : Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023 a

    work page 2023

  6. [6]

    Altclip: Altering the language encoder in clip for extended language capabilities

    Chen, Z., Liu, G., Zhang, B.-W., Ye, F., Yang, Q., and Wu, L. Altclip: Altering the language encoder in clip for extended language capabilities. 2022. doi:10.48550/ARXIV.2211.06679. URL https://arxiv.org/abs/2211.06679

    arXiv doi:10.48550/arxiv.2211.06679 2022

  7. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Muyan, Z., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023 b

    arXiv 2023

  8. [8]

    Unsupervised cross-lingual representation learning at scale

    Conneau, A. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019

    arXiv 1911

Show all 52 references
  1. [9]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/abs/1810.04805

    2019 arXiv

  2. [10]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., et al. The llama 3 herd of models. ArXiv, abs/2407.21783, 2024. URL https://api.semanticscholar.org/CorpusID:271571434

    2024 arXiv

  3. [11]

    Scaling rectified flow transformers for high-resolution image synthesis

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., M \"u ller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206, 2024

    2024 arXiv

  4. [12]

    Efficient diffusion training via min-snr weighting strategy

    Hang, T., Gu, S., Li, C., Bao, J., Chen, D., Hu, H., Geng, X., and Guo, B. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 7441--7451, 2023

    2023

  5. [13]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  6. [14]

    V., Sung, Y., Li, Z., and Duerig, T

    Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision, 2021. URL https://arxiv.org/abs/2102.05918

    2021 arXiv

  7. [15]

    Clip-vit-h-14-frozen-xlm-roberta-large-laion5b-s13b-b90k, 2023

    LAION. Clip-vit-h-14-frozen-xlm-roberta-large-laion5b-s13b-b90k, 2023. Accessed: 2024-11-15

    2023

  8. [16]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019

    Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019. URL https://arxiv.org/abs/1910.13461

    2019 arXiv

  9. [17]

    Playground v2

    Li, D., Kamko, A., Akhgari, E., Sabet, A., Xu, L., and Doshi, S. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245, 2024 a

    2024 arXiv

  10. [18]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding, 2024 b

    Li, Z., Zhang, J., Lin, Q., Xiong, J., Long, Y., Deng, X., Zhang, Y., Liu, X., Huang, M., Xiao, Z., Chen, D., He, J., Li, J., Li, W., Zhang, C., Quan, R., Lu, J., Huang, J., Yuan, X., Zheng, X., Li, Y., Zhang, J., Zhang, C., Chen, M., Liu, J., Fang, Z., Wang, W., Xue, J., Tao,...

    2024

  11. [19]

    L., and Dollár, P

    Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., and Dollár, P. Microsoft coco: Common objects in context, 2015. URL https://arxiv.org/abs/1405.0312

    2015 arXiv

  12. [20]

    Evaluating text-to-visual generation with image-to-text generation

    Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., and Ramanan, D. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, 2024. URL https://api.semanticscholar.org/CorpusID:268857167

    2024

  13. [21]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  14. [22]

    Pangu-draw: Advancing resource-efficient text-to-image synthesis with time-decoupled training and reusable coop-diffusion

    Lu, G., Guo, Y., Han, J., Niu, M., Zeng, Y., Xu, S., Huang, Z., Zhong, Z., Zhang, W., and Xu, H. Pangu-draw: Advancing resource-efficient text-to-image synthesis with time-decoupled training and reusable coop-diffusion. arXiv preprint arXiv:2312.16486, 2023

    2023 arXiv

  15. [23]

    Latent consistency models: Synthesizing high-resolution images with few-step inference

    Luo, S., Tan, Y., Huang, L., Li, J., and Zhao, H. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023

    2023 arXiv

  16. [24]

    Pea-diffusion: Parameter-efficient adapter with knowledge distillation in non-english text-to-image generation

    Ma, J., Chen, C., Xie, Q., and Lu, H. Pea-diffusion: Parameter-efficient adapter with knowledge distillation in non-english text-to-image generation. arXiv preprint arXiv:2311.17086, 2023

    2023 arXiv

  17. [25]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022

    Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022. URL https://arxiv.org/abs/2112.10741

    2022 arXiv

  18. [26]

    NLLB Team , Costa-jussà, M. R., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., Sun, A., Wang, S., Wenzek, G., Youngblood, A., Akula, B., Barrault, L., Mejia-Gonzalez, G., Hansanti, P., Hoffman, J., Jarrett, S., ...

    2022

  19. [27]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  20. [28]

    Journeydb: A benchmark for generative image understanding, 2023

    Pan, J., Sun, K., Ge, Y., Li, H., Duan, H., Wu, X., Zhang, R., Zhou, A., Qin, Z., Wang, Y., Dai, J., Qiao, Y., and Li, H. Journeydb: A benchmark for generative image understanding, 2023

    2023

  21. [29]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., M \"u ller, J., Penna, J., and Rombach, R. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  22. [30]

    Gluegen: Plug and play multi-modal encoders for x-to-image generation

    Qin, C., Yu, N., Xing, C., Zhang, S., Chen, Z., Ermon, S., Fu, Y., Xiong, C., and Xu, R. Gluegen: Plug and play multi-modal encoders for x-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 23085--23096, 2023

    2023

  23. [31]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PMLR, 2021

    2021

  24. [32]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  25. [33]

    Zero-shot text-to-image generation

    Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp.\ 8821--8831. PMLR, 2021

    2021

  26. [34]

    Hierarchical text-conditional image generation with clip latents

    Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022

    2022 arXiv

  27. [35]

    High-resolution image synthesis with latent diffusion models

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 10684--10695, 2022

    2022

  28. [36]

    Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022

    2022 arXiv

  29. [37]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022

    arXiv

  30. [38]

    Ccmatrix: Mining billions of high-quality parallel sentences on the web, 2020

    Schwenk, H., Wenzek, G., Edunov, S., Grave, E., and Joulin, A. Ccmatrix: Mining billions of high-quality parallel sentences on the web, 2020. URL https://arxiv.org/abs/1911.04944

    2020 arXiv

  31. [39]

    Mvdream: Multi-view diffusion for 3d generation

    Shi, Y., Wang, P., Ye, J., Mai, L., Li, K., and Yang, X. Mvdream: Multi-view diffusion for 3d generation. arXiv:2308.16512, 2023

    2023 arXiv

  32. [40]

    and Akiba, T

    Shing, M. and Akiba, T. Japanese stable diffusion xl. URL [https://huggingface.co/stabilityai/japanese-stable-diffusion-xl](https://huggingface.co/stabilityai/japanese-stable-diffusion-xl)

  33. [41]

    Recent advances in implicit representation-based 3d shape generation

    Sun, J.-M., Wu, T., and Gao, L. Recent advances in implicit representation-based 3d shape generation. Visual Intelligence, 2 0 (1): 0 9, Mar 2024. ISSN 2731-9008. doi:10.1007/s44267-024-00042-1. URL https://doi.org/10.1007/s44267-024-00042-1

    2024 doi

  34. [42]

    Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis

    Team, K. Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis. arXiv preprint, 2024

    2024

  35. [43]

    V., Pont-Tuset, J., Chen, X., and Soricut, R

    Thapliyal, A. V., Pont-Tuset, J., Chen, X., and Soricut, R. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. ArXiv, abs/2205.12522, 2022. URL https://api.semanticscholar.org/CorpusID:249062751

    2022 arXiv

  36. [44]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  37. [45]

    and Hinton, G

    van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9 0 (Nov): 0 2579--2605, 2008

    2008

  38. [46]

    Taiyi-diffusion-xl: Advancing bilingual text-to-image generation with large vision-language model support

    Wu, X., Zhang, D., Gan, R., Lu, J., Wu, Z., Sun, R., Zhang, J., Zhang, P., and Song, Y. Taiyi-diffusion-xl: Advancing bilingual text-to-image generation with large vision-language model support. arXiv preprint arXiv:2401.14688, 2024

    2024 arXiv

  39. [47]

    mt5: A massively multilingual pre-trained text-to-text transformer, 2021

    Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. mt5: A massively multilingual pre-trained text-to-text transformer, 2021. URL https://arxiv.org/abs/2010.11934

    2021 arXiv

  40. [48]

    Dialoguenerf: towards realistic avatar face-to-face conversation video generation

    Yan, Y., Zhou, Z., Wang, Z., Gao, J., and Yang, X. Dialoguenerf: towards realistic avatar face-to-face conversation video generation. Visual Intelligence, 2 0 (1): 0 24, Aug 2024. ISSN 2731-9008. doi:10.1007/s44267-024-00057-8. URL https://doi.org/10.1007/s44267-024-00057-8

    2024 doi

  41. [49]

    Altdiffusion: A multilingual text-to-image diffusion model

    Ye, F., Liu, G., Wu, X., and Wu, L. Altdiffusion: A multilingual text-to-image diffusion model. arXiv preprint arXiv:2308.09991, 2023 a

    2023 arXiv

  42. [50]

    Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models

    Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023 b

    2023

  43. [51]

    Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence

    Zhang, J., Gan, R., Wang, J., Zhang, Y., Zhang, L., Yang, P., Gao, X., Wu, Z., Dong, X., He, J., Zhuo, J., Yang, Q., Huang, Y., Li, X., Wu, Y., Lu, J., Zhu, X., Chen, W., Han, T., Pan, K., Wang, R., Wang, H., Wu, X., Zeng, Z., and Chen, C. Fengshenbang 1.0: Being the foundatio...

    2022 arXiv

  44. [52]

    Adding conditional control to text-to-image diffusion models

    Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 3836--3847, 2023

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.