Pith. sign in

REVIEW 5 major objections 5 minor 48 references

NeoBabel: A Multilingual Open Tower for Visual Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NeoBabel is a 2-billion-parameter model that generates images directly from prompts in six languages, bypassing the translation stage that prior systems require.

desk verdict A genuinely useful open multilingual text-to-image package whose headline SOTA claim is not yet proven because the evaluation lacks a translation-first baseline and reports no uncertainty. read the letter →

arxiv 2507.06137 v1 pith:NYLLVVGK submitted 2025-07-08 cs.CL cs.AIcs.CV

Mohammad Mahdi Derakhshani , Dheeraj Varghese , Marzieh Fadaee , Cees G. M. Snoek This is my paper
classification cs.CLcs.AIcs.CV
keywords multilingualtext-to-imagegenerationdiscretediffusionmaskedimagetokenpredictionmachinetranslationpivotcross-lingualgeneralizationcode-switchedpromptsmodelmergingbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NeoBabel is a 2-billion-parameter text-to-image model that turns prompts in English, Chinese, Dutch, French, Hindi, and Persian directly into pictures, with no translation step at generation time. The paper's central claim is that a model trained on 124 million captions machine-translated into six languages can match the English performance of systems two to four times its size while beating them on multilingual benchmarks, scoring 0.75 on m-GenEval and 0.68 on m-DPG, gains of +0.11 and +0.09 over the strongest prior system. If correct, this makes translation-free multilingual image generation practical at small scale, eliminating the latency, memory overhead, and meaning loss of translate-then-generate pipelines. The paper also contributes multilingual extensions of two standard benchmarks and two metrics, CLC and CSS, for measuring cross-lingual consistency and robustness to code-mixed prompts.

What carries the argument

The engine is a unified multimodal transformer built on the Gemma-2 multilingual LLM, whose embedding table is extended with 8,192 discrete image-token embeddings from a MAGVIT-v2 visual quantizer; text tokens receive causal attention while image tokens receive full bidirectional attention, and the only training loss is a masked-token discrete-diffusion loss over image tokens, following the Show-o recipe. The other half of the machinery is data: a curation pipeline that recaptions 39 million images with InternVL, filters them for length, language, visual-text alignment, and safety, and translates the captions into five languages with NLLB for pretraining and Gemini for instruction tuning, yielding 124 million multilingual pairs. A five-stage curriculum, three pretraining stages ascending from class labels to large-scale and then curated multilingual data, followed by two instruction-tuning stages at $512\times512$ resolution, plus averaging of twenty checkpoints, carries the cross-lingual generalization reported in the results.

What would settle it

Collect natively written prompts in the five non-English languages covering culture-specific concepts, such as Dutch gezellig, Persian Nowruz imagery, or Hindi diya lamps, and compare NeoBabel's images to those it produces from machine-translated versions of the same prompts; if the two sets are indistinguishable under the paper's own CLC embeddings, the model has learned translationese rather than cultural grounding. A quantitative companion test is to have professional translators recreate the m-GenEval prompts and re-run the benchmark, asking whether the +0.11 margin over BLIP3-o survives.

Watch

Extended reading notes

Core claim

The central discovery claim is that multilingual capability is not a trade-off: training one unified 2B model directly on translated captions yields top scores on multilingual extensions of GenEval and DPG-Bench while preserving English-level performance (0.83 on English GenEval, on par with BLIP3-o 8B). Across the five non-English languages NeoBabel outperforms all baselines, most sharply in low-resource languages, where its Hindi and Persian scores exceed larger models by up to 0.3, and it does so while processing prompts 2.8$\times$ faster and with 59% less memory than a translation-then-generation pipeline. The authors present this as evidence that targeted multilingual alignment training, rather than parameter count or a multilingual base LLM, is what produces cross-lingual generalization in image generation.

Load-bearing premise

The model's entire non-English knowledge comes from machine-translated captions, so the claim of native, culturally faithful multilingual generation stands or falls on those translations being faithful and culturally appropriate rather than English-shaped translationese.

Editorial extensions

If this is right

  • Multilingual image generation no longer needs a translation model in front of the generator: NeoBabel reports 2.8$\times$ faster prompt processing and 59% lower memory than translate-then-generate systems, which is what makes multilingual support practical for real-time deployment.
  • The released m-GenEval and m-DPG benchmarks, together with the CLC and CSS metrics, give researchers standardized tools for comparing multilingual generators instead of evaluating only English prompts.
  • Because the same architecture processes text and image tokens in one sequence, the model also performs multilingual inpainting and extrapolation without extra fine-tuning, enabling collaborative native-language editing of a single image.
  • Multilingual alignment training appears to confer robustness beyond language coverage: NeoBabel posts the highest cross-lingual consistency (0.79/0.61 under EVA-CLIP/DINOv2) and the highest code-switch similarity scores, with lower variance than larger baselines.
  • The open release of weights, code, and the 124M-pair dataset means the recipe and its reported efficiency gains can be reproduced and extended by others.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same translate-then-pretrain recipe, with NLLB as the translator, could be lifted to additional languages, to video generation, or to the visual-question-answering tasks the paper lists as future work; nothing in the pipeline is specific to the six chosen languages.
  • The paper's cultural-fidelity framing outruns its evidence: because every non-English caption is machine-translated from English, the cultural claim would be established only by showing that culture-specific terms generate images distinct from their English glosses, a test the paper does not run.
  • The CLC metric cuts both ways: near-perfect cross-lingual consistency is exactly what a model would show if it collapsed every language onto a shared English-derived concept space, so high consistency alone does not prove native multilingual understanding.
  • A testable extension would be to train the identical pipeline on a subset of human-written native captions instead of translations; if scores then rise on m-DPG's natural prompts, the remaining gap measures exactly how much translationese the current data pipeline leaves in the model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. NeoBabel is a 2B-parameter multilingual text-to-image model built on Gemma-2 with a Show-o-style discrete diffusion transformer, trained in three pretraining and two instruction-tuning stages on 124M image-text pairs. The English captions are generated by InternVL and then machine-translated into Chinese, Dutch, French, Hindi, and Persian using NLLB for pretraining and Gemini for instruction tuning. The authors translate GenEval and DPG-Bench into m-GenEval and m-DPG, propose two new metrics (Cross-Lingual Consistency and Code-Switching Similarity), and report that NeoBabel scores 0.75 on m-GenEval and 0.68 on m-DPG, outperforming larger models such as BLIP3-o 8B on non-English prompts while matching English-only GenEval/DPG performance. The paper releases code, checkpoints, datasets, and multilingual evaluation protocols.

Significance. If the claims hold, this is an important result: a 2B model trained mostly on machine-translated captions could match much larger English-trained unified models and beat them on translated benchmarks, while also providing open data and benchmarks that would help the community. The released toolkit, the 124M-pair dataset, the translated benchmarks, and the standardized evaluation scripts are concrete and valuable assets. However, the headline comparisons currently rest on single-pass evaluations, a missing translation-first baseline, and benchmarks/metrics whose construction overlaps with the training signal. The scientific significance is therefore conditional until these load-bearing points are addressed; the contribution would be strengthened substantially by a translation-first baseline, seeded evaluations, and an independently authored native-language validation set.

major comments (5)
  1. [§6, Tables 2 and 3, Figure 3] No translation-first baseline is run. The paper's motivation is that translation pipelines introduce semantic drift, computational overhead, and cultural misalignment, and the contributions list claims 2.8x faster processing and 59% lower memory versus translation-then-generation pipelines. Yet no experiment translates the m-GenEval or m-DPG prompts back to English with NLLB or Gemini and then generates with a strong English-only model such as Show-o, BLIP3-o 8B, or SD3. Because NeoBabel's non-English supervision is machine translation of English captions (Section 3.1) and the benchmarks are translated with Gemini (Section 5.1), a translate-then-generate baseline could match or exceed NeoBabel's multilingual scores simply by matching the machine-translation distribution. This baseline is load-bearing for the claim that translation-free generation outperforms translation pipelines; without it the central comparison is incomplete.
  2. [§5.1 and §3.1] The multilingual evaluation shares its translation origin with the training data. m-GenEval and m-DPG are produced by translating English prompts with Gemini, and the instruction-tuning data are also translated with Gemini (gemini-2.0-flash-lite), while pretraining uses NLLB. Consequently, high scores on these benchmarks may reflect exposure to translationese rather than native multilingual comprehension. The paper's cultural-fidelity claim is not tested by any prompt written natively in the target languages, nor by human judgments of cultural appropriateness. I request either an independently authored or natively written evaluation set, or a human evaluation on culturally grounded prompts, together with a discussion of translationese effects on the reported numbers.
  3. [Tables 2–6] All headline numbers are single-pass scores with no seeds, confidence intervals, or per-language sample counts. For a generative model, especially at 2B scale, differences of +0.11 and +0.09 between systems can fall within run-to-run variance, particularly on translated prompts where machine translation may introduce ambiguity. Reporting multiple seeds, or at least bootstrapped confidence intervals over the prompt set, is necessary to support the state-of-the-art claim and to make the cross-model comparisons in Tables 2, 3, 5, and 6 interpretable.
  4. [§5.2, Eqs. (6)–(8)] CLC and CSS can be high for a prompt-ignoring generator. CLC measures visual similarity between images generated from English and translated prompts, and CSS measures similarity between code-switched and English-reference images; a model that produces nearly identical images irrespective of the textual input would score well on both metrics. Without a null or control baseline (for example, a prompt-agnostic generation condition, a fixed-image baseline, or a model evaluated on scrambled prompts), and without reporting per-language generation quality alongside CLC and CSS, the introduced metrics do not yet establish cross-lingual grounding.
  5. [§7.3, Table 4] The final model's merging strategy is selected on English benchmark scores. Table 4 is explicitly an English-prompt ablation (its caption states 'This ablation uses English prompts'), and SMA is chosen as the best strategy; the same merged checkpoint then produces the headline multilingual m-GenEval and m-DPG results. Because no multilingual validation split or final-model selection protocol is described, the multilingual gains could partly reflect selection on the English variant of the same benchmark. Please report multilingual results for the last checkpoint versus the merged model, or select the merge strategy on a held-out multilingual set.
minor comments (5)
  1. [§3.2] There is a typo, 'obtian' for 'obtain', and Table 1 says all modified datasets are prefixed with 'm-', but m-CC12M and m-SA-1B are recaptioned English-only datasets and are not translated, so the prefix convention is inconsistent.
  2. [Appendix A, Table 7] The hyperparameter table omits batch size, gradient accumulation steps, GPU count, and number of epochs per dataset, which makes exact training cost and reproduction difficult despite the otherwise useful release.
  3. [§1, contribution list] The claim of 2.8x faster inference and 59% lower memory is not accompanied by a measurement protocol (hardware, translation model, batch size, image resolution, or number of samples); please report these details.
  4. [§5.2, Eqs. (7)–(8)] The EF/ES notation is under-specified; the paper should define how the 'first half' of a prompt is determined for prompts with odd token counts, and clarify whether CSS is averaged per language before the overall average is computed.
  5. [§5.1] The statement that translations are 'followed by human verification and manual corrections' needs detail: how many prompts per language, how many annotators, and what inter-annotator agreement was reached; otherwise the benchmark validity is hard to assess.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the multilingual benchmark scores are empirical measurements rather than reductions to training inputs or self-citation chains.

full rationale

NeoBabel's central claims are empirical: a model is trained and then evaluated on benchmark suites, so no equation defines a predicted quantity in terms of a fitted parameter and no load-bearing claim rests on a self-citation. The nearest concern is that the non-English training captions are NLLB/Gemini translations of English captions (Section 3.1) while the m-GenEval and m-DPG prompts are also Gemini translations (Section 5.1), creating distribution overlap between training and evaluation, and no translation-first baseline is reported. That is a legitimate external-validity limitation for the 'native multilingual/cultural fidelity' interpretation, but it is not circularity: the benchmark scores are held-out measurements on translated versions of established GenEval/DPG tasks, all baselines are scored on the same translated prompts, and the final model-merging choice (SMA) is selected on English prompts in Section 7.3 rather than on the multilingual benchmarks. The introduced CLC and CSS metrics are measurements of trained behavior, not quantities that the paper derives from its own outputs by definition. No quoted equation or construction makes a claimed result equivalent to an input, so no circular step is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. m-GenEval, m-DPG, CLC, and CSS are evaluation constructs rather than invented mechanisms. The main uncharged assumptions are the quality of machine-translated captions and the validity of embedding-based consistency metrics.

free parameters (4)
  • Instruction-tuning dataset mixture weights = Stage 1: 60/30/10; Stage 2: 25/60/15 for m-LAION / m-JourneyDB / m-BLIP3o-Instruct
    Chosen by hand without a sweep, as acknowledged in Section 7.2; the large instruction-tuning gains depend on these ratios.
  • Model merging strategy and weights = SMA equal weights; WMA w_i=i; EMA decay alpha unspecified
    The merging strategy was selected after comparing English m-GenEval/m-DPG scores (Section 7.3, Table 4), so the headline result partly depends on this post-hoc selection.
  • Caption filtering thresholds = Length 5-500 tokens; fastText language confidence above 90%; VQAScore threshold not stated
    These thresholds shape the 124M pretraining set, are not tuned systematically, and the VQAScore cutoff is not reported, making the data curation partly ad hoc.
  • Resolution and sequence-length schedule = Sequence length 128 to 512; image resolution 256x256 to 512x512
    The staged schedule is chosen by hand and affects both cost and final quality, though it is a standard progressive-training choice.
assumptions (5)
  • domain assumption Gemma-2's tokenizer gives adequate coverage of all six languages, including Hindi and Persian, without modification.
    Invoked in Section 2.1.1; if tokenization fragments low-resource scripts, multilingual gains would be limited.
  • domain assumption Machine-translated captions preserve semantics and cultural fidelity across the five non-English languages.
    Invoked throughout Section 3.1; the entire non-English training signal is translated captions.
  • domain assumption Human-verified Gemini translations of GenEval and DPG prompts are faithful enough to serve as ground-truth multilingual benchmarks.
    Invoked in Section 5.1; benchmark scores are only as valid as these translations.
  • domain assumption Cosine similarity in EVA-CLIP and DINOv2 embedding spaces measures semantic and visual equivalence across languages for CLC and CSS.
    Invoked in Section 5.2; no validation against human judgments is reported, and high consistency can be achieved by prompt-agnostic models.
  • standard math Masked discrete diffusion objective from Show-o is a suitable objective for text-to-image generation.
    Adopted without modification from prior work (Xie et al., 2025b); treated as a valid background result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NeoBabel: A Multilingual Open Tower for Visual Generation." pith.science (2026). https://pith.science/paper/NYLLVVGK

@misc{pith2026250706137,
  author       = {Pith},
  title        = {Pith review of: NeoBabel: A Multilingual Open Tower for Visual Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NYLLVVGK}},
  note         = {Machine review of arXiv:2507.06137}
}
read the original abstract

Text-to-image generation advancements have been predominantly English-centric, creating barriers for non-English speakers and perpetuating digital inequities. While existing systems rely on translation pipelines, these introduce semantic drift, computational overhead, and cultural misalignment. We introduce NeoBabel, a novel multilingual image generation framework that sets a new Pareto frontier in performance, efficiency and inclusivity, supporting six languages: English, Chinese, Dutch, French, Hindi, and Persian. The model is trained using a combination of large-scale multilingual pretraining and high-resolution instruction tuning. To evaluate its capabilities, we expand two English-only benchmarks to multilingual equivalents: m-GenEval and m-DPG. NeoBabel achieves state-of-the-art multilingual performance while retaining strong English capability, scoring 0.75 on m-GenEval and 0.68 on m-DPG. Notably, it performs on par with leading models on English tasks while outperforming them by +0.11 and +0.09 on multilingual benchmarks, even though these models are built on multilingual base LLMs. This demonstrates the effectiveness of our targeted alignment training for preserving and extending crosslingual generalization. We further introduce two new metrics to rigorously assess multilingual alignment and robustness to code-mixed prompts. Notably, NeoBabel matches or exceeds English-only models while being 2-4x smaller. We release an open toolkit, including all code, model checkpoints, a curated dataset of 124M multilingual text-image pairs, and standardized multilingual evaluation protocols, to advance inclusive AI research. Our work demonstrates that multilingual capability is not a trade-off but a catalyst for improved robustness, efficiency, and cultural fidelity in generative AI.

Figures

Figures reproduced from arXiv: 2507.06137 by the authors.

Figure 1
Figure 1. NeoBabel establishes a new Pareto frontier in multilingual image generation performance, efficiency, and inclusivity. Left: GenEval English-only scores show that NeoBabel matches state-of-the-art models despite being 2–4× smaller. Right: On our multilingual benchmark extensions, m-GenEval and m-DPG, NeoBabel outperforms the second-best model, demonstrating strong multilingual generalization. NeoBabel is fully open (… view at source ↗
Figure 2
Figure 2. NeoBabel: A Multilingual Open Tower for Visual Generation. Regardless of modality, all input data is first tokenized and embedded into a unified input sequence. NeoBabel then applies causal attention to text tokens and full attention within a discrete denoising diffusion framework for image tokens, ultimately generating the desired image. This design enables NeoBabel to support a wide range of tasks, including text-… view at source ↗
Figure 3
Figure 3. m-GenEval benchmark comparison. Models such as Janus Pro and BLIP3-o rely on multilingual base LLMs but are trained solely on English image-generation data, leading to a sharp performance drop in non-English languages. In contrast, NeoBabel maintains strong and consistent results across all six languages, demonstrating robust cross-lingual generalization. Here baseline models are ordered by parameter count. designed… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Qualitative evaluation of NeoBabel. Each row is based on a single concept expressed in six different languages. For clarity, we show only one of the prompts (in one language) and present six images generated from its translated prompts in the other five languages. Acro…
Figure 5
Figure 5. Figure 5: Qualitative evaluation of NeoBabel. Each row is based on a single concept expressed in six different languages. For clarity, we show only one of the prompts (in one language) and present six images generated from its translated prompts in the other five languages. No m…
Figure 6
Figure 6. Figure 6: Multilingual image inpainting. NeoBabel supports multilingual text-guided image inpainting, highlighting its potential for interactive and language-inclusive visual editing across diverse user groups. 7.1 Effect of Progressive Pretraining We first analyze the impact of…
Figure 7
Figure 7. Figure 7: Multilingual image extrapolation. NeoBabel successfully performs text-guided image extrapola￾tion using multilingual prompts. Given the middle image and two different multilingual prompts (for the left and right extensions), NeoBabel generates coherent visual completio…
Figure 8
Figure 8. Figure 8: Cross-Lingual Prompt Generation. Examples of code-switched prompts mixing three languages, along with images generated by NeoBabel. Top: English, Dutch and French. Bottom: Hindi, Persian and Chinese. English translations are shown below each prompt for reader convenien…
Figure 9
Figure 9. Figure 9: Effect of Progressive Pretraining and Instruction Tuning. Performance on m-GenEval (top) and m-DPG (bottom) improves steadily across pretraining and instruction tuning stages. Pretraining at 256×256 yields significant gains—especially on m-DPG—when scaling to large mul…
Figure 10
Figure 10. Figure 10: Cross-Lingual Consistency (CLC) Score Distributions across Models. We show the distribution of CLC scores computed using EVA-CLIP (left column) and DINOv2 (right column), where higher values reflect greater consistency across languages. EVA-CLIP captures semantic simi…
Figure 11
Figure 11. Figure 11: Variation in Code Switching Similarity (CSS) Scores across Models. We report CSS scores for code-mixed prompts under two settings: English-first (left column) and English-second (right column), using EVA-CLIP (top row) and DINOv2 (bottom row) as backbones. Higher scor…
Figure 12
Figure 12. Figure 12: Qualitative Evaluation of NeoBabel. Each row corresponds to a single concept expressed in six different languages: English, Chinese, Dutch, French, Hindi, and Persian. Although prompts are not shown for readability, all images were generated using translated versions …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 5 canonical work pages

  1. [1]

    Maya: 25 An instruction finetuned multilingual multimodal model.arXiv preprint arXiv:2412.07112,

    Nahid Alam, Karthik Reddy Kanjula, Surya Guthikonda, Timothy Chung, Bala Krishna S Vegesna, AbhipshaDas, AnthonySusevski, RyanSze-YinChan, SMUddin, ShayekhBinIslam, etal. Maya: 25 An instruction finetuned multilingual multimodal model.arXiv preprint arXiv:2412.07112,

    arXiv

  2. [4]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923,

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report.ar...

    arXiv

  3. [10]

    Unveiling encoder-free vision-language models.arXiv preprint arXiv:2406.11832,

    Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision-language models.arXiv preprint arXiv:2406.11832,

    arXiv

  4. [11]

    Evev2: Improved baselines for encoder-free vision-language models.arXiv preprint arXiv:2502.06788,

    Haiwen Diao, Xiaotong Li, Yufeng Cui, Yueze Wang, Haoge Deng, Ting Pan, Wenxuan Wang, Huchuan Lu, and Xinlong Wang. Evev2: Improved baselines for encoder-free vision-language models.arXiv preprint arXiv:2502.06788,

    arXiv

  5. [12]

    Unified autoregressive visual generation and understanding with continuous tokens.arXiv preprint arXiv:2503.13436,

    Lijie Fan, Luming Tang, Siyang Qin, Tianhong Li, Xuan Yang, Siyuan Qiao, Andreas Steiner, Chen Sun, Yuanzhen Li, Tao Zhu, et al. Unified autoregressive visual generation and understanding with continuous tokens.arXiv preprint arXiv:2503.13436,

    arXiv

  6. [13]

    Multilingual text-to-image generation magnifies gender stereotypes and prompt engineering may not help you.arXiv preprint arXiv:2401.16092,

    27 Felix Friedrich, Katharina Hammerl, Patrick Schramowski, Manuel Brack, Jindrich Libovicky, Kris- tian Kersting, and Alexander Fraser. Multilingual text-to-image generation magnifies gender stereotypes and prompt engineering may not help you.arXiv preprint arXiv:2401.16092,

    arXiv

  7. [14]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396,

    arXiv

  8. [15]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    arXiv

Show all 48 references
  1. [16]

    Query-key normalization for transformers.arXiv preprint arXiv:2010.04245,

    Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers.arXiv preprint arXiv:2010.04245,

    2010 arXiv

  2. [17]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598,

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598,

    arXiv

  3. [18]

    Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135,

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135,

    arXiv

  4. [19]

    Can machine trans- lation bridge multilingual pretraining and cross-lingual transfer learning?arXiv preprint arXiv:2403.16777,

    Shaoxiong Ji, Timothee Mickus, Vincent Segonne, and Jörg Tiedemann. Can machine trans- lation bridge multilingual pretraining and cross-lingual transfer learning?arXiv preprint arXiv:2403.16777,

    arXiv

  5. [20]

    Uni- token: Harmonizing multimodal understanding and generation through unified visual encoding

    Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Uni- token: Harmonizing multimodal understanding and generation through unified visual encoding. arXiv preprint arXiv:2504.04423,

    arXiv

  6. [21]

    Fasttext.zip: Compressing text classification models.arXiv preprint arXiv:1612.03651,

    Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models.arXiv preprint arXiv:1612.03651,

    arXiv

  7. [22]

    Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125,

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125,

    arXiv

  8. [23]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024a

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024a. Haopeng Li, Jinyue Yang, Guoqi Li, and Huan Wang. Autoregressive image gen...

    arXiv

  9. [24]

    World model on million-length video and language with blockwise ringattention.arXiv preprint arXiv:2402.08268, 2024a

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention.arXiv preprint arXiv:2402.08268, 2024a. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances on Neural In...

    arXiv

  10. [25]

    Unitok: A unified tokenizer for visual generation and understanding.arXiv preprint arXiv:2502.20321,

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xi- aojuan Qi. Unitok: A unified tokenizer for visual generation and understanding.arXiv preprint arXiv:2502.20321,

  11. [26]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193,

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193,

    arXiv

  12. [27]

    Transfer between modalities with metaqueries.arXiv preprint arXiv:2504.06256,

    Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Ji Hou, and Saining Xie. Transfer between modalities with metaqueries.arXiv preprint arXiv:2504.06256,

    arXiv

  13. [28]

    Freeman, and Yu-Xiong Wang

    29 Ziqi Pang, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T. Freeman, and Yu-Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders. arXiv preprint arXiv:2412.01827,

    arXiv

  14. [29]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952,

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952,

    arXiv

  15. [30]

    Lumina-image 2.0: A unified and efficient image generative framework.arXiv preprint arXiv:2503.21758,

    Qi Qin, Le Zhuo, Yi Xin, Ruoyi Du, Zhen Li, Bin Fu, Yiting Lu, Jiakang Yuan, Xinyue Li, Dongyang Liu, et al. Lumina-image 2.0: A unified and efficient image generative framework.arXiv preprint arXiv:2503.21758,

    arXiv

  16. [31]

    Hierarchical text- conditional image generation with clip latents.arXiv preprint arXiv:2204.06125,

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents.arXiv preprint arXiv:2204.06125,

    arXiv

  17. [32]

    Kaleidoscope: In-language exams for massively multilingual vision evaluation.arXiv preprint arXiv:2504.07072,

    Israfel Salazar, Manuel Fernández Burda, Shayekh Bin Islam, Arshia Soltani Moakhar, Shivalika Singh, Fabian Farestam, Angelika Romanou, Danylo Boiko, Dipika Khullar, Mike Zhang, et al. Kaleidoscope: In-language exams for massively multilingual vision evaluation.arXiv preprint ...

    arXiv

  18. [33]

    Seaweed-7b: Cost-effective training of video generation foundation model.arXiv preprint arXiv:2504.08685,

    Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al. Seaweed-7b: Cost-effective training of video generation foundation model.arXiv preprint arXiv:2504.08685,

    arXiv

  19. [34]

    A post-trainer’s guide to multilingual training data: Uncovering cross-lingual transfer dynamics.arXiv preprint arXiv:2504.16677,

    Luisa Shimabucoro, Ahmet Ustun, Marzieh Fadaee, and Sebastian Ruder. A post-trainer’s guide to multilingual training data: Uncovering cross-lingual transfer dynamics.arXiv preprint arXiv:2504.16677,

    arXiv

  20. [35]

    Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evalu- ation.arXiv preprint arXiv:2412.03304,

    30 Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, et al. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingu...

    arXiv

  21. [36]

    Dualtoken: Towards unifying visual understanding and generation with dual visual vocabularies.arXiv preprint arXiv:2503.14324,

    Wei Song, Yuran Wang, Zijia Song, Yadong Li, Haoze Sun, Weipeng Chen, Zenan Zhou, Jianhua Xu, Jiaqi Wang, and Kaicheng Yu. Dualtoken: Towards unifying visual understanding and generation with dual visual vocabularies.arXiv preprint arXiv:2503.14324,

    arXiv

  22. [37]

    Journeydb: A benchmark for generative image understanding

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. Advances on Neural Information Processing Systems, 2023a. Peize Sun, Yi Jiang, Shoufa Chen, S...

    arXiv

  23. [38]

    Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023b

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023b. Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion.Ad...

    arXiv

  24. [39]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,

    arXiv

  25. [40]

    Metamorph: Multimodal understanding and generation via instruction tuning.arXiv preprint arXiv:2412.14164,

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning.arXiv preprint arXiv:2412.14164,

    arXiv

  26. [41]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869,

    arXiv

  27. [42]

    Lost in translationese? reducing translation effect using abstract meaning representation.arXiv preprint arXiv:2304.11501,

    Shira Wein and Nathan Schneider. Lost in translationese? reducing translation effect using abstract meaning representation.arXiv preprint arXiv:2304.11501,

    arXiv

  28. [43]

    Tune-a-video: One-shot tuning of image diffusion 31 models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion 31 models for text-to-video generation. InIEEE International Conference on Computer Vision, ...

    arXiv

  29. [44]

    Sana 1.5: Efficient scaling of training-time and inference- time compute in linear diffusion transformer.arXiv preprint arXiv:2501.18427, 2025a

    Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, et al. Sana 1.5: Efficient scaling of training-time and inference- time compute in linear diffusion transformer.arXiv preprint arXiv:2501.18427, 2025a. Ji...

    arXiv

  30. [45]

    Show-o: One single trans- former to unify multimodal understanding and generation.International Conference on Learning Representations, 2025b

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation.International Conference on Learning Represen...

    arXiv

  31. [46]

    Language model beats diffusion– tokenizer is key to visual generation.arXiv preprint arXiv:2310.05737,

    Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion– tokenizer is key to visual generation.arXiv preprint arXiv:2310.05737,

    arXiv

  32. [47]

    Show-1: Marrying pixel and latent diffusion models for text-to-video generation.arXiv preprint arXiv:2309.15818,

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation.arXiv preprint arXiv:2309.15818,

    arXiv

  33. [48]

    Table 7 outlines the key hyperparameters used across the three pretraining stages and two instruction tuning stages ofNeoBabel

    32 Appendix A This appendix provides additional training details and qualitative results to supplement the main paper. Table 7 outlines the key hyperparameters used across the three pretraining stages and two instruction tuning stages ofNeoBabel. Figure 12 presents representat...

    2000

  34. [2019]

    No language left behind: Scaling human-centered machine translation.arXiv preprint arXiv:2207.04672,

    Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffer- nan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation.arXiv preprint arXiv:2207.04672,

    arXiv

  35. [2020]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025b

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025b. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zha...

    2019 arXiv

  36. [2021]

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025a

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025a. Junsong Che...

    arXiv

  37. [2022]

    Aya vision: Advancing the frontier of multilingual multimodality.arXiv preprint arXiv:2505.08751,

    Saurabh Dash, Yiyang Nan, John Dang, Arash Ahmadian, Shivalika Singh, Madeline Smith, Bharat Venkitesh, Vlad Shmyhlo, Viraat Aryabumi, Walter Beller-Morales, Jeremy Pekmez, Ja- son Ozuzu, Pierre Richemond, Acyr Locatelli, Nick Frosst, Phil Blunsom, Aidan Gomez, Ivan Zhang, Mar...

    arXiv

  38. [2023]

    The ai gap: How socioeconomic status affects language technology interactions.arXiv preprint arXiv:2505.12158,

    Elisa Bassignana, Amanda Cercas Curry, and Dirk Hovy. The ai gap: How socioeconomic status affects language technology interactions.arXiv preprint arXiv:2505.12158,

    arXiv

  39. [2024]

    Behind maya: Building a multilingual vision language model.arXiv preprint arXiv:2505.08910,

    Nahid Alam, Karthik Reddy Kanjula, Surya Guthikonda, Timothy Chung, Bala Krishna S Vegesna, Abhipsha Das, Anthony Susevski, Ryan Sze-Yin Chan, SM Uddin, Shayekh Bin Islam, et al. Behind maya: Building a multilingual vision language model.arXiv preprint arXiv:2505.08910,

    arXiv

  40. [2025]

    The translation barrier hypothesis: Multilingual generation with large language models suffers from implicit translation failure.arXiv preprint arXiv:2506.22724,

    Niyati Bafna, Tianjian Li, Kenton Murray, David R Mortensen, David Yarowsky, Hale Sirin, and Daniel Khashabi. The translation barrier hypothesis: Multilingual generation with large language models suffers from implicit translation failure.arXiv preprint arXiv:2506.22724,

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.