Pith. sign in

REVIEW 2 major objections 5 minor 212 references

In unified multimodal models, training a skill on image understanding can transfer into generation without retraining generation itself.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 19:16 UTC pith:Z2EMTUBK

load-bearing objection Solid empirical isolation of capability-level und↔gen transfer in UMMs, with a usable post-training recipe; architecture attribution is correlational, not causal. the 2 major comments →

arxiv 2607.04423 v2 pith:Z2EMTUBK submitted 2026-07-05 cs.CV cs.AI

Transferability Between Understanding and Generation in Unified Multimodal Models

Jiwon Kang , Heeji Yoon , Jaewoo Jung , Jaewon Min , Minkyeong Jeon , Biyeon Hwang , Sangwon Jung , Seungryong Kim This is my paper
classification cs.CV cs.AI
keywords unified multimodal modelscross-task transferabilityimage understandingimage generationshared transformerunified visual encoderdistribution shiftcapability-specific fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Unified Multimodal Models put image understanding and image generation inside one system, but it has been unclear whether a skill learned on one side actually helps the other. This paper treats that question as transferability: if you train only counting, spatial layout, or text recognition on the understanding side, does generation of the same skill improve without any generation supervision? Controlled fine-tuning experiments across four open models show that the answer depends on architecture. Models that share a single transformer backbone and a single visual encoder transfer consistently in both directions; designs that keep separate encoders or separate transformers transfer little or not at all. The practical payoff is a safer way to fix weak generative skills: train the matching understanding task and let the capability migrate into generation. Direct generation fine-tuning can match or beat the accuracy gains but shifts the image distribution and hurts visual quality; understanding-side training improves the same skills while keeping generation closer to the original model.

Core claim

Cross-task transfer of concrete visual capabilities exists in unified multimodal models and is architecture-dependent: fully shared transformer backbones with a unified visual encoder show consistent understanding-to-generation and generation-to-understanding transfer on counting, spatial relations, and text, while loosely coupled designs show little or none. That transfer can be used as a practical post-training strategy that improves targeted generative accuracy while producing less distribution shift than direct generation fine-tuning.

What carries the argument

Transferability: the controlled test that training a named capability on only one task (understanding or generation) improves the same capability on the other task without explicit supervision for that task. The paper uses it both as an analysis probe across architectures and as the basis for an und-to-gen training recipe.

Load-bearing premise

The claim that architecture is the main reason transfer appears or fails rests on a small set of open models being representative of their design families rather than of their particular training data or objectives.

What would settle it

Take another model with a fully shared transformer and unified visual encoder, and another with separate pathways, train only on counting understanding under the same protocol, and check whether generation counting accuracy and mean absolute deviation improve only for the shared design.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies cross-task transferability in Unified Multimodal Models: whether training a specific capability (counting, spatial relation, text recognition/generation) on understanding improves the same capability on generation without generation supervision, and vice versa. Controlled LoRA fine-tunes on four open UMMs spanning shared-transformer/unified-encoder, shared-transformer/separate-encoder, MoT, and decoupled designs (plus MMaDA in the appendix) show that transfer is strongest and most bidirectional for fully shared transformer + unified visual encoder models (Lumina-DiMOO, MMaDA), weaker or absent for loosely coupled designs. The authors then exploit und o gen transfer on Lumina-DiMOO as a practical alternative to direct gen o gen fine-tuning, reporting comparable capability gains with lower FID/IS degradation and preserved general multimodal benchmarks (POPE, MMBench, MMMU, MME). Attention-map and compositionality analyses, plus transfer-strength ratios, further characterize when and how transfer occurs.

Significance. If the architecture-transfer link and the und o gen recipe hold more broadly, the work supplies a concrete, capability-level lens on UMM interactions that aggregate benchmarks obscure, and a practical post-training recipe that improves targeted generative skills while limiting distribution shift relative to direct generation fine-tuning. Strengths include matched image sets for und/gen, independent evaluation protocols (detector counts, OCR, spatial centroids), quantitative quality metrics (FID, IS), attention visualizations, a second shared-architecture model (MMaDA), and explicit discussion of joint-training oscillation and capability-dependent transfer direction. These make the empirical claims falsifiable and useful for post-training practice even if the causal architecture story remains correlational.

major comments (2)
  1. Section 3.2, Table 1, and Appendix C.1: The central scientific claim attributes consistent bidirectional transfer primarily to “fully shared transformer backbone + unified visual encoder.” Only Lumina-DiMOO (and MMaDA, same family) show strong und o gen and gen o und on counting; Janus-Pro (shared transformer, separate encoders) shows none/negative und o gen; BAGEL/BLIP3-o are weaker/asymmetric. Encoders and heads are frozen and LoRA is applied only to the backbone (Appendix A), so residual differences in pre-training data, discrete vs continuous generation objective, tokenizer, or scale remain uncontrolled. Without a controlled swap (same pretrain/data/objective, only encoder unification or backbone sharing changed), the architecture o transfer link is correlational. This link is load-bearing for both the scientific claim and the practical und o gen recipe, which is demonstrated almost
  2. Section 4 and Tables 2–4: The practical claim that und o gen “improves capability-specific generative performance while minimizing distribution shift” is validated almost entirely on Lumina-DiMOO. Appendix C.1 shows transfer exists for MMaDA on counting/spatial, but the FID/IS quality comparison and the three-capability suite are not repeated. Given that the recipe is offered as architecture-dependent, the quality-preserving benefit should be shown on at least one additional shared-architecture model, or the claim should be scoped more narrowly to models already known to exhibit strong transfer.
minor comments (5)
  1. Figure 2 and qualitative panels: random seeds are matched, which is good; still note detector/OCR failure rates and how failed detections are handled in accuracy denominators (Appendix B mentions exclusion for spatial, but main tables do not report rates).
  2. Table 6 transfer-strength ratios: the “upper bound = direct training” assumption is reasonable but should be caveated when gen o gen already degrades quality (counting FID 52.51), so the ratio can exceed 100% for MAD.
  3. Section 5.2 joint-training oscillation: the oscillatory pattern is interesting; a short note on whether early-stopping or loss weighting changes the outcome would help readers who might try und+gen.
  4. Appendix A: learning rates differ substantially across models (3e-6 vs 1e-4); a brief sensitivity check or justification would reduce the free-parameter concern.
  5. Typos/clarity: “und o gen” / “gen o und” notation is clear once introduced; ensure consistent bolding of transferability in the abstract and first use.

Circularity Check

0 steps flagged

No circularity: purely empirical transfer measurements with independent evaluation protocols; architecture claims are correlational observations, not definitional reductions.

full rationale

The paper's central claims rest on controlled fine-tuning experiments (LoRA on shared backbones) followed by independent evaluation: object-detector counts for generation counting (GenEval-style), centroid geometry for spatial relations, and OCR metrics (WER/CER/etc.) for text. These metrics are not algebraically forced by the understanding or generation losses, nor by any fitted scalar that is later re-labeled a prediction. No uniqueness theorem, ansatz, or self-citation is load-bearing for the transfer numbers; model citations (Lumina-DiMOO, Janus-Pro, BAGEL, BLIP3-o, MMaDA) merely identify the architectures under test. The practical und o gen recipe is likewise an empirical comparison of FID/IS and task accuracy against direct gen o gen fine-tuning, not a derivation that collapses to its inputs. Architecture dependence is reported as an observed correlation across four (plus one) open models; any residual confounds of pre-training data or objective are external validity concerns, not circularity. The derivation chain is therefore self-contained experimental measurement with score 0.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The work is empirical; load-bearing premises are standard ML practice plus the modeling choice that four open models represent the architectural space and that synthetic probes measure the intended capabilities.

free parameters (3)
  • LoRA rank = 128
    Fixed at 128 for main runs; ablation shows transfer strength rises with rank (8→32→128). Choice affects measured transfer magnitude.
  • learning rates per model = model-dependent
    Model-specific LRs (1e-4 BAGEL, 4e-5 Janus-Pro, 3e-6 Lumina-DiMOO, 2e-5 BLIP3-o) chosen by authors; not derived.
  • counting range filter = 0–20
    Objects >20 discarded from PixMo to keep difficulty manageable; changes the training distribution.
axioms (3)
  • domain assumption Capability-level accuracy/MAD/WER measured by off-the-shelf detectors and OCR models is a valid proxy for the intended visual skill.
    Used throughout evaluation (GenEval-style detection, GLM-OCR); detector/OCR errors could inflate or deflate transfer scores.
  • domain assumption LoRA updates on QKV and MLP layers of the shared transformer are sufficient to reveal architectural transfer effects.
    Full fine-tuning collapsed quality; authors therefore rely on LoRA as the intervention.
  • ad hoc to paper The four selected open models plus MMaDA adequately sample the three architectural families defined in Section 2.1.
    Central architecture claim rests on this sampling; acknowledged as a limitation in Appendix E.

pith-pipeline@v1.1.0-grok45 · 33462 in / 2628 out tokens · 35414 ms · 2026-07-11T19:16:32.579383+00:00 · methodology

0 comments
read the original abstract

Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate $\boldsymbol{\mathsf{transferability}}$ in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.

Figures

Figures reproduced from arXiv: 2607.04423 by Biyeon Hwang, Heeji Yoon, Jaewon Min, Jaewoo Jung, Jiwon Kang, Minkyeong Jeon, Sangwon Jung, Seungryong Kim.

Figure 1
Figure 1. Figure 1: Architecture of chosen unified multimodal models for cross-task [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative results of und → gen transferability across models. For each prompt, we compare the baseline model with the model trained only on counting understanding data (und → gen) and evaluated on generation. All images are generated using the same random seed for fair comparison. swering, where it answers how many objects appear in an image. For generation, the model learns counting through text-to-imag… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison of understanding-to-generation transfer ( [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example of synthetic dataset for spatial relation. Experimental setting. We observe that pretrained models already handle simple spa￾tial relations such as left, right, top, and bottom reasonably well. We therefore focus on a more challenging setting: diagonal rela￾tions between two objects (e.g., top-left, top￾right, bottom-right, bottom-left). Since exist￾ing spatial reasoning datasets [24,29,41,74,98, 1… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison between und → gen and gen → gen on spatial relation tasks. While generation-only training (gen → gen) significantly degrades im￾age quality reflecting traits of synthetic training data, und → gen preserves generation quality while achieving improved spatial relation accuracy. Train → Test Acc. ↑ IS [71] ↑ FID [39] ↓ Baseline 67.0 16.86 - und → gen 74.0 (+7.0) 17.50 30.95 gen → gen 80… view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison between und → gen and gen → gen on text generation task. und → gen generates clear and accurate text, preserving rendering fidelity. Train → Test WER ↓ CER ↓ Edit Dist. ↓ BLEU ↑ METEOR ↑ F1 ↑ Baseline 0.654 0.362 36.812 0.277 0.403 0.446 und → gen 0.641 (-0.013) 0.351 (-0.011) 35.636 (-1.176) 0.274 (-0.003) 0.410 (+0.007) 0.452 (+0.006) gen → gen 0.641 (-0.013) 0.367 (+0.005) 37.312 … view at source ↗
Figure 6
Figure 6. Figure 6: Synthetic dataset for text recognition and generation. We next examine text-contained image generation, where the model must render legible characters with accurate strokes, spacing, and glyph structure. Un￾like counting or spatial relation, this requires the model to capture highly localized and fine-grained visual details. We investigate whether such fine￾grained capabilities can be introduced through un… view at source ↗
Figure 8
Figure 8. Figure 8: Trade-off in joint und+gen training. (a) Counting understanding accuracy and mean absolute deviation (MAD) over training epochs. (b) Counting generation accuracy and MAD over training epochs. Blue lines denote accuracy and red lines denote MAD. 5.2 Does joint training lead to stronger transferability? Throughout this work, we examined transferability in UMMs and showed that training a capability through on… view at source ↗
Figure 9
Figure 9. Figure 9: Full list of spatial relation evaluation prompts. [PITH_FULL_IMAGE:figures/full_fig_p032_9.png] view at source ↗
Figure 1
Figure 1. Figure 1: Among them, Lumina-DiMOO, which belongs to the shared transformer [PITH_FULL_IMAGE:figures/full_fig_p033_1.png] view at source ↗
Figure 10
Figure 10. Figure 10: Attention analysis on Lumina-DiMOO and Janus-Pro on counting. [PITH_FULL_IMAGE:figures/full_fig_p034_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Additional qualitative comparisons for Lumina-DiMOO between [PITH_FULL_IMAGE:figures/full_fig_p039_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Additional qualitative comparisons for Lumina-DiMOO between [PITH_FULL_IMAGE:figures/full_fig_p040_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Additional qualitative comparisons for Lumina-DiMOO between [PITH_FULL_IMAGE:figures/full_fig_p041_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

212 extracted references · 38 linked inside Pith

  1. [1]

    AI, Z.: Glm-ocr.https://github.com/zai-org/GLM-OCR(2026), accessed: 2026- 07-01

    2026

  2. [2]

    arXiv preprint arXiv:2209.15571 (2022)

    Albergo, M.S., Vanden-Eijnden, E.: Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209.15571 (2022)

    Pith/arXiv arXiv 2022

  3. [3]

    anthropic.com/news/claude-3-family, accessed: 2026-07-01

    Anthropic: Introducing the next generation of claude (2024),https://www. anthropic.com/news/claude-3-family, accessed: 2026-07-01

    2024

  4. [4]

    Austin, J., Johnson, D.D., Ho, J., Tarlow, D., Van Den Berg, R.: Structured denoisingdiffusionmodelsindiscretestate-spaces.Advancesinneuralinformation processing systems34, 17981–17993 (2021)

    2021

  5. [5]

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao,C.,Ge,C.,etal.:Qwen3-vltechnicalreport.arXivpreprintarXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  6. [6]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv e-prints pp. arXiv–2502 (2025)

    2025

  7. [7]

    In: Proceedings of the sixth National conference on Artificial intelligence-Volume 1

    Ballard, D.H.: Modular learning in neural networks. In: Proceedings of the sixth National conference on Artificial intelligence-Volume 1. pp. 279–284 (1987)

    1987

  8. [8]

    In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization

    Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. pp. 65–72 (2005)

    2005

  9. [9]

    In: Pro- ceedings of the Computer Vision and Pattern Recognition Conference

    Binyamin, L., Tewel, Y., Segev, H., Hirsch, E., Rassin, R., Chechik, G.: Make it count: Text-to-image generation with an accurate number of objects. In: Pro- ceedings of the Computer Vision and Pattern Recognition Conference. pp. 13242– 13251 (2025)

    2025

  10. [10]

    In: International Conference on Learning Represen- tations

    Black, K., Janner, M., Du, Y., Kostrikov, I., Levine, S.: Training diffusion models with reinforcement learning. In: International Conference on Learning Represen- tations. vol. 2024, pp. 4965–4987 (2024)

    2024

  11. [11]

    Black Forest Labs: FLUX.2: Analyzing and enhancing the latent space of FLUX – representation comparison (2025),https://bfl.ai/research/representation- comparison, accessed: 2026-07-01

    2025

  12. [12]

    arXiv preprint arXiv:2511.11253 (2025)

    Boo, H., Kim, H., Lee, M., Lee, S., Lee, J., Choi, J.H., Cho, H.: Countsteer: Steering attention for object counting in diffusion models. arXiv preprint arXiv:2511.11253 (2025)

    arXiv 2025

  13. [13]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Chae, J., Kim, J., Choi, J., Kim, K., Hwang, S.: Apt: Adaptive personalized training for diffusion models with limited data. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 28619–28628 (2025)

    2025

  14. [14]

    URL https://arxiv.org/abs/2505.09568

    Chen, J., Xu, Z., Pan, X., Hu, Y., Qin, C., Goldstein, T., Huang, L., Zhou, T., Xie, S., Savarese, S., et al.: Blip3-o: A family of fully open unified multimodal models- architecture, training and dataset, 2025. URL https://arxiv.org/abs/2505.09568

    Pith/arXiv arXiv 2025

  15. [15]

    In: Proceedings of the IEEE/CVF winter conference on applications of computer vision

    Chen, M., Laina, I., Vedaldi, A.: Training-free layout control with cross-attention guidance. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 5343–5353 (2024)

    2024

  16. [16]

    arXiv preprint arXiv:2501.17811 (2025)

    Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C.: Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 (2025)

    Pith/arXiv arXiv 2025

  17. [17]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning Transferability in Unified Multimodal Models 17 for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 24185–24198 (2024)

    2024

  18. [18]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290– 1299 (2022)

    2022

  19. [19]

    5: Native multimodal models are world learners

    Cui, Y., Chen, H., Deng, H., Huang, X., Li, X., Liu, J., Liu, Y., Luo, Z., Wang, J., Wang, W., et al.: Emu3. 5: Native multimodal models are world learners. arXiv preprint arXiv:2510.26583 (2025)

    Pith/arXiv arXiv 2025

  20. [20]

    arXiv preprint arXiv:2309.15807 (2023)

    Dai, X., Hou, J., Ma, C.Y., Tsai, S., Wang, J., Wang, R., Zhang, P., Vandenhende, S., Wang, X., Dubey, A., et al.: Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807 (2023)

    Pith/arXiv arXiv 2023

  21. [21]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J.S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., et al.: Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 91–104 (2025)

    2025

  22. [22]

    arXiv preprint arXiv:2505.14683 (2025)

    Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., et al.: Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683 (2025)

    Pith/arXiv arXiv 2025

  23. [23]

    In: 2009 IEEE conference on computer vision and pattern recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)

    2009

  24. [24]

    arXiv preprint arXiv:2506.18385 (2025)

    Deng, N., Gu, L., Ye, S., He, Y., Chen, Z., Li, S., Wang, H., Wei, X., Yang, T., Dou, M., et al.: Internspatial: A comprehensive dataset for spatial reasoning in vision-language models. arXiv preprint arXiv:2506.18385 (2025)

    Pith/arXiv arXiv 2025

  25. [25]

    In: International Conference on Learning Representations

    Dong, R., Peng, Y., Qi, Z., Ge, Z., Yang, J., Zhao, L., Sun, J., Zhou, H., Wei, H., Kong, X., et al.: Dreamllm: Synergistic multimodal comprehension and creation. In: International Conference on Learning Representations. vol. 2024, pp. 6666– 6702 (2024)

    2024

  26. [26]

    In: Forty-first international conference on machine learning (2024)

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz,D.,Sauer,A.,Boesel,F.,etal.:Scalingrectifiedflowtransformersforhigh- resolution image synthesis. In: Forty-first international conference on machine learning (2024)

    2024

  27. [27]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution im- age synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12873–12883 (2021)

    2021

  28. [28]

    Advances in Neural Information Processing Sys- tems36, 79858–79885 (2023)

    Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., Lee, K.: Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Sys- tems36, 79858–79885 (2023)

    2023

  29. [29]

    Fan, Y., He, X., Yang, D., Zheng, K., Kuo, C.C., Zheng, Y., Narayanaraju, S.J., Guan, X., Wang, X.E.: Grit: Teaching mllms to think with images (2025),https: //arxiv.org/abs/2505.15879, accessed: 2026-07-01

    Pith/arXiv arXiv 2025

  30. [30]

    arXiv preprint arXiv:2212.05032 (2022)

    Feng, W., He, X., Fu, T.J., Jampani, V., Akula, A., Narayana, P., Basu, S., Wang, X.E., Wang, W.Y.: Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032 (2022)

    Pith/arXiv arXiv 2022

  31. [31]

    arXiv preprint arXiv:2306.13394 (2023) 18 Kang et al

    Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al.: Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 (2023) 18 Kang et al

    Pith/arXiv arXiv 2023

  32. [32]

    arXiv preprint arXiv:2510.13080 (2025)

    Fu, S., Zhou, J., Chen, Q., Jing, H., Nguyen, H.A., Liu, X., Zeng, Z., Ma, L., Zhang, Q., Wu, Q.: Counting hallucinations in diffusion models. arXiv preprint arXiv:2510.13080 (2025)

    arXiv 2025

  33. [33]

    arXiv preprint arXiv:2404.14396 (2024)

    Ge, Y., Zhao, S., Zhu, J., Ge, Y., Yi, K., Song, L., Li, C., Ding, X., Shan, Y.: Seed- x: Multimodal models with unified multi-granularity comprehension and genera- tion. arXiv preprint arXiv:2404.14396 (2024)

    Pith/arXiv arXiv 2024

  34. [34]

    arXiv preprint arXiv:2312.11805 (2023)

    Gemini Team Google: Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    Pith/arXiv arXiv 2023

  35. [35]

    Advances in Neural Information Processing Systems36, 52132–52152 (2023)

    Ghosh, D., Hajishirzi, H., Schmidt, L.: Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems36, 52132–52152 (2023)

    2023

  36. [36]

    arXiv preprint arXiv:2602.17871 (2026)

    Ghosh, D., Zhang, Y., Schmidt, L.: Understanding the fine-grained knowledge capabilities of vision-language models. arXiv preprint arXiv:2602.17871 (2026)

    arXiv 2026

  37. [37]

    arXiv preprint arXiv:2212.10015 (2022)

    Gokhale, T., Palangi, H., Nushi, B., Vineet, V., Horvitz, E., Kamar, E., Baral, C., Yang, Y.: Benchmarking spatial relationships in text-to-image generation. arXiv preprint arXiv:2212.10015 (2022)

    Pith/arXiv arXiv 2022

  38. [38]

    arXiv preprint arXiv:2501.12948 (2025)

    Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al.: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    Pith/arXiv arXiv 2025

  39. [39]

    Advances in neural information processing systems30(2017)

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems30(2017)

    2017

  40. [40]

    Advances in neural information processing systems33, 6840–6851 (2020)

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems33, 6840–6851 (2020)

    2020

  41. [41]

    In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition. pp. 6700–6709 (2019)

    2019

  42. [42]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Jung, W., Go, J., Jeon, M., Yoon, S., Kim, J.: Visual funnel: Resolving contextual blindness in multimodal large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8962–8971 (2026)

    2026

  43. [43]

    In: Interna- tional Conference on Learning Representations

    Khayatkhoei, M., Chhikara, P., Ilievski, F., et al.: Mllms know where to look: Training-free perception of small visual details with multimodal llms. In: Interna- tional Conference on Learning Representations. vol. 2025, pp. 68194–68213 (2025)

    2025

  44. [44]

    arXiv preprint arXiv:1312.6114 (2013)

    Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

    Pith/arXiv arXiv 2013

  45. [45]

    In: International Conference on Machine Learning

    Kou, S., Jin, J., Liu, Z., Liu, C., Ma, Y., Jia, J., Chen, Q., Jiang, P., Deng, Z.: Orthus: Autoregressive interleaved image-text generation with modality-specific heads. In: International Conference on Machine Learning. pp. 31619–31633. PMLR (2025)

    2025

  46. [46]

    1 kontext: Flow match- ing for in-context image generation and editing in latent space

    Labs, B.F., Batifol, S., Blattmann, A., Boesel, F., Consul, S., Diagne, C., Dock- horn, T., English, J., English, Z., Esser, P., et al.: Flux. 1 kontext: Flow match- ing for in-context image generation and editing in latent space. arXiv preprint arXiv:2506.15742 (2025)

    Pith/arXiv arXiv 2025

  47. [47]

    arXiv preprint arXiv:2302.12192 (2023)

    Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Gu, S.S.: Aligning text-to-image models using human feed- back. arXiv preprint arXiv:2302.12192 (2023)

    Pith/arXiv arXiv 2023

  48. [48]

    Advances in neural information processing systems37, 103269–103304 (2024) Transferability in Unified Multimodal Models 19

    Lee, K., Kwak, S., Sohn, K., Shin, J.: Direct consistency optimization for robust customization of text-to-image diffusion models. Advances in neural information processing systems37, 103269–103304 (2024) Transferability in Unified Multimodal Models 19

    2024

  49. [49]

    Advances in Neural Information Processing Systems37, 24897–24925 (2024)

    Li, S., Kallidromitis, K., Gokul, A., Kato, Y., Kozuka, K.: Aligning diffusion models by optimizing human utility. Advances in Neural Information Processing Systems37, 24897–24925 (2024)

    2024

  50. [50]

    arXiv preprint arXiv:2506.17202 (2025)

    Li, T., Lu, Q., Zhao, L., Li, H., Zhu, X., Qiao, Y., Zhang, J., Shao, W.: Unifork: Exploring modality alignment for unified multimodal understanding and genera- tion. arXiv preprint arXiv:2506.17202 (2025)

    Pith/arXiv arXiv 2025

  51. [51]

    arXiv preprint arXiv:2505.04347 (2025)

    Li, Y., Wan, P., Han, L., Wang, Y., Nie, L., Zhang, M.: Countdiffusion: Text- to-image synthesis with training-free counting-guidance diffusion. arXiv preprint arXiv:2505.04347 (2025)

    Pith/arXiv arXiv 2025

  52. [52]

    In: Proceedings of the 2023 conference on empirical methods in natural language processing

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hal- lucination in large vision-language models. In: Proceedings of the 2023 conference on empirical methods in natural language processing. pp. 292–305 (2023)

    2023

  53. [53]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Li, Z., Li, H., Shi, Y., Farimani, A.B., Kluger, Y., Yang, L., Wang, P.: Dual diffusion for unified image generation and understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 2779–2790 (2025)

    2025

  54. [54]

    In: European confer- ence on computer vision

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European confer- ence on computer vision. pp. 740–755. Springer (2014)

    2014

  55. [55]

    arXiv preprint arXiv:2210.02747 (2022)

    Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M., Le, M.: Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 (2022)

    Pith/arXiv arXiv 2022

  56. [56]

    Advances in neural information processing systems36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023)

    2023

  57. [57]

    In: European conference on computer vision

    Liu, N., Li, S., Du, Y., Torralba, A., Tenenbaum, J.B.: Compositional visual gen- eration with composable diffusion models. In: European conference on computer vision. pp. 423–439. Springer (2022)

    2022

  58. [58]

    arXiv preprint arXiv:2209.03003 (2022)

    Liu, X., Gong, C., Liu, Q.: Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 (2022)

    Pith/arXiv arXiv 2022

  59. [59]

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al.: Mmbench: Is your multi-modal model an all-around player? In: European conference on computer vision. pp. 216–233. Springer (2024)

    2024

  60. [60]

    arXiv preprint arXiv:2310.16834 (2023)

    Lou, A., Meng, C., Ermon, S.: Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834 (2023)

    Pith/arXiv arXiv 2023

  61. [61]

    In: Proceedings of the 32nd ACM International Confer- ence on Multimedia

    Lv, H., Xiao, J., Li, L.: Pick-and-draw: Training-free semantic guidance for text- to-image personalization. In: Proceedings of the 32nd ACM International Confer- ence on Multimedia. pp. 10535–10543 (2024)

    2024

  62. [62]

    In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition

    Marsili, D., Mehta, A., Lin, R.Y., Gkioxari, G.: Same or not? enhancing visual perception in vision-language models. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition. pp. 17303–17315 (2026)

    2026

  63. [63]

    NVIDIA: Llama-nemotron-vlm-dataset-v1.https://huggingface.co/datasets/ nvidia/Llama-Nemotron-VLM-Dataset-v1(2025), accessed: 2026-07-01

    2025

  64. [64]

    arXiv preprint arXiv:2303.08774 (2023)

    OpenAI: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  65. [65]

    arXiv preprint arXiv:2504.06256 (2025)

    Pan, X., Shukla, S.N., Singh, A., Zhao, Z., Mishra, S.K., Wang, J., Xu, Z., Chen, J., Li, K., Juefei-Xu, F., et al.: Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.06256 (2025)

    Pith/arXiv arXiv 2025

  66. [66]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Phung, Q., Ge, S., Huang, J.B.: Grounded text-to-image synthesis with attention refocusing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7932–7942 (2024)

    2024

  67. [67]

    In: International Conference on Learning Representations

    Podell,D.,English,Z.,Lacey,K.,Blattmann,A.,Dockhorn,T.,Müller,J.,Penna, J.,Rombach,R.:Sdxl:Improvinglatentdiffusionmodelsforhigh-resolutionimage synthesis. In: International Conference on Learning Representations. vol. 2024, pp. 1862–1874 (2024) 20 Kang et al

    2024

  68. [68]

    In: International conference on machine learning

    Radford,A.,Kim,J.W.,Hallacy,C.,Ramesh,A.,Goh,G.,Agarwal,S.,Sastry,G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  69. [69]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)

    2022

  70. [70]

    Advances in Neural Information Processing Systems37, 130136–130184 (2024)

    Sahoo, S.S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J.T., Rush, A., Kuleshov, V.: Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems37, 130136–130184 (2024)

    2024

  71. [71]

    Advances in neural information processing systems29(2016)

    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. Advances in neural information processing systems29(2016)

    2016

  72. [72]

    arXiv preprint arXiv:2505.23606 (2025)

    Shi, Q., Bai, J., Zhao, Z., Chai, W., Yu, K., Wu, J., Tong, Y., Li, X., Li, X., Yan, S.: Muddit: Liberating generation beyond text-to-image with a unified discrete diffusion model. arXiv preprint arXiv:2505.23606 (2025)

    Pith/arXiv arXiv 2025

  73. [73]

    In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion

    Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., Hassner, T.: Textocr: To- wards large-scale end-to-end reasoning for arbitrary-shaped scene text. In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion. pp. 8802–8812 (2021)

    2021

  74. [74]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Song, C.H., Blukis, V., Tremblay, J., Tyree, S., Su, Y., Birchfield, S.: Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 15768–15780 (2025)

    2025

  75. [75]

    arXiv preprint arXiv:2010.02502 (2020)

    Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)

    Pith/arXiv arXiv 2010

  76. [76]

    Advances in neural information processing systems32(2019)

    Song, Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems32(2019)

    2019

  77. [77]

    In: International conference on learning representations

    Sun, Q., Yu, Q., Cui, Y., Zhang, F., Zhang, X., Wang, Y., Gao, H., Liu, J., Huang, T., Wang, X.: Emu: Generative pretraining in multimodality. In: International conference on learning representations. vol. 2024, pp. 12352–12380 (2024)

    2024

  78. [78]

    arXiv preprint arXiv:2405.09818 (2024)

    Team, C.: Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 (2024)

    Pith/arXiv arXiv 2024

  79. [79]

    Advances in Neural Information Processing Systems 37, 87310–87356 (2024)

    Tong, P., Brown, E., Wu, P., Woo, S., Iyer, A.J.V., Akula, S.C., Yang, S., Yang, J., Middepogu, M., Wang, Z., et al.: Cambrian-1: A fully open, vision-centric ex- ploration of multimodal llms. Advances in Neural Information Processing Systems 37, 87310–87356 (2024)

    2024

  80. [80]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Tong, S., Fan, D., Li, J., Xiong, Y., Chen, X., Sinha, K., Rabbat, M., LeCun, Y., Xie, S., Liu, Z.: Metamorph: Multimodal understanding and generation via instruction tuning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 17001–17012 (2025)

    2025

Showing first 80 references.