Pith. sign in

REVIEW 4 major objections 6 minor 10 cited by

Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One image space unifies understanding, generation, and editing by replacing token-by-token feedback with prefilled placeholders, and the paper reports state-of-the-art scores on all three tasks.

desk verdict The prefilled autoregression trick is a genuinely novel idea with promising numbers, but the paper's evidence for it is thinner than it looks and the abstract oversells the generation result; worth a serious referee but needs revision. read the letter →

arxiv 2504.21356 v3 pith:6E2K2UBH submitted 2025-04-30 cs.CV cs.AI

Hong Zhang , Zhongjie Duan , Xingjun Wang , Yuze Zhao , Weiyi Lu , Zhipeng Di , Yixuan Xu , Yingda Chen
show 1 more author
Yu Zhang
This is my paper · ORCID
classification cs.CVcs.AI
keywords unifiedmultimodalmodelprefilledautoregressionsharedimageembeddingspaceunderstandinggenerationeditingcontinuousembeddingsflowmatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Nexus-Gen is a unified model that claims one architecture can understand, generate, and edit images by routing all visual information through a single continuous embedding space. The paper's central assertion is that an autoregressive language model, a diffusion decoder, and a vision encoder can share this space without forcing the language model to predict discrete tokens, and that a prefilled autoregression strategy prevents the cascade of prediction errors that normally plagues continuous embedding generation. If true, this offers a practical route to a single system that both reasons about images and produces or modifies them, trained jointly on interleaved text-image data. The paper reports state-of-the-art numbers across understanding, generation, and editing benchmarks, including an overall 0.81 on GenEval for the instruction-tuned variant.

What carries the argument

The prefilled autoregression mechanism, together with the shared embedding space. During training, input sequences are prefilled with $N_E$ learnable embeddings plus positional encodings at the image-token positions; during inference, the same learned embeddings are inserted as soon as the beginning-of-image token is predicted. This makes training and inference compute over the same input distribution, so errors from one predicted token never get recycled into the next prediction. The composite image loss is mean squared error plus cosine similarity against ground-truth embeddings, and the vision decoder is conditioned on these predicted embeddings through a projector. A second conditioning stream for editing uses 324-token embeddings of the input image to preserve details.

What would settle it

Train a small autoregressive model on the same continuous embedding space with the prefilled strategy and with naive teacher-forcing, then evaluate generation quality and per-token MSE at inference on a fixed prompt set. If the prefilled variant does not reduce inference loss compared to naive autoregression, or if shuffling the order of ground-truth image embeddings during training changes the predicted embeddings, the permutation-invariance assumption is falsified. A direct test: encode the same image with its tokens in the natural order and in a shuffled order, then check whether the autoregressive model's predictions for subsequent tokens differ measurably; if they do, the embedding space is order-dependent and the prefilling logic collapses.

Watch

Extended reading notes

Core claim

The core claim is that prefilled autoregression in a shared continuous image embedding space removes the error-accumulation bottleneck that limits previous autoregressive image generators. Instead of predicting image embeddings token by token and feeding those predictions back as input, the model fills the image-token positions with learned placeholder embeddings during both training and inference. This relies on the invariance of image token prediction to ordering: each embedding can be predicted from the text and its positional encoding alone, so the prefilled placeholders lose no information while making train and test behavior identical. Nexus-Gen ties this to a unified embedding space where image understanding, generation, and editing all operate: understanding encodes input images into the space, generation predicts semantic embedding tokens that a diffusion decoder turns into images, and editing feeds both semantic and fine-grained embeddings into a dual-conditioned decoder. Empirical sections report state-of-the-art results on MMMU (45.7), GenEval (0.81 after instruction tuning), and editing metrics on the ImagePulse test set.

Load-bearing premise

The whole prefilling design depends on the claim that image-token prediction is permutation-invariant, meaning each image embedding can be predicted from the caption and its position alone; if that invariance does not hold in the continuous embedding space, prefilled placeholders would discard order information the model needs.

Editorial extensions

If this is right

  • A single 7B-parameter model can match or surpass specialized single-task models on image understanding, generation, and editing without task-specific heads, because all tasks share one embedding interface.
  • Interleaved text-image corpora become directly usable for joint training, since visual inputs and outputs are expressed in the same continuous space as the language model's own representations.
  • The prefilled autoregression strategy promises to remove the training-inference gap for any continuous-valued autoregressive prediction, not just images, since it eliminates the feedback of biased predictions.
  • For editing, dual conditioning that combines coarse semantic tokens with fine-grained detail tokens preserves unedited regions better than a single-stream generation decoder, pointing to a reusable two-stream design.
  • Bilingual generation and editing emerge naturally from re-annotating a portion of the training data, indicating that the unified space adapts to language diversity without architectural changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the permutation-invariance assumption holds in this continuous embedding space, the same prefilling trick could be applied to other continuous token domains such as audio or video, where autoregressive feedback drift is even more severe; this is a testable extension the paper does not explore.
  • The observed trade-off between 81 tokens and 324 tokens suggests a capacity limit in autoregressive prediction of dense embeddings; one could adaptively choose token budgets based on image complexity or saliency rather than a fixed count.
  • The paper's own limitations section admits sensitivity to prompt variations and unexplored visual reasoning; a natural next step is to evaluate whether the unified embedding space serves as a scratchpad for step-by-step vision-language reasoning, where prefilled placeholder tokens could act as intermediate reasoning states.
  • The editing decoder's joint attention layer for fusing semantic and detail streams is a reusable architectural pattern for other multi-condition diffusion tasks, such as guided inpainting or subject-driven generation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Nexus-Gen, a unified multimodal model that combines an autoregressive LLM (Qwen2.5-VL-7B) with a diffusion decoder (FLUX.1-Dev) through a shared continuous image embedding space. The core technical contribution is 'prefilled autoregression': during both training and inference, the sequence positions for image tokens are filled with NE learnable embeddings instead of previously predicted image embeddings, which the authors argue eliminates autoregressive error accumulation. The model is trained in three stages on a curated 26.3M-sample dataset spanning understanding, generation, and editing, and is evaluated on standard understanding benchmarks (MME, SEED-Bench, MMMU, TextVQA, VQAv2, RealWorldQA), on GenEval for generation, and on a self-built ImagePulse benchmark for editing. The paper reports strong results, including GenEval 0.81 for the instruction-tuned variant, and releases models, data, and code.

Significance. If validated, the prefilled autoregression idea is a simple and potentially useful alternative to standard teacher-forced autoregressive training for continuous image embeddings, and the unified embedding space connecting an LLM to a diffusion decoder is a plausible direction for joint understanding-generation-editing models. The paper contributes a large-scale training dataset and releases all artifacts, which are valuable for reproducibility and follow-up work. However, the central prefilling claim currently rests on an unverified strong independence assumption and is supported only by a single per-token-MSE plot, while the editing benchmark is constructed by the authors themselves. The claimed state-of-the-art status also needs qualification because the base model outperforms the proposed model on understanding benchmarks, and because the reported GenEval state-of-the-art requires additional instruction tuning. These issues are fixable with additional experiments and more careful framing, so the work is a plausible candidate for publication after major revision.

major comments (4)
  1. [Approach, 'Prefilled Autoregression' and Fig. 6] The justification for prefilled autoregression is that 'Prior research (Li et al. 2024) demonstrates that image token prediction is permutation-invariant,' but permutation invariance is not equivalent to the conditional independence p(e_i | text, position_i) = p(e_i | text, e_1..e_{i-1}) that prefilling implements by discarding all real image tokens from the input. Figure 6 actually undercuts the argument: teacher-forced training with real previous tokens achieves per-token MSE 0.2574, while prefilled training achieves 0.4300, showing the model can predict embeddings much more accurately when it conditions on other image tokens. The paper never measures final image quality (e.g., GenEval or FID) for naive versus prefilled autoregression, so the claimed benefit of 'eliminating error accumulation' is not distinguished from the effect of removing conditioning information. Please add an ablation that compares final generation quality under both training-inference schemes, and either verify the conditional independence assumption on Qwen2.5-VL embeddings or revise the theoretical claim.
  2. [Experiments, 'Image Editing' and Table 3] The editing evaluation is performed on a 1,000-sample test set randomly sampled from the same ImagePulse dataset used to train the editing decoder, making the editing claim circular: the decoder is tuned and evaluated on the same constructed distribution. The paper should also report results on external editing benchmarks such as MagicBrush, HQ-Edit, or Emu-Edit (or at least on independently collected test data), and should compare against unified models such as Emu3, Janus-Pro, or SEED-X on editing if such comparisons are feasible. Without this, the 'state-of-the-art' editing claim in Table 3 is not established.
  3. [Abstract and 'Main Results', Table 2] The abstract and conclusion state that Nexus-Gen 'achieves state-of-the-art performance' without qualification, but Table 2 shows that the jointly trained Nexus-Gen (0.77) is below Janus-Pro 7B (0.80) on GenEval; only the instruction-tuned Nexus-Gen* reaches 0.81. Similarly, Table 1 shows that the underlying base model Qwen2.5-VL-Instruct 7B outperforms Nexus-Gen on every understanding benchmark (e.g., MMMU 50.6 vs. 45.7; VQAv2 82.3 vs. 79.3), so the understanding 'state-of-the-art' claim holds only among unified models and at the cost of nontrivial degradation relative to the base. Please qualify the abstract and conclusion to state which comparisons support the SOTA claim and to acknowledge the base-model degradation.
  4. [Experiments, 'The Impact of Prefilled Autoregression', Fig. 6] Figure 6 reports a single training/inference loss trajectory with no error bars, no multiple seeds, and no downstream image-quality metric, so the claimed advantage of prefilled autoregression over naive autoregression is not statistically supported. The ablation also omits a comparison of generation and editing results using the naive autoregressive model at inference time, which is necessary to determine whether the reduced per-token MSE translates into better final images. Please provide repeated-run statistics and end-to-end metrics, or explicitly state the limitations of the current evidence.
minor comments (6)
  1. [Introduction, contributions list] The contributions list contains a tense inconsistency: 'We proposed a prefilling strategy' should read 'We propose a prefilling strategy' to match the other bullets.
  2. [Appendix, 'Dataset Construction Details', Image Editing paragraph] The paragraph contains a duplicated sentence: 'However, these datasets exhibit notable limitations' appears twice in consecutive sentences; please remove the duplicate.
  3. [Figure 2 and Equation (1)] The notation 'BOI' and 'EOI' in Figure 2 is not defined in the text; please define these tokens in the caption or in Section 3.2.
  4. [Table 1] The table uses abbreviations MME-P, MME-C, RWQA without expansion; please define all metric acronyms either in the caption or in the text before the table.
  5. [Experiments, 'Image Generation' and Table 2] The description of Nexus-Gen* says it is fine-tuned on Blip3o-60k, but the appendix reports this dataset contains only 0.06M samples; please clarify whether the fine-tuning data includes other sources or whether the 60k size is correct.
  6. [Approach, 'Unified Image Embedding Space' and Eq. (1)] The formula NE = (H/P) * (W/P) assumes a simple patch grid, but Qwen2.5-VL uses dynamic resolution with additional merging/subsampling; please specify whether NE is exactly this product or an approximation for the actual vision encoder.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation is present: the prefilled autoregression scheme is an empirically evaluated training/inference design, and the only minor issue is a non-load-bearing self-citation to the authors' EliGen dataset.

full rationale

The paper's central chain is architectural and empirical rather than derivational: it proposes a shared embedding space, a prefilled autoregression strategy, and a multi-stage training recipe, then measures performance on external benchmarks such as GenEval, MMMU, TextVQA, VQAv2, and SEED-Bench. The prefilled autoregression strategy is justified by an external citation to Li et al. 2024 for permutation invariance of image token prediction; even if that citation is over-read as implying full conditional independence of image embeddings, this is a correctness or validity concern, not a circular reduction, because the paper does not define its conclusion into the premise and instead reports empirical comparisons in Figure 6 and downstream benchmarks. The editing evaluation uses the authors' in-house ImagePulse dataset, with a test set explicitly stated to be non-overlapping with training data; while this weakens external generalizability, it is a benchmark-construction caveat, not a circular derivation. The only self-citations are data and tooling references, notably the EliGen dataset from Zhang et al. 2025, which shares authors with this paper, and ModelScope software frameworks; these are used as training resources, not as load-bearing theorems, uniqueness claims, or fitted predictions. No equation reduces to its own input, no fitted parameter is renamed as a prediction, and no central claim is forced by a self-citation chain. The score of 2 reflects the minor self-cited EliGen data source, not substantive circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central method rests on the permutation-invariance of image token prediction (external citation) and on the compatibility of the Qwen2.5-VL embedding space with the FLUX decoder. Loss weights and token count are hand-tuned hyperparameters.

free parameters (3)
  • loss weights λ1, λ2, λ3 = 3, 1.5, 1.5
    Hand-chosen weights in Eq. 3 balancing text CE, image MSE, and cosine losses; affect training dynamics but not core derivation.
  • image token count NE = 81
    Selected via ablations (Figures 3-4) as the optimal trade-off between reconstruction fidelity and AR generation difficulty.
  • CFG scale = 3.0
    Classifier-free guidance scale for generation decoder at inference; affects sample quality, not core method.
assumptions (4)
  • domain assumption Image token prediction is permutation-invariant (Li et al. 2024)
    Used to justify prefilled autoregression where all image tokens are replaced with learnable placeholders; not independently verified for Qwen2.5-VL embeddings.
  • domain assumption The Qwen2.5-VL vision encoder's embedding space can serve as a unified conditioning space for the FLUX diffusion decoder via a learned projector
    Central architectural assumption; the projectors must map between modalities so that the diffusion decoder can reconstruct images from these embeddings.
  • standard math Standard loss functions (CE, MSE, cosine, flow matching) are appropriate for the tasks
    These are standard objectives; no new mathematical axioms introduced.
  • ad hoc to paper The ImagePulse dataset construction pipeline yields high-quality editing pairs representative of real editing tasks
    The editing evaluation uses a held-out set from this self-constructed dataset; results depend on the quality and distribution of this data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space." pith.science (2026). https://pith.science/paper/6E2K2UBH

@misc{pith2026250421356,
  author       = {Pith},
  title        = {Pith review of: Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6E2K2UBH}},
  note         = {Machine review of arXiv:2504.21356}
}
read the original abstract

Unified multimodal generative models aim to integrate image understanding and generation abilities, offering significant advantages in harnessing multimodal corpora, particularly interleaved text-image data. However, existing unified models exhibit limitations in image synthesis quality, autoregressive error accumulation, and image editing capability. In this work, we propose Nexus-Gen, a novel architecture that unifies image understanding, generation, and editing tasks in a shared image embedding space. This shared space serves as a bridge for the autoregressive and diffusion models, which seamlessly integrates their complementary strengths in cross-modal modeling. To mitigate the severe error accumulation during autoregressive embedding prediction, we propose a novel prefilled autoregression strategy that aligns training-inference dynamics by prefilling input sequences with learnable embeddings. After multi-stage and multi-task training on our constructed large-scale dataset with 26.3 million samples, Nexus-Gen achieves state-of-the-art performance on the evaluation benchmarks spanning image understanding, generation and editing tasks. All models, datasets, and source codes are released in https://github.com/modelscope/Nexus-Gen to facilitate further advancements across the field.

Figures

Figures reproduced from arXiv: 2504.21356 by the authors.

Figure 1
Figure 1. The architecture and the multi-stage training recipe for Nexus-Gen. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) The naive autoregressive approach exhibits in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Image reconstruction results of our generation de [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Image generation results from Nexus-Gen trained [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: MSE loss comparison between the naive and pre [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Dataset distribution of our Nexus-Gen dataset. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: The dataset construction pipeline for three subsets of ImagePulse. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Qualitative image generation results of Nexus-Gen. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Qualitative image editing results of Nexus-Gen. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An MLLM-conditioned next-scale VAR decoder handles 15+ unified visual generation tasks with competitive quality and substantially lower latency than diffusion baselines.

  2. Twins: Learn to Predict Unified Representations with Focal Loss

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Channel-wise concatenation of SigLIP2 and Flux VAE features into one token, trained with a focal-style flow-matching loss, yields a unified representation with 1.59 gFID on ImageNet 256 and VAE-level reconstruction.

  3. UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy

    cs.CV 2026-03 conditional novelty 6.0 of 10

    A six-level capability taxonomy plus UniICL-760K and a lightweight CAPM module improve unified multimodal few-shot learning and beat larger MLLMs on most understanding ICL tasks.

  4. Reconstruction Alignment Improves Unified Multimodal Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    RECA, a self-supervised post-training objective that conditions unified multimodal models on their own visual understanding embeddings to reconstruct input images, improves text-to-image and editing benchmarks across ...

  5. ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A 91K GPT-4o-generated image and editing dataset, and a fine-tuned open model Janus-4o, report improved text-to-image scores and new editing ability.

  6. Image Editing As Programs with Diffusion Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    IEAP decomposes complex editing instructions into atomic operations executed sequentially on a diffusion transformer, and reports state-of-the-art results on MagicBrush and AnyEdit.

  7. Physical Plausibility Reasoning via HCM-GRPO: Empowering Compact Model for Superior Performance

    cs.CV 2025-11 conditional novelty 5.0 of 10

    A 2B multimodal model trained with HCM-GRPO, a GRPO variant with partial-credit rewards and hard-case oversampling, outperforms larger models on the authors' private physical-plausibility test set.

  8. FramePrompt: In-context Controllable Animation with Zero Structural Changes

    cs.GR 2025-06 conditional novelty 5.0 of 10

    FramePrompt turns character animation into a video-continuation task by concatenating reference image, skeleton frames, and target frames into one sequence, then training the pretrained Wan-I2V model to generate only ...

  9. Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generation

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A new benchmark and agent framework for complex text-to-image generation, with an unvalidated AI-judge evaluation and claims that the agent outperforms GPT-4o on the authors' own benchmark.

  10. Instruction-based Image Editing: A Survey on Data, Models, Evaluation, and Applications

    cs.CV 2026-07 conditional novelty 4.0 of 10

    A survey of instruction-based image editing plus a new 21-task benchmark, CDD-IIE, on which ten open models are scored by human experts.

Reference graph

Works this paper leans on

59 extracted references · 21 canonical work pages · cited by 10 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  4. [4]

    Brooks, T.; Holynski, A.; and Efros, A. A. 2023. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 18392--18402

    work page 2023

  5. [5]

    Caron, M.; Touvron, H.; Misra, I.; J\'egou, H.; Mairal, J.; Bojanowski, P.; and Joulin, A. 2021. Emerging Properties in Self-Supervised Vision Transformers. In Proceedings of the International Conference on Computer Vision (ICCV)

    2021

  6. [6]

    Chen, J.; Xu, Z.; Pan, X.; Hu, Y.; Qin, C.; Goldstein, T.; Huang, L.; Zhou, T.; Xie, S.; Savarese, S.; et al. 2025 a . Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset. arXiv preprint arXiv:2505.09568

    arXiv 2025

  7. [7]

    Chen, X.; Wu, Z.; Liu, X.; Pan, Z.; Liu, W.; Xie, Z.; Yu, X.; and Ruan, C. 2025 b . Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling. arXiv preprint arXiv:2501.17811

    arXiv 2025

  8. [8]

    Creative, A. 2024. FLUX-Controlnet-Inpainting. https://github.com/alimama-creative/FLUX-Controlnet-Inpainting.git

    work page 2024

Show all 59 references
  1. [9]

    Esser, P.; Kulal, S.; Blattmann, A.; Entezari, R.; M \"u ller, J.; Saini, H.; Levi, Y.; Lorenz, D.; Sauer, A.; Boesel, F.; et al. 2024. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206

    2024 arXiv

  2. [10]

    Fu, C.; Chen, P.; Shen, Y.; Qin, Y.; Zhang, M.; Lin, X.; Yang, J.; Zheng, X.; Li, K.; Sun, X.; Wu, Y.; and Ji, R. 2024. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv:2306.13394

    2024 arXiv

  3. [11]

    Ge, Y.; Zhao, S.; Zhu, J.; Ge, Y.; Yi, K.; Song, L.; Li, C.; Ding, X.; and Shan, Y. 2024. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396

    2024 arXiv

  4. [12]

    Ghosh, D.; Hajishirzi, H.; and Schmidt, L. 2023. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132--52152

    2023

  5. [13]

    gogoduan. 2025. flux-laion-aes

    2025

  6. [14]

    Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; and Parikh, D. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, 6904--6913

    2017

  7. [15]

    Gu, T.; Yang, K.; Feng, Z.; Wang, X.; Zhang, Y.; Long, D.; Chen, Y.; Cai, W.; and Deng, J. 2025. Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs. arXiv preprint arXiv:2504.17432

    2025

  8. [16]

    Han, Z.; Mao, C.; Jiang, Z.; Pan, Y.; and Zhang, J. 2024. StyleBooth: Image Style Editing with Multimodal Instruction. arXiv preprint arXiv:2404.12154

    2024 arXiv

  9. [17]

    Hui, M.; Yang, S.; Zhao, B.; Shi, Y.; Wang, H.; Wang, P.; Zhou, Y.; and Xie, C. 2024. Hq-edit: A high-quality dataset for instruction-based image editing. arXiv preprint arXiv:2404.09990

    2024 arXiv

  10. [18]

    jackyhate. 2024. text-to-image-2M

    2024

  11. [19]

    Jiang, L.; Yan, Q.; Jia, Y.; Liu, Z.; Kang, H.; and Lu, X. 2025. InfiniteYou : Flexible Photo Recrafting While Preserving Your Identity. In ICCV

    2025

  12. [20]

    P.; Welling, M.; et al

    Kingma, D. P.; Welling, M.; et al. 2013. Auto-encoding variational bayes

    2013

  13. [21]

    Labs, B. F. 2024. FLUX. https://blackforestlabs.ai/announcing-black-forest-labs

    2024

  14. [22]

    LAION. 2024. laion-high-resolution

    2024

  15. [23]

    Li, B.; Wang, R.; Wang, G.; Ge, Y.; Ge, Y.; and Shan, Y. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125

    2023 arXiv

  16. [24]

    Li, T.; Tian, Y.; Li, H.; Deng, M.; and He, K. 2024. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37: 56424--56445

    2024

  17. [25]

    Liu, S.; Han, Y.; Xing, P.; Yin, F.; Wang, R.; Cheng, W.; Liao, J.; Wang, Y.; Fu, H.; Han, C.; et al. 2025. Step1x-edit: A practical framework for general image editing. arXiv preprint arXiv:2504.17761

    2025 arXiv

  18. [26]

    Liu, Z.; Luo, P.; Wang, X.; and Tang, X. 2015. Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision (ICCV)

    2015

  19. [27]

    Ma, Y.; Liu, X.; Chen, X.; Liu, W.; Wu, C.; Wu, Z.; Pan, Z.; Xie, Z.; Zhang, H.; Yu, X.; et al. 2025. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Con...

    2025

  20. [28]

    ModelScope. 2025. Diffsynth-Studio

    2025

  21. [29]

    OpenAI. 2025. Introducing 4o Image Generation

    2025

  22. [30]

    Pan, J.; Sun, K.; Ge, Y.; Li, H.; Duan, H.; Wu, X.; Zhang, R.; Zhou, A.; Qin, Z.; Wang, Y.; Dai, J.; Qiao, Y.; and Li, H. 2023. JourneyDB: A Benchmark for Generative Image Understanding. arXiv:2307.00716

    2023 arXiv

  23. [31]

    N.; Singh, A.; Zhao, Z.; Mishra, S

    Pan, X.; Shukla, S. N.; Singh, A.; Zhao, Z.; Mishra, S. K.; Wang, J.; Xu, Z.; Chen, J.; Li, K.; Juefei-Xu, F.; et al. 2025. Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.06256

    2025 arXiv

  24. [32]

    Podell, D.; English, Z.; Lacey, K.; Blattmann, A.; Dockhorn, T.; M \"u ller, J.; Penna, J.; and Rombach, R. 2024. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. In ICLR

    2024

  25. [33]

    K.; Yuan, Z.; and Wu, X

    Qu, L.; Zhang, H.; Liu, Y.; Wang, X.; Jiang, Y.; Gao, Y.; Ye, H.; Du, D. K.; Yuan, Z.; and Wu, X. 2025. Tokenflow: Unified image tokenizer for multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2545--2555

    2025

  26. [34]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PmLR

    2021

  27. [35]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1--67

    2020

  28. [36]

    V.; Zettlemoyer, L.; and Yu, L

    Shi, W.; Han, X.; Zhou, C.; Liang, W.; Lin, X. V.; Zettlemoyer, L.; and Yu, L. 2024. LlamaFusion: Adapting Pretrained Language Models for Multimodal Generation. arXiv preprint arXiv:2412.15188

    2024 arXiv

  29. [37]

    Singh, A.; Natarajan, V.; Shah, M.; Jiang, Y.; Chen, X.; Batra, D.; Parikh, D.; and Rohrbach, M. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8317--8326

    2019

  30. [38]

    Sun, P.; Jiang, Y.; Chen, S.; Zhang, S.; Peng, B.; Luo, P.; and Yuan, Z. 2024 a . Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525

    2024 arXiv

  31. [39]

    Sun, Q.; Cui, Y.; Zhang, X.; Zhang, F.; Yu, Q.; Wang, Y.; Rao, Y.; Liu, J.; Huang, T.; and Wang, X. 2024 b . Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14398--14409

    2024

  32. [40]

    Team, C. 2024. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818

    2024 arXiv

  33. [41]

    C.; Yang, J.; Yang, S.; Iyer, A.; Pan, X.; et al

    Tong, S.; Brown, E.; Wu, P.; Woo, S.; Middepogu, M.; Akula, S. C.; Yang, J.; Yang, S.; Iyer, A.; Pan, X.; et al. 2024 a . Cambrian-1: A fully open, vision-centric exploration of multimodal llms. In NeurIPS

    2024

  34. [42]

    Tong, S.; Fan, D.; Zhu, J.; Xiong, Y.; Chen, X.; Sinha, K.; Rabbat, M.; LeCun, Y.; Xie, S.; and Liu, Z. 2024 b . MetaMorph: Multimodal Understanding and Generation via Instruction Tuning. arXiv preprint arXiv:2412.14164

    2024 arXiv

  35. [43]

    Tuo, Y.; Xiang, W.; He, J.-Y.; Geng, Y.; and Xie, X. 2023. Anytext: Multilingual visual text generation and editing. arXiv preprint arXiv:2311.03054

    2023 arXiv

  36. [44]

    Wang, X.; Zhang, X.; Luo, Z.; Sun, Q.; Cui, Y.; Wang, J.; Zhang, F.; Wang, Y.; Li, Z.; Yu, Q.; et al. 2024. Emu3: Next-Token Prediction is All You Need. arXiv preprint arXiv:2409.18869

    2024 arXiv

  37. [45]

    J.; Montoya, E.; Munechika, D.; Yang, H.; Hoover, B.; and Chau, D

    Wang, Z. J.; Montoya, E.; Munechika, D.; Yang, H.; Hoover, B.; and Chau, D. H. 2022. DiffusionDB : A Large-Scale Prompt Gallery Dataset for Text-to-Image Generative Models. arXiv:2210.14896 [cs]

    2022 arXiv

  38. [46]

    Wei, C.; Xiong, Z.; Ren, W.; Du, X.; Zhang, G.; and Chen, W. 2024. OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision. arXiv preprint arXiv:2411.07199

    2024 arXiv

  39. [47]

    Wu, C.; Chen, X.; Wu, Z.; Ma, Y.; Liu, X.; Pan, Z.; Liu, W.; Xie, Z.; Yu, X.; Ruan, C.; et al. 2025. Janus: Decoupling visual encoding for unified multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, 12966--12977

    2025

  40. [48]

    Wu, Y.; Zhang, Z.; Chen, J.; Tang, H.; Li, D.; Fang, Y.; Zhu, L.; Xie, E.; Yin, H.; Yi, L.; et al. 2024. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429

    2024 arXiv

  41. [49]

    XAI. 2024. RealWorldQA. https://huggingface.co/datasets/visheratin/realworldqa

    2024

  42. [50]

    Xiao, S.; Wang, Y.; Zhou, J.; Yuan, H.; Xing, X.; Yan, R.; Li, C.; Wang, S.; Huang, T.; and Liu, Z. 2025. Omnigen: Unified image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, 13294--13304

    2025

  43. [51]

    J.; Wang, W.; Lin, K

    Xie, J.; Mao, W.; Bai, Z.; Zhang, D. J.; Wang, W.; Lin, K. Q.; Gu, Y.; Chen, Z.; Yang, Z.; and Shou, M. Z. 2024. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528

    2024 arXiv

  44. [52]

    Yu, Q.; Chow, W.; Yue, Z.; Pan, K.; Wu, Y.; Wan, X.; Li, J.; Tang, S.; Zhang, H.; and Zhuang, Y. 2025. Anyedit: Mastering unified high-quality image editing for any idea. In Proceedings of the Computer Vision and Pattern Recognition Conference, 26125--26135

    2025

  45. [53]

    Yue, X.; Ni, Y.; Zhang, K.; Zheng, T.; Liu, R.; Zhang, G.; Stevens, S.; Jiang, D.; Ren, W.; Sun, Y.; Wei, C.; Yu, B.; Yuan, R.; Sun, R.; Yin, M.; Zheng, B.; Yang, Z.; Liu, Y.; Huang, W.; Sun, H.; Su, Y.; and Chen, W. 2024. MMMU: A Massive Multi-discipline Multimodal Understand...

    2024

  46. [54]

    Zhai, X.; Mustafa, B.; Kolesnikov, A.; and Beyer, L. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, 11975--11986

    2023

  47. [55]

    Zhang, H.; Duan, Z.; Wang, X.; Chen, Y.; and Zhang, Y. 2025. EliGen: Entity-Level Controlled Image Generation with Regional Attention. arXiv preprint arXiv:2501.01097

    2025 arXiv

  48. [56]

    Zhang, K.; Mo, L.; Chen, W.; Sun, H.; and Su, Y. 2023. Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems, 36: 31428--31449

    2023

  49. [57]

    S.; Chen, L.; Si, S.; Wu, R.; An, K.; Yu, P.; Zhang, M.; Li, Q.; and Chang, B

    Zhao, H.; Ma, X. S.; Chen, L.; Si, S.; Wu, R.; An, K.; Yu, P.; Zhang, M.; Li, Q.; and Chang, B. 2024 a . Ultraedit: Instruction-based fine-grained image editing at scale. Advances in Neural Information Processing Systems, 37: 3058--3093

    2024

  50. [58]

    Zhao, Y.; Huang, J.; Hu, J.; Wang, X.; Mao, Y.; Zhang, D.; Jiang, Z.; Wu, Z.; Ai, B.; Wang, A.; Zhou, W.; and Chen, Y. 2024 b . SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning. arXiv:2408.05517

    2024 arXiv

  51. [59]

    Zhou, C.; Yu, L.; Babu, A.; Tirumala, K.; Yasunaga, M.; Shamis, L.; Kahn, J.; Ma, X.; Zettlemoyer, L.; and Levy, O. 2025. Transfusion: Predict the next token and diffuse images with one multi-modal model. In ICLR

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.