Pith. sign in

REVIEW 3 major objections 5 minor 71 references

The paper argues that any-to-any generation across diverse modalities can be realized in a single decoder-only model without modality-specific heads, losses, or task pipelines, and reports competitive zero-shot results across depth, normals

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:01 UTC pith:62MDZFM7

load-bearing objection Real recipe-level finding; zero-shot numbers need a verifiable contamination check before the 'competitive with specialists' claim holds. the 3 major comments →

arxiv 2607.25948 v1 pith:62MDZFM7 submitted 2026-07-28 cs.CV cs.AIcs.LG

MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

Mingqiao Ye , Zhaochong An , Zhitong Gao , Xian Liu , Fran\c{c}ois Fleuret , Chuan Li , Amir Zadeh , Serge Belongie
show 6 more authors
Afshin Dehghan Jesse Allardice David Mizrahi O\u{g}uzhan Fatih Kar Roman Bachmann Amir Zamir
This is my paper
classification cs.CV cs.AIcs.LG
keywords any-to-any modelingdecoder-only multimodal generationflow matchingunified tokenizationmodality confusionchained generationself-verificationpseudo-label dataset
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MODUS sets out to show that any-to-any multimodal generation does not require encoder–decoder or diffusion backbones trained from scratch. The paper argues that one decoder-only transformer, initialized from a pretrained image–text model, can be extended to 15 modalities—depth, surface normals, edges, segmentation, grounding, learned visual features, and more—using a single unified token sequence and two training objectives inside one decoder. If correct, this means strong pretrained decoder-only priors can be reused for arbitrary modality-to-modality mappings with a cheap recipe, rather than building any-to-any models from scratch. The paper reports zero-shot results competitive with specialists on depth, normals, grounding, and retrieval, and it demonstrates chained generation, cross-modal self-verification, and representation composition as consequences of the unified design. The paper also flags two scope limits: it does not support iterative editing within the same modality, and it is pretraining-only without dedicated reasoning post-training (Appendix F).

Core claim

MODUS claims that any-to-any multimodal generation can be achieved in a single decoder-only transformer: all modalities are tokenized into one shared sequence, discrete modalities are generated by next-token prediction, continuous spatial modalities are generated by flow matching in latent space, and both share one causal attention context. No modality-specific heads, losses, or task pipelines are used. Starting from a pretrained image–text decoder and training on a 29M-sample pseudo-labelled corpus with uniform timestep sampling and a three-stage curriculum, MODUS reports zero-shot performance competitive with specialist models on depth, surface normal, grounding, and retrieval, while addit

What carries the argument

The load-bearing mechanism is the unified token sequence with two experts inside one decoder: a 1D Expert (causal attention, next-token prediction) for text, grounding boxes, detection, and feature tokens, and a 2D Expert (bidirectional attention within each modality, flow matching in a VAE latent space) for RGB, depth, normals, segmentation, and edges. Cross-modality attention remains causal so any token can condition later tokens. A second mechanism is uniform timestep sampling for flow matching, which keeps early denoising steps—where the target modality is decided—well represented and prevents modality mixing. A third is a staged curriculum that first learns 1D modalities, then adds 2D m

Load-bearing premise

The training corpus is machine-annotated by a fixed set of pretrained teachers, and the zero-shot results assume those teachers were not trained on the evaluation benchmarks and that their pseudo-labels are accurate enough to teach true cross-modal structure; the contamination check in Appendix C.5 is self-reported and cannot be independently verified from the text.

What would settle it

Compare the teacher models' training sets against the evaluation benchmarks used for zero-shot depth, normals, grounding, and retrieval; if any overlap exists, or if a fresh run with provably uncontaminated teachers reproduces the reported numbers, that would settle whether the results reflect genuine cross-modal modeling or imitation of the teachers.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single MODUS model performs zero-shot depth estimation, surface-normal prediction, grounding, and retrieval at levels near single-task specialists (Table 1).
  • Chaining through a spatially aligned intermediate (e.g., RGB→Canny→Surface Normal) improves the final target, while redundant or non-aligned intermediates do not (Table 2).
  • The same model can score its own outputs: best-of-4 self-verification lifts GenEval from 0.81 to 0.84 without an external verifier (Table 3).
  • The training recipe transfers: applying the same pipeline to a different 1.3B base model adds depth and surface-normal abilities at roughly 13× less compute than the base model's original pretraining (Appendix B.5).
  • Multi-conditioning (e.g., RGB+Depth→Surface Normal) beats single-condition RGB→Surface Normal, and Edge+Depth→Surface Normal nearly matches the RGB-based baseline without RGB input (Appendix C.4).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the pseudo-label supervision is clean and the contamination check holds, the results suggest that any-to-any modeling can be obtained by fine-tuning a pretrained vision-language decoder rather than training a new architecture from scratch, which could lower the cost of entering any-to-any domains like genomics or astronomy.
  • The modality-confusion story—early timesteps decide the target modality—implies that any multimodal flow-matching system sharing a single noise source may need early-timestep weighting; this is testable by reproducing the ablation on other unified diffusion or flow models.
  • The self-verification result suggests a general design principle: a model that generates all modalities can grade its own samples, which may extend to other modalities such as audio and to reasoning-style evaluation.
  • Because the released dataset is built from pseudo-labels produced by fixed teacher models, downstream users of the open-source release may inherit biases encoded by those teachers; this is a caution about the resource, not a claim the paper makes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MODUS, a decoder-only any-to-any multimodal model built on BAGEL-7B. It supports 15 modalities by mapping all inputs and outputs into a unified token sequence, with a 1D expert for discrete tokens (next-token prediction) and a 2D expert for continuous spatial latents (flow matching). Training uses uniform timestep sampling and a three-stage curriculum on MODUS-DATASET, a 29M-sample corpus with pseudo-labels from pretrained teachers. The paper reports competitive zero-shot results on depth, surface normals, grounding, and retrieval, and demonstrates chained generation, cross-modal self-verification, and visual representation composition. The central claim is that any-to-any generation across diverse modalities can be realized in a single, fully decoder-only model without modality-specific heads, losses, or task pipelines.

Significance. If the central claim holds, this is a significant result: it shows that strong pretrained decoder-only priors can be extended to arbitrary modality-to-modality mappings with a relatively cheap training recipe (roughly 5.7k GH200 hours), rather than training any-to-any models from scratch. The paper's strengths include controlled ablations of timestep sampling, staged training, and unified vs decoupled heads; a generalization experiment to a different base model (Janus-Flow); a self-verification demonstration scored on an external benchmark; and open release of the dataset and checkpoints. The main risk is the contamination check in Appendix C.5, which is self-reported and contains a likely incorrect assertion about DINOv2's training data. If evaluation images overlap with MODUS-DATASET, the zero-shot claims in Table 1 would be invalid. The ablation evidence otherwise supports the proposed recipe, though the 'competitive' claim is not uniformly true across all benchmarks.

major comments (3)
  1. [Appendix C.5 / Table 1] The contamination check is a brief assertion with no quantitative evidence. It states 'DINOv2 is not trained on the ImageNet splits used for our retrieval evaluation,' but DINOv2 was pretrained on LVD-142M, which includes ImageNet-22k; ImageNet-1k is a subset, so this claim is almost certainly false. The check also asserts that SA-1B, JourneyDB, and CC12M do not overlap evaluation benchmarks without reporting any hashing, duplicate-detection method, or overlap statistics. This is load-bearing: RefCOCO is built on COCO images, and CC12M is a web-scraped corpus that can plausibly contain COCO images. If any evaluation image appears in MODUS-DATASET, the zero-shot results in Table 1 are invalid. The authors must provide a verifiable contamination analysis (exact and near-duplicate overlap statistics) and correct the teacher-training-data claims.
  2. [Abstract / Sec. 3.2 / Fig. 2] The claim 'without modality-specific heads, losses, or task pipelines' is overstated as written. The architecture uses two experts with independent parameters and two different training objectives: cross-entropy for 1D tokens (Eq. 3) and flow-matching MSE for 2D latents (Eq. 4). It also relies on modality-specific encoders and tokenizers (SigLIP-2 ViT, FLUX VAE, DINOv2/CLIP/ImageBind tokenizers). This does not invalidate the unified-decoder idea, but the contribution statement should be qualified, for example as 'no task-specific output heads' or 'no per-task pipelines, but with modality-specific input encoders and two training objectives.'
  3. [Table 1 / Sec. 4.2] The claim of being 'competitive with specialist and multitask baselines' is only partially supported. On DIODE, MODUS (0.285) is worse than DepthAnything2 (0.249); on NYUv2 surface normals, MODUS (19.92) is worse than Marigold (16.40); on ImageNet retrieval, MODUS (77.9/92.5) is worse than DINOv2 (82.1/93.9). Only on RefCOCO grounding does MODUS clearly exceed the specialist (54.5 vs 50.4). The authors should either provide error bars/statistical tests or temper the claim to avoid overstating the comparison.
minor comments (5)
  1. [Sec. 3.3 / Table 5] 'Mode' timestep sampling is not defined. Please define it or cite the source.
  2. [Sec. 4.4 / Table 3] The VQA-based verifier is not described: what prompt is used, and how are VQA answer likelihoods aggregated into a score? Provide details.
  3. [Sec. 3.1 / Fig. 2] The 'modality instructions' in Figure 2(b) are mentioned but not exemplified. Provide concrete examples of the instruction format used to specify the target modality.
  4. [Appendix C.5] The claim that the GLaMM-based grounding pipeline 'remains zero-shot on COCO' should clarify whether GLaMM's own training data includes COCO images. If it does, the grounding pseudo-labels are not zero-shot with respect to COCO-based benchmarks.
  5. [Eqs. (1) and (2)] The arrow notation in Eqs. (1) and (2) is informal; define it as denoting the training or inference mapping from conditioning modalities to the target.

Circularity Check

0 steps flagged

No derivation-level circularity; the central benchmarks are external and the design choices are ablated rather than fitted. Minor self-citations exist but are not load-bearing.

full rationale

MODUS's training is a supervised mapping from condition modalities to target modalities on MODUS-DATASET (Section 3.4), with targets generated by external teachers (DepthAnything V2, Marigold, GLaMM, ViTDet, DINOv2, CLIP, ImageBind). The headline comparisons (Table 1) are measured against external benchmarks (MMMU, GenEval, DIODE, NYUv2, RefCOCO, ImageNet), not against the training targets or the teacher outputs, so the reported predictions are not fitted values renamed. The ablations (Tables 5-8) are run under fixed token budgets and compare design choices; none fits a parameter to the evaluation set. Self-verification (Table 3) selects among GenEval samples using the model's own grounding/VQA confidence, but the reported metric is external GenEval, so the selection signal is not the evaluation objective by construction. Self-citations (e.g., the 4M-style MLP tokenizer following Mizrahi et al. 2023) are implementation borrowings, not load-bearing premises; no uniqueness theorem or ansatz is imported from the authors' prior work. The only caveat is Appendix C.5's contamination check, a self-reported assertion without overlap statistics; if false it would weaken the zero-shot claim, but that is a validity/robustness issue, not a circular reduction. The derivation chain is therefore self-contained, with no step equivalent to its inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

No new physical or conceptual entities are introduced. The axiom burden is mostly domain assumptions: pretrained priors transfer, the RGB VAE represents non-RGB maps, teacher pseudo-labels are reliable and uncontaminated, and the two-expert causal-routing design preserves cross-modal information. The free parameters are hand-chosen training/inference settings; none are fitted to the evaluation benchmarks in a way that would invalidate the external comparisons.

free parameters (5)
  • Timestep sampling distribution = Uniform over [0,1] (vs logit-normal)
    Choice of uniform timestep sampling was made after observing modality mixing with logit-normal sampling (Section 3.3, Table 5). It is a hand-chosen distributional design decision, validated on target benchmarks.
  • Stage token budgets = 30B / 20B / 15B tokens
    The three-stage curriculum lengths (Table 17) are chosen by hand without a stated search procedure.
  • Conditioning modality count = 1 → 1 → 3
    Number of conditioning modalities per sample in each stage (Table 17) is an ad hoc curriculum choice.
  • LLaVA-OneVision data-mix ratio = Unspecified
    MODUS mixes LLaVA-OneVision into MODUS-DATASET to preserve image-text capabilities (Appendix C.3), but the exact mixture ratio is not stated, making the recipe incompletely specified.
  • Inference denoising steps and CFG scale = 5–50 steps; CFG scale not stated
    Section 3.5 says classifier-free guidance is applied for 2D outputs; the strength and step counts used for each reported number are not consistently listed.
axioms (6)
  • standard math Flow-matching objective (Lipman et al. 2022) is a valid generative objective for 2D latents.
    Equation (4) uses the standard flow-matching loss; no new math is derived.
  • domain assumption BAGEL-7B pretrained weights provide a useful prior that transfers to new modalities.
    The whole recipe initializes from BAGEL-7B (Section 3.2) and Appendix B.3 shows a from-scratch baseline converges slower and performs worse.
  • domain assumption FLUX VAE latents and SigLIP-2 ViT features faithfully represent non-RGB 2D maps (depth, normals, segmentation, edges).
    Section 3.1 uses a VAE trained on RGB images to encode all 2D modalities; if reconstruction of these maps is lossy, the 2D generation ceiling is set by the VAE. Table 4 supports the pairing empirically.
  • domain assumption Teacher pseudo-labels (DepthAnything, Marigold, GLaMM, SAM, DINOv2, CLIP, ImageBind) are accurate and uncontaminated for the evaluation benchmarks.
    Section 3.4 builds MODUS-DATASET from off-the-shelf teachers; the contamination check (Appendix C.5) is self-reported and cannot be verified from the manuscript.
  • ad hoc to paper Uniform timestep sampling reduces modality confusion in multimodal flow matching.
    Claimed in Section 3.3 and supported by Table 5 and Figure 3, but it is an empirical design principle rather than a derived theorem.
  • domain assumption A two-expert decoder with shared causal cross-modality attention suffices for any-to-any generation.
    Section 3.2: each token routes to the 1D or 2D expert; cross-modality attention is causal. There is no formal proof that this routing preserves all cross-modal information.

pith-pipeline@v1.3.0-alltime-deepseek · 31149 in / 15761 out tokens · 145162 ms · 2026-08-01T01:01:03.383945+00:00 · methodology

0 comments
read the original abstract

Any-to-any models predict any modality from any combination of others within a single network, a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named Modus, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. Modus demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced at https://modus-multimodal.epfl.ch/.

Figures

Figures reproduced from arXiv: 2607.25948 by Afshin Dehghan, Amir Zadeh, Amir Zamir, Chuan Li, David Mizrahi, Fran\c{c}ois Fleuret, Jesse Allardice, Mingqiao Ye, O\u{g}uzhan Fatih Kar, Roman Bachmann, Serge Belongie, Xian Liu, Zhaochong An, Zhitong Gao.

Figure 1
Figure 1. Figure 1: (1) Bottom left: Decoder-Only. MODUS is a decoder-only model, where all modalities are processed and generated within a single Transformer decoder using a unified autoregressive token sequence without modality-specific heads. (2) Top left: Any-to-Any. MODUS performs any-to-any modeling, i.e., predicting any modality given any other, and supports diverse 1D and 2D modalities. (3) Right: Multimodal Capabilit… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the MODUS framework. (a) Modalities. MODUS supports a diverse and extensible set of modalities. 1D sequential modalities include text, visual grounding boxes, object detection, and feature tokens such as DINOv2, CLIP, and ImageBind, in both global and spatially-local forms. 2D spatial modalities include RGB images, depth, surface normals, segmentation, and canny edges, together with SAM-based s… view at source ↗
Figure 3
Figure 3. Figure 3: Top: Multimodal Flow Matching. Early timesteps play a crucial role in determining the target modality, whereas later timesteps primarily refine visual quality. Bottom: Timestep Sam￾pling. Logit-normal sampling undersamples these early steps and oversamples intermediate ones, which causes the model to drift between modalities and produce mixed outputs. Uniform timestep sampling ensures balanced exposure acr… view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of Zero-shot Grounding Results. Given an image and a text query, MODUS predicts the corresponding re￾gion in a zero-shot manner within the unified decoder-only model. Input Depth Input Normal [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Zero-shot Depth and Surface Normal Estimation. Visualization on the NYUv2 Dataset. herent predictions. System-level comparison. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Any-to-Any Chained Generation. Rows are the input modality (leftmost column) and columns are the target; the diagonal (input = target) is left blank. Each target is generated by chaining: every newly generated modality is appended to the context used for the next. Because each output is conditioned on the previously generated ones, all targets describe the same underlying scene and stay mutually consistent… view at source ↗
Figure 7
Figure 7. Figure 7: Visual Representation Composition. ViT-only conditioning produces plausible but less faithful outputs with structural changes. VAE features supply low-level details important for geometric consistency. Conditioning on both ViT and VAE yields the most accurate results [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Modality confusion and timestep sampling for depth prediction. We compare inference trajectories across timesteps for models trained with uniform and logit-normal sampling. Uniform sampling covers early timesteps well, allowing the model to establish the correct depth modality early and produce stable predictions. Logit-normal sampling undersamples this regime, leading to early-stage modality mixing and fa… view at source ↗
Figure 9
Figure 9. Figure 9: Modality confusion and timestep sampling for surface normal prediction. We compare inference trajectories from models trained with uniform and logit-normal sampling. With uniform sampling, the model consistently locks onto the surface normal modality in the early timesteps and refines details thereafter. In contrast, logit-normal sampling provides little supervision in this early region, causing the model … view at source ↗
Figure 10
Figure 10. Figure 10: Training losses across 1D and 2D modalities. Loss curves for representative modalities during MODUS training. All modalities converge stably under the unified training regime, and initializing from BAGEL accelerates convergence across the board. As shown in [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Any-to-Any Independent Generation. Rows are the input modality (leftmost column) and columns are the target; each off-diagonal cell shows MODUS generating the column’s modality directly from the row’s modality within a single unified architecture (the diagonal, input = target, is left blank). Each target is predicted independently from the input, so the outputs along a row are free to vary and need not ag… view at source ↗
Figure 12
Figure 12. Figure 12: Chained text-to-image generation. Text prompts are transformed into intermediate 2D modalities, including Canny edges, depth maps, and surface normals, before producing the final RGB image. The examples demonstrate high visual quality and strong cross-modality consistency throughout the chained generation process. D.2. Chained Generation In Section 4.3 of the main paper, we discussed chained generation, w… view at source ↗
Figure 13
Figure 13. Figure 13: Chained image-to-surface normal prediction. The input image is first transformed into intermediate modalities, such as depth or Canny edges, and the resulting representations are then used to produce the final surface normal map. The examples illustrate consistent and coherent predictions across the chained modalities. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Text-to-image generation with self-verification. We apply the grounding capability of MODUS to evaluate the quality of its own text-to-image outputs and select the sample with the highest verification score. As shown, this simple test-time search, using a task already supported by MODUS, leads to improved image quality and better alignment with the input prompt. In Sec. 4.4 of the main paper, we discussed… view at source ↗
Figure 15
Figure 15. Figure 15: Illustration of hallucination with 2D modality feature representations. Each 2D modality in MODUS is represented using a ViT-based semantic feature and a VAE-based reconstruction feature. Conditioning the generation on the semantic feature can produce plausible but structurally incorrect depth or surface normal maps, showing changes in object shape, number, or fine details. GPT-4o exhibits similar behavio… view at source ↗
Figure 16
Figure 16. Figure 16: MODUS-DATASET examples. Each row shows one image together with its aligned annotations across the supported modalities: RGB, caption, depth, surface normal, canny and SAM-based edges, segmentation and SAM-based masks, detection, grounding, and DINOv2, CLIP, and ImageBind features (in both global and local forms). 34 [PITH_FULL_IMAGE:figures/full_fig_p034_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 12 linked inside Pith

  1. [1]

    Onestory: Coherent multi-shot video generation with adaptive memory

    An, Z., Jia, M., Qiu, H., Zhou, Z., Huang, X., Liu, Z., Ren, W., Kahatapitiya, K., Liu, D., He, S., et al. Onestory: Coherent multi-shot video generation with adaptive memory. CVPR, 2026 a

    2026

  2. [2]

    An, Z., Kupyn, O., Uscidda, T., Colaco, A., Ahuja, K., Belongie, S., Gonzalez-Franco, M., and Gazulla, M. T. Vggrpo: Towards world-consistent video generation with 4d latent reward. arXiv preprint arXiv:2603.26599, 2026 b

    Pith/arXiv arXiv 2026

  3. [3]

    F., Mizrahi, D., Garjani, A., Gao, M., Griffiths, D., Hu, J., Dehghan, A., and Zamir, A

    Bachmann, R., Kar, O. F., Mizrahi, D., Garjani, A., Gao, M., Griffiths, D., Hu, J., Dehghan, A., and Zamir, A. 4m-21: An any-to-any vision model for tens of tasks and modalities. NeurIPS, 37: 0 61872--61911, 2024

    2024

  4. [4]

    F., Amirloo, E., El-Nouby, A., Zamir, A., and Dehghan, A

    Bachmann, R., Allardice, J., Mizrahi, D., Fini, E., Kar, O. F., Amirloo, E., El-Nouby, A., Zamir, A., and Dehghan, A. Flextok: Resampling images into 1d token sequences of flexible length. In ICML, 2025

    2025

  5. [5]

    Qwen technical report

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint, 2023

    2023

  6. [6]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint, 2025

    2025

  7. [7]

    Batifol, S., Blattmann, A., Boesel, F., Consul, S., Diagne, C., Dockhorn, T., English, J., English, Z., Esser, P., Kulal, S., et al. Flux. 1 kontext: Flow matching for in-context image generation and editing in latent space. arXiv e-prints, pp.\ arXiv--2506, 2025

    2025

  8. [8]

    V., R \'e , C., and Mirhoseini, A

    Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., R \'e , C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Hunyuanimage 3.0 technical report

    Cao, S., Chen, H., Chen, P., Cheng, Y., Cui, Y., Deng, X., Dong, Y., Gong, K., Gu, T., Gu, X., et al. Hunyuanimage 3.0 technical report. arXiv preprint, 2025

    2025

  10. [10]

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset

    Chen, J., Xu, Z., Pan, X., Hu, Y., Qin, C., Goldstein, T., Huang, L., Zhou, T., Xie, S., Savarese, S., et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset. arXiv preprint, 2025 a

    2025

  11. [11]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., and Ruan, C. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint, 2025 b

    2025

  12. [12]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pp.\ 24185--24198, 2024

    2024

  13. [13]

    Tts-var: A test-time scaling framework for visual auto-regressive generation

    Chen, Z., Chu, R., Chen, Y., Zhang, S., Wei, Y., Zhang, Y., and Liu, X. Tts-var: A test-time scaling framework for visual auto-regressive generation. arXiv preprint arXiv:2507.18537, 2025 c

    arXiv 2025

  14. [14]

    Emerging properties in unified multimodal pretraining

    Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Scaling rectified flow transformers for high-resolution image synthesis

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., M \"u ller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024

    2024

  16. [16]

    Eva-02: A visual representation for neon genesis

    Fang, Y., Sun, Q., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 2024

    2024

  17. [17]

    Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image

    Fu, X., Yin, W., Hu, M., Wang, K., Ma, Y., Tan, P., Shen, S., Lin, D., and Long, X. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In ECCV, pp.\ 241--258. Springer, 2024

    2024

  18. [18]

    T., Genova, K., Kannen, N., Ben, S., Li, Y., Guo, M., and Yogin, S

    Gabeur, V., Long, S., Peng, S., Voigtlaender, P., Sun, S., Bao, Y., Truong, K., Wang, Z., Zhou, W., Barron, J. T., Genova, K., Kannen, N., Ben, S., Li, Y., Guo, M., and Yogin, S. Image generators are generalist vision learners. arXiv preprint arXiv:2604.20329, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Gao, Z., Rezaei, P., Cy, A., Ye, M., Jovanovi\' c , N., Allardice, J., Dehghan, A., Zamir, A., Bachmann, R., and Kar, O. F. (1d) ordered tokens enable efficient test-time search. arXiv preprint arXiv:2604.15453, 2026

    Pith/arXiv arXiv 2026

  20. [20]

    V., Joulin, A., and Misra, I

    Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  21. [21]

    A., Hu, V

    Gui, M., Schusterbauer, J., Prestel, U., Ma, P., Kotovenko, D., Grebenkova, O., Baumann, S. A., Hu, V. T., and Ommer, B. Depthfm: Fast generative monocular depth estimation with flow matching. In AAAI, volume 39, pp.\ 3203--3211, 2025

    2025

  22. [22]

    Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis

    Han, J., Liu, J., Jiang, Y., Yan, B., Zhang, Y., Yuan, Z., Peng, B., and Liu, X. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In CVPR, pp.\ 15733--15744, 2025 a

    2025

  23. [23]

    Infgen: A resolution-agnostic paradigm for scalable image synthesis

    Han, T., Xu, W., Gong, J., Yue, X., Guo, S., Zhou, L., and Bai, L. Infgen: A resolution-agnostic paradigm for scalable image synthesis. In ICCV, pp.\ 17941--17950, 2025 b

    2025

  24. [24]

    Lotus: Diffusion-based visual foundation model for high-quality dense prediction

    He, J., Li, H., Yin, W., Liang, Y., Li, L., Zhou, K., Zhang, H., Liu, B., and Chen, Y.-C. Lotus: Diffusion-based visual foundation model for high-quality dense prediction. ICLR, 2025

    2025

  25. [25]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint, 2024

    2024

  26. [26]

    F., Yeo, T., Atanov, A., and Zamir, A

    Kar, O. F., Yeo, T., Atanov, A., and Zamir, A. 3d common corruptions and data augmentation. In CVPR, pp.\ 18963--18974, 2022

    2022

  27. [27]

    C., and Schindler, K

    Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R. C., and Schindler, K. Repurposing diffusion-based image generators for monocular depth estimation. In CVPR, pp.\ 9492--9502, 2024

    2024

  28. [28]

    Marigold: Affordable adaptation of diffusion-based image generators for image analysis

    Ke, B., Qu, K., Wang, T., Metzger, N., Huang, S., Li, B., Obukhov, A., and Schindler, K. Marigold: Affordable adaptation of diffusion-based image generators for image analysis. TPAMI, 2025

    2025

  29. [29]

    C., Lo, W.-Y., Doll\'ar, P., and Girshick, R

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Doll\'ar, P., and Girshick, R. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023

    2023

  30. [30]

    H., Pham, T., Lee, S., Clark, C., Kembhavi, A., Mandt, S., Krishna, R., and Lu, J

    Le, D. H., Pham, T., Lee, S., Clark, C., Kembhavi, A., Mandt, S., Krishna, R., and Lu, J. One diffusion to generate them all. In CVPR, pp.\ 2671--2682, 2025

    2025

  31. [31]

    Llava-onevision: Easy visual task transfer

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava-onevision: Easy visual task transfer. arXiv preprint, 2024

    2024

  32. [32]

    Human Motion Instruction Tuning

    Li, L., Jia, S., Wang, J., Jiang, Z., Zhou, F., Dai, J., Zhang, T., Wu, Z., and Hwang, J.-N. Human Motion Instruction Tuning . In CVPR, 2025

    2025

  33. [33]

    Multiple human motion understanding

    Li, L., Jia, S., and Hwang, J.-N. Multiple human motion understanding. In AAAI, volume 40, pp.\ 6297--6305, 2026

    2026

  34. [34]

    Exploring plain vision transformer backbones for object detection

    Li, Y., Mao, H., Girshick, R., and He, K. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision (ECCV), 2022

    2022

  35. [35]

    Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models

    Lin, Z., Liu, C., Zhang, R., Gao, P., Qiu, L., Xiao, H., Qiu, H., Lin, C., Shao, W., Chen, K., et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint, 2023

    2023

  36. [36]

    T., Ben-Hamu, H., Nickel, M., and Le, M

    Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint, 2022

    2022

  37. [37]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. NeurIPS, 36: 0 34892--34916, 2023

    2023

  38. [38]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, pp.\ 38--55. Springer, 2024

    2024

  39. [39]

    S., Yang, S., Wang, Y., Yang, J., and Cheng, M.-M

    Liu, T., Wang, K., Li, S., van de Weijer, J., Khan, F. S., Yang, S., Wang, Y., Yang, J., and Cheng, M.-M. One-prompt-one-story: Free-lunch consistent text-to-image generation using a single prompt. ICLR, 2025

    2025

  40. [40]

    Unified-io: A unified model for vision, language, and multi-modal tasks

    Lu, J., Clark, C., Zellers, R., Mottaghi, R., and Kembhavi, A. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint, 2022

    2022

  41. [41]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In CVPR, pp.\ 26439--26455, 2024

    2024

  42. [42]

    Inference-time scaling for diffusion models beyond scaling denoising steps

    Ma, N., Tong, S., Jia, H., Hu, H., Su, Y.-C., Zhang, M., Yang, X., Li, Y., Jaakkola, T., Jia, X., and Xie, S. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.09732, 2025 a

    Pith/arXiv arXiv 2025

  43. [43]

    Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation

    Ma, Y., Liu, X., Chen, X., Liu, W., Wu, C., Wu, Z., Pan, Z., Xie, Z., Zhang, H., Yu, X., et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. In CVPR, pp.\ 7739--7751, 2025 b

    2025

  44. [44]

    4m: Massively multimodal masked modeling

    Mizrahi, D., Bachmann, R., Kar, O., Yeo, T., Gao, M., Dehghan, A., and Zamir, A. 4m: Massively multimodal masked modeling. NeurIPS, 36: 0 58363--58408, 2023

    2023

  45. [45]

    C., Scalia, G., and Eraslan, G

    Nair, S., Hajiramezanali, E., Tseng, A., Diamant, N., Hingerl, J., Lal, A., Biancalani, T., Bravo, H. C., Scalia, G., and Eraslan, G. Nona: A unifying multimodal masking framework for functional genomics. bioRxiv, pp.\ 2025--11, 2025

    2025

  46. [46]

    Dinov2: Learning robust visual features without supervision

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint, 2023

    2023

  47. [47]

    Aion-1: Omnimodal foundation model for astronomical sciences

    Parker, L., Lanusse, F., Shen, J., Liu, O., Hehir, T., Sarra, L., Meyer, L., Bowles, M., Wagner-Carena, S., Qu, H., et al. Aion-1: Omnimodal foundation model for astronomical sciences. NeurIPS, 2025

    2025

  48. [48]

    Kosmos-2: Grounding multimodal large language models to the world

    Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint, 2023

    2023

  49. [49]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, pp.\ 8748--8763. PmLR, 2021

    2021

  50. [50]

    F., and Zamir, A

    Ramachandran, R., Garjani, A., Bachmann, R., Atanov, A., Kar, O. F., and Zamir, A. How well does gpt-4o understand vision? evaluating multimodal foundation models on standard computer vision tasks. arXiv preprint, 2025

    2025

  51. [51]

    M., Xing, E., Yang, M.-H., and Khan, F

    Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E., Yang, M.-H., and Khan, F. S. Glamm: Pixel grounding large multimodal model. In CVPR, pp.\ 13009--13018, 2024

    2024

  52. [52]

    Grounded sam: Assembling open-world models for diverse visual tasks

    Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint, 2024

    2024

  53. [53]

    Prom3e: Probabilistic masked multimodal embedding model for ecology

    Sastry, S., Khanal, S., Dhakal, A., Lin, J., Cher, D., Jarosz, P., and Jacobs, N. Prom3e: Probabilistic masked multimodal embedding model for ecology. arXiv preprint arXiv:2511.02946, 2025

    arXiv 2025

  54. [54]

    A general framework for inference-time scaling and steering of diffusion models

    Singhal, R., Horvitz, Z., Teehan, R., Ren, M., Yu, Z., McKeown, K., and Ranganath, R. A general framework for inference-time scaling and steering of diffusion models. arXiv preprint arXiv:2501.06848, 2025

    Pith/arXiv arXiv 2025

  55. [55]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    Pith/arXiv arXiv 2024

  56. [56]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    Chameleon: Mixed-modal early-fusion foundation models

    Team, C. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint, 2024

    2024

  58. [58]

    M., Hauth, A., Millican, K., et al

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint, 2023

    2023

  59. [59]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang, L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NeurIPS, 37: 0 84839--84865, 2024

    2024

  60. [60]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint, 2023

    2023

  61. [61]

    F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., Hénaff, O., Harmsen, J., Steiner, A., and Zhai, X

    Tschannen, M., Gritsenko, A., Wang, X., Naeem, M. F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., Hénaff, O., Harmsen, J., Steiner, A., and Zhai, X. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  62. [62]

    Emu3: Next-token prediction is all you need

    Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q., Zhao, Y., Ao, Y., Min, X., Li, T., Wu, B., Zhao, B., Zhang, B., Wang, L., Liu, G., He, Z., Yang, X., Liu, J., Lin, Y., Huang, T., and Wang, Z. Emu3: Next-token prediction is all you need. arXiv preprint, 2024

    2024

  63. [63]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint, 2024 a

    2024

  64. [64]

    Next-gpt: Any-to-any multimodal llm

    Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. Next-gpt: Any-to-any multimodal llm. In Forty-first International Conference on Machine Learning, 2024 b

    2024

  65. [65]

    DeepSeek-VL2 : Mixture-of-experts vision-language models for advanced multimodal understanding

    Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al. DeepSeek-VL2 : Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024 c

    Pith/arXiv arXiv 2024

  66. [66]

    J., Wang, W., Lin, K

    Xie, J., Mao, W., Bai, Z., Zhang, D. J., Wang, W., Lin, K. Q., Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint, 2024

    2024

  67. [67]

    Depth anything v2

    Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., and Zhao, H. Depth anything v2. NeurIPS, 37: 0 21875--21911, 2024

    2024

  68. [68]

    Anygpt: Unified multimodal llm with discrete sequence modeling

    Zhan, J., Dai, J., Ye, J., Zhou, Y., Zhang, D., Liu, Z., Zhang, X., Yuan, R., Zhang, G., Li, L., et al. Anygpt: Unified multimodal llm with discrete sequence modeling. In ACL, pp.\ 9637--9662, 2024

    2024

  69. [69]

    Inference-time scaling of diffusion models through classical search

    Zhang, X., Lin, H., Ye, H., Zou, J., Ma, J., Liang, Y., and Du, Y. Inference-time scaling of diffusion models through classical search. arXiv preprint arXiv:2505.23614, 2025

    arXiv 2025

  70. [70]

    thinking with images

    Zheng, Z., Yang, M., Hong, J., Zhao, C., Xu, G., Yang, L., Shen, C., and Yu, X. Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505.14362, 2025

    Pith/arXiv arXiv 2025

  71. [71]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint, 2023

    2023