REVIEW 4 major objections 5 minor 299 references
Twins: Learn to Predict Unified Representations with Focal Loss
T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read This paper claims that a diffusion transformer can learn one token space that fuses semantic and pixel-detail features, and that a focal reweighting of the regression loss is what makes joint modeling work.
desk verdict Simple, useful recipe: channel-concat SigLIP2 and Flux.2 VAE tokens, train a DiT with a focal-style loss, and get a real gFID gain. The empirical result stands; the paper's explanation of why it works is under-tested, and the L^3 flow-matching objective needs scrutiny. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the focal regression weight w_i = |v_i − v_θ(z,t)_i|^(2γ) applied channel-wise to the VAE dimensions of the flow-matching MSE (Eqs. 5–6). Multiplying the squared error of a VAE channel by its own residual magnitude raised to power 2γ upweights the hard, high-frequency VAE dimensions during training, countering the network's spectral bias toward the smooth SigLIP component. The concatenation itself matters too: because the two feature types share a token grid, the unified token lengthens the channel dimension only, leaving sequence length and quadratic attention cost unchanged.
What would settle it
Train the same Twins model with the focal objective and estimate the conditional expectation E[v|x_t] by Monte Carlo on a held-out set along the interpolation path; if the focal-trained model's predicted velocity deviates systematically from E[v|x_t] at several noise levels, the reweighted loss distorts the generative flow rather than merely rebalancing optimization.
Extended reading notes
Core claim
The paper claims that a Diffusion Transformer can learn to predict a single continuous token formed by concatenating SigLIP2 semantic features with Flux.2 VAE latents on the same grid, and that the 'impossible triangle' among understanding, reconstruction, and generation is breakable. The obstacle is not the concatenation itself but the loss: with plain MSE the transformer settles into modeling the low-frequency, low-dimensional, condition-aligned ViT component and leaves the high-frequency, high-dimensional VAE component underfit. The authors trace this to three measurable heterogeneities and counter it by reweighting the per-dimension regression error on VAE channels with a focal term |err
Load-bearing premise
The upweighted training loss still points the model at the same target velocity field that plain MSE would, so the generated images come from the intended distribution; the paper does not analyze whether the L^(2+2γ) loss on VAE channels changes the conditional velocity optimum.
Editorial extensions
If this is right
- A single token space can serve both understanding and generation without a decode-encode round trip, reducing computation and engineering complexity for unified multimodal models.
- Jointly predicting semantic and VAE latents is feasible in continuous space, offering an alternative to discrete codebooks that lose visual information.
- The focal reweighting carries over to any flow-matching model that regresses heterogeneous target channels, not just ViT+VAE concatenations.
- With classifier-free guidance, the unified model reaches a competitive FID of 1.59 while retaining high reconstruction fidelity (PSNR 31.46, rFID 0.11), narrowing the gap between understanding- and generation-oriented representations.
Reading between the lines
- If the focal reweighting is truly a general calibration, it should transfer to other heterogeneous regression targets in flow matching (e.g., video latents, depth maps, audio features), but the paper does not test that; a natural next experiment would apply the same weighting scheme to those targets.
- The three diagnostic measures (frequency energy distribution, intrinsic dimension via Two-NN, and conditional PCA collapse) could serve as a cheap pre-training check for whether a proposed unified representation is learnable, before committing to expensive DiT runs.
- The paper never examines whether the reweighted loss still yields the conditional velocity E[v|x_t] that flow matching relies on; a targeted probe comparing focal-trained and MSE-trained velocity fields along the interpolation path would settle whether the gains come from rebalancing or from a biased flow.
- Because the loss upweights only VAE channels, the optimal downweighting of SigLIP channels might depend on dataset and resolution; testing γ sensitivity across different class-conditional and text-conditional settings would clarify how robust the reported 10.57 gFID gain is.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Twins, a unified continuous visual token space formed by channel-wise concatenating SigLIP2 ViT features and Flux.2 VAE latents on the same token grid, so token sequence length is unchanged. A Diffusion Transformer is trained to predict this concatenated representation with a flow-matching objective. The authors observe an optimization imbalance: the model fits the SigLIP component well but underfits the VAE component, which they attribute to spectral bias, higher intrinsic dimensionality, and condition-independent uncertainty of VAE latents. To correct this, they replace the MSE loss on VAE channels with a focal reweighting that upweights large residual errors, setting γ=0.5 in Eq. (5)-(6). They report large ImageNet gFID gains over MSE (up to 10.57 at 256x256 without guidance), competitive multimodal understanding, and reconstruction metrics. The central claim is that the focal objective balances heterogeneous feature learning and that this explains the improvement.
Significance. If the empirical results hold, the paper offers a simple, attention-cost-neutral way to unify semantic and pixel-level representations in a continuous space, with substantial generation gains over naive MSE. The paper ships code, uses external baselines, and provides component-wise FID trajectories, which are useful. However, the load-bearing mechanistic claim — that focal reweighting is a principled flow-matching objective that corrects an optimization imbalance — is not established. The L^{2+2γ} loss changes the statistical functional being estimated, and the paper neither analyzes this nor ablates it on single-encoder baselines. The reconstruction comparison is also overstated because Twins' PSNR matches the VAE decoder's own reconstruction. With additional analysis and ablations, the contribution could be significant; in its current form the theoretical framing and attribution of the gain are not fully supported.
major comments (4)
- [Section 3.3, Eqs. (5)-(6)] The focal loss replaces the flow-matching MSE with an L^{2+2γ} loss on VAE channels. The standard flow-matching guarantee that the MSE minimizer equals the conditional velocity E[v|x_t] is an L^2 property; for a general L^p loss the minimizer is a different functional of the conditional distribution. Since v = x1 - x0 contains an independent Gaussian component, even the optimal MSE solution has irreducible per-sample residuals on VAE channels, and the focal weight |residual|^{2γ} upweights exactly those noisy dimensions. The paper does not analyze what velocity field this objective actually learns, nor show that it still defines a valid generative flow. This gap is load-bearing for the claimed 'gradient balancing' explanation and for transferability of γ. Please provide a derivation or empirical diagnostics (e.g., comparing the learned velocity to the conditional mean on VAE channels, or
- [Table 3 and Table 5] The 10.57 gFID gain is attributed to focal loss, but there is no ablation of focal loss on the Flux.2 VAE-only baseline or the SigLIP2-only baseline. Without this, the gain could come from the concatenation itself, from the changed loss geometry, or from interactions, rather than from the stated imbalance correction. A target-standardization baseline (e.g., normalizing each channel to unit variance before MSE) is also missing; such a baseline would test whether the effect is just scale rebalancing rather than the proposed focal mechanism. These ablations are necessary to support the central claim.
- [Table 1 and Table 3] Twins is reported as achieving state-of-the-art reconstruction with PSNR 31.46, but this is identical to the Flux.2 VAE baseline in Table 3. Since decoding is performed with the VAE decoder on the VAE channels, the SigLIP component does not contribute to pixel reconstruction; the reported PSNR/SSIM/rFID are simply those of the frozen VAE. The claim that Twins 'improves reconstruction fidelity' relative to the VAE is therefore misleading. Please clarify that reconstruction comes solely from the VAE part and reposition this result.
- [Section 2, Fig. 5] The analysis asserts that single-class intrinsic dimension is the appropriate proxy for conditional generation difficulty. The empirical support in Fig. 5 (right) is a correlation across three feature types, not a controlled test of the causal claim. Since Section 2 is used to motivate the focal loss, this assertion needs a more direct test, e.g., varying class-conditioned ID while controlling other factors, or showing that single-class ID predicts per-component FID within the same model. Without this, the 'three sources of heterogeneity' analysis is suggestive but not established.
minor comments (5)
- [Abstract and Section 4.3] The phrase 'up to 10.57 gFID gain' is used without specifying the baseline configuration (epochs, guidance, resolution). Please state the exact comparison in the abstract or at first use.
- [Table 4] The column header says 'Generation@512' while the table body lists gFID etc.; this is fine, but the baseline Flux.2 VAE row at 80 epochs has no 'w/ guidance' entry, making the comparison to Twins with guidance incomplete. Please note whether guidance was not used for that row.
- [Section 3.3] The weighting is applied only to VAE dimensions; the SigLIP dimensions use plain MSE. This asymmetry is not discussed. If focal weighting is meant to balance the two components, an alternative would be a separate scale parameter per component; please justify the choice.
- [Appendix A.1] The Two-NN estimation is described for the global dataset, but Fig. 5 reports 'Class 1' intrinsic dimensions. Please state how the single-class ID is computed (e.g., subsample size, number of points) to ensure the estimate is meaningful.
- [Table 2] The Twins row with SigLIP2+Flux.2 VAE is compared to a SigLIP2-only baseline, but the VAE component adds 128 dimensions per token, increasing the projector input size. Please clarify whether the VLM training budget and projector capacity were matched.
Circularity Check
No significant circularity: the claimed gFID gains are empirical comparisons against an MSE baseline, gamma is hand-set, and the Section 2 analysis is post-hoc diagnosis rather than a prediction derived from a fitted input.
full rationale
The paper's central claims are benchmark results (gFID, PSNR, understanding scores) measured against external baselines, not quantities derived from fitted parameters. The focal loss constant gamma=0.5 is stated in Sec. 3.3 as a fixed experimental choice rather than fitted to the reported FID values; no equation is constructed to make a 'prediction' equal to an input. The Section 2 diagnosis (spectral bias, intrinsic dimensionality, conditional dependency) is a post-hoc explanation of an observed failure mode, not a derivation whose conclusion is assumed in its premises. Citations to prior work are standard (flow matching, Focal Loss, Two-NN, spectral bias) and do not function as self-referential justifications; the few papers by the authors' group (e.g., DA-VAE, 'Vision as a dialect') appear only as contextual related-work mentions. The one substantive concern is that the L^{2+2γ} objective may not preserve flow matching's L^2 conditional-velocity optimality; that is a correctness/validity risk, not a circularity, because the paper's empirical results would still be evidence for the method even if the theoretical explanation is incomplete.
Assumptions & free parameters
free parameters (2)
- focal exponent γ =
0.5
- implicit SigLIP/VAE loss balance =
1:1 per-dimension weighting, no λ tuned
assumptions (7)
- standard math MSE flow-matching regression learns the conditional velocity E[v|x_t]
- domain assumption Spectral bias of neural networks (Rahaman et al. 2019) governs DiT optimization priority
- domain assumption Learning difficulty scales with intrinsic dimension (Pope et al. 2021)
- domain assumption Two-NN estimator gives accurate intrinsic dimension for SigLIP/VAE features
- domain assumption SigLIP2 and Flux.2 VAE can be configured to share the same token grid (patch size P)
- ad hoc to paper The L^{2+2γ} focal objective still yields a valid flow-matching velocity model
- ad hoc to paper Single-class intrinsic dimension is the appropriate proxy for conditional generation difficulty
Cite this review
Pith. "Pith review of Twins: Learn to Predict Unified Representations with Focal Loss." pith.science (2026). https://pith.science/paper/2BDI7RYE
@misc{pith2026260722531,
author = {Pith},
title = {Pith review of: Twins: Learn to Predict Unified Representations with Focal Loss},
year = {2026},
howpublished = {\url{https://pith.science/paper/2BDI7RYE}},
note = {Machine review of arXiv:2607.22531}
}
read the original abstract
Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation. Discrete methods unify the interface via a shared codebook, whereas continuous pipelines often rely on two disparate representations -- semantic features (e.g., ViT) for understanding and low-level latents (e.g., VAE) for synthesis -- resulting in mismatched latent spaces. We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so the sequence length is unchanged and attention cost does not increase. However, jointly modeling Twins in a Diffusion Transformer exposes a severe optimization imbalance: the model fits the ViT component well but struggles to match the VAE latent distribution. We trace this imbalance to three sources of heterogeneity: frequency bias, intrinsic dimensionality, and condition-aligned vs condition-independent uncertainty. To address it, we adapt a focal regression objective for flow matching that upweights large-error VAE dimensions, better balancing optimization across the ViT and VAE components. On ImageNet, this yields up to 10.57 gFID gain over naive MSE loss without classifier-free guidance. Twins also performs competitively on multimodal understanding benchmarks and improves reconstruction fidelity, narrowing the gap between understanding- and generation-oriented representations.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2210.02747 , year=
Flow matching for generative modeling , author=. arXiv preprint arXiv:2210.02747 , year=
-
[2]
arXiv preprint arXiv:2209.14577 , year=
Rectified flow: A marginal preserving approach to optimal transport , author=. arXiv preprint arXiv:2209.14577 , year=
-
[3]
Advances in neural information processing systems , volume=
Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps , author=. Advances in neural information processing systems , volume=
-
[4]
Advances in neural information processing systems , volume=
Elucidating the design space of diffusion-based generative models , author=. Advances in neural information processing systems , volume=
-
[5]
arXiv preprint arXiv:2409.00588 , year=
Diffusion policy policy optimization , author=. arXiv preprint arXiv:2409.00588 , year=
-
[6]
Advances in Neural Information Processing Systems , volume=
Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models , author=. Advances in Neural Information Processing Systems , volume=
-
[7]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Align your latents: High-resolution video synthesis with latent diffusion models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[8]
International conference on machine learning , pages=
Trust region policy optimization , author=. International conference on machine learning , pages=. 2015 , organization=
2015
Show all 299 references
-
[9]
arXiv preprint arXiv:1707.06347 , year=
Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=
-
[10]
arXiv preprint arXiv:1312.5602 , year=
Playing atari with deep reinforcement learning , author=. arXiv preprint arXiv:1312.5602 , year=
-
[11]
arXiv preprint arXiv:2511.13649 , year=
Distribution Matching Distillation Meets Reinforcement Learning , author=. arXiv preprint arXiv:2511.13649 , year=
-
[12]
Proceedings of the 37th International Conference on Neural Information Processing Systems , pages=
ImageReward: learning and evaluating human preferences for text-to-image generation , author=. Proceedings of the 37th International Conference on Neural Information Processing Systems , pages=
-
[13]
arXiv preprint arXiv:2505.05470 , year=
Flow-grpo: Training flow matching models via online rl , author=. arXiv preprint arXiv:2505.05470 , year=
-
[14]
arXiv preprint arXiv:2505.07818 , year=
DanceGRPO: Unleashing GRPO on Visual Generation , author=. arXiv preprint arXiv:2505.07818 , year=
-
[15]
arXiv preprint arXiv:2305.13301 , year=
Training diffusion models with reinforcement learning , author=. arXiv preprint arXiv:2305.13301 , year=
-
[16]
arXiv preprint arXiv:2501.13918 , year=
Improving video generation with human feedback , author=. arXiv preprint arXiv:2501.13918 , year=
-
[17]
arXiv preprint arXiv:2501.12948 , year=
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=
-
[18]
arXiv preprint arXiv:2402.03300 , year=
Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=
-
[19]
European Conference on Computer Vision , pages=
Diffusion reward: Learning rewards via conditional video diffusion , author=. European Conference on Computer Vision , pages=. 2024 , organization=
2024
-
[20]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Diffusion model alignment using direct preference optimization , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[21]
arXiv preprint arXiv:2511.21541 , year=
Video Generation Models Are Good Latent Reward Models , author=. arXiv preprint arXiv:2511.21541 , year=
-
[22]
arXiv preprint arXiv:2512.04678 , year=
Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation , author=. arXiv preprint arXiv:2512.04678 , year=
-
[23]
arXiv preprint arXiv:2412.21059 , year=
Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation , author=. arXiv preprint arXiv:2412.21059 , year=
-
[25]
arXiv preprint arXiv:2511.18870 , year=
Hunyuanvideo 1.5 technical report , author=. arXiv preprint arXiv:2511.18870 , year=
-
[26]
arXiv preprint arXiv:2506.09113 , year=
Seedance 1.0: Exploring the Boundaries of Video Generation Models , author=. arXiv preprint arXiv:2506.09113 , year=
-
[27]
arXiv preprint arXiv:2511.19356 , year=
Growing with the Generator: Self-paced GRPO for Video Generation , author=. arXiv preprint arXiv:2511.19356 , year=
-
[28]
arXiv preprint arXiv:2509.05952 , year=
Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching , author=. arXiv preprint arXiv:2509.05952 , year=
-
[29]
arXiv preprint arXiv:2006.09359 , year=
Awac: Accelerating online reinforcement learning with offline datasets , author=. arXiv preprint arXiv:2006.09359 , year=
2006 arXiv
-
[30]
arXiv preprint arXiv:2309.17425 , year=
Data filtering networks , author=. arXiv preprint arXiv:2309.17425 , year=
-
[31]
Proceedings of the IEEE/CVF international conference on computer vision , pages=
An empirical study of training self-supervised vision transformers , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=
-
[32]
arXiv preprint arXiv:2106.08254 , year=
Beit: Bert pre-training of image transformers , author=. arXiv preprint arXiv:2106.08254 , year=
-
[33]
Proceedings of the IEEE/CVF international conference on computer vision , pages=
Vision transformers for dense prediction , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=
-
[34]
arXiv preprint arXiv:1907.10326 , year=
From big to small: Multi-scale local planar guidance for monocular depth estimation , author=. arXiv preprint arXiv:1907.10326 , year=
1907 arXiv
-
[35]
arXiv preprint arXiv:2111.11429 , year=
Benchmarking detection transfer learning with vision transformers , author=. arXiv preprint arXiv:2111.11429 , year=
-
[36]
Proceedings of the IEEE international conference on computer vision , pages=
Mask r-cnn , author=. Proceedings of the IEEE international conference on computer vision , pages=
-
[37]
ArXiv , year=
Zero-Shot Text-to-Image Generation , author=. ArXiv , year=
-
[38]
ArXiv , year=
Cosmos World Foundation Model Platform for Physical AI , author=. ArXiv , year=
-
[39]
2024 , eprint=
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers , author=. 2024 , eprint=
2024
-
[40]
2025 , eprint=
Wan: Open and Advanced Large-Scale Video Generative Models , author=. 2025 , eprint=
2025
-
[41]
arXiv preprint arXiv:2503.20314 , year=
Wan: Open and Advanced Large-Scale Video Generative Models , author=. arXiv preprint arXiv:2503.20314 , year=
-
[42]
2025 , eprint=
AToken: A Unified Tokenizer for Vision , author=. 2025 , eprint=
2025
-
[43]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
High-resolution image synthesis with latent diffusion models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[44]
arXiv preprint arXiv:2307.01952 , year=
Sdxl: Improving latent diffusion models for high-resolution image synthesis , author=. arXiv preprint arXiv:2307.01952 , year=
-
[45]
1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space , author=
FLUX. 1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space , author=. arXiv e-prints , pages=
-
[46]
arXiv preprint arXiv:2508.02324 , year=
Qwen-image technical report , author=. arXiv preprint arXiv:2508.02324 , year=
-
[47]
ArXiv , year=
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation , author=. ArXiv , year=
-
[48]
generation: Taming optimization dilemma in latent diffusion models , author=
Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[49]
ArXiv , year=
Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generation , author=. ArXiv , year=
-
[50]
Advances in Neural Information Processing Systems , volume=
Omnitokenizer: A joint image-video tokenizer for visual generation , author=. Advances in Neural Information Processing Systems , volume=
-
[51]
arXiv preprint arXiv:2507.15856 , year=
Latent Denoising Makes Good Visual Tokenizers , author=. arXiv preprint arXiv:2507.15856 , year=
-
[52]
Image Processing Algorithms and Techniques II , volume=
Residual VQ (vector quantizaton) with state prediction: a new method for image coding , author=. Image Processing Algorithms and Techniques II , volume=. 1991 , organization=
1991
-
[53]
International Conference on Learning Representations , volume=
Language model beats diffusion-tokenizer is key to visual generation , author=. International Conference on Learning Representations , volume=
-
[54]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Softvq-vae: Efficient 1-dimensional continuous tokenizer , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[55]
arXiv preprint arXiv:2410.01756 , year=
Imagefolder: Autoregressive image generation with folded tokens , author=. arXiv preprint arXiv:2410.01756 , year=
-
[56]
2022 , eprint=
Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow , author=. 2022 , eprint=
2022
-
[57]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Diffusion autoencoders: Toward a meaningful and decodable representation , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[58]
arXiv preprint arXiv:2405.09818 , year=
Chameleon: Mixed-modal early-fusion foundation models , author=. arXiv preprint arXiv:2405.09818 , year=
-
[59]
2025 , eprint=
Towards Semantic Equivalence of Tokenization in Multimodal LLM , author=. 2025 , eprint=
2025
-
[60]
arXiv preprint arXiv:2309.11499 , year=
Dreamllm: Synergistic multimodal comprehension and creation , author=. arXiv preprint arXiv:2309.11499 , year=
-
[61]
arXiv preprint arXiv:2309.04669 , year=
Unified language-vision pretraining in llm with dynamic discrete visual tokenization , author=. arXiv preprint arXiv:2309.04669 , year=
-
[62]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Janus: Decoupling visual encoding for unified multimodal understanding and generation , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[63]
arXiv preprint , year=
World Model on Million-Length Video and Language with RingAttention , author=. arXiv preprint , year=
-
[64]
arXiv preprint arXiv:2404.14396 , year=
Seed-x: Multimodal models with unified multi-granularity comprehension and generation , author=. arXiv preprint arXiv:2404.14396 , year=
-
[65]
arXiv preprint arXiv:2310.01218 , year=
Making llama see and draw with seed tokenizer , author=. arXiv preprint arXiv:2310.01218 , year=
-
[66]
arXiv preprint arXiv:2408.12528 , year=
Show-o: One single transformer to unify multimodal understanding and generation , author=. arXiv preprint arXiv:2408.12528 , year=
-
[67]
arXiv preprint arXiv:2409.18869 , year=
Emu3: Next-token prediction is all you need , author=. arXiv preprint arXiv:2409.18869 , year=
-
[68]
arXiv preprint arXiv:2409.04429 , year=
Vila-u: a unified foundation model integrating visual understanding and generation , author=. arXiv preprint arXiv:2409.04429 , year=
-
[69]
arXiv preprint arXiv:2412.06673 , year=
Illume: Illuminating your llms to see, draw, and self-enhance , author=. arXiv preprint arXiv:2412.06673 , year=
-
[70]
arXiv preprint arXiv:2412.00127 , year=
Orthus: Autoregressive interleaved image-text generation with modality-specific heads , author=. arXiv preprint arXiv:2412.00127 , year=
-
[71]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[72]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Tokenflow: Unified image tokenizer for multimodal understanding and generation , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[73]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Vila: On pre-training for visual language models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[74]
arXiv preprint arXiv:2411.17762 , year=
Muse-vl: Modeling unified vlm through semantic discrete encoding , author=. arXiv preprint arXiv:2411.17762 , year=
-
[75]
2023 , eprint=
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action , author=. 2023 , eprint=
2023
-
[76]
2025 , eprint=
LMFusion: Adapting Pretrained Language Models for Multimodal Generation , author=. 2025 , eprint=
2025
-
[77]
2025 , eprint=
Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling , author=. 2025 , eprint=
2025
-
[78]
2025 , eprint=
UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding , author=. 2025 , eprint=
2025
-
[79]
2025 , eprint=
Show-o2: Improved Native Unified Multimodal Models , author=. 2025 , eprint=
2025
-
[80]
arXiv preprint arXiv:2502.05178 , year=
Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation , author=. arXiv preprint arXiv:2502.05178 , year=
-
[81]
Proceedings of the IEEE international conference on computer vision , pages=
Focal loss for dense object detection , author=. Proceedings of the IEEE international conference on computer vision , pages=
-
[82]
arXiv preprint arXiv:2412.04332 , year=
Liquid: Language models are scalable and unified multi-modal generators , author=. arXiv preprint arXiv:2412.04332 , year=
-
[83]
arXiv preprint arXiv:2502.20321 , year=
Unitok: A unified tokenizer for visual generation and understanding , author=. arXiv preprint arXiv:2502.20321 , year=
-
[84]
arXiv preprint arXiv:2505.05422 , year=
Toklip: Marry visual tokens to clip for multimodal comprehension and generation , author=. arXiv preprint arXiv:2505.05422 , year=
-
[85]
arXiv preprint arXiv:2504.01934 , year=
Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement , author=. arXiv preprint arXiv:2504.01934 , year=
-
[86]
arXiv preprint arXiv:2412.01762 , year=
Xq-gan: An open-source image tokenization framework for autoregressive generation , author=. arXiv preprint arXiv:2412.01762 , year=
-
[87]
arXiv preprint arXiv:2503.08354 , year=
Robust latent matters: Boosting image generation with sampling error synthesis , author=. arXiv preprint arXiv:2503.08354 , year=
-
[88]
arXiv preprint arXiv:2506.01144 , year=
FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation , author=. arXiv preprint arXiv:2506.01144 , year=
-
[89]
arXiv preprint arXiv:2505.07538 , year=
Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning , author=. arXiv preprint arXiv:2505.07538 , year=
-
[90]
Forty-second International Conference on Machine Learning , year=
FlexTok: Resampling Images into 1D Token Sequences of Flexible Length , author=. Forty-second International Conference on Machine Learning , year=
-
[91]
arXiv preprint arXiv:2308.02490 , year=
Mm-vet: Evaluating large multimodal models for integrated capabilities , author=. arXiv preprint arXiv:2308.02490 , year=
-
[92]
arXiv preprint arXiv:2305.10355 , year=
Evaluating object hallucination in large vision-language models , author=. arXiv preprint arXiv:2305.10355 , year=
-
[93]
Proceedings of the IEEE conference on computer vision and pattern recognition , pages=
Making the v in vqa matter: Elevating the role of image understanding in visual question answering , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=
-
[94]
2025 , url=
Gemini 2.5 Flash , author=. 2025 , url=
2025
-
[95]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Gqa: A new dataset for real-world visual reasoning and compositional question answering , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[96]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Towards vqa models that can read , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[97]
European conference on computer vision , pages=
Mmbench: Is your multi-modal model an all-around player? , author=. European conference on computer vision , pages=. 2024 , organization=
2024
-
[98]
arXiv preprint arXiv:2408.03326 , year=
Llava-onevision: Easy visual task transfer , author=. arXiv preprint arXiv:2408.03326 , year=
-
[99]
nature , volume=
Deep learning , author=. nature , volume=. 2015 , publisher=
2015
-
[100]
Proceedings of the European conference on computer vision (ECCV) , pages=
Unified perceptual parsing for scene understanding , author=. Proceedings of the European conference on computer vision (ECCV) , pages=
-
[101]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[102]
arXiv preprint arXiv:2306.13394 , year=
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models , author=. arXiv preprint arXiv:2306.13394 , year=
-
[103]
European conference on computer vision , pages=
End-to-end object detection with transformers , author=. European conference on computer vision , pages=. 2020 , organization=
2020
-
[104]
International conference on machine learning , pages=
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation , author=. International conference on machine learning , pages=. 2022 , organization=
2022
-
[105]
arXiv preprint arXiv:2205.01917 , year=
Coca: Contrastive captioners are image-text foundation models , author=. arXiv preprint arXiv:2205.01917 , year=
-
[106]
arXiv preprint arXiv:2505.14683 , year=
Emerging properties in unified multimodal pretraining , author=. arXiv preprint arXiv:2505.14683 , year=
-
[107]
arXiv preprint arXiv:2505.05472 , year=
Mogao: An omni foundation model for interleaved multi-modal generation , author=. arXiv preprint arXiv:2505.05472 , year=
-
[108]
Advances in Neural Information Processing Systems , volume=
Depth anything v2 , author=. Advances in Neural Information Processing Systems , volume=
-
[109]
Proceedings of the IEEE/CVF international conference on computer vision , pages=
Segment anything , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=
-
[110]
arXiv preprint arXiv:2503.14324 , year=
Dualtoken: Towards unifying visual understanding and generation with dual visual vocabularies , author=. arXiv preprint arXiv:2503.14324 , year=
-
[111]
arXiv preprint arXiv:2503.06764 , year=
Semhitok: A unified image tokenizer via semantic-guided hierarchical codebook for multimodal understanding and generation , author=. arXiv preprint arXiv:2503.06764 , year=
-
[112]
arXiv preprint arXiv:2504.21356 , year=
Nexus-gen: A unified model for image understanding, generation, and editing , author=. arXiv preprint arXiv:2504.21356 , year=
-
[113]
Taming Transformers for High-Resolution Image Synthesis , author=
-
[114]
arXiv preprint arXiv:2303.08774 , year=
Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=
-
[115]
Advances in neural information processing systems , volume=
Language models are few-shot learners , author=. Advances in neural information processing systems , volume=
-
[116]
International conference on machine learning , pages=
Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=
2021
-
[117]
Advances in Neural Information Processing Systems , volume=
Unit: Unifying image and text recognition in one vision encoder , author=. Advances in Neural Information Processing Systems , volume=
-
[118]
arXiv preprint arXiv:2505.09568 , year=
Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset , author=. arXiv preprint arXiv:2505.09568 , year=
-
[119]
Liu, Haotian and Li, Chunyuan and Li, Yuheng and Lee, Yong Jae , title=
-
[120]
and Stoica, Ion and Xing, Eric P
Chiang, Wei-Lin and Li, Zhuohan and Lin, Zi and Sheng, Ying and Wu, Zhanghao and Zhang, Hao and Zheng, Lianmin and Zhuang, Siyuan and Zhuang, Yonghao and Gonzalez, Joseph E. and Stoica, Ion and Xing, Eric P. , month =. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\ url =
-
[121]
arXiv preprint arXiv:2412.15115 , year =
Qwen2.5 Technical Report , author =. arXiv preprint arXiv:2412.15115 , year =
-
[122]
Advances in neural information processing systems , volume=
Instructblip: Towards general-purpose vision-language models with instruction tuning , author=. Advances in neural information processing systems , volume=
-
[123]
arXiv preprint arXiv:2304.10592 , year=
Minigpt-4: Enhancing vision-language understanding with advanced large language models , author=. arXiv preprint arXiv:2304.10592 , year=
-
[124]
2024 , eprint=
What matters when building vision-language models? , author=. 2024 , eprint=
2024
-
[125]
Proceedings of the ieee/cvf conference on computer vision and pattern recognition , pages=
mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration , author=. Proceedings of the ieee/cvf conference on computer vision and pattern recognition , pages=
-
[126]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[127]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Improved baselines with visual instruction tuning , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[128]
arXiv preprint arXiv:2409.12191 , year=
Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution , author=. arXiv preprint arXiv:2409.12191 , year=
-
[129]
arXiv preprint arXiv:2503.10200 , year=
Lvagent: Long video understanding by multi-round dynamical collaboration of mllm agents , author=. arXiv preprint arXiv:2503.10200 , year=
-
[130]
arXiv preprint arXiv:2506.07576 , year=
Super Encoding Network: Recursive Association of Multi-Modal Encoders for Video Understanding , author=. arXiv preprint arXiv:2506.07576 , year=
-
[131]
arXiv preprint arXiv:2507.02626 , year=
VRAgent-R1: Boosting Video Recommendation with MLLM-based Agents via Reinforcement Learning , author=. arXiv preprint arXiv:2507.02626 , year=
-
[132]
arXiv preprint arXiv:2508.05709 , year=
G-UBS: Towards Robust Understanding of Implicit Feedback via Group-Aware User Behavior Simulation , author=. arXiv preprint arXiv:2508.05709 , year=
-
[133]
arXiv preprint arXiv:2506.06097 , year=
VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoning , author=. arXiv preprint arXiv:2506.06097 , year=
-
[134]
arXiv preprint arXiv:2402.18951 , year=
Percept, chat, and then adapt: Multimodal knowledge transfer of foundation models for open-world video recognition , author=. arXiv preprint arXiv:2402.18951 , year=
-
[135]
European Conference on Computer Vision , year=
Microsoft COCO: Common Objects in Context , author=. European Conference on Computer Vision , year=
-
[136]
ECCV , year =
Nathan Silberman, Derek Hoiem, Pushmeet Kohli and Rob Fergus , title =. ECCV , year =
-
[137]
International Journal of Computer Vision , volume=
Semantic understanding of scenes through the ade20k dataset , author=. International Journal of Computer Vision , volume=. 2019 , publisher=
2019
-
[138]
2025 , eprint=
TokBench: Evaluating Your Visual Tokenizer before Visual Generation , author=. 2025 , eprint=
2025
-
[139]
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer
Ren\' e Ranftl and Katrin Lasinger and David Hafner and Konrad Schindler and Vladlen Koltun. Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2022
2022
-
[140]
arXiv preprint arXiv:2510.10575 , year=
Uniflow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation , author=. arXiv preprint arXiv:2510.10575 , year=
-
[141]
arXiv preprint arXiv:2510.11690 , year=
Diffusion Transformers with Representation Autoencoders , author=. arXiv preprint arXiv:2510.11690 , year=
-
[142]
arXiv preprint arXiv:2510.15301 , year=
Latent Diffusion Model Without Variational Autoencoder , author=. arXiv preprint arXiv:2510.15301 , year=
-
[143]
arXiv preprint arXiv:2601.16208 , year=
Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders , author=. arXiv preprint arXiv:2601.16208 , year=
-
[144]
Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
Dc-ae 1.5: Accelerating diffusion model convergence with structured latent space , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
-
[145]
arXiv preprint arXiv:2512.19693 , year=
The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding , author=. arXiv preprint arXiv:2512.19693 , year=
-
[146]
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model , author=
-
[147]
LMFusion: Adapting Pretrained Language Models for Multimodal Generation , author=
-
[148]
Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations , author=
-
[149]
ICML , year=
StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis , author=. ICML , year=
-
[150]
ICCV , year=
Emerging Properties in Self-Supervised Vision Transformers , author=. ICCV , year=
-
[151]
2024 , howpublished=
Black Forest Labs , title=. 2024 , howpublished=
2024
-
[152]
2021 , booktitle=
Taming Transformers for High-Resolution Image Synthesis , author=. 2021 , booktitle=
2021
-
[153]
NeurIPS , year=
Differentiable Augmentation for Data-Efficient GAN Training , author=. NeurIPS , year=
-
[154]
2024 , booktitle=
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis , author=. 2024 , booktitle=
2024
-
[155]
CVPR , year=
High-Resolution Image Synthesis with Latent Diffusion Models , author=. CVPR , year=
-
[156]
2024 , eprint=
Massive Activations in Large Language Models , author=. 2024 , eprint=
2024
-
[157]
2020 , eprint=
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains , author=. 2020 , eprint=
2020
-
[158]
2022 , eprint=
Classifier-Free Diffusion Guidance , author=. 2022 , eprint=
2022
-
[159]
NeurIPS , year=
Guiding a Diffusion Model with a Bad Version of Itself , author=. NeurIPS , year=
-
[160]
NeurIPS , year=
Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models , author=. NeurIPS , year=
-
[161]
arXiv preprint arXiv:2406.06525 , year=
Autoregressive model beats diffusion: Llama for scalable image generation , author=. arXiv preprint arXiv:2406.06525 , year=
-
[162]
ICLR , year=
Language Model Beats Diffusion--Tokenizer is Key to Visual Generation , author=. ICLR , year=
-
[163]
NeurIPS , year=
Autoregressive image generation without vector quantization , author=. NeurIPS , year=
-
[164]
arXiv preprint arXiv:2303.14389 , year=
Mdtv2: Masked diffusion transformer is a strong image synthesizer , author=. arXiv preprint arXiv:2303.14389 , year=
-
[165]
ICCV , year=
Masked diffusion transformer is a strong image synthesizer , author=. ICCV , year=
-
[166]
ICCV , year=
REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers , author=. ICCV , year=
-
[167]
NeurIPS , volume=
Fasterdit: Towards faster diffusion transformers training without architecture modification , author=. NeurIPS , volume=
-
[168]
ECCV , year=
Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers , author=. ECCV , year=
-
[169]
TMLR , year=
Fast training of diffusion models with masked transformers , author=. TMLR , year=
-
[170]
ICLR , year=
Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think , author=. ICLR , year=
-
[171]
NeurIPS , year=
Visual autoregressive modeling: Scalable image generation via next-scale prediction , author=. NeurIPS , year=
-
[172]
ICCV , year=
Scalable diffusion models with transformers , author=. ICCV , year=
-
[173]
2025 , eprint=
SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features , author=. 2025 , eprint=
2025
-
[174]
2021 , booktitle=
Masked Autoencoders Are Scalable Vision Learners , author=. 2021 , booktitle=
2021
-
[175]
TMLR , year=
Dinov2: Learning robust visual features without supervision , author=. TMLR , year=
-
[176]
2025 , booktitle=
Vision Transformers Need Registers , author=. 2025 , booktitle=
2025
-
[177]
2025 , eprint=
DDT: Decoupled Diffusion Transformer , author=. 2025 , eprint=
2025
-
[178]
ICML , year=
Improving the Diffusability of Autoencoders , author=. ICML , year=
-
[179]
2025 , eprint=
Learnings from Scaling Visual Tokenizers for Reconstruction and Generation , author=. 2025 , eprint=
2025
-
[180]
2025 , eprint=
Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models , author=. 2025 , eprint=
2025
-
[181]
2022 , eprint=
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness , author=. 2022 , eprint=
2022
-
[182]
arXiv preprint arXiv:2503.21979 , year=
Harmonizing visual representations for unified multimodal understanding and generation , author=. arXiv preprint arXiv:2503.21979 , year=
-
[183]
ICML , year=
Eq-vae: Equivariance regularized latent space for improved generative image modeling , author=. ICML , year=
-
[184]
International journal of computer vision , volume=
Imagenet large scale visual recognition challenge , author=. International journal of computer vision , volume=. 2015 , publisher=
2015
-
[185]
ICLR , year=
Score-Based Generative Modeling through Stochastic Differential Equations , author=. ICLR , year=
-
[186]
Heusel, Martin and Ramsauer, Hubert and Unterthiner, Thomas and Nessler, Bernhard and Hochreiter, Sepp , booktitle=
-
[187]
ICML , year=
Generating Images with Sparse Representations , author=. ICML , year=
-
[188]
Improved Techniques for Training
Salimans, Tim and Goodfellow, Ian and Zaremba, Wojciech and Cheung, Vicki and Radford, Alec and Chen, Xi , booktitle=. Improved Techniques for Training
-
[189]
NeurIPS , year=
Improved Precision and Recall Metric for Assessing Generative Models , author=. NeurIPS , year=
-
[190]
Rethinking the
Szegedy, Christian and Vanhoucke, Vincent and Ioffe, Sergey and Shlens, Jon and Wojna, Zbigniew , booktitle=. Rethinking the
-
[191]
Diffusion models beat
Dhariwal, Prafulla and Nichol, Alexander , booktitle=. Diffusion models beat
-
[192]
Understanding Diffusion Objectives as the
Kingma, Diederik and Gao, Ruiqi , journal=NeurIPS, year=. Understanding Diffusion Objectives as the
-
[193]
ICML , year=
Simple Diffusion: End-to-End Diffusion for High Resolution Images , author=. ICML , year=
-
[194]
Journal of Machine Learning Research , volume=
Cascaded Diffusion Models for high fidelity image generation , author=. Journal of Machine Learning Research , volume=
-
[195]
All are Worth Words: A
Bao, Fan and Nie, Shen and Xue, Kaiwen and Cao, Yue and Li, Chongxuan and Su, Hang and Zhu, Jun , booktitle = CVPR, year=. All are Worth Words: A
-
[196]
Hatamizadeh, Ali and Song, Jiaming and Liu, Guilin and Kautz, Jan and Vahdat, Arash , booktitle=
-
[197]
Zhu, Rui and Pan, Yingwei and Li, Yehao and Yao, Ting and Sun, Zhenglong and Mei, Tao and Chen, Chang Wen , booktitle=
-
[198]
Progressive Growing of
Karras, Tero and Aila, Timo and Laine, Samuli and Lehtinen, Jaakko , booktitle=. Progressive Growing of
-
[199]
2025 , booktitle=
Masked Autoencoders Are Effective Tokenizers for Diffusion Models , author=. 2025 , booktitle=
2025
-
[200]
2025 , eprint=
DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Space , author=. 2025 , eprint=
2025
-
[201]
2025 , eprint=
Latent Denoising Makes Good Visual Tokenizers , author=. 2025 , eprint=
2025
-
[202]
NeurIPS , year=
An Image is Worth 32 Tokens for Reconstruction and Generation , author=. NeurIPS , year=
-
[203]
2025 , booktitle=
Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models , author=. 2025 , booktitle=
2025
-
[204]
2024 , eprint=
Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Models , author=. 2024 , eprint=
2024
-
[205]
arXiv e-prints, art , author=
Momentum contrast for unsupervised visual representation learning. arXiv e-prints, art , author=. arXiv preprint arXiv:1911.05722 , volume=
1911 arXiv
-
[206]
ICCV , year=
Emerging properties in self-supervised vision transformers , author=. ICCV , year=
-
[207]
ICML , year=
A simple framework for contrastive learning of visual representations , author=. ICML , year=
-
[208]
2025 , eprint=
Representation Entanglement for Generation:Training Diffusion Transformers Is Much Easier Than You Think , author=. 2025 , eprint=
2025
-
[209]
NeurIPS , year=
Boosting Generative Image Modeling via Joint Image-Feature Synthesis , author=. NeurIPS , year=
-
[210]
ICLR , year=
Auto-encoding variational bayes , author=. ICLR , year=
-
[211]
ICML , year=
Extracting and composing robust features with denoising autoencoders , author=. ICML , year=
-
[212]
2025 , eprint=
UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing , author=. 2025 , eprint=
2025
-
[213]
arXiv preprint arXiv:2303.08797 , year=
Stochastic interpolants: A unifying framework for flows and diffusions , author=. arXiv preprint arXiv:2303.08797 , year=
-
[214]
NeurIPS , year=
Elucidating the design space of diffusion-based generative models , author=. NeurIPS , year=
-
[215]
NeurIPS , year=
Denoising diffusion probabilistic models , author=. NeurIPS , year=
-
[216]
ICLR , year=
Flow matching for generative modeling , author=. ICLR , year=
-
[217]
ICLR , year=
Flow straight and fast: Learning to generate and transfer data with rectified flow , author=. ICLR , year=
-
[218]
arXiv preprint arXiv:2011.13456 , year=
Score-based generative modeling through stochastic differential equations , author=. arXiv preprint arXiv:2011.13456 , year=
2011 arXiv
-
[219]
2025 , eprint=
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models , author=. 2025 , eprint=
2025
-
[220]
2025 , eprint=
BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset , author=. 2025 , eprint=
2025
-
[221]
2025 , eprint=
Transfer between Modalities with MetaQueries , author=. 2025 , eprint=
2025
-
[222]
ICCV , year=
MetaMorph: Multimodal Understanding and Generation via Instruction Tuning , author=. ICCV , year=
-
[223]
CVPR , year=
Generative Multimodal Models are In-Context Learners , author=. CVPR , year=
-
[224]
NeurIPS , year=
Generating Diverse High-Fidelity Images with VQ-VAE-2 , author=. NeurIPS , year=
-
[225]
ICML , year=
Improved Denoising Diffusion Probabilistic Models , author=. ICML , year=
-
[226]
2022 , eprint=
Vector Quantized Diffusion Model for Text-to-Image Synthesis , author=. 2022 , eprint=
2022
-
[227]
TMLR , year=
Enhancing Sample Generation of Diffusion Models using Noise Level Correction , author=. TMLR , year=
-
[228]
ICLR , year=
Density estimation using real nvp , author=. ICLR , year=
-
[229]
ICML , year=
Flow++: Improving flow-based generative models with variational dequantization and architecture design , author=. ICML , year=
-
[230]
ICML , year=
Normalizing flows are capable generative models , author=. ICML , year=
-
[231]
ICLR , year=
Relay diffusion: Unifying diffusion process across resolutions for image synthesis , author=. ICLR , year=
-
[232]
arXiv preprint arXiv:2301.10972 , year=
On the importance of noise scheduling for diffusion models , author=. arXiv preprint arXiv:2301.10972 , year=
-
[233]
ICML , year=
Scalable adaptive computation for iterative generation , author=. ICML , year=
-
[234]
CVPR , year=
Simpler diffusion (sid2): 1.5 fid on imagenet512 with pixel-space diffusion , author=. CVPR , year=
-
[235]
arXiv preprint arXiv:2504.07963 , year=
PixelFlow: Pixel-Space Generative Models with Flow , author=. arXiv preprint arXiv:2504.07963 , year=
-
[236]
arXiv preprint arXiv:2507.23268 , year=
PixNerd: Pixel Neural Field Diffusion , author=. arXiv preprint arXiv:2507.23268 , year=
-
[237]
ICLR , year=
An image is worth 16x16 words: Transformers for image recognition at scale , author=. ICLR , year=
-
[238]
CVPR , year=
Analyzing and improving the training dynamics of diffusion models , author=. CVPR , year=
-
[239]
NeurIPS , year=
Flowdcn: Exploring dcn-like architectures for fast image generation with arbitrary resolution , author=. NeurIPS , year=
-
[240]
CVPR , year=
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , author=. CVPR , year=
-
[241]
2019 , booktitle=
Large Scale GAN Training for High Fidelity Natural Image Synthesis , author=. 2019 , booktitle=
2019
-
[242]
science , volume=
Reducing the dimensionality of data with neural networks , author=. science , volume=
-
[243]
NeurIPS , year=
Neural discrete representation learning , author=. NeurIPS , year=
-
[244]
NeurIPS , year=
Generative adversarial nets , author=. NeurIPS , year=
-
[245]
ICLR , year=
Vector-quantized image modeling with improved vqgan , author=. ICLR , year=
-
[246]
ICCV , year=
Efficient-VQGAN: Towards High-Resolution Image Generation with Efficient Vision Transformers , author=. ICCV , year=
-
[247]
CVPR , year=
Autoregressive image generation using residual quantization , author=. CVPR , year=
-
[248]
NeurIPS , year=
Movq: Modulating quantized vectors for high-fidelity image generation , author=. NeurIPS , year=
-
[249]
Finite Scalar Quantization:
Fabian Mentzer and David Minnen and Eirikur Agustsson and Michael Tschannen , booktitle=. Finite Scalar Quantization:
-
[250]
2025 , eprint=
Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens , author=. 2025 , eprint=
2025
-
[251]
2025 , booktitle=
Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens , author=. 2025 , booktitle=
2025
-
[252]
arXiv preprint arXiv:2204.06125 , volume=
Hierarchical text-conditional image generation with clip latents , author=. arXiv preprint arXiv:2204.06125 , volume=
-
[253]
TMLR , year=
Scaling autoregressive models for content-rich text-to-image generation , author=. TMLR , year=
-
[254]
ICML , year=
Image transformer , author=. ICML , year=
-
[255]
ICML , year=
Generative pretraining from pixels , author=. ICML , year=
-
[256]
CVPR , year=
Maskgit: Masked generative image transformer , author=. CVPR , year=
-
[257]
CVPR , year=
Mage: Masked generative encoder to unify representation learning and image synthesis , author=. CVPR , year=
-
[258]
arXiv preprint arXiv:2409.16211 , year=
MaskBit: Embedding-free Image Generation via Bit Tokens , author=. arXiv preprint arXiv:2409.16211 , year=
-
[259]
ICLR , year=
Image and Video Tokenization with Binary Spherical Quantization , author=. ICLR , year=
-
[260]
2025 , eprint=
GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation , author=. 2025 , eprint=
2025
-
[261]
NeurIPS , year=
When Worse is Better: Navigating the compression-generation tradeoff in visual tokenization , author=. NeurIPS , year=
-
[262]
ICLR , year=
LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior , author=. ICLR , year=
-
[263]
2025 , eprint=
Epsilon-VAE: Denoising as Visual Decoding , author=. 2025 , eprint=
2025
-
[264]
NeurIPS , year=
Visual Instruction Tuning , author=. NeurIPS , year=
-
[265]
ICLR , year=
Demystifying clip data , author=. ICLR , year=
-
[266]
ICCV , year=
Motion-guided masking for spatiotemporal representation learning , author=. ICCV , year=
-
[267]
ICML , year=
Data determines distributional robustness in contrastive language image pre-training (clip) , author=. ICML , year=
-
[268]
ICCV , year=
Revisiting unreasonable effectiveness of data in deep learning era , author=. ICCV , year=
-
[269]
Incomplete Ideas (blog) , year=
The bitter lesson , author=. Incomplete Ideas (blog) , year=
-
[270]
arXiv preprint arXiv:2412.16334 , year=
DINOv2 Meets Text: A Unified Framework for Image-and Pixel-Level Vision-Language Alignment , author=. arXiv preprint arXiv:2412.16334 , year=
-
[271]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Lit: Zero-shot transfer with locked-image text tuning , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[272]
Langley , title =
P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =
2000
-
[273]
T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980
1980
-
[274]
M. J. Kearns , title =
-
[275]
Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983
1983
-
[276]
R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000
2000
-
[277]
Suppressed for Anonymity , author=
-
[278]
Newell and P
A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981
1981
-
[279]
A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959
1959
-
[280]
arXiv preprint arXiv:2201.02177 , year=
Grokking: Generalization beyond overfitting on small algorithmic datasets , author=. arXiv preprint arXiv:2201.02177 , year=
-
[281]
arXiv preprint arXiv:2311.07574 , year=
To see is to believe: Prompting gpt-4v for better visual instruction tuning , author=. arXiv preprint arXiv:2311.07574 , year=
-
[282]
arXiv preprint arXiv:2306.17107 , year=
Llavar: Enhanced visual instruction tuning for text-rich image understanding , author=. arXiv preprint arXiv:2306.17107 , year=
-
[283]
CVPR , year=
A convnet for the 2020s , author=. CVPR , year=
-
[284]
ACL , year=
Chartqa: A benchmark for question answering about charts with visual and logical reasoning , author=. ACL , year=
-
[285]
WACV , year=
Docvqa: A dataset for vqa on document images , author=. WACV , year=
-
[286]
CVPR , year=
Dvqa: Understanding data visualizations via question answering , author=. CVPR , year=
-
[287]
AAAI , year=
TallyQA: Answering complex counting questions , author=. AAAI , year=
-
[288]
CVPR , year=
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning , author=. CVPR , year=
-
[289]
arXiv preprint arXiv:2311.16101 , year=
How many unicorns are in this image? a safety evaluation benchmark for vision llms , author=. arXiv preprint arXiv:2311.16101 , year=
-
[290]
CVPR , year=
Vizwiz grand challenge: Answering visual questions from blind people , author=. CVPR , year=
-
[291]
EMNLP , year=
Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation , author=. EMNLP , year=
-
[292]
arXiv preprint arXiv:2402.11684 , year=
ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model , author=. arXiv preprint arXiv:2402.11684 , year=
-
[293]
Proceedings of the National Academy of Sciences , volume=
Coherent concepts are computed in the anterior temporal lobes , author=. Proceedings of the National Academy of Sciences , volume=. 2010 , publisher=
2010
-
[294]
arXiv e-prints, art , author=
Momentum Contrast for Unsupervised Visual Representation Learning. arXiv e-prints, art , author=. CVPR , year=
-
[295]
TMLR , year=
Revisiting feature prediction for learning visual representations from video , author=. TMLR , year=
-
[296]
arXiv preprint arXiv:1606.08415 , year=
Gaussian error linear units (gelus) , author=. arXiv preprint arXiv:1606.08415 , year=
-
[297]
arXiv preprint arXiv:2404.01258 , year=
Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward , author=. arXiv preprint arXiv:2404.01258 , year=
-
[298]
ICLR , year=
Decoupled weight decay regularization , author=. ICLR , year=
-
[299]
NeurIPS , year=
Layer normalization , author=. NeurIPS , year=
-
[300]
CVPR , year=
Diffusion autoencoders: Toward a meaningful and decodable representation , author=. CVPR , year=
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.