Pith. sign in

REVIEW 4 major objections 5 minor 299 references

This paper claims that a diffusion transformer can learn one token space that fuses semantic and pixel-detail features, and that a focal reweighting of the regression loss is what makes joint modeling work.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:24 UTC pith:2BDI7RYE

load-bearing objection Simple, useful recipe: channel-concat SigLIP2 and Flux.2 VAE tokens, train a DiT with a focal-style loss, and get a real gFID gain. The empirical result stands; the paper's explanation of why it works is under-tested, and the L^3 flow-matching objective needs scrutiny. the 4 major comments →

arxiv 2607.22531 v1 pith:2BDI7RYE submitted 2026-07-24 cs.CV

Twins: Learn to Predict Unified Representations with Focal Loss

Kaixiong Gong , Xin Cai , Bin Lin , Hao Wang , Yunlong Lin , Mingzhe Zheng , Bohao Li , Jian-Wei Zhang
show 4 more authors
Miles Yang Zhao Zhong Liefeng Bo Xiangyu Yue
This is my paper
classification cs.CV
keywords unified representationflow matchingfocal lossdiffusion transformervisual tokenizationimage generationmultimodal understandinglatent space
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Twins tries to break the apparent trade-off among image understanding, reconstruction, and generation by channel-wise concatenating SigLIP2 semantic features with Flux.2 VAE latents into a single continuous token space. The paper identifies why a standard diffusion transformer fails at this joint modeling: with plain MSE loss, the model quickly fits the easy, low-frequency, low-dimensional semantic component and neglects the harder, high-frequency VAE latents. To fix this, they adapt a focal regression objective for flow matching that upweights the large-error VAE dimensions, rebalancing the optimization. On ImageNet they report up to 10.57 gFID improvement over MSE without classifier-free guidance, a 1.59 FID with guidance, and understanding scores comparable to or better than a strong SigLIP2 baseline. A sympathetic reader would care because a truly shared continuous token space would let one model both perceive and generate images without extra encode-decode round trips.

Core claim

The paper claims that a Diffusion Transformer can learn to predict a single continuous token formed by concatenating SigLIP2 semantic features with Flux.2 VAE latents on the same grid, and that the 'impossible triangle' among understanding, reconstruction, and generation is breakable. The obstacle is not the concatenation itself but the loss: with plain MSE the transformer settles into modeling the low-frequency, low-dimensional, condition-aligned ViT component and leaves the high-frequency, high-dimensional VAE component underfit. The authors trace this to three measurable heterogeneities and counter it by reweighting the per-dimension regression error on VAE channels with a focal term |err

What carries the argument

The mechanism is the focal regression weight w_i = |v_i − v_θ(z,t)_i|^(2γ) applied channel-wise to the VAE dimensions of the flow-matching MSE (Eqs. 5–6). Multiplying the squared error of a VAE channel by its own residual magnitude raised to power 2γ upweights the hard, high-frequency VAE dimensions during training, countering the network's spectral bias toward the smooth SigLIP component. The concatenation itself matters too: because the two feature types share a token grid, the unified token lengthens the channel dimension only, leaving sequence length and quadratic attention cost unchanged.

Load-bearing premise

The upweighted training loss still points the model at the same target velocity field that plain MSE would, so the generated images come from the intended distribution; the paper does not analyze whether the L^(2+2γ) loss on VAE channels changes the conditional velocity optimum.

What would settle it

Train the same Twins model with the focal objective and estimate the conditional expectation E[v|x_t] by Monte Carlo on a held-out set along the interpolation path; if the focal-trained model's predicted velocity deviates systematically from E[v|x_t] at several noise levels, the reweighted loss distorts the generative flow rather than merely rebalancing optimization.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single token space can serve both understanding and generation without a decode-encode round trip, reducing computation and engineering complexity for unified multimodal models.
  • Jointly predicting semantic and VAE latents is feasible in continuous space, offering an alternative to discrete codebooks that lose visual information.
  • The focal reweighting carries over to any flow-matching model that regresses heterogeneous target channels, not just ViT+VAE concatenations.
  • With classifier-free guidance, the unified model reaches a competitive FID of 1.59 while retaining high reconstruction fidelity (PSNR 31.46, rFID 0.11), narrowing the gap between understanding- and generation-oriented representations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the focal reweighting is truly a general calibration, it should transfer to other heterogeneous regression targets in flow matching (e.g., video latents, depth maps, audio features), but the paper does not test that; a natural next experiment would apply the same weighting scheme to those targets.
  • The three diagnostic measures (frequency energy distribution, intrinsic dimension via Two-NN, and conditional PCA collapse) could serve as a cheap pre-training check for whether a proposed unified representation is learnable, before committing to expensive DiT runs.
  • The paper never examines whether the reweighted loss still yields the conditional velocity E[v|x_t] that flow matching relies on; a targeted probe comparing focal-trained and MSE-trained velocity fields along the interpolation path would settle whether the gains come from rebalancing or from a biased flow.
  • Because the loss upweights only VAE channels, the optimal downweighting of SigLIP channels might depend on dataset and resolution; testing γ sensitivity across different class-conditional and text-conditional settings would clarify how robust the reported 10.57 gFID gain is.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Twins, a unified continuous visual token space formed by channel-wise concatenating SigLIP2 ViT features and Flux.2 VAE latents on the same token grid, so token sequence length is unchanged. A Diffusion Transformer is trained to predict this concatenated representation with a flow-matching objective. The authors observe an optimization imbalance: the model fits the SigLIP component well but underfits the VAE component, which they attribute to spectral bias, higher intrinsic dimensionality, and condition-independent uncertainty of VAE latents. To correct this, they replace the MSE loss on VAE channels with a focal reweighting that upweights large residual errors, setting γ=0.5 in Eq. (5)-(6). They report large ImageNet gFID gains over MSE (up to 10.57 at 256x256 without guidance), competitive multimodal understanding, and reconstruction metrics. The central claim is that the focal objective balances heterogeneous feature learning and that this explains the improvement.

Significance. If the empirical results hold, the paper offers a simple, attention-cost-neutral way to unify semantic and pixel-level representations in a continuous space, with substantial generation gains over naive MSE. The paper ships code, uses external baselines, and provides component-wise FID trajectories, which are useful. However, the load-bearing mechanistic claim — that focal reweighting is a principled flow-matching objective that corrects an optimization imbalance — is not established. The L^{2+2γ} loss changes the statistical functional being estimated, and the paper neither analyzes this nor ablates it on single-encoder baselines. The reconstruction comparison is also overstated because Twins' PSNR matches the VAE decoder's own reconstruction. With additional analysis and ablations, the contribution could be significant; in its current form the theoretical framing and attribution of the gain are not fully supported.

major comments (4)
  1. [Section 3.3, Eqs. (5)-(6)] The focal loss replaces the flow-matching MSE with an L^{2+2γ} loss on VAE channels. The standard flow-matching guarantee that the MSE minimizer equals the conditional velocity E[v|x_t] is an L^2 property; for a general L^p loss the minimizer is a different functional of the conditional distribution. Since v = x1 - x0 contains an independent Gaussian component, even the optimal MSE solution has irreducible per-sample residuals on VAE channels, and the focal weight |residual|^{2γ} upweights exactly those noisy dimensions. The paper does not analyze what velocity field this objective actually learns, nor show that it still defines a valid generative flow. This gap is load-bearing for the claimed 'gradient balancing' explanation and for transferability of γ. Please provide a derivation or empirical diagnostics (e.g., comparing the learned velocity to the conditional mean on VAE channels, or
  2. [Table 3 and Table 5] The 10.57 gFID gain is attributed to focal loss, but there is no ablation of focal loss on the Flux.2 VAE-only baseline or the SigLIP2-only baseline. Without this, the gain could come from the concatenation itself, from the changed loss geometry, or from interactions, rather than from the stated imbalance correction. A target-standardization baseline (e.g., normalizing each channel to unit variance before MSE) is also missing; such a baseline would test whether the effect is just scale rebalancing rather than the proposed focal mechanism. These ablations are necessary to support the central claim.
  3. [Table 1 and Table 3] Twins is reported as achieving state-of-the-art reconstruction with PSNR 31.46, but this is identical to the Flux.2 VAE baseline in Table 3. Since decoding is performed with the VAE decoder on the VAE channels, the SigLIP component does not contribute to pixel reconstruction; the reported PSNR/SSIM/rFID are simply those of the frozen VAE. The claim that Twins 'improves reconstruction fidelity' relative to the VAE is therefore misleading. Please clarify that reconstruction comes solely from the VAE part and reposition this result.
  4. [Section 2, Fig. 5] The analysis asserts that single-class intrinsic dimension is the appropriate proxy for conditional generation difficulty. The empirical support in Fig. 5 (right) is a correlation across three feature types, not a controlled test of the causal claim. Since Section 2 is used to motivate the focal loss, this assertion needs a more direct test, e.g., varying class-conditioned ID while controlling other factors, or showing that single-class ID predicts per-component FID within the same model. Without this, the 'three sources of heterogeneity' analysis is suggestive but not established.
minor comments (5)
  1. [Abstract and Section 4.3] The phrase 'up to 10.57 gFID gain' is used without specifying the baseline configuration (epochs, guidance, resolution). Please state the exact comparison in the abstract or at first use.
  2. [Table 4] The column header says 'Generation@512' while the table body lists gFID etc.; this is fine, but the baseline Flux.2 VAE row at 80 epochs has no 'w/ guidance' entry, making the comparison to Twins with guidance incomplete. Please note whether guidance was not used for that row.
  3. [Section 3.3] The weighting is applied only to VAE dimensions; the SigLIP dimensions use plain MSE. This asymmetry is not discussed. If focal weighting is meant to balance the two components, an alternative would be a separate scale parameter per component; please justify the choice.
  4. [Appendix A.1] The Two-NN estimation is described for the global dataset, but Fig. 5 reports 'Class 1' intrinsic dimensions. Please state how the single-class ID is computed (e.g., subsample size, number of points) to ensure the estimate is meaningful.
  5. [Table 2] The Twins row with SigLIP2+Flux.2 VAE is compared to a SigLIP2-only baseline, but the VAE component adds 128 dimensions per token, increasing the projector input size. Please clarify whether the VLM training budget and projector capacity were matched.

Circularity Check

0 steps flagged

No significant circularity: the claimed gFID gains are empirical comparisons against an MSE baseline, gamma is hand-set, and the Section 2 analysis is post-hoc diagnosis rather than a prediction derived from a fitted input.

full rationale

The paper's central claims are benchmark results (gFID, PSNR, understanding scores) measured against external baselines, not quantities derived from fitted parameters. The focal loss constant gamma=0.5 is stated in Sec. 3.3 as a fixed experimental choice rather than fitted to the reported FID values; no equation is constructed to make a 'prediction' equal to an input. The Section 2 diagnosis (spectral bias, intrinsic dimensionality, conditional dependency) is a post-hoc explanation of an observed failure mode, not a derivation whose conclusion is assumed in its premises. Citations to prior work are standard (flow matching, Focal Loss, Two-NN, spectral bias) and do not function as self-referential justifications; the few papers by the authors' group (e.g., DA-VAE, 'Vision as a dialect') appear only as contextual related-work mentions. The one substantive concern is that the L^{2+2γ} objective may not preserve flow matching's L^2 conditional-velocity optimality; that is a correctness/validity risk, not a circularity, because the paper's empirical results would still be evidence for the method even if the theoretical explanation is incomplete.

Axiom & Free-Parameter Ledger

2 free parameters · 7 axioms · 0 invented entities

The central experiment depends on two hand-set hyperparameters (γ=0.5 and the implicit SigLIP/VAE loss balance). The diagnostic measurements (spectral energy, Two-NN ID, PCA) are explanatory and are not used to set constants in the loss. No new entities are introduced. The main unexamined assumption is that L^p flow matching (p=3) remains a valid generative objective.

free parameters (2)
  • focal exponent γ = 0.5
    Set by hand with no ablation or sensitivity analysis; controls the degree of upweighting of large-residual VAE dimensions in the flow-matching loss (Eq. 6).
  • implicit SigLIP/VAE loss balance = 1:1 per-dimension weighting, no λ tuned
    The total loss is asserted as MSE on SigLIP dims plus focal loss on VAE dims without stating the relative scale; the balance is a hidden choice that directly affects the optimization balance the paper claims to fix.
axioms (7)
  • standard math MSE flow-matching regression learns the conditional velocity E[v|x_t]
    Sec 3.1 Eq (2); underpins why the DiT can generate at all. The focal loss modifies this without re-establishing the property.
  • domain assumption Spectral bias of neural networks (Rahaman et al. 2019) governs DiT optimization priority
    Sec 2.1; used to claim the network fits low-frequency SigLIP features first, delaying VAE high-frequency learning.
  • domain assumption Learning difficulty scales with intrinsic dimension (Pope et al. 2021)
    Sec 2.2; used to claim the high-ID VAE manifold is harder to learn than the low-ID SigLIP manifold.
  • domain assumption Two-NN estimator gives accurate intrinsic dimension for SigLIP/VAE features
    Sec 2.2 / Appendix A.1; the ID values (≈15 vs ≈35) are load-bearing for the second cause of imbalance.
  • domain assumption SigLIP2 and Flux.2 VAE can be configured to share the same token grid (patch size P)
    Sec 3.2; if the token grids differ, channel-wise concatenation is invalid. The paper asserts this configuration without a derivation.
  • ad hoc to paper The L^{2+2γ} focal objective still yields a valid flow-matching velocity model
    Sec 3.3 Eqs (5)-(6); the standard flow-matching MSE theory does not transfer to L^p losses without an argument, and the paper supplies none.
  • ad hoc to paper Single-class intrinsic dimension is the appropriate proxy for conditional generation difficulty
    Sec 2.2; they use single-class ID rather than global ID to argue SigLIP is easy, justified only by a brief remark about 'semantic islands'.

pith-pipeline@v1.3.0-alltime-deepseek · 21149 in / 21801 out tokens · 211383 ms · 2026-08-01T04:24:48.698823+00:00 · methodology

0 comments
read the original abstract

Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation. Discrete methods unify the interface via a shared codebook, whereas continuous pipelines often rely on two disparate representations -- semantic features (e.g., ViT) for understanding and low-level latents (e.g., VAE) for synthesis -- resulting in mismatched latent spaces. We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so the sequence length is unchanged and attention cost does not increase. However, jointly modeling Twins in a Diffusion Transformer exposes a severe optimization imbalance: the model fits the ViT component well but struggles to match the VAE latent distribution. We trace this imbalance to three sources of heterogeneity: frequency bias, intrinsic dimensionality, and condition-aligned vs condition-independent uncertainty. To address it, we adapt a focal regression objective for flow matching that upweights large-error VAE dimensions, better balancing optimization across the ViT and VAE components. On ImageNet, this yields up to 10.57 gFID gain over naive MSE loss without classifier-free guidance. Twins also performs competitively on multimodal understanding benchmarks and improves reconstruction fidelity, narrowing the gap between understanding- and generation-oriented representations.

Figures

Figures reproduced from arXiv: 2607.22531 by Bin Lin, Bohao Li, Hao Wang, Jian-Wei Zhang, Kaixiong Gong, Liefeng Bo, Miles Yang, Mingzhe Zheng, Xiangyu Yue, Xin Cai, Yunlong Lin, Zhao Zhong.

Figure 1
Figure 1. Figure 1: Breaking the “Impossible Triangle” of visual tokeniza￾tion. Existing approaches (edges) are forced to trade off between Understanding, Reconstruction, and Generation. Twins (Ours) con￾structs a unified representation that explicitly fuses semantic-rich ViT features with detail-preserving VAE features, simultaneously satisfying all three objectives. 1. Introduction Unified multimodal models (Zhou et al., 20… view at source ↗
Figure 2
Figure 2. Figure 2: FID trajectories of the SigLIP and VAE components within the Twins representation. In the baseline setting, a critical failure mode emerges where the DiT fits SigLIP features well but struggles to model VAE latents. Focal Loss significantly mitigates this imbalance, leading to a substantial reduction in VAE FID (∆ = 5.36 at 80 epochs) while maintaining performance on the SigLIP component. semantics (e.g., … view at source ↗
Figure 5
Figure 5. Figure 5: (Left) Comparison of physical and intrinsic dimen￾sions (ID) estimated via Two-NN. A dimensionality paradox is observed: SigLIP has a higher physical dimension but a sig￾nificantly lower ID than VAE, indicating a highly compressed manifold. (Right): Generation performance (FID) across train￾ing epochs for features with varying IDs. Features with higher ID (e.g., Flux.2) exhibit significantly higher FID and… view at source ↗
Figure 6
Figure 6. Figure 6: PCA cumulative explained variance. The drastic col￾lapse of SigLIP’s effective dimension under single-class conditions (dashed line) reveals its strong conditional dependency, whereas the consistently high dimensionality of VAE features indicates significant condition-independent uncertainty. rich in texture and noise. According to the Spectral Bias theory (Rahaman et al., 2019), neural networks naturally … view at source ↗
Figure 7
Figure 7. Figure 7: Overview: (Left) Construction: The Twins Tokenizer creates a unified visual token by concatenating semantic features from the SigLIP encoder and fine-grained latents from the Flux.2 VAE encoder along the channel dimension. (Middle) Generation: During DiT training, we employ Focal Loss within the Flow Matching objective to balance the learning of heterogeneous feature spaces. (Right) Understanding: For VLM … view at source ↗
Figure 8
Figure 8. Figure 8: Generation: MSE vs. Focal. MSE Loss generates blurred and distorted images [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

299 extracted references · 79 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2210.02747 , year=

    Flow matching for generative modeling , author=. arXiv preprint arXiv:2210.02747 , year=

    Pith/arXiv arXiv

  2. [2]

    arXiv preprint arXiv:2209.14577 , year=

    Rectified flow: A marginal preserving approach to optimal transport , author=. arXiv preprint arXiv:2209.14577 , year=

    Pith/arXiv arXiv

  3. [3]

    Advances in neural information processing systems , volume=

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps , author=. Advances in neural information processing systems , volume=

  4. [4]

    Advances in neural information processing systems , volume=

    Elucidating the design space of diffusion-based generative models , author=. Advances in neural information processing systems , volume=

  5. [5]

    arXiv preprint arXiv:2409.00588 , year=

    Diffusion policy policy optimization , author=. arXiv preprint arXiv:2409.00588 , year=

    Pith/arXiv arXiv

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models , author=. Advances in Neural Information Processing Systems , volume=

  7. [7]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Align your latents: High-resolution video synthesis with latent diffusion models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  8. [8]

    International conference on machine learning , pages=

    Trust region policy optimization , author=. International conference on machine learning , pages=. 2015 , organization=

    2015

  9. [9]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:1312.5602 , year=

    Playing atari with deep reinforcement learning , author=. arXiv preprint arXiv:1312.5602 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2511.13649 , year=

    Distribution Matching Distillation Meets Reinforcement Learning , author=. arXiv preprint arXiv:2511.13649 , year=

    Pith/arXiv arXiv

  12. [12]

    Proceedings of the 37th International Conference on Neural Information Processing Systems , pages=

    ImageReward: learning and evaluating human preferences for text-to-image generation , author=. Proceedings of the 37th International Conference on Neural Information Processing Systems , pages=

  13. [13]

    arXiv preprint arXiv:2505.05470 , year=

    Flow-grpo: Training flow matching models via online rl , author=. arXiv preprint arXiv:2505.05470 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2505.07818 , year=

    DanceGRPO: Unleashing GRPO on Visual Generation , author=. arXiv preprint arXiv:2505.07818 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv preprint arXiv:2305.13301 , year=

    Training diffusion models with reinforcement learning , author=. arXiv preprint arXiv:2305.13301 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:2501.13918 , year=

    Improving video generation with human feedback , author=. arXiv preprint arXiv:2501.13918 , year=

    Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  19. [19]

    European Conference on Computer Vision , pages=

    Diffusion reward: Learning rewards via conditional video diffusion , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  20. [20]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Diffusion model alignment using direct preference optimization , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  21. [21]

    arXiv preprint arXiv:2511.21541 , year=

    Video Generation Models Are Good Latent Reward Models , author=. arXiv preprint arXiv:2511.21541 , year=

    arXiv

  22. [22]

    arXiv preprint arXiv:2512.04678 , year=

    Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation , author=. arXiv preprint arXiv:2512.04678 , year=

    Pith/arXiv arXiv

  23. [23]

    arXiv preprint arXiv:2412.21059 , year=

    Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation , author=. arXiv preprint arXiv:2412.21059 , year=

    Pith/arXiv arXiv

  24. [25]

    arXiv preprint arXiv:2511.18870 , year=

    Hunyuanvideo 1.5 technical report , author=. arXiv preprint arXiv:2511.18870 , year=

    Pith/arXiv arXiv

  25. [26]

    arXiv preprint arXiv:2506.09113 , year=

    Seedance 1.0: Exploring the Boundaries of Video Generation Models , author=. arXiv preprint arXiv:2506.09113 , year=

    Pith/arXiv arXiv

  26. [27]

    arXiv preprint arXiv:2511.19356 , year=

    Growing with the Generator: Self-paced GRPO for Video Generation , author=. arXiv preprint arXiv:2511.19356 , year=

    Pith/arXiv arXiv

  27. [28]

    arXiv preprint arXiv:2509.05952 , year=

    Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching , author=. arXiv preprint arXiv:2509.05952 , year=

    arXiv

  28. [29]

    arXiv preprint arXiv:2006.09359 , year=

    Awac: Accelerating online reinforcement learning with offline datasets , author=. arXiv preprint arXiv:2006.09359 , year=

    Pith/arXiv arXiv 2006

  29. [30]

    arXiv preprint arXiv:2309.17425 , year=

    Data filtering networks , author=. arXiv preprint arXiv:2309.17425 , year=

    Pith/arXiv arXiv

  30. [31]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    An empirical study of training self-supervised vision transformers , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  31. [32]

    arXiv preprint arXiv:2106.08254 , year=

    Beit: Bert pre-training of image transformers , author=. arXiv preprint arXiv:2106.08254 , year=

    Pith/arXiv arXiv

  32. [33]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Vision transformers for dense prediction , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  33. [34]

    arXiv preprint arXiv:1907.10326 , year=

    From big to small: Multi-scale local planar guidance for monocular depth estimation , author=. arXiv preprint arXiv:1907.10326 , year=

    Pith/arXiv arXiv 1907

  34. [35]

    arXiv preprint arXiv:2111.11429 , year=

    Benchmarking detection transfer learning with vision transformers , author=. arXiv preprint arXiv:2111.11429 , year=

    Pith/arXiv arXiv

  35. [36]

    Proceedings of the IEEE international conference on computer vision , pages=

    Mask r-cnn , author=. Proceedings of the IEEE international conference on computer vision , pages=

  36. [37]

    ArXiv , year=

    Zero-Shot Text-to-Image Generation , author=. ArXiv , year=

  37. [38]

    ArXiv , year=

    Cosmos World Foundation Model Platform for Physical AI , author=. ArXiv , year=

  38. [39]

    2024 , eprint=

    SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers , author=. 2024 , eprint=

    2024

  39. [40]

    2025 , eprint=

    Wan: Open and Advanced Large-Scale Video Generative Models , author=. 2025 , eprint=

    2025

  40. [41]

    arXiv preprint arXiv:2503.20314 , year=

    Wan: Open and Advanced Large-Scale Video Generative Models , author=. arXiv preprint arXiv:2503.20314 , year=

    Pith/arXiv arXiv

  41. [42]

    2025 , eprint=

    AToken: A Unified Tokenizer for Vision , author=. 2025 , eprint=

    2025

  42. [43]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    High-resolution image synthesis with latent diffusion models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  43. [44]

    arXiv preprint arXiv:2307.01952 , year=

    Sdxl: Improving latent diffusion models for high-resolution image synthesis , author=. arXiv preprint arXiv:2307.01952 , year=

    Pith/arXiv arXiv

  44. [45]

    1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space , author=

    FLUX. 1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space , author=. arXiv e-prints , pages=

  45. [46]

    arXiv preprint arXiv:2508.02324 , year=

    Qwen-image technical report , author=. arXiv preprint arXiv:2508.02324 , year=

    Pith/arXiv arXiv

  46. [47]

    ArXiv , year=

    Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation , author=. ArXiv , year=

  47. [48]

    generation: Taming optimization dilemma in latent diffusion models , author=

    Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  48. [49]

    ArXiv , year=

    Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generation , author=. ArXiv , year=

  49. [50]

    Advances in Neural Information Processing Systems , volume=

    Omnitokenizer: A joint image-video tokenizer for visual generation , author=. Advances in Neural Information Processing Systems , volume=

  50. [51]

    arXiv preprint arXiv:2507.15856 , year=

    Latent Denoising Makes Good Visual Tokenizers , author=. arXiv preprint arXiv:2507.15856 , year=

    arXiv

  51. [52]

    Image Processing Algorithms and Techniques II , volume=

    Residual VQ (vector quantizaton) with state prediction: a new method for image coding , author=. Image Processing Algorithms and Techniques II , volume=. 1991 , organization=

    1991

  52. [53]

    International Conference on Learning Representations , volume=

    Language model beats diffusion-tokenizer is key to visual generation , author=. International Conference on Learning Representations , volume=

  53. [54]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Softvq-vae: Efficient 1-dimensional continuous tokenizer , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  54. [55]

    arXiv preprint arXiv:2410.01756 , year=

    Imagefolder: Autoregressive image generation with folded tokens , author=. arXiv preprint arXiv:2410.01756 , year=

    Pith/arXiv arXiv

  55. [56]

    2022 , eprint=

    Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow , author=. 2022 , eprint=

    2022

  56. [57]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Diffusion autoencoders: Toward a meaningful and decodable representation , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  57. [58]

    arXiv preprint arXiv:2405.09818 , year=

    Chameleon: Mixed-modal early-fusion foundation models , author=. arXiv preprint arXiv:2405.09818 , year=

    Pith/arXiv arXiv

  58. [59]

    2025 , eprint=

    Towards Semantic Equivalence of Tokenization in Multimodal LLM , author=. 2025 , eprint=

    2025

  59. [60]

    arXiv preprint arXiv:2309.11499 , year=

    Dreamllm: Synergistic multimodal comprehension and creation , author=. arXiv preprint arXiv:2309.11499 , year=

    Pith/arXiv arXiv

  60. [61]

    arXiv preprint arXiv:2309.04669 , year=

    Unified language-vision pretraining in llm with dynamic discrete visual tokenization , author=. arXiv preprint arXiv:2309.04669 , year=

    arXiv

  61. [62]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Janus: Decoupling visual encoding for unified multimodal understanding and generation , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  62. [63]

    arXiv preprint , year=

    World Model on Million-Length Video and Language with RingAttention , author=. arXiv preprint , year=

  63. [64]

    arXiv preprint arXiv:2404.14396 , year=

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation , author=. arXiv preprint arXiv:2404.14396 , year=

    Pith/arXiv arXiv

  64. [65]

    arXiv preprint arXiv:2310.01218 , year=

    Making llama see and draw with seed tokenizer , author=. arXiv preprint arXiv:2310.01218 , year=

    Pith/arXiv arXiv

  65. [66]

    arXiv preprint arXiv:2408.12528 , year=

    Show-o: One single transformer to unify multimodal understanding and generation , author=. arXiv preprint arXiv:2408.12528 , year=

    Pith/arXiv arXiv

  66. [67]

    arXiv preprint arXiv:2409.18869 , year=

    Emu3: Next-token prediction is all you need , author=. arXiv preprint arXiv:2409.18869 , year=

    Pith/arXiv arXiv

  67. [68]

    arXiv preprint arXiv:2409.04429 , year=

    Vila-u: a unified foundation model integrating visual understanding and generation , author=. arXiv preprint arXiv:2409.04429 , year=

    Pith/arXiv arXiv

  68. [69]

    arXiv preprint arXiv:2412.06673 , year=

    Illume: Illuminating your llms to see, draw, and self-enhance , author=. arXiv preprint arXiv:2412.06673 , year=

    Pith/arXiv arXiv

  69. [70]

    arXiv preprint arXiv:2412.00127 , year=

    Orthus: Autoregressive interleaved image-text generation with modality-specific heads , author=. arXiv preprint arXiv:2412.00127 , year=

    Pith/arXiv arXiv

  70. [71]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  71. [72]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Tokenflow: Unified image tokenizer for multimodal understanding and generation , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  72. [73]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Vila: On pre-training for visual language models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  73. [74]

    arXiv preprint arXiv:2411.17762 , year=

    Muse-vl: Modeling unified vlm through semantic discrete encoding , author=. arXiv preprint arXiv:2411.17762 , year=

    Pith/arXiv arXiv

  74. [75]

    2023 , eprint=

    Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action , author=. 2023 , eprint=

    2023

  75. [76]

    2025 , eprint=

    LMFusion: Adapting Pretrained Language Models for Multimodal Generation , author=. 2025 , eprint=

    2025

  76. [77]

    2025 , eprint=

    Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling , author=. 2025 , eprint=

    2025

  77. [78]

    2025 , eprint=

    UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding , author=. 2025 , eprint=

    2025

  78. [79]

    2025 , eprint=

    Show-o2: Improved Native Unified Multimodal Models , author=. 2025 , eprint=

    2025

  79. [80]

    arXiv preprint arXiv:2502.05178 , year=

    Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation , author=. arXiv preprint arXiv:2502.05178 , year=

    Pith/arXiv arXiv

  80. [81]

    Proceedings of the IEEE international conference on computer vision , pages=

    Focal loss for dense object detection , author=. Proceedings of the IEEE international conference on computer vision , pages=

Showing first 80 references.