Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A soft codebook lets 32 tokens match 1024-token image generation

desk verdict Solid systems paper on 1D continuous tokenizers with real efficiency wins; the DINOv2 alignment is doing much of the work behind the headline few-token FIDs, and the KL framing needs a fix. read the letter →

arxiv 2412.10958 v3 pith:6YELJ4GO submitted 2024-12-14 cs.CV cs.AIcs.LG

Hao Chen , Ze Wang , Xiang Li , Ximeng Sun , Fangyi Chen , Jiang Liu , Jindong Wang , Bhiksha Raj
show 2 more authors
Zicheng Liu Emad Barsoum
This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords softcategoricalposteriorcontinuoustokenizerimagegenerationrepresentationalignmentvectorquantizationhighcompressionratiolatentdiffusionNet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SoftVQ-VAE claims that replacing a vector-quantized tokenizer's hard codeword assignment with a soft categorical posterior — a weighted blend of many codewords per latent token — yields a fully differentiable, continuous tokenizer that compresses 256×256 and 512×512 images to just 32 or 64 one-dimensional tokens. On ImageNet, this lets diffusion (DiT), flow (SiT), and autoregressive (MAR) generators match or beat the quality of systems using 256–1024 tokens, with SiT-XL reaching FID 1.78 at 256×256 and 2.21 at 512×512 while raising inference throughput up to 18× and 55×. The paper argues that the soft posterior removes the broken gradients of VQ, letting the latent space be directly aligned to pre-trained semantic features, and that this semantic richness — not reconstruction quality alone — is what makes few-token generation work. If correct, token count stops being the main cost driver for transformer-based generation, and the same recipe could carry over to video, audio, and multimodal generation.

What carries the argument

The central object is the soft categorical posterior, $q_{\phi}(z|x) = \operatorname{Softmax}(-\|\hat{z} - C\|^2/\tau)$, which maps the encoder output $\hat{z}$ to a convex combination of codebook vectors $C$. This turns the discrete VQ lookup into soft K-means, makes the whole tokenizer end-to-end differentiable, and — together with the representation-alignment loss of Eq. (6), which replicates each latent token $N/L$ times and matches a pre-trained encoder's patch features through an MLP projector — creates a latent space whose semantics carry the generation task at very few tokens. The KL term is the entropy of the soft posterior minus the entropy of its batch average, regularizing codebook usage without the VQ codebook or commitment losses.

What would settle it

Train the identical SoftVQ-B/64 pipeline on ImageNet with a frozen randomly-initialized or non-semantic alignment target (or with the alignment loss removed and no DINOv2 init), and check whether the final SiT-XL gFID stays near 1.78; a rise above about 5 would confirm that the DINOv2 teacher, not the soft posterior alone, carries the few-token generation quality. A second check: the throughput figures count generative-model FLOPs only, so recompute end-to-end latency including tokenizer decoding; if the 55× advantage shrinks below 10×, the practical speedup claim needs qualification.

Watch

Extended reading notes

Core claim

The central claim is that a VQ-VAE can be turned continuous by replacing the arg-min codeword selection with a softmax over negative squared distances to the codebook (temperature τ=0.07), then taking the latent as the probability-weighted sum of codewords. Because every step is differentiable, the codebook and encoder learn directly from reconstruction, perceptual, adversarial, and KL losses without codebook or commit losses, and — crucially — a cosine-similarity loss can align each latent token to the corresponding patch features of a pre-trained self-supervised ViT (DINOv2), with a projector MLP and replicated tokens to match resolution. The paper shows that with this alignment and DINOv2-initialized encoder, SoftVQ variants hold rFID at 0.61–0.89 with only 32–64 tokens, and trained downstream generators (SiT-XL with 64 tokens) obtain gFID 1.78 on ImageNet 256 and 2.21 on 512, using 2.3× fewer training steps than the vanilla baselines.

Load-bearing premise

The headline generation numbers rest on the assumption that aligning the tokenizer's latents to a pre-trained self-supervised vision teacher (DINOv2), and initializing the encoder from it, is a sufficient and transferable inductive bias for downstream generative modeling; Table 4 shows the same 64-token SoftVQ-B drops from gFID 10.13 to 17.20 when the alignment is removed.

Editorial extensions

If this is right

  • With 32 and 64 tokens, DiT-XL, SiT-XL, and MAR-H reach generation quality comparable to or better than 256–1024-token baselines on ImageNet 256×256 and 512×512, at a fraction of the GFLOPs.
  • Inference throughput for generating a 512×512 image rises up to 55× with 64 tokens, and training iterations drop by roughly 2.3× for comparable FID.
  • The soft posterior removes the need for codebook and commitment losses, making the tokenizer a drop-in continuous alternative compatible with existing VQ techniques such as product and residual quantization.
  • Reconstruction quality stays high at high compression (rFID 0.61–0.89 at 256×256), unlike hard VQ and plain AE baselines whose rFID degrades sharply when tokens drop below 128.
  • The tokenizer's linear-probing accuracy transfers to the trained generative model, indicating that the latent space is semantically organized for downstream generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the alignment premise transfers across domains, the same soft-posterior recipe could push video and audio tokenizers to similarly tiny token budgets, where the quadratic attention cost of transformers is even more prohibitive.
  • The paper's Table 4 shows that choosing a different teacher (CLIP or EVA) keeps quality high, suggesting the method may be teacher-agnostic; a testable extension is to see whether a small, task-specific teacher can substitute DINOv2 and reduce tokenizer training cost further.
  • Since better reconstruction (rFID) does not track better generation (gFID) here, an implicit design rule for tokenizers is to optimize semantic alignment rather than pixel fidelity at high compression — a principle that, if true, could reshape how tokenizers are evaluated.
  • The CFG gains with SoftVQ are smaller than for the REPA baseline, an observation the paper leaves open; investigating why classifier-free guidance saturates at very low token counts could yield methods that further close the gap.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SoftVQ-VAE proposes replacing VQ-VAE's hard argmin assignment with a soft categorical posterior over a learnable codebook, making the tokenizer fully differentiable and allowing each 1D latent token to aggregate multiple codewords. On ImageNet 256x256 and 512x512, the authors train ViT-based tokenizers with 32 or 64 latent tokens, align the latent space with DINOv2 (or other) features via Eq. (6), and then train DiT, SiT, and MAR generative models on these latents. They report reconstruction rFID around 0.6-0.9 at 32-64 tokens, generation gFID as low as 1.78 (256x256, SiT-XL, CFG) and 2.21 (512x512), and large inference throughput gains (up to 55x), claiming state-of-the-art few-token generation while reducing training iterations. The paper includes extensive system-level tables, tokenizer ablations (codebook size, temperature, PQ/RQ/GMM variants), linear-probing analyses, and visualizations.

Significance. If the results are reproducible and correctly attributed, this is a practically important contribution: it shows that a soft, continuous tokenizer can compress images to 32-64 tokens without collapsing generation quality, directly attacking the token-count bottleneck of transformer-based generative models. The study is unusually thorough on the empirical side, covering two resolutions, three generative backbones, multiple tokenizer scales, and a wide ablation set including codebook size, softmax temperature, product/residual quantization, and GMM variants. The paper also ships code and model releases, and its linear-probing analysis adds evidence about latent-space quality. However, the central attribution of the headline gains to the soft posterior is not yet clean: the few-token FID results are obtained with a DINOv2-initialized encoder and the Eq. (6) alignment loss, and the factorial design in Table 4 shows that this external teacher is load-bearing (gFID moves from 20.33 to 10.13). The scale-matched VQ/AE controls in Table 3 are at the small (46M) scale only. The theoretical framing also needs correction: the quantity called a KL divergence in Eq.

major comments (3)
  1. [Section 3.2, Eq. (4); Appendix B.1] The term Lkl in Eq. (4), presented as the 'KL divergence term in the ELBO', is not the KL divergence between the posterior and the uniform codebook prior. For a uniform prior over K codewords, KL(q||p) = log K - H(q). The implemented expression H(q) - H(E[q]) is, after batch averaging, approximately the negative of the Jensen gap between the average posterior entropy and the conditional entropy; it is not equal to log K - H(q) and is not even a valid KL divergence as written. The derivation in Appendix B.1 substitutes E[q] into the cross-entropy term without justification, and the first entropy term is not averaged over x. Since the method's core formulation is built on this term, the equation and derivation should be corrected or the term should be explicitly reframed as a diversity regularizer rather than a KL divergence. The empirical results are unlikely to be invalidated because lambda4 is small (0.01), but the description is mathematically incorrect.
  2. [Section 4.3, Table 3; Section 4.4, Table 4] The headline few-token generation numbers (e.g., FID 1.78 on 256x256 and 2.21 on 512x512) are reported for SoftVQ-B, SoftVQ-BL, and SoftVQ-L with DINOv2 encoder initialization and the Eq. (6) alignment loss, yet the only scale-matched VQ and AE controls are the 46M-parameter SoftVQ-S variants in Table 3. Table 4 shows that DINOv2 initialization plus alignment changes SoftVQ-B 64-token gFID from 20.33 to 10.13, with alignment alone giving 10.96 and initialization alone 17.20. This makes the external teacher load-bearing, and without a VQ-B or AE-B baseline trained with the same DINOv2 initialization and alignment objective, the paper cannot yet attribute the central generation gains to the softness of the posterior. Please add such controlled baselines at the scale of the headline results (or clearly state if the Table 3 controls already include DINOv2 alignment and initialization, in which case that should be stated explicitly in the text).
  3. [Section 4.1, Tables 1-2; efficiency claims] The throughput speedup claims (up to 18x for 256x256, 55x for 512x512) are presented as system-level comparisons, but the measurement protocol is not fully specified for baseline rows. The text says throughput is measured 'on a single AMD MI250', yet Table 1 includes throughput values for many baseline systems (LDM-4, U-ViT, DiT-XL/2, SiT-XL/2, REPA) that are typically quoted from their original papers, which used different hardware and software stacks. If those baseline numbers are not re-measured on the same MI250 setup, the speedup ratios are not controlled comparisons. Please clarify in the tables or text which throughput values were measured in this work and which are taken from other sources, and if possible report same-hardware measurements for the main baselines.
minor comments (5)
  1. [Section 4.1 and Appendix C.1] The loss weights are written as 'lambda1 = 1.0, lambda1 = 0.2'; the second should presumably be lambda2 = 0.2, matching Eq. (7). Please fix this typo in both places.
  2. [Abstract and Introduction, bullet 2] The unqualified phrase 'state-of-the-art generation results' overclaims: in Table 1, SiT-XL/2 + REPA achieves gFID 1.42 with CFG, which is better than the reported SoftVQ best of 1.78. The claim is defensible if restricted to 'with only 32 or 64 tokens' or 'among high-compression tokenizers', and the bullet already includes that qualifier, so please make the abstract and introduction equally precise.
  3. [Section 3.3, Eq. (6); Section 4.1] The similarity function 'sim' in Eq. (6) is not defined; later text and Table 4 suggest cosine similarity, but the reader should not have to infer this. Also, the notation z_r[n] is garbled in Eq. (5).
  4. [Throughout] There are several typographical errors: 'Gflops' vs 'GLOPs' (the latter is not a standard unit), 'Similarly to RPEA' should read 'Similarly to REPA' (Appendix C.2), and '1.41 rFID of of' and 'an 1.34 rFID of of' appear in Appendix A.1/Table 5 text.
  5. [Section 4.4, Table 4] The sentence 'The encoder initialization and alignment with DINOv2-B achieve superior performance with rFID 0.88 and IS 103.4, compared to using either component alone' is accurate, but it could be clearer that CLIP-B and EVA-02-B achieve even better reconstruction and IS; the current wording may mislead readers into thinking DINOv2 is best on all metrics.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: FID, rFID, and throughput are external measurements; the only overlapping-author citations (ImageFolder) are training-detail references, not the source of the central soft-posterior claim.

full rationale

The paper's central claims are empirical system-level measurements against the external ImageNet benchmark: the gFID values (1.78 / 2.21), rFID values, and throughput numbers are measured outputs of trained tokenizers and generative models, not quantities defined to equal their inputs. The soft categorical posterior in Eq. (4) is a model definition; the paper's experiments then test whether this definition helps, rather than deriving the results from it. Hyperparameters such as the softmax temperature (tau = 0.07), loss weights, and CFG scales are selected by ablation or grid search on validation data, which is standard model selection rather than a fitted parameter being relabeled as a prediction. The strongest dependency, DINOv2 initialization plus the Eq. (6) alignment loss, is external, pretrained, and explicitly ablated in Table 4; the large effect of this component is an attribution gap (no VQ-B/AE-B controls at the 173M-608M scale), not circularity. The paper does cite ImageFolder [62] (overlapping authors) for the PQ/RQ combination and for the frozen DINO discriminator recipe, and it cites it alongside the external works [99] and [116]; these citations concern implementation details and do not supply the central soft-VQ idea, which is introduced and ablated within the paper. Appendix A.1 contains a pipeline artifact stating that Table 5 support 'will add in the revision,' and Appendix C.2 concedes that MAR training used a lower learning rate; these are completeness/limitation caveats, not circular steps. Overall, no derivation reduces to its inputs; score 2 reflects minor overlapping-author citations that are not load-bearing.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard empirical deep learning components: a learnable codebook, a pretrained semantic teacher (DINOv2), and several tuned hyperparameters. No new physical or conceptual entities are introduced. The main ledger items are the tuning parameters and the external assumption that DINOv2 features are the right inductive bias for generation.

free parameters (5)
  • Softmax temperature tau = 0.07
    Set by ablation (Appendix D.1); controls sharpness of the soft categorical posterior in Eq. (4).
  • CFG guidance scales for DiT and SiT = 1.35/1.45 (DiT, 64/32 tokens), 1.75/2.25 (SiT, 64/32 tokens)
    Grid-searched per model and token count (Appendix C.2); headline FID numbers use these values.
  • Loss weights lambda1, lambda2, lambda3, lambda4 = 1.0, 0.2, 0.1, 0.01
    Set following prior practice (Section 4.1); lambda1 is listed twice in the text, lambda2 inferred.
  • MAR maximum learning rate = 2e-4
    Chosen to avoid NaN; differs from the original 8e-4, making the MAR comparison uncontrolled (Appendix C.2).
  • Codebook size K and latent dimension D = K=8192, D=32 for SoftVQ-S; ablations vary
    Chosen by ablation; larger K helps up to a point, and larger D improves rFID but hurts gFID (Table 6).
assumptions (4)
  • ad hoc to paper The implemented regularizer H(q) minus H(E[q]) is a valid KL divergence to a uniform codebook prior.
    Appendix B.1 substitutes the marginal entropy for the cross-entropy term; the resulting objective is an entropy-based diversity regularizer, not the KL divergence stated in Eq. (4).
  • domain assumption DINOv2 features are a suitable semantic target for generation quality.
    The alignment loss Eq. (6) assumes tokenizer latents should match DINOv2 patch features; Table 4 shows gFID is strongly dependent on this choice.
  • domain assumption A frozen DINO-S discriminator with StyleGAN-style training is an effective adversarial regularizer.
    Used in the training objective Eq. (7) following Tian et al. and Li et al.; no ablation against other discriminators is reported.
  • domain assumption FID on ImageNet is the relevant measure of generation quality.
    All headline claims are benchmarked on ImageNet rFID and gFID.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer." pith.science (2026). https://pith.science/paper/6YELJ4GO

@misc{pith2026241210958,
  author       = {Pith},
  title        = {Pith review of: SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6YELJ4GO}},
  note         = {Machine review of arXiv:2412.10958}
}
read the original abstract

Efficient image tokenization with high compression ratios remains a critical challenge for training generative models. We present SoftVQ-VAE, a continuous image tokenizer that leverages soft categorical posteriors to aggregate multiple codewords into each latent token, substantially increasing the representation capacity of the latent space. When applied to Transformer-based architectures, our approach compresses 256x256 and 512x512 images using as few as 32 or 64 1-dimensional tokens. Not only does SoftVQ-VAE show consistent and high-quality reconstruction, more importantly, it also achieves state-of-the-art and significantly faster image generation results across different denoising-based generative models. Remarkably, SoftVQ-VAE improves inference throughput by up to 18x for generating 256x256 images and 55x for 512x512 images while achieving competitive FID scores of 1.78 and 2.21 for SiT-XL. It also improves the training efficiency of the generative models by reducing the number of training iterations by 2.3x while maintaining comparable performance. With its fully-differentiable design and semantic-rich latent space, our experiment demonstrates that SoftVQ-VAE achieves efficient tokenization without compromising generation quality, paving the way for more efficient generative models. Code and model are released.

Figures

Figures reproduced from arXiv: 2412.10958 by the authors.

Figure 1
Figure 1. ImageNet-1K 256×256 and 512×512 generation results of generative models trained on SoftVQ-VAE with 32 and 64 tokens. Beyond the compression challenge, the quality of the learned representations of the tokenizer poses another crit￾ical limitation. While latent representations are crucial for both understanding [31, 109] and generation tasks [30, 62, 64, 116], current tokenizers usually struggle to cap￾ture more discr… view at source ↗
Figure 2
Figure 2. Illustration of SoftVQ-VAE. Left: Transformer encoder [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Linear probing accuracy of ImageNet-1K val. set on (a) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (25 more)
Figure 4
Figure 4. Figure 4: Visualization of zˆ (top), i.e., encoder output, and z (bottom), i.e., decoder input, of (a) VQ-S 64; (b) SoftVQ-S 64; (c) SoftVQ-S 32; (d) SoftVQ-L 32, trained with latent space alignment. based architectures [55, 66, 98, 105, 114] to enhance scal￾ability. Latest work…
Figure 5
Figure 5. Figure 5: , where SoftVQ learns code embeddings uniformly across the entire distribution. B. Additional Details of SoftVQ-VAE and its Variants In this section, we present more details on the posterior of SoftVQ-VAE, its variant GMMVQ-VAE with the latent space as a GMM model, and…
Figure 6
Figure 6. Figure 6: Illustration of product quantization (left) and residual quantization (right) with SoftVQ. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Reconstruction results of SoftVQ-L with 32 tokens. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Reconstruction results of SoftVQ-L with 64 tokens. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Uncurated 256×256 generation results of DiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = ”balloon” (417) [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Uncurated 256×256 generation results of DiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “baseball” (429) [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Uncurated 256×256 generation results of DiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “dog sled” (537) [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Uncurated 256×256 generation results of DiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “space shuttle” (812) [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Uncurated 256×256 generation results of DiT-XL with SoftVQ-L 32 tokens. We use CFG with 4.0. Class label = “panda” (388) [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Uncurated 256×256 generation results of DiT-XL with SoftVQ-L 32 tokens. We use CFG with 4.0. Class label = “geyser” (974) [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Uncurated 256×256 generation results of SiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “golden retriever” (207) [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Uncurated 256×256 generation results of SiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “arctic wolf” (270) [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Uncurated 256×256 generation results of SiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “lion” (291) [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Uncurated 256×256 generation results of SiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “loggerhead sea turtle” (33) [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: Uncurated 256×256 generation results of SiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “fire truck” (555) [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: Uncurated 256×256 generation results of SiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “laptop” (620) [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Uncurated 256×256 generation results of SiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “macaw” (88) [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22: Uncurated 256×256 generation results of SiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “sulphur-crested cockatoo” (89) [PITH_FULL_IMAGE:figures/full_fig_p026_22.png]
Figure 23
Figure 23. Figure 23: Uncurated 256×256 generation results of SiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “ice cream” (928) [PITH_FULL_IMAGE:figures/full_fig_p027_23.png]
Figure 24
Figure 24. Figure 24: Uncurated 256×256 generation results of SiT-XL with SoftVQ-L 64 tokens. We use CFG with 4.0. Class label = “cheeseburger” (933) [PITH_FULL_IMAGE:figures/full_fig_p027_24.png]
Figure 25
Figure 25. Figure 25: Uncurated 256×256 generation results of SiT-XL with SoftVQ-L 32 tokens. We use CFG with 4.0. Class label = “coral reef” (973) [PITH_FULL_IMAGE:figures/full_fig_p028_25.png]
Figure 26
Figure 26. Figure 26: Uncurated 256×256 generation results of SiT-XL with SoftVQ-L 32 tokens. We use CFG with 4.0. Class label = “lake shore” (975) [PITH_FULL_IMAGE:figures/full_fig_p028_26.png]
Figure 27
Figure 27. Figure 27: Uncurated 512×512 generation results of SiT-XL with SoftVQ-L 64 tokens. We use CFG with 2.0. Class label = “cliff drop-off” (972) [PITH_FULL_IMAGE:figures/full_fig_p029_27.png]
Figure 28
Figure 28. Figure 28: Uncurated 512×512 generation results of SiT-XL with SoftVQ-L 64 tokens. We use CFG with 2.0. Class label = “volcano” (980) [PITH_FULL_IMAGE:figures/full_fig_p030_28.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking Discrete Tokens: Treating Them as Conditions for Continuous Autoregressive Image Synthesis

    cs.CV 2025-07 conditional novelty 6.0 of 10

    DisCon treats discrete image tokens as conditioning signals rather than targets, letting a continuous autoregressive model refine details and reach gFID 1.38 on ImageNet-256.

  2. Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    1D binary image latents reduce a 1024x1024 image to 128 discrete tokens and support text-to-image generation with diffusion and autoregressive models.

  3. Masked Autoencoders Are Effective Tokenizers for Diffusion Models

    cs.CV 2025-02 conditional novelty 6.0 of 10

    MAETok shows that a masked-autoencoder-trained plain autoencoder, without variational constraints, reaches state-of-the-art ImageNet generation quality using only 128 latent tokens.

  4. DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizer

    cs.CV 2025-07 conditional novelty 5.0 of 10

    DC-AR generates 512x512 images in 12 masked autoregressive steps plus 20 diffusion refinement steps, using a 32x compressed 2D tokenizer, and reports gFID 5.49 on MJHQ-30K.

Reference graph

Works this paper leans on

126 extracted references · 28 canonical work pages · cited by 4 Pith papers

  1. [1]

    Stochastic interpolants: A unifying framework for flows and diffusions

    Michael S Albergo, Nicholas M Boffi, and Eric Vanden- Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797,

    arXiv

  2. [2]

    Reverse-time diffusion equation mod- els

    Brian DO Anderson. Reverse-time diffusion equation mod- els. Stochastic Processes and their Applications , 12(3): 313–326, 1982. 3

    1982

  3. [3]

    wav2vec 2.0: A framework for self- supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self- supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460,

  4. [4]

    All are worth words: A vit backbone for diffusion models

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 22669–22679, 2023. 6, 7, 5

    2023

  5. [5]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. 4

    arXiv 2021

  6. [6]

    Lumiere: A space- time diffusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Her- rmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, et al. Lumiere: A space- time diffusion model for video generation. arXiv preprint arXiv:2401.12945, 2024. 1

    arXiv 2024

  7. [7]

    Estimating or propagating gradients through stochastic neurons for conditional computation

    Yoshua Bengio, Nicholas L ´eonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. 1, 3

    arXiv 2013

  8. [8]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1

    arXiv 2023

Show all 126 references
  1. [9]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021. 5, 2

    2021

  2. [10]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer,

  3. [11]

    Deep compression autoencoder for efficient high-resolution diffu- sion models

    Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffu- sion models. arXiv preprint arXiv:2410.10733, 2024. 1, 6, 7, 5

    2024 arXiv

  4. [12]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Jun- yang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models. arXiv preprint arXiv:2403.06764, 2024. 1

    2024 arXiv

  5. [13]

    Variational lossy autoencoder

    Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Pra- fulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv:1611.02731, 2016. 1, 2, 3

    2016 arXiv

  6. [14]

    De- constructing denoising diffusion models for self-supervised learning

    Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. De- constructing denoising diffusion models for self-supervised learning. arXiv preprint arXiv:2401.14404, 2024. 1, 2, 3

    2024 arXiv

  7. [15]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 5, 2

    2009

  8. [16]

    Diffusion models beat gans on image synthesis, 2021

    Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021. 1, 7, 8, 5

    2021

  9. [17]

    Deep unsupervised clustering with gaussian mixture variational autoencoders

    Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.02648, 2016. 4

    2016 arXiv

  10. [18]

    Cogview: Mastering text-to-image generation via transformers

    Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in neural information processing systems, 34:19822–19835, 2021. 8

    2021

  11. [19]

    Peco: Perceptual codebook for bert pre-training of vision transformers

    Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu, and Baining Guo. Peco: Perceptual codebook for bert pre-training of vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 552–560,

  12. [20]

    Generating images with perceptual similarity metrics based on deep networks

    Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems , 29,

  13. [21]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  14. [22]

    A fuzzy relative of the isodata process and its use in detecting compact well-separated clusters

    Joseph C Dunn. A fuzzy relative of the isodata process and its use in detecting compact well-separated clusters. 1973. 2, 4

    1973

  15. [23]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 1, 2, 3, 5, 6, 8

    2021

  16. [24]

    Fast timing-conditioned latent audio diffusion

    Zach Evans, CJ Carr, Josiah Taylor, Scott H Hawley, and Jordi Pons. Fast timing-conditioned latent audio diffusion. arXiv preprint arXiv:2402.04825, 2024. 1

    2024 arXiv

  17. [25]

    Fluid: Scaling autoregressive text-to-image generative models with continuous tokens

    Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410.13863, 2024. 1

    arXiv

  18. [26]

    Eva-02: A visual representation for neon genesis

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xin- long Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149:105171,

  19. [27]

    Restructuring vector quantization with the rotation trick

    Christopher Fifty, Ronald G Junkins, Dennis Duan, Aniketh Iger, Jerry W Liu, Ehsan Amid, Sebastian Thrun, and Christopher R´e. Restructuring vector quantization with the rotation trick. arXiv preprint arXiv:2410.06424, 2024. 2, 3

    2024 arXiv

  20. [28]

    Make-a-scene: Scene- based text-to-image generation with human priors

    Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene- based text-to-image generation with human priors. In Eu- ropean Conference on Computer Vision , pages 89–106. Springer, 2022. 8

    2022

  21. [29]

    Mdtv2: Masked diffusion transformer is a strong image synthesizer

    Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer is a strong image synthesizer. arXiv preprint arXiv:2303.14389,

    arXiv

  22. [30]

    Planting a seed of vision in large language model,

    Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model,

  23. [31]

    Making llama see and draw with seed tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218, 2023. 2

    2023 arXiv

  24. [32]

    Diffuseq: Sequence to sequence text generation with diffusion models

    Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933, 2022. 1

    2022 arXiv

  25. [33]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 63(11):139–144, 2020. 5

    2020

  26. [34]

    Rethinking the objectives of vector-quantized tokenizers for image synthesis, 2023

    Yuchao Gu, Xintao Wang, Yixiao Ge, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Rethinking the objectives of vector-quantized tokenizers for image synthesis, 2023. 2, 8

    2023

  27. [35]

    Masked autoencoders are scal- able vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scal- able vision learners. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 16000–16009, 2022. 1, 3, 4

    2022

  28. [36]

    Rotary position embedding for vision transformer

    Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, pages 289–305. Springer, 2025. 4

    2025

  29. [37]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bern- hard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems , 30,

  30. [38]

    beta-vae: Learning basic visual concepts with a constrained variational framework

    Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mo- hamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR, 3, 2017. 1, 2, 3

    2017

  31. [39]

    Reducing the dimensionality of data with neural networks

    Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504–507, 2006. 8

    2006

  32. [40]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 5

    2022 arXiv

  33. [41]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  34. [42]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 1

    2022

  35. [43]

    Straightening out the straight-through estimator: Over- coming optimization challenges in vector quantized net- works

    Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight-through estimator: Over- coming optimization challenges in vector quantized net- works. In International Conference on Machine Learning, pages 14096–14113. PMLR, 2023. 3, 4

    2023

  36. [44]

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adver- sarial networks, 2018. 5

    2018

  37. [45]

    Product quantization for nearest neighbor search

    Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence , 33(1):117– 128, 2010. 2, 8

    2010

  38. [46]

    Percep- tual losses for real-time style transfer and super-resolution

    Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Percep- tual losses for real-time style transfer and super-resolution. In Computer Vision–ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 694–711. Springer, 2016. 5

    2016

  39. [47]

    Composing graphical models with neural networks for structured representations and fast inference

    Matthew J Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta. Composing graphical models with neural networks for structured representations and fast inference. Advances in neural information process- ing systems, 29, 2016. 4

    2016

  40. [48]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019. 5, 7, 2

    2019

  41. [49]

    Analyzing and improv- ing the image quality of stylegan

    Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improv- ing the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 8110–8119, 2020. 5, 2

    2020

  42. [50]

    Understanding diffusion objectives as the elbo with simple data augmentation

    Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Ad- vances in Neural Information Processing Systems, 36, 2024. 3

    2024

  43. [51]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 1, 2, 3

    2013 arXiv

  44. [52]

    Improved precision and recall metric for assessing generative models

    Tuomas Kynk¨a¨anniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019. 5

    2019

  45. [53]

    Applying guidance in a limited interval improves sample and distribution quality in diffusion models

    Tuomas Kynk¨a¨anniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724, 2024. 2

    2024 arXiv

  46. [54]

    Autoencoding beyond pixels using a learned similarity metric

    Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. InInternational conference on machine learning, pages 1558–1566. PMLR, 2016. 5

    2016

  47. [55]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523–11532, 2022. 2, 6, 8, 5

    2022

  48. [56]

    Autoregressive image generation using residual quantization, 2022

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization, 2022. 1

    2022

  49. [57]

    Scalable autoregressive image generation with mamba

    Haopeng Li, Jinyue Yang, Kexin Wang, Xuerui Qiu, Yuhong Chou, Xin Li, and Guoqi Li. Scalable autoregressive image generation with mamba. arXiv preprint arXiv:2408.12245,

  50. [58]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Interna- tional conference on machine learning, pages 19730–19742. PMLR, 2023. 1, 2

    2023

  51. [59]

    Mage: Masked generative encoder to unify representation learning and im- age synthesis, 2023

    Tianhong Li, Huiwen Chang, Shlok Kumar Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and im- age synthesis, 2023. 3, 6, 5

    2023

  52. [60]

    Autoregressive image generation without vec- tor quantization, 2024

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vec- tor quantization, 2024. 1, 2, 3, 5, 6, 8

    2024

  53. [61]

    Tokenpacker: Effi- cient visual projector for multimodal llm

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. Tokenpacker: Effi- cient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392, 2024. 1

    2024 arXiv

  54. [62]

    Imagefolder: Autoregres- sive image generation with folded tokens

    Xiang Li, Hao Chen, Kai Qiu, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregres- sive image generation with folded tokens. arXiv preprint arXiv:2410.01756, 2024. 1, 2, 4, 5, 8

    2024 arXiv

  55. [63]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  56. [64]

    Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023. 2

    2023 arXiv

  57. [65]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maxim- ilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. 3

    2022 arXiv

  58. [66]

    Customize your visual autoregressive recipe with set autoregressive modeling

    Wenze Liu, Le Zhuo, Yi Xin, Sheng Xia, Peng Gao, and Xiangyu Yue. Customize your visual autoregressive recipe with set autoregressive modeling. arXiv preprint arXiv:2410.10511, 2024. 8

    2024 arXiv

  59. [67]

    Least squares quantization in pcm

    Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 28(2):129–137, 1982. 2, 4

    1982

  60. [68]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 2

    2017 arXiv

  61. [69]

    Simplifying, stabilizing and scaling continuous-time consistency models

    Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint arXiv:2410.11081, 2024. 1

    2024 arXiv

  62. [70]

    Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xi- aoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024. 1

    2024 arXiv

  63. [71]

    Open-magvit2: An open-source project toward democratizing auto-regressive visual genera- tion

    Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual genera- tion. arXiv preprint arXiv:2409.04410, 2024. 2

    2024 arXiv

  64. [72]

    Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers

    Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08740,

    arXiv

  65. [73]

    Finite scalar quantization: Vq-vae made simple, 2023

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple, 2023. 2, 8

    2023

  66. [74]

    Midjourney

    MidJourney Inc. Midjourney. Software available from MidJourney Inc., 2022. 8

    2022

  67. [75]

    Improved denoising diffusion probabilistic models, 2021

    Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021. 8

    2021

  68. [76]

    Glide: Towards photorealistic image genera- tion and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image genera- tion and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021. 8

    2021 arXiv

  69. [77]

    Certain topics in telegraph transmission theory

    Harry Nyquist. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers, 47(2):617–644, 1928. 2

    1928

  70. [78]

    Video generation models as world simulators

    OpenAI. Video generation models as world simulators. Ope- nAI Blog, 2024. 8

    2024

  71. [79]

    Maxime Oquab, Timoth´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Rus- sell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nic...

    2023

  72. [80]

    Scalable diffusion models with transformers, 2023

    William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. 1, 2, 3, 5, 6, 7, 8

    2023

  73. [81]

    Grad-tts: A diffusion probabilistic model for text-to-speech

    Vadim Popov, Ivan V ovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Con- ference on Machine Learning , pages 8599–8608. PMLR,

  74. [82]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International conference on machine learning...

    2021

  75. [83]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning, pages 8821–8831. Pmlr, 2021. 1, 8

    2021

  76. [84]

    Gener- ating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Gener- ating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019. 1, 8

    2019

  77. [85]

    Gen- erating diverse high-fidelity images with vq-vae-2, 2019

    Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Gen- erating diverse high-fidelity images with vq-vae-2, 2019. 8

    2019

  78. [86]

    Gaussian mixture models

    Douglas A Reynolds et al. Gaussian mixture models. Ency- clopedia of biometrics, 741(659-663), 2009. 2, 4

    2009

  79. [87]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 8

    2022

  80. [88]

    High-resolution image synthesis with latent diffusion models, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models, 2022. 1, 6, 8, 5

    2022

  81. [89]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- ti...

    2022

  82. [90]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Process- ing Systems, 29, 2016. 5

    2016

  83. [91]

    Communication in the presence of noise

    Claude Elwood Shannon. Communication in the presence of noise. Proceedings of the IRE, 37(1):10–21, 1949. 2

    1949

  84. [92]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 3, 8

    2015

  85. [93]

    Weiss, Niru Mah- eswaranathan, and Surya Ganguli

    Jascha Sohl-Dickstein, Eric A. Weiss, Niru Mah- eswaranathan, and Surya Ganguli. Deep unsupervised learn- ing using nonequilibrium thermodynamics, 2015. 8

    2015

  86. [94]

    Less is more: A sim- ple yet effective token reduction method for efficient multi- modal llms

    Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, and Benyou Wang. Less is more: A sim- ple yet effective token reduction method for efficient multi- modal llms. arXiv preprint arXiv:2409.10994, 2024. 1

    2024 arXiv

  87. [95]

    Denois- ing diffusion implicit models, 2022

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models, 2022. 8

    2022

  88. [96]

    Sd vae ft ema, 2023

    stabilityai. Sd vae ft ema, 2023. Accessed: 2023. 1, 3, 7

    2023

  89. [97]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  90. [98]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. 1, 5, 6, 8

    2024 arXiv

  91. [99]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction, 2024

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction, 2024. 3, 5, 8, 2

    2024

  92. [100]

    Givt: Generative infinite-vocabulary transformers

    Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infinite-vocabulary transformers. In Eu- ropean Conference on Computer Vision , pages 292–309. Springer, 2025. 3

    2025

  93. [101]

    Regularizing generative adversarial networks under limited data, 2021

    Hung-Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data, 2021. 5, 2

    2021

  94. [102]

    Score-based generative modeling in latent space, 2021

    Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space, 2021. 8

    2021

  95. [103]

    Conditional image genera- tion with pixelcnn decoders

    Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image genera- tion with pixelcnn decoders. Advances in neural information processing systems, 29, 2016. 8

    2016

  96. [104]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. 1, 2, 3, 8

    2017

  97. [105]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. 1, 8

    2023

  98. [106]

    Extracting and composing ro- bust features with denoising autoencoders

    Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing ro- bust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096–1103, 2008. 8

    2008

  99. [107]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 1

    2024 arXiv

  100. [108]

    Maskbit: Embedding-free image generation via bit tokens

    Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. Maskbit: Embedding-free image generation via bit tokens. arXiv preprint arXiv:2409.16211, 2024. 2

    2024 arXiv

  101. [109]

    Vila-u: a unified foundation model inte- grating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model inte- grating visual understanding and generation. arXiv preprint arXiv:2409.04429, 2024. 1, 2, 4, 8

    2024 arXiv

  102. [110]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024. 1

    2024 arXiv

  103. [111]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021. 2, 3

    2021 arXiv

  104. [112]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 1, 2, 8

    arXiv

  105. [113]

    Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms

    Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolf- gang Macherey, Yanping Huang, David Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, et al. Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms. Advances in Neural Information Processing Systems, 3...

    2024

  106. [114]

    Randomized autoregressive visual generation

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chieh Chen. Randomized autoregressive visual generation. arXiv preprint arXiv:2411.00776, 2024. 8

    2024 arXiv

  107. [115]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. arxiv: 2406.07550, 2024. 1, 2, 4, 6, 7, 8, 5

    2024 arXiv

  108. [116]

    Representation alignment for generation: Training diffu- sion transformers is easier than you think

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffu- sion transformers is easier than you think. arXiv preprint arXiv:2410.06940, 2024. 1, 2, 4, 5, 6

    arXiv

  109. [117]

    Soundstream: An end-to- end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to- end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2021. 1

    2021

  110. [118]

    Codebook transfer with part-of-speech for vector-quantized image modeling

    Baoquan Zhang, Huaibin Wang, Chuyao Luo, Xutao Li, Guotao Liang, Yunming Ye, Xiaochen Qi, and Yao He. Codebook transfer with part-of-speech for vector-quantized image modeling. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 7757–7...

    2024

  111. [119]

    Consistency regularization for generative adversarial networks

    Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. arXiv preprint arXiv:1910.12027, 2019. 5, 2

    1910 arXiv

  112. [120]

    Efros, Eli Shecht- man, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric, 2018. 5

    2018

  113. [121]

    Differentiable augmentation for data-efficient gan training

    Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in neural information processing systems, 33:7559–7570, 2020. 5, 2

    2020

  114. [122]

    A reparameterized discrete diffusion model for text generation

    Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737, 2023. 1

    2023 arXiv

  115. [123]

    Transfusion: Pre- dict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024. 1

    2024 arXiv

  116. [124]

    Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%

    Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%. arXiv preprint arXiv:2406.11837, 2024. 2, 4, 8

    2024 arXiv

  117. [125]

    Deformable detr: Deformable transformers for end-to-end object detec- tion

    X Zhu, W Su, L Lu, B Li, X Wang, and J Dai. Deformable detr: Deformable transformers for end-to-end object detec- tion. arxiv 2020. arXiv preprint arXiv:2010.04159, 2010. 8

    2020 arXiv

  118. [126]

    baseball

    Yongxin Zhu, Bocheng Li, Yifei Xin, and Linli Xu. Address- ing representation collapse in vector quantized models with one linear layer. arXiv preprint arXiv:2411.02038, 2024. 2 SoftVQ-V AE: Efficient 1-Dimensional Continuous Tokenizer Supplementary Material A. Posterior of KL...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.