Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantization

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A discrete image tokenizer beats continuous VAEs when its codebook is split into sub-codebooks.

desk verdict A well-ablated multi-group VQ tokenizer with a real reconstruction gain, but the 'beat VAE' claim is undermined by a contaminated zero-shot benchmark and mismatched SD-VAE comparisons. read the letter →

arxiv 2507.07997 v2 pith:24K3T3ZR submitted 2025-07-10 cs.CV

Mingkai Jia , Wei Yin , Xiaotao Hu , Jiaxin Guo , Xiaoyang Guo , Qian Zhang , Xiao-Xiao Long , Ping Tan This is my paper
classification cs.CV
keywords VQ-VAEimagetokenizervectorquantizationmulti-groupcodebookcapacityreconstructionzero-shotbenchmarkdiscretelatentrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to close the reconstruction gap between vector-quantized (discrete) and continuous image tokenizers. It argues that the limiting factor is not codebook size but representation capacity, and that a multi-group quantizer which keeps the full latent dimension while splitting it into G sub-tokens, each quantized by its own small codebook, expands capacity to K^G while making the codebooks easier to optimize. If this is right, discrete tokenizers can outperform continuous VAEs such as SD-VAE at the same downsampling ratio, which would change how latents are produced for image generation and compression. The paper reports state-of-the-art reconstruction metrics on ImageNet and on eight zero-shot high-resolution benchmarks, including a lower rFID (0.49 vs 0.91) than SD-VAE at 16x downsampling.

What carries the argument

Multi-group quantization: the latent embedding z(x) of dimension Cl is uniformly split into G sub-tokens, each quantized by its own non-shared sub-codebook of size K and dimension Cl/G, then the quantized sub-tokens are concatenated to form zq(x), giving a discrete representation capacity of K^G while keeping each sub-codebook small and easy to train. A nested masking training strategy randomly keeps Mkeep of the G quantized sub-tokens during training, forcing the encoder to order information from coarse to fine and preventing sub-codebooks from collapsing onto each other.

What would settle it

Train a continuous SD-VAE-style tokenizer from scratch at 16x downsampling on ImageNet-only data, or evaluate MGVQ on a held-out dataset entirely disjoint from its training corpora; if the continuous model matches or beats MGVQ, or if MGVQ's zero-shot lead disappears without the overlapping training domains, the central claim would be falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a VQ-VAE can beat a continuous VAE in reconstruction quality when quantization is organized as multiple independent sub-codebooks over a preserved latent dimension. Instead of shrinking the latent channel count to avoid codebook collapse, MGVQ keeps a larger latent dimension, splits each latent vector into G sub-tokens, and quantizes each sub-token with its own small codebook; concatenating the quantized sub-tokens gives a combined capacity of K^G. The paper reports that this removes the dead-code problem, yields 100% codebook usage, and on ImageNet 256x256 at 16x downsampling reaches rFID 0.49 and PSNR 24.70, surpassing the continuous SD-VAE tokenizer (rFID 0.91, PSNR 22.65) and all compared VQ-VAEs. The same pattern appears on zero-shot 512p and 2k benchmarks, where MGVQ leads in PSNR across the board.

Load-bearing premise

The headline comparison assumes SD-VAE is evaluated in a fair setting (16x downsampling on ImageNet, with unknown training data) and that the zero-shot test images are not present in the large collection of datasets MGVQ was trained on.

Editorial extensions

If this is right

  • If discrete tokenizers can beat continuous VAEs at equal downsampling, autoregressive image generators can use the same tokenizer for both high-fidelity reconstruction and generation, without switching to continuous latents.
  • The K^G capacity means a modest per-group codebook of a few thousand entries suffices to cover an enormous discrete space, keeping memory and optimization manageable where a monolithic codebook of equivalent capacity would collapse.
  • At 8x downsampling, MGVQ-G8 reaches rFID 0.27 and PSNR 29.96 on ImageNet, suggesting the reconstruction gap to continuous tokenizers shrinks further as group capacity grows.
  • Because MGVQ preserves the latent dimension instead of reducing it, the tokenizer can serve as a drop-in replacement in pipelines built on LlamaGen-style VQGAN backbones.
  • The zero-shot results at 512p and 2k indicate that the multi-group quantizer generalizes beyond ImageNet, which matters for real-world HD image processing and compression.
  • Nested masking enforces a coarse-to-fine ordering in the sub-tokens, so the model supports progressive decoding where early sub-tokens give a rough reconstruction and later ones add detail.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline comparison at 16x downsampling places SD-VAE at a non-native setting; a fairer test would retrain SD-VAE at 16x on ImageNet-only data, so the 'VQ beats VAE' conclusion should be read as conditional on the evaluation protocol.
  • The nested masking mechanism is effectively a matryoshka-style ordered representation, which could transfer to other discrete representation learners such as audio or video tokenizers as a way to enforce coarse-to-fine structure while training sub-codebooks.
  • The paper's ablation finding that G=16 is worse than G=8 points to a capacity-versus-per-dimension trade-off; a testable extension would be to use variable-size sub-codebooks or a learned split instead of a uniform split, which might shift the optimum beyond G=8.
  • If the 100% codebook usage claim transfers to low-resource training settings, MGVQ could improve reconstruction quality on smaller datasets, but the paper does not test that regime.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes MGVQ, a VQ-VAE-style image tokenizer that keeps a larger latent dimension (e.g., 32 channels) and splits each latent embedding into G sub-tokens, quantizing each sub-token with its own smaller sub-codebook. The authors argue that this design increases the discrete representation capacity from K to K^G while avoiding codebook collapse, and they introduce a nested masking training strategy to impose a coarse-to-fine ordering across sub-tokens. The method is evaluated on ImageNet 256x256 reconstruction at 8x and 16x downsampling, on eight claimed zero-shot benchmarks at 512p and 2K, and on ImageNet class-to-image generation. The headline results show MGVQ-G8 at 16x downsampling achieving rFID 0.49 and PSNR 24.70 on ImageNet, beating SD-VAE (rFID 0.91, PSNR 22.65), and MGVQ-G4 ranking first on PSNR in the zero-shot tables. The paper also reports ablations on the sub-codebook design, nested masking, group count, and codebook size.

Significance. If the headline claims held under matched conditions, the paper would be significant: it would demonstrate that a discrete tokenizer can beat strong continuous VAE tokenizers on reconstruction while retaining a discrete latent, which is directly relevant to autoregressive generation and to high-fidelity compression. The core decomposition idea—capacity via a product of independent sub-codebooks rather than a single large codebook—is simple, clearly explained, and supported by internal consistency in the ablations: the gains from sub-codebooks, nested masking, larger group counts (up to a point), and codebook size follow the paper's stated rationale. The ImageNet reconstruction comparisons against other VQ-VAE variants (VQGAN, LlamaGen, Open-MAGVIT2, VAR) are also credible as controlled comparisons because those tokenizers were trained under the same ImageNet setting. The paper's strengths include a concrete and checkable capacity argument, honest reporting of codebook usage, and a clear ablation table (Table 5) showing the contribution of each component.

major comments (5)
  1. [Sec. 4, Tables 2-3] The 'zero-shot' claim is undermined by a direct training/evaluation overlap: the zero-shot models are trained on OpenImages (Sec. 4), and TextOCR (cited as Singh et al., CVPR 2021) is a scene-text dataset built from Open Images images. Therefore the TextOCR row in Table 2 is not zero-shot; the model has very likely seen the same images in training. This contamination affects the claim in the abstract and in Sec. 5.1 that MGVQ achieves 'superior PSNR on all zero-shot benchmarks.' The remaining seven benchmarks are not audited for overlap with FFHQ, SA-1B, Argoverse2, Mapillary, or LAION-HQ, so their zero-shot status is also unverified. The paper should either remove contaminated benchmarks, provide overlap statistics for all eight, or re-run the evaluation on a curated non-overlapping subset.
  2. [Table 1, SD-VAE comparison] The headline comparison against SD-VAE is confounded by training-data and evaluation-protocol mismatch. SD-VAE is trained on unknown large-scale data and its native downsampling ratio is 8, not 16, yet the paper evaluates it at 16x downsampling in the first block of Table 1 and reports a PSNR of 22.65. MGVQ is trained on ImageNet and evaluated at its training resolution, so the comparison is not apples-to-apples. To support the claim that 'VQ beats VAE,' the paper should report MGVQ numbers under the same evaluation protocol as SD-VAE (e.g., at 8x downsampling with native SD-VAE input processing) or clearly state that the comparison is against an off-the-shelf SD-VAE applied outside its intended setting. As written, the claimed superiority over a continuous VAE is not established by Table 1.
  3. [Sec. 4, Sec. 5.2; Fig. 8 and Table 8] Key hyperparameters (group count G, sub-codebook size K, nested masking probabilities, loss weights lambda_1...lambda_6) appear to be selected on the same ImageNet validation set used for the headline numbers, with no separate validation split or hyperparameter sensitivity analysis reported. The paper states in Sec. 5.2.1 that 'We ablation on ImageNet,' and Fig. 8 shows PSNR versus G and codebook size on ImageNet. This raises a selection-on-test concern for the main ImageNet results, even though the ablations themselves are internally consistent. The authors should state explicitly whether the validation set was held out from model selection, or provide a separate validation split for hyperparameter tuning and report the headline numbers on a disjoint test split.
  4. [Sec. 3.3, Eq. (4) and Fig. 3] The 'representation capacity' argument is a direct combinatorial count (each sub-token is drawn from K choices, giving K^G possibilities), which is sound. However, the paper's framing that this capacity is 'exploitable by the decoder' is an assumption that is only indirectly tested: the ablations show that increasing G helps, but they do not measure whether the actual distribution of used sub-token combinations covers the claimed product space or whether the decoder learns to exploit the ordering imposed by nested masking. Table 6 shows that using only Mkeep=1 gives very poor performance (rFID 196.43, PSNR 12.83), which suggests the sub-tokens are not independently informative. The authors should add a quantitative analysis, e.g., token usage per sub-codebook and co-occurrence statistics, to substantiate the claim that the K^G product capacity is actually usable rather than a theoretical upper bound.
  5. [Table 3 and Fig. 5] In Table 3, MGVQ-G4 outperforms SD-VAE on PSNR on UHDBench (28.27 vs. 26.86) and DA VIS (29.50 vs. 28.02) but the rFID of MGVQ-G4 is worse than SD-VAE on both (1.59 vs. 1.07 on UHDBench; 2.77 vs. 1.92 on DA VIS). The paper's claim of 'superior PSNR on all zero-shot benchmarks' is therefore not matched by a uniform rFID improvement. The text in Sec. 5.1 says 'our performance ranks the first on both rFID and PSNR metrics,' but Table 3 contradicts this: SD-VAE ranks first on rFID for both 2K datasets. The authors should correct this overstatement and discuss why the discrete tokenizer wins on PSNR but loses on rFID at 2K resolution.
minor comments (6)
  1. [Abstract and Sec. 1] The abstract says '7 zero-shot HD datasets' in the contributions list but the experiments report 8 benchmarks (6 at 512x512 plus 2 at 2K). The count should be made consistent.
  2. [Eq. (5) and text] The sentence after Eq. (5) says 'lambda_1, lambda_2, lambda_3, lambda_4, lambda_5 denote the weights' but the equation and the values listed include lambda_6. Please correct the enumeration.
  3. [Sec. 3.3, line after Eq. (4)] The phrase 'the total codebook size is set to 32768 at most' is ambiguous: for a sub-codebook size of 8192 with 4 groups, the total number of stored vectors is 4 x 8192 = 32768, which is also the number of distinct parameter vectors, but the representation capacity is 8192^4. The paper should clarify the distinction between stored codebook parameters and representation capacity.
  4. [Fig. 3 caption] The caption refers to points '(i.a)', '(i.b)', '(i.c)', '(i.d)' but the figure itself is rendered in black and white and the visual distinction between 'used' and 'dead' points is hard to discern. A color legend or higher-contrast markers would improve readability.
  5. [References] The reference list has an irregular citation pattern: [49] and [50] are both Van Den Oord et al. entries, and [4] (Bengio et al., 2013) is a preprint version that could be replaced by the NeurIPS 2013 version. The authors should standardize the bibliography.
  6. [Table 4] The table reports gFID and IS for generation. The paper should state how many samples were used for gFID/IS computation, since generative metrics are sensitive to sample count.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: MGVQ's capacity gain is a direct combinatorial definition, and performance claims are empirical comparisons against external baselines; the only self-citation is peripheral.

full rationale

The paper's central derivation is the multi-group quantization construction (Eq. 4): the latent z is split into G sub-tokens, each quantized with an independent sub-codebook, and the quantized sub-tokens are concatenated. The claimed capacity K^G is not a fitted prediction or a consequence of a circular definition: it is the number of joint lookup outcomes from G independent sub-codebooks of size K, equivalently the cardinality of the Cartesian product of the sub-codebooks. The paper states this directly ('representation capacity of 8192^4'), and it is a definitional counting statement, not a derived empirical result. No uniqueness theorem or load-bearing result is imported from the authors' prior work. The only self-citation is [16] (DrivingWorld) for the Charbonnier loss, and it is cited together with [24] (Lai et al.) for a standard reconstruction loss; it is not load-bearing for the central claims. Hyperparameters such as the group number G and codebook size are empirical choices evaluated on the same benchmark, which is standard tuning practice and not a 'prediction' derived from a fit. The zero-shot protocol may have an external-validity concern (TextOCR is built from OpenImages, which is in the training mixture), but that is a benchmark-contamination / evaluation-protocol issue, not a circular derivation: it does not make the capacity argument or the ImageNet results reduce to their own inputs. Therefore no circular step is identified.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The load-bearing choices are hyperparameters (latent dimension, group count, codebook sizes, masking distribution, loss weights) and two modeling assumptions about codebook optimization and ordered representation. All of these are evaluated empirically but not derived.

free parameters (5)
  • latent dimension Cl = 32
    Chosen and fixed in all main experiments; the paper ablates sub-codebook dimension and group count but not Cl.
  • number of sub-codebooks G = 4 or 8
    Central design variable; the best ImageNet results use G=8, and G is varied in ablations.
  • sub-codebook size K = 8192 for G4, 2048 for G8
    Chosen per group; ablations in Fig. 8 show performance saturates above 16384 total entries.
  • nested masking probabilities = {0.1, 0.1, 0.1, 0.7}
    Hand set for G=4, not ablated, and it controls the ordered information assignment during training.
  • loss weights lambda1 to lambda6 = 2, 1, 0.25, 1, 0.5, 1
    Set by hand following LlamaGen with Charbonnier and L2 losses; no sensitivity analysis is provided.
assumptions (5)
  • standard math Nearest-neighbor assignment with a straight-through gradient estimator is an effective training signal for discrete codebooks.
    Invoked in Sec. 3.3 following LlamaGen and Oord et al.; not proven, but standard practice.
  • domain assumption Larger latent dimension preserves more encoded information, so retaining Cl=32 instead of reducing to 8 reduces reconstruction loss.
    Taken from VAE literature in Sec. 3.2; the paper does not derive this trade-off.
  • ad hoc to paper Independent small sub-codebooks avoid codebook collapse and their product capacity is exploitable by the decoder.
    This is the core design assumption behind MGVQ; supported only by ablations in Fig. 3 and Tab. 5, not by a formal argument.
  • ad hoc to paper Nested masking enforces an ordered coarse-to-fine encoding across sub-codebooks.
    Sec. 3.3 and Fig. 6 show reconstruction improves with more kept groups, but no metric directly proves the ordering.
  • domain assumption The zero-shot evaluation sets are not contaminated by the extra training data (FFHQ, OpenImages, SA-1B, Argoverse2, Mapillary, LAION-HQ).
    Sec. 4 describes the training mixture and Sec. 5 evaluates on external benchmarks; no overlap analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantization." pith.science (2026). https://pith.science/paper/24K3T3ZR

@misc{pith2026250707997,
  author       = {Pith},
  title        = {Pith review of: MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/24K3T3ZR}},
  note         = {Machine review of arXiv:2507.07997}
}
read the original abstract

Vector Quantized Variational Autoencoders (VQ-VAEs) are fundamental models that compress continuous visual data into discrete tokens. Existing methods have tried to improve the quantization strategy for better reconstruction quality, however, there still exists a large gap between VQ-VAEs and VAEs. To narrow this gap, we propose MGVQ, a novel method to augment the representation capability of discrete codebooks, facilitating easier optimization for codebooks and minimizing information loss, thereby enhancing reconstruction quality. Specifically, we propose to retain the latent dimension to preserve encoded features and incorporate a set of sub-codebooks for quantization. Furthermore, we construct comprehensive zero-shot benchmarks featuring resolutions of 512p and 2k to evaluate the reconstruction performance of existing methods rigorously. MGVQ achieves the state-of-the-art performance on both ImageNet and 8 zero-shot benchmarks across all VQ-VAEs. Notably, compared with SD-VAE, we outperform them on ImageNet significantly, with rFID 0.49 v.s. 0.91, and achieve superior PSNR on all zero-shot benchmarks. These results highlight the superiority of MGVQ in reconstruction and pave the way for preserving fidelity in HD image processing tasks. Code will be publicly available at https://github.com/MKJia/MGVQ.

Figures

Figures reproduced from arXiv: 2507.07997 by the authors.

Figure 1
Figure 1. Reconstruction performance comparison between VQ-VAEs and SD-VAE on ImageNet 256×256 benchmark with 16× downsampling. The size of bubbles indicates the capacity, that is, the possibility of a token by sampling from the codebook. MGVQ-G8 with 8 groups achieves a PSNR of 24.70, evidently surpassing all others, with a large capacity of 2 88. Qualitative results are illustrated where details are zoomed in for a better v… view at source ↗
Figure 2
Figure 2. An overview of MGVQ framework. MGVQ keeps a larger dimension Cl of latent z and split it into G sub-tokens, where each sub-token is quantized individually with sub-codebook Ei. Sub-tokens are then combined to compose zq and for decoding. representation not only reduces computational complexity but also retains critical semantic information, enabling more efficient data processing and analysis. As a key component in … view at source ↗
Figure 3
Figure 3. (i) Codebook points of well-trained VQ-VAE models, that have different codebook sizes or latent dimensions. (ii) Sub-codebook points in our proposed MGVQ . The group size is 4. Used points are shown in red, while dead points shown in blue. (i.a) A larger dimension and smaller size may lead to an anisotropic distribution and low usage. (i.b) A larger dimension and larger size show favor of certain directions, resulti… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The process of nested masking. Gray blocks represents the masked last tokens, and other colors show active sub-groups. training and unused in inference. In contrast, the codebook usage of the baseline ( [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Qualitative reconstruction images with 16× downsampling on 2560 × 1440 UHDBench dataset. We crop a 360 × 360 sub-region, and zoom in detailed textures using blue and yellow for better view. of L1 loss, defined as LChar = p (f(x) − x) 2 + ϵ 2 ≈ |f(x) − x| = L1, and L2 l…
Figure 6
Figure 6. Figure 6: Qualitative ablation of our nested masking strategy. The first figure is the input image. Others show reconstruction results using first Mkeep sub-groups for G = 4 from left to right. The output images gradually gain detailed textures using more sub-groups. The yellow …
Figure 7
Figure 7. Figure 7: Visualization of class-to-image generation. The classes are dog(Golden Retriever and Husky), cliff, and bald eagle of ImageNet￾1k dataset, for each row. enriched features. Image Generation Results. We strictly follow LlamaGen’s code [46] and settings in the ImageNet ge…
Figure 8
Figure 8. Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. 3D and 4D World Modeling: A Survey

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A survey that defines 3D/4D world modeling, organizes methods into VideoGen, OccGen, and LiDARGen categories, and compiles datasets, metrics, and benchmark numbers.

  2. Pixel-Space Diffusion Transformers

    cs.CV 2026-07 conditional novelty 3.0 of 10

    A systematic review of pixel-space diffusion transformers, categorizing architectures and challenges for end-to-end image generation without latent compression.

Reference graph

Works this paper leans on

68 extracted references · 36 canonical work pages · cited by 2 Pith papers

  1. [1]

    Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575,

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575,

    arXiv

  2. [2]

    Soft-to-hard vector quantization for end-to-end learn- ing compressible representations.NeurIPS, 30, 2017

    Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Luc V Gool. Soft-to-hard vector quantization for end-to-end learn- ing compressible representations.NeurIPS, 30, 2017. 2

    work page 2017

  3. [3]

    Factorized visual to- kenization and generation.arXiv preprint arXiv:2411.16681,

    Zechen Bai, Jianxiong Gao, Ziteng Gao, Pichao Wang, Zheng Zhang, Tong He, and Mike Zheng Shou. Factorized visual to- kenization and generation.arXiv preprint arXiv:2411.16681,

    arXiv

  4. [4]

    Es- timating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432,

    Yoshua Bengio, Nicholas L´eonard, and Aaron Courville. Es- timating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432,

    arXiv

  5. [5]

    Matryoshka multimodal models

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. InWorkshop on Video- Language Models@ NeurIPS 2024, 2024. 5

    work page 2024

  6. [6]

    Deep com- pression autoencoder for efficient high-resolution diffusion models.arXiv preprint arXiv:2410.10733, 2024

    Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep com- pression autoencoder for efficient high-resolution diffusion models.arXiv preprint arXiv:2410.10733, 2024. 4

    arXiv 2024

  7. [7]

    Od- vae: An omni-dimensional video compressor for improving la- tent video diffusion model.arXiv preprint arXiv:2409.01199,

    Liuhan Chen, Zongjian Li, Bin Lin, Bin Zhu, Qian Wang, Shenghai Yuan, Xing Zhou, Xinhua Cheng, and Li Yuan. Od- vae: An omni-dimensional video compressor for improving la- tent video diffusion model.arXiv preprint arXiv:2409.01199,

    arXiv

  8. [8]

    Emu: Enhancing image generation models using photogenic needles in a haystack

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiao- fang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023. 4

    arXiv 2023

Show all 68 references
  1. [9]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InCVPR, pages 248–255. Ieee, 2009. 7, 9

    2009

  2. [10]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InCVPR, pages 12873–12883, 2021. 2, 3, 4, 5, 7, 8

    2021

  3. [11]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim En- tezari, Jonas M¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InICML,

  4. [12]

    Dynamical variational autoencoders: A comprehensive review.arXiv preprint arXiv:2008.12595, 2020

    Laurent Girin, Simon Leglaive, Xiaoyu Bie, Julien Diard, Thomas Hueber, and Xavier Alameda-Pineda. Dynamical variational autoencoders: A comprehensive review.arXiv preprint arXiv:2008.12595, 2020. 3

    2008 arXiv

  5. [13]

    Vector quantization.IEEE Assp Magazine, 1 (2):4–29, 1984

    Robert Gray. Vector quantization.IEEE Assp Magazine, 1 (2):4–29, 1984. 1

    1984

  6. [14]

    Dome: Tam- ing diffusion model into high-fidelity controllable occupancy world model.arXiv preprint arXiv:2410.10429, 2024

    Songen Gu, Wei Yin, Bu Jin, Xiaoyang Guo, Junming Wang, Haodong Li, Qian Zhang, and Xiaoxiao Long. Dome: Tam- ing diffusion model into high-fidelity controllable occupancy world model.arXiv preprint arXiv:2410.10429, 2024. 2

    2024 arXiv

  7. [15]

    Learnings from scaling visual tokenizers for reconstruction and generation.arXiv preprint arXiv:2501.09755, 2025

    Philippe Hansen-Estruch, David Yan, Ching-Yao Chung, Orr Zohar, Jialiang Wang, Tingbo Hou, Tao Xu, Sriram Vish- wanath, Peter Vajda, and Xinlei Chen. Learnings from scaling visual tokenizers for reconstruction and generation.arXiv preprint arXiv:2501.09755, 2025. 4

    2025 arXiv

  8. [16]

    Driving- world: Constructingworld model for autonomous driving via video gpt.arXiv preprint arXiv:2412.19505, 2024

    Xiaotao Hu, Wei Yin, Mingkai Jia, Junyuan Deng, Xiaoyang Guo, Qian Zhang, Xiaoxiao Long, and Ping Tan. Driving- world: Constructingworld model for autonomous driving via video gpt.arXiv preprint arXiv:2412.19505, 2024. 5

    2024 arXiv

  9. [17]

    Image-to-image translation with conditional adversarial net- works

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial net- works. InCVPR, pages 1125–1134, 2017. 5

    2017

  10. [18]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pages 4401–4410, 2019. 7

    2019

  11. [19]

    Auto-encoding variational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013. 2, 3

    2013

  12. [20]

    An introduction to variational autoencoders.Foundations and Trends® in Machine Learning, 12(4):307–392, 2019

    Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders.Foundations and Trends® in Machine Learning, 12(4):307–392, 2019. 3

    2019

  13. [21]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InICCV, pages 4015–4026, 2023. 7

    2023

  14. [22]

    Matryoshka representation learning.NeurIPS, 35: 30233–30249, 2022

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning.NeurIPS, 35: 30233–30249, 2022. 5

    2022

  15. [23]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open im- ages dataset v4: Unified image classification, object detection, and visual relationship detection...

    1956

  16. [24]

    Fast and accurate image super-resolution with deep laplacian pyramid networks.PAMI, 41(11):2599–2613,

    Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming- Hsuan Yang. Fast and accurate image super-resolution with deep laplacian pyramid networks.PAMI, 41(11):2599–2613,

  17. [25]

    Autoregressive image generation using resid- ual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using resid- ual quantization. InCVPR, pages 11523–11532, 2022. 2, 8

    2022

  18. [26]

    Unimo-2: End-to- end unified vision-language grounded learning.arXiv preprint arXiv:2203.09067, 2022

    Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. Unimo-2: End-to- end unified vision-language grounded learning.arXiv preprint arXiv:2203.09067, 2022. 2

    2022 arXiv

  19. [27]

    Efficient neural radiance fields for interactive free-viewpoint video

    Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Efficient neural radiance fields for interactive free-viewpoint video. InSIGGRAPH Asia Conference Proceedings, 2022. 7

    2022

  20. [28]

    Cross- modal discrete representation learning.arXiv preprint arXiv:2106.05438, 2021

    Alexander H Liu, SouYoung Jin, Cheng-I Jeff Lai, An- drew Rouditchenko, Aude Oliva, and James Glass. Cross- modal discrete representation learning.arXiv preprint arXiv:2106.05438, 2021. 2

    2021 arXiv

  21. [29]

    J. Liu, D. Liu, W. Yang, S. Xia, X. Zhang, and Y . Dai. A comprehensive benchmark for single image compression arti- fact reduction.IEEE Transactions on Image Processing, 29: 7845–7860, 2020. 7

    2020

  22. [30]

    Deep learning face attributes in the wild

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. InICCV, pages 3730–3738, 2015. 7

    2015

  23. [31]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  24. [32]

    Open-magvit2: An open-source project toward democratizing auto-regressive visual generation.arXiv preprint arXiv:2409.04410, 2024

    Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation.arXiv preprint arXiv:2409.04410, 2024. 2, 5, 7, 8

    2024 arXiv

  25. [33]

    Uavid: A semantic segmentation dataset for uav imagery.ISPRS journal of photogrammetry and remote sensing, 165:108–119, 2020

    Ye Lyu, George V osselman, Gui-Song Xia, Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic segmentation dataset for uav imagery.ISPRS journal of photogrammetry and remote sensing, 165:108–119, 2020. 7

    2020

  26. [34]

    Unitok: A unified tokenizer for visual generation and understanding

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generation and understanding. arXiv preprint arXiv:2502.20321, 2025. 2

    2025

  27. [35]

    Discrete representations strengthen vision transformer robustness.arXiv preprint arXiv:2111.10493, 2021

    Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl V ondrick, Rahul Sukthankar, and Irfan Essa. Discrete representations strengthen vision transformer robustness.arXiv preprint arXiv:2111.10493, 2021. 2

    2021 arXiv

  28. [36]

    Spring: A high-resolution high- detail dataset and benchmark for scene flow, optical flow and stereo

    Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nali- vayko, and Andr´es Bruhn. Spring: A high-resolution high- detail dataset and benchmark for scene flow, optical flow and stereo. InCVPR, pages 4981–4991, 2023. 7

    2023

  29. [37]

    Finite scalar quantization: Vq-vae made simple.arXiv preprint arXiv:2309.15505, 2023

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple.arXiv preprint arXiv:2309.15505, 2023. 3

    2023 arXiv

  30. [38]

    The mapillary vistas dataset for semantic understanding of street scenes

    Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. InICCV, pages 4990–4999,

  31. [39]

    A benchmark dataset and evaluation methodology for video object segmentation

    Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. InCVPR, pages 724–732, 2016. 7

    2016

  32. [40]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023. 2, 3, 5, 7

    2023 arXiv

  33. [41]

    Generat- ing diverse high-fidelity images with vq-vae-2.NeurIPS, 32,

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generat- ing diverse high-fidelity images with vq-vae-2.NeurIPS, 32,

  34. [42]

    Learning ordered representations with nested dropout

    Oren Rippel, Michael Gelbart, and Ryan Adams. Learning ordered representations with nested dropout. InICML, pages 1746–1754. PMLR, 2014. 5

    2014

  35. [43]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InCVPR, pages 10684– 10695, 2022. 2, 3, 4, 5, 7, 8

    2022

  36. [44]

    Laion-5b: An open large-scale dataset for training next gener- ation image-text models.NeurIPS, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next gener- ation image-text models.NeurIPS, 35:25278–25294, 2022. 7

    2022

  37. [45]

    Textocr: Towards large- scale end-to-end reasoning for arbitrary-shaped scene text

    Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wo- jciech Galuba, and Tal Hassner. Textocr: Towards large- scale end-to-end reasoning for arbitrary-shaped scene text. In CVPR, pages 8802–8812, 2021. 7

    2021

  38. [46]

    Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024. 2, 3, 4, 5, 7, 8

    2024 arXiv

  39. [47]

    Sq-vae: Variational bayes on discrete representation with self-annealed stochastic quantization.arXiv preprint arXiv:2205.07547, 2022

    Yuhta Takida, Takashi Shibuya, WeiHsiang Liao, Chieh- Hsin Lai, Junki Ohmura, Toshimitsu Uesaka, Naoki Murata, Shusuke Takahashi, Toshiyuki Kumakura, and Yuki Mitsu- fuji. Sq-vae: Variational bayes on discrete representation with self-annealed stochastic quantization.arXiv pre...

    2022 arXiv

  40. [48]

    Visual autoregressive modeling: Scalable image gen- eration via next-scale prediction.NeurIPS, 37:84839–84865,

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image gen- eration via next-scale prediction.NeurIPS, 37:84839–84865,

  41. [49]

    Neural discrete representation learning.NeurIPS, 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.NeurIPS, 30, 2017. 5

    2017

  42. [50]

    Neural discrete representation learning.NeurIPS, 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.NeurIPS, 30, 2017. 2, 3, 5

    2017

  43. [51]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024. 2, 4, 5

    2024 arXiv

  44. [52]

    Hierarchical quantized autoen- coders.NeurIPS, 33:4524–4535, 2020

    Will Williams, Sam Ringer, Tom Ash, David MacLeod, Jamie Dougherty, and John Hughes. Hierarchical quantized autoen- coders.NeurIPS, 33:4524–4535, 2020. 2

    2020

  45. [53]

    Argoverse 2: Next generation datasets for self-driving perception and forecasting.arXiv preprint arXiv:2301.00493,

    Benjamin Wilson, William Qi, Tanmay Agarwal, John Lam- bert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Rat- nesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting.arXiv preprint arXiv:...

    arXiv

  46. [54]

    Vfhq: A high-quality dataset and benchmark for video face super-resolution

    Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and benchmark for video face super-resolution. InCVPR, pages 657–666,

  47. [55]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024. 2

    2024 arXiv

  48. [56]

    Locally hierarchical auto-regressive modeling for image generation.NeurIPS, 35:16360–16372, 2022

    Tackgeun You, Saehoon Kim, Chiheon Kim, Doyup Lee, and Bohyung Han. Locally hierarchical auto-regressive modeling for image generation.NeurIPS, 35:16360–16372, 2022. 2

    2022

  49. [58]

    Vector-quantized image modeling with improved vqgan.arXiv preprint arXiv:2110.04627, 2021

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan.arXiv preprint arXiv:2110.04627, 2021. 2, 3

    2021 arXiv

  50. [59]

    Language model beats diffusion–tokenizer is key to visual generation.arXiv preprint arXiv:2310.05737, 2023

    Lijun Yu, Jos´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation.arXiv preprint arXiv:2310.05737, 2023. 2, 3, 7

    arXiv

  51. [60]

    Towards efficient and scale-robust ultra- high-definition image demoir´eing

    Xin Yu, Peng Dai, Wenbo Li, Lan Ma, Jiajun Shen, Jia Li, and Xiaojuan Qi. Towards efficient and scale-robust ultra- high-definition image demoir´eing. InECCV, pages 646–662. Springer, 2022. 7

    2022

  52. [61]

    Towards high-resolution salient object detection

    Yi Zeng, Pingping Zhang, Jianming Zhang, Zhe Lin, and Huchuan Lu. Towards high-resolution salient object detection. InICCV, pages 7234–7243, 2019. 7

    2019

  53. [62]

    Regularized vector quantization for tokenized image synthesis

    Jiahui Zhang, Fangneng Zhan, Christian Theobalt, and Shijian Lu. Regularized vector quantization for tokenized image synthesis. InCVPR, pages 18467–18476, 2023. 2

    2023

  54. [63]

    Epona: Autoregressive dif- fusion world model for autonomous driving.arXiv preprint arXiv:2506.24113, 2025

    Kaiwen Zhang, Zhenyu Tang, Xiaotao Hu, Xingang Pan, Xiaoyang Guo, Yuan Liu, Jingwei Huang, Li Yuan, Qian Zhang, Xiao-Xiao Long, et al. Epona: Autoregressive dif- fusion world model for autonomous driving.arXiv preprint arXiv:2506.24113, 2025. 2

    2025 arXiv

  55. [64]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InCVPR, pages 586–595,

  56. [65]

    Cv-vae: A compatible video vae for latent generative video models

    Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cv-vae: A compatible video vae for latent generative video models. NeurIPS, 37:12847–12871, 2025. 2

    2025

  57. [66]

    Online clustered code- book

    Chuanxia Zheng and Andrea Vedaldi. Online clustered code- book. InICCV, pages 22798–22807, 2023. 3, 4, 5

    2023

  58. [67]

    Movq: Modulating quantized vectors for high-fidelity image generation.NeurIPS, 35:23412–23425, 2022

    Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation.NeurIPS, 35:23412–23425, 2022. 2

    2022

  59. [68]

    Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024. 2

    2024 arXiv

  60. [69]

    Address- ing representation collapse in vector quantized models with one linear layer.arXiv preprint arXiv:2411.02038, 2024

    Yongxin Zhu, Bocheng Li, Yifei Xin, and Linli Xu. Address- ing representation collapse in vector quantized models with one linear layer.arXiv preprint arXiv:2411.02038, 2024. 3

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.