Pith. sign in

REVIEW 7 cited by

Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.11837 v1 pith:OF3N5QAQ submitted 2024-06-17 cs.CV

Lei Zhu , Fangyun Wei , Yanye Lu , Dong Chen This is my paper
classification cs.CV
keywords codebookimagesizeutilizationmodelratevqganvqgan-lc
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In the realm of image quantization exemplified by VQGAN, the process encodes images into discrete tokens drawn from a codebook with a predefined size. Recent advancements, particularly with LLAMA 3, reveal that enlarging the codebook significantly enhances model performance. However, VQGAN and its derivatives, such as VQGAN-FC (Factorized Codes) and VQGAN-EMA, continue to grapple with challenges related to expanding the codebook size and enhancing codebook utilization. For instance, VQGAN-FC is restricted to learning a codebook with a maximum size of 16,384, maintaining a typically low utilization rate of less than 12% on ImageNet. In this work, we propose a novel image quantization model named VQGAN-LC (Large Codebook), which extends the codebook size to 100,000, achieving an utilization rate exceeding 99%. Unlike previous methods that optimize each codebook entry, our approach begins with a codebook initialized with 100,000 features extracted by a pre-trained vision encoder. Optimization then focuses on training a projector that aligns the entire codebook with the feature distributions of the encoder in VQGAN-LC. We demonstrate the superior performance of our model over its counterparts across a variety of tasks, including image reconstruction, image classification, auto-regressive image generation using GPT, and image creation with diffusion- and flow-based generative models. Code and models are available at https://github.com/zh460045050/VQGAN-LC.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HiTokSR: A Coarse-to-Fine Tokenizer with Hierarchical Codebooks for High-Fidelity Real-World Image Super-Resolution

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    HiTokSR uses a coarse-to-fine hierarchical tokenizer with frequency-aware sub-codebooks, vision foundation model priors, and index perturbation to achieve state-of-the-art perceptual quality and fidelity in real-world...

  2. Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment

    cs.SD 2026-06 unverdicted novelty 6.0 of 10

    Self-guidance adds a lightweight feature-mapping loss to align decoder manifolds in VQ-VAE speech codecs, raising reconstruction metrics and allowing 4x codebook reduction with no fidelity loss.

  3. IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    IDEAL improves discrete representation autoencoders by jointly aligning quantized tokens with shallow and deep VFM features, reporting 0.61 rFID on ImageNet and 1.89 gFID for autoregressive image generation.

  4. Vision Foundation Models as Generalist Tokenizers for Image Generation

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    VFMTok builds a generalist image tokenizer on frozen VFMs using adaptive quantization and semantic alignment, delivering gFID 1.36 for autoregressive and 1.25 for continuous generation on ImageNet with 3x faster convergence.

  5. Scalable Training for Vector-Quantized Networks with 100% Codebook Utilization

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A training-only ViT-based projector, VQBridge, combined with learning annealing, achieves full codebook utilization in vector-quantized networks at large codebook sizes, improving reconstruction and autoregressive ima...

  6. NSVQ: Mitigating Codebook Collapse by Stabilizing Encoder Drift in Vector Quantization

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    NSVQ mitigates codebook collapse in large-codebook VQ by addressing encoder drift via non-stationary loss, replacement, and staged freezing, improving rFID from 2.39 to 2.10 on ImageNet-1k while achieving 100% utilization.

  7. Mutual Enhancement Between Global Tokens and Patch Tokens: From Theory to Practice

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    TaTok is a theoretically grounded adaptive tokenization method that uses global tokens and cumulative conditional entropy filtering to reduce redundancy while improving reconstruction quality over fixed-rate patch tok...

Pith tools