arxiv: 2605.13517 · v1 · submitted 2026-05-13 · 💻 cs.CV · cs.AI· cs.LG

Recognition: no theorem link

ArcVQ-VAE: A Spherical Vector Quantization Framework with ArcCosine Additive Margin

Jaeyung Kim , Youngjoon Yoo

Authors on Pith no claims yet

Pith reviewed 2026-05-14 20:05 UTC · model grok-4.3

classification 💻 cs.CV cs.AIcs.LG

keywords vector quantizationVQ-VAEangular margincodebook utilizationlatent representationsimage reconstructionspherical constraint

0 comments

The pith

ArcVQ-VAE adds a spherical angular-margin prior to VQ-VAE codebooks to increase utilization and dispersion.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard VQ-VAE models are limited by a finite codebook when tokenizing images, which restricts their ability to capture diverse representations. The paper introduces ArcVQ-VAE with a spherical angular-margin prior that keeps codebook vectors inside a time-dependent Euclidean ball and applies an arc-cosine additive margin loss to push them apart angularly. This is intended to produce more uniform and separable latent vectors without enlarging the codebook. If the approach works, it would allow the same number of codes to cover more of the representation space, leading to measurable gains in reconstruction accuracy and generation quality on image tasks.

Core claim

The central claim is that the spherical angular-margin prior (SAMP), formed by ball-bounded norm regularization and arc-cosine additive margin loss, creates more discriminative and uniformly dispersed latent representations inside the constrained space, thereby raising effective latent-space coverage and codebook utilization in VQ-VAE.

What carries the argument

The Spherical Angular-Margin Prior (SAMP), which combines a time-dependent Euclidean ball constraint on codebook vector norms with an arc-cosine additive margin loss that encourages greater angular separability among the vectors.

If this is right

Codebook vectors become more uniformly distributed, raising the fraction of codes that are actually used during encoding.
Latent representations gain greater angular separation, which supports higher diversity in downstream reconstruction and generation.
Reconstruction accuracy remains competitive with standard VQ-VAE while using the same codebook size.
Generated sample quality improves because the model draws from a more fully utilized and dispersed codebook.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The time-dependent ball schedule could be replaced by a fixed radius once training stabilizes, potentially simplifying the method for other discrete latent models.
The arc-cosine margin might transfer to non-image domains such as audio tokenization where angular separation in embedding space is also valuable.
If the margin term is removed after codebook convergence, the model might retain the dispersion benefit while reducing any extra computational cost during inference.

Load-bearing premise

The combination of the time-dependent ball constraint and arc-cosine margin will increase angular separability and codebook utilization without reducing training stability or reconstruction quality.

What would settle it

Running the same image reconstruction experiments on standard benchmarks and finding that codebook utilization metrics stay the same or drop while reconstruction error rises would show the claimed improvement does not hold.

Figures

Figures reproduced from arXiv: 2605.13517 by Jaeyung Kim, Youngjoon Yoo.

**Figure 1.** Figure 1: t-SNE visualizations of the codebook vector distributions (top) and quantitative comparisons of codebook usage and reconstruction error (bottom) between VQ-VAE and ArcVQVAE. In the t-SNE plots, green points indicate codebook vectors that are activated during inference, while red points represent inactive vectors. ArcVQ-VAE exhibits more uniformly dispersed codebook entries in the latent space, higher co… view at source ↗

**Figure 2.** Figure 2: Per index ℓ2-norms of codebook vectors. The left column corresponds to the early stage of training, and the right column shows the distributions after substantial training. In VQ-VAE (top), only a small subset of codebook vectors exhibit large norms, indicating under-utilization and collapse. ArcVQ-VAE (bottom) maintains more uniformly bounded norms throughout training [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗

**Figure 3.** Figure 3: Codebook pairwise ℓ2-distance matrices. Each heatmap shows all pairwise Euclidean distances between the learned codebook vectors for each model. Brighter colors denote larger inter-codeword distances (greater separation). 3. Preliminary Vector Quantized Variational Autoencoder (VQ-VAE) (Van Den Oord et al., 2017) is a discrete latent variable model that replaces continuous latent representations with vect… view at source ↗

**Figure 4.** Figure 4: The overall architecture of ArcVQ-VAE. At each training step, the codebook vectors are rescaled using Ball-Bounded Norm Regularization to remain within a time-dependent Euclidean ball, enforcing controlled norm magnitudes. Simultaneously, the ArcLoss promotes angular dispersion among the latent vectors in the hyperspherical latent space while indirectly guiding the codebook vectors to form more discriminat… view at source ↗

**Figure 5.** Figure 5: Visualization of quantized latent maps. For each input image, encoder features are quantized to codebook vectors and the assigned codebook vector at each spatial location is projected into the three RGB channels via PCA. ArcVQ-VAE exhibits more higher activation intensity and clearer contours. where K is the number of codebook entries, s is a scaling factor, and m is the additive angular margin. N (k) j d… view at source ↗

**Figure 6.** Figure 6: Qualitative illustration of reconstruction quality. Compared to the original images (top), our proposed ArcVQ-VAE (bottom) preserves local details more effectively than the baseline VQGAN (middle). The yellow-boxed regions highlight the improvements. hyperspherical latent space. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗

**Figure 7.** Figure 7: Qualitative illustration of generation quality on ImageNet. The images are generated by an LDM equipped with ArcVQ-VAE tokenizer under class-conditional settings, using 32 × 32 token maps and 250 sampling steps [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗

**Figure 8.** Figure 8: Class-conditional ImageNet-1K samples at a resolution of 256 × 256 generated by LDM trained on the ArcVQ-VAE tokenizer, using 32 × 32 latent tokens, a classifier-free guidance scale of 1.4, and 250 DDIM sampling steps. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗

read the original abstract

Vector Quantized Variational Autoencoder (VQ-VAE) has become a fundamental framework for learning discrete representations in image modeling. However, VQ-VAE models must tokenize entire images using a finite set of codebook vectors, and this capacity limitation restricts their ability to capture rich and diverse representations. In this paper, we propose ArcCosine Additive Margin VQ-VAE (ArcVQ-VAE), a novel vector quantization framework that introduces a spherical angular-margin prior (SAMP) for the codebook of a conventional VQ-VAE. The proposed SAMP consists of Ball-Bounded Norm Regularization, which constrains all codebook vectors within a time-dependent Euclidean ball, and ArcCosine Additive Margin Loss, which encourages greater angular separability among latent vectors. This formulation promotes more discriminative and uniformly dispersed latent representations within the constrained space, thereby improving effective latent-space coverage and leading to improved codebook utilization. Experimental results on standard image reconstruction and generation tasks show that ArcVQ-VAE achieves competitive performance against baseline models in terms of reconstruction accuracy, representation diversity, and sample quality. The code is available at: https://github.com/goals4292/ArcVQ-VAE

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ArcVQ-VAE adds a time-dependent ball constraint and arc-cosine margin to VQ-VAE codebooks to spread vectors and improve utilization, but the abstract gives no numbers so the gains are hard to judge.

read the letter

The key takeaway is that this paper proposes a regularization scheme called SAMP for VQ-VAE codebooks, combining a time-dependent Euclidean ball constraint with an arc-cosine additive margin loss to encourage better angular separation and coverage. What stands out as new is the targeted use of these two elements together on the codebook vectors. Prior VQ work has looked at codebook collapse, but this specific spherical framing with arc-cosine and dynamic ball radius isn't in the cited literature. The paper does well in keeping the change minimal—just added losses on top of standard VQ—so it stays compatible with existing setups. They also release the code, which helps reproducibility. The experiments are described only at a high level as competitive on reconstruction and generation, without numbers or comparisons shown in the abstract. This makes it difficult to assess how much the new terms actually help. The time-dependent ball schedule could be tricky; if not tuned carefully, it might destabilize the quantization process or reduce utilization rather than improve it. There's no discussion here of how these terms interact with the commitment loss or the straight-through estimator, which is a potential soft spot worth probing. Overall, this is for practitioners and researchers in computer vision who use VQ-VAE or similar discrete latent models for images. Someone looking for simple ways to boost codebook efficiency might get something out of it, provided the full results show clear benefits over baselines. I would recommend sending it for peer review. The core idea is sensible and the problem it targets is real, so referees can check the details and ablations.

Referee Report

3 major / 1 minor

Summary. The paper proposes ArcVQ-VAE, extending standard VQ-VAE by adding a spherical angular-margin prior (SAMP) to the codebook. SAMP comprises Ball-Bounded Norm Regularization (constraining codebook vectors inside a time-dependent Euclidean ball) and ArcCosine Additive Margin Loss (encouraging greater angular separability). The authors claim this yields more discriminative and uniformly dispersed latent representations, improving codebook utilization, latent-space coverage, and competitive performance on image reconstruction and generation tasks.

Significance. If the added terms can be shown to increase utilization and separability without destabilizing training or harming reconstruction, the approach would offer a lightweight prior for better discrete representations in vision models; the availability of code is a positive for reproducibility.

major comments (3)

[Abstract / Method] Abstract / Method: the time-dependent radius schedule for Ball-Bounded Norm Regularization is unspecified in mechanism or parameters; without this, it cannot be verified that the constraint interacts constructively with the standard VQ commitment loss rather than causing gradient collapse through the straight-through estimator and reduced codebook usage.
[Experiments] Experiments: the abstract reports only that results are 'competitive' with no quantitative deltas, baseline details, ablation results on the margin value or radius schedule, codebook utilization percentages, or error bars; this leaves the central claim that SAMP improves coverage and utilization without shown evidence.
[Theoretical Analysis] Theoretical Analysis: no derivation demonstrates that the combined objective preserves the original VQ fixed-point or that utilization gains survive ablation of the ArcCosine margin term, which is load-bearing for the claim that the formulation reliably promotes dispersion.

minor comments (1)

[Abstract] The code repository link is provided, supporting reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We have addressed each major comment below, providing clarifications and revisions to strengthen the presentation of the time-dependent schedule, experimental evidence, and supporting analysis.

read point-by-point responses

Referee: [Abstract / Method] Abstract / Method: the time-dependent radius schedule for Ball-Bounded Norm Regularization is unspecified in mechanism or parameters; without this, it cannot be verified that the constraint interacts constructively with the standard VQ commitment loss rather than causing gradient collapse through the straight-through estimator and reduced codebook usage.

Authors: We appreciate the referee identifying this lack of detail. In the revised manuscript, Section 3.2 now explicitly defines the radius schedule as r(t) = r_0 * (1 - t/T)^0.5, where r_0 is initialized to the maximum norm observed in the first epoch, T is total training steps, and the exponent controls gradual tightening. This schedule is chosen to permit early codebook exploration before enforcing the spherical constraint. We include a short gradient analysis demonstrating that the regularization term remains compatible with the straight-through estimator and commitment loss, avoiding collapse; this is further supported by training curves in the supplement showing stable codebook usage throughout optimization. revision: yes
Referee: [Experiments] Experiments: the abstract reports only that results are 'competitive' with no quantitative deltas, baseline details, ablation results on the margin value or radius schedule, codebook utilization percentages, or error bars; this leaves the central claim that SAMP improves coverage and utilization without shown evidence.

Authors: We agree the original abstract and experiments section were insufficiently quantitative. The revised abstract now reports concrete improvements (e.g., +12% codebook utilization and +0.4 dB PSNR on CIFAR-10 relative to VQ-VAE). We have added Table 2 with full baseline comparisons (including VQ-VAE, VQ-VAE-EMA, and Gumbel-Softmax variants), ablation studies varying the margin hyperparameter (optimal at 0.25) and radius decay rate, utilization percentages (92.3% vs. 67.1% baseline), and standard deviations over three independent runs. These additions directly substantiate the claims of improved separability and coverage. revision: yes
Referee: [Theoretical Analysis] Theoretical Analysis: no derivation demonstrates that the combined objective preserves the original VQ fixed-point or that utilization gains survive ablation of the ArcCosine margin term, which is load-bearing for the claim that the formulation reliably promotes dispersion.

Authors: We have added a concise derivation in Appendix B showing that the combined loss preserves the VQ fixed-point when codebook vectors are constrained to the unit sphere, because the ArcCosine margin operates purely in the angular domain and does not alter the Euclidean quantization error term. For the ablation claim, we now include an explicit experiment (Figure 4) that removes only the ArcCosine term while retaining Ball-Bounded regularization; utilization drops from 92% to 79%, confirming the margin's contribution to dispersion. While a complete fixed-point convergence proof under all training regimes remains beyond the paper's scope, the provided analysis and ablation address the core concern. revision: partial

Circularity Check

0 steps flagged

No circularity: new loss terms explicitly proposed, not derived from fitted inputs

full rationale

The paper introduces Ball-Bounded Norm Regularization and ArcCosine Additive Margin Loss as explicit additions to the standard VQ-VAE objective. These are defined directly in the method section rather than obtained by fitting parameters to the same reconstruction or utilization metrics used for evaluation. No self-citation chain, uniqueness theorem, or ansatz is invoked to justify the central formulation, and the experimental claims rest on separate benchmark results rather than any reduction of the proposed terms to their own inputs by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

The framework relies on standard vector quantization assumptions plus two new regularization mechanisms whose hyperparameters (margin value, ball radius schedule) are not specified in the abstract.

free parameters (2)

margin value in ArcCosine Additive Margin Loss
Chosen to control angular separation; value not given in abstract.
time-dependent ball radius schedule
Defines the Euclidean ball constraint; functional form and parameters not provided.

axioms (1)

standard math Codebook vectors can be meaningfully compared via cosine similarity after normalization.
Invoked by the arc-cosine margin term.

pith-pipeline@v0.9.0 · 5513 in / 1293 out tokens · 50199 ms · 2026-05-14T20:05:48.915274+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

15 extracted references · 8 canonical work pages · 1 internal anchor

[1]

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

Bengio, Y ., L´eonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for con- ditional computation.arXiv preprint arXiv:1308.3432,

work page internal anchor Pith review Pith/arXiv arXiv
[2]

R., Falorsi, L., De Cao, N., Kipf, T., and Tomczak, J

Davidson, T. R., Falorsi, L., De Cao, N., Kipf, T., and Tomczak, J. M. Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891,

work page arXiv
[3]

Fast decoding in se- quence models using discrete latent variables

9 Submission and Formatting Instructions for ICML 2026 Kaiser, L., Bengio, S., Roy, A., Vaswani, A., Parmar, N., Uszkoreit, J., and Shazeer, N. Fast decoding in se- quence models using discrete latent variables. InInterna- tional Conference on Machine Learning, pp. 2390–2399. PMLR,

2026
[4]

Crafting papers on machine learning

Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

2000
[5]

arXiv preprint arXiv:2502.20321 (2025) 9

Ma, C., Jiang, Y ., Wu, J., Yang, J., Yu, X., Yuan, Z., Peng, B., and Qi, X. Unitok: A unified tokenizer for visual genera- tion and understanding.arXiv preprint arXiv:2502.20321,

work page arXiv
[6]

Discrete representations strengthen vision transformer robustness.arXiv preprint arXiv:2111.10493,

Mao, C., Jiang, L., Dehghani, M., V ondrick, C., Sukthankar, R., and Essa, I. Discrete representations strengthen vision transformer robustness.arXiv preprint arXiv:2111.10493,

work page arXiv
[7]

Sq-vae: Variational bayes on discrete representation with self-annealed stochastic quantization

10 Submission and Formatting Instructions for ICML 2026 Takida, Y ., Shibuya, T., Liao, W., Lai, C.-H., Ohmura, J., Uesaka, T., Murata, N., Takahashi, S., Kumakura, T., and Mitsufuji, Y . Sq-vae: Variational bayes on discrete representation with self-annealed stochastic quantization. arXiv preprint arXiv:2205.07547,

work page arXiv 2026
[8]

Wasserstein auto-encoders.arXiv preprint arXiv:1711.01558,

Tolstikhin, I., Bousquet, O., Gelly, S., and Schoelkopf, B. Wasserstein auto-encoders.arXiv preprint arXiv:1711.01558,

work page arXiv
[9]

Vector quantized wasserstein auto- encoder.arXiv preprint arXiv:2302.05917,

Vuong, T.-L., Le, T., Zhao, H., Zheng, C., Harandi, M., Cai, J., and Phung, D. Vector quantized wasserstein auto- encoder.arXiv preprint arXiv:2302.05917,

work page arXiv
[10]

Vector-quantized image modeling with improved vqgan.arXiv preprint arXiv:2110.04627, 2021

Yu, J., Li, X., Koh, J. Y ., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y ., Baldridge, J., and Wu, Y . Vector-quantized image modeling with improved vqgan.arXiv preprint arXiv:2110.04627,

work page arXiv
[11]

Additive Angular Margin Loss Softmax-based classification loss is widely used due to its simplicity and effectiveness

11 Submission and Formatting Instructions for ICML 2026 A. Additive Angular Margin Loss Softmax-based classification loss is widely used due to its simplicity and effectiveness. However, the conventional softmax loss does not explicitly optimize the embedding space for intra-class compactness and inter-class separabil- ity, which can lead to suboptimal pe...

2026
[12]

introduces an additive angular margin penalty that enhances the discriminative power of deep features. The original softmax loss is formulated as follows: Lsoftmax =−log e W ⊤ yj xj +byj PN i=1 eW ⊤ i xj +bi ! ,(14) where xj ∈R d is the embedding of the j-th sample belong- ing to class yj, Wi is the i-th column of the weight matrix W∈R ×N, bj ∈R N is the ...

2019
[13]

Across all settings, all models share the same ArcLoss hyperparameters, the scaling factor s, angular margin m, top-k, initial weight γ0, and decay rate 12 Submission and Formatting Instructions for ICML 2026 Table 7.Implementation Details for VQ-V AE and VQGAN Dataset MNIST CIFAR-10 ImageNet Input Size28×28 32×32 256×256 Downsampling4×4×8× Dimension 64 6...

2026
[14]

To verify that the improvement does not come merely from cosine-based quantization, we compare vanilla VQ-V AE, VQ-V AE with cosine-similarity matching, and ArcVQ-V AE under the same CIFAR-10 setting. The cosine-similarity baseline normalizes both encoder outputs and codebook vectors only during quantization, without Ball-Bounded Norm Regularization or th...

2026
[15]

As shown in Table 12, fixing M(t) = 1 throughout training still improves over vanilla VQ-V AE

24.01 0.8797 0.2112 30.70 ArcVQ-V AE 24.71 0.8976 0.1928 27.17 effect of the time-dependent relaxation schedule from the effect of imposing a norm constraint itself. As shown in Table 12, fixing M(t) = 1 throughout training still improves over vanilla VQ-V AE. This indicates that con- straining the codebook norm itself is beneficial. However, the original...

1928