Pith. sign in

REVIEW 5 cited by

Restructuring Vector Quantization with the Rotation Trick

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.06424 v2 pith:BSKHYYNE submitted 2024-10-08 cs.LG cs.CV

Christopher Fifty , Ronald G. Junkins , Dennis Duan , Aniketh Iyengar , Jerry W. Liu , Ehsan Amid , Sebastian Thrun , Christopher Ré This is my paper
classification cs.LGcs.CV
keywords vectorquantizationcodebookencoderlayeroutputrotationapproximation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Vector Quantized Variational AutoEncoders (VQ-VAEs) are designed to compress a continuous input to a discrete latent space and reconstruct it with minimal distortion. They operate by maintaining a set of vectors -- often referred to as the codebook -- and quantizing each encoder output to the nearest vector in the codebook. However, as vector quantization is non-differentiable, the gradient to the encoder flows around the vector quantization layer rather than through it in a straight-through approximation. This approximation may be undesirable as all information from the vector quantization operation is lost. In this work, we propose a way to propagate gradients through the vector quantization layer of VQ-VAEs. We smoothly transform each encoder output into its corresponding codebook vector via a rotation and rescaling linear transformation that is treated as a constant during backpropagation. As a result, the relative magnitude and angle between encoder output and codebook vector becomes encoded into the gradient as it propagates through the vector quantization layer and back to the encoder. Across 11 different VQ-VAE training paradigms, we find this restructuring improves reconstruction metrics, codebook utilization, and quantization error. Our code is available at https://github.com/cfifty/rotation_trick.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation

    cs.RO 2025-12 conditional novelty 6.0 of 10

    An imitation-learning system that segments demonstrations into VLM-labeled atomic skills, aligns them with contrastive learning, and uses keypose prediction to chain skills, outperforming prior baselines in multi-task...

  2. Generative Recommendation with Semantic IDs: A Practitioner's Handbook

    cs.IR 2025-07 conditional novelty 6.0 of 10

    An open-source framework and ablation study showing which semantic-ID generative recommendation components actually matter, with results that challenge several standard defaults.

  3. Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    1D binary image latents reduce a 1024x1024 image to 128 discrete tokens and support text-to-image generation with diffusion and autoregressive models.

  4. Learning Reusable Hybrid Motion Priors for Humanoid Locomotion from Motion Imitation

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A frozen hybrid motion prior, distilled from a humanoid motion-imitation expert via residual vector quantization, lets task policies choose discrete codebook entries to perform multiple locomotion tasks and transfers ...

  5. LGQ: Learnable Geometric Quantization for Image Tokenization

    cs.CV 2026-02 reject novelty 4.0 of 10

    LGQ reports better ImageNet reconstruction FID than FSQ/SimVQ using soft-to-hard learnable-codebook quantization, but its abstract's generation and utilization claims are contradicted by the body.

Pith tools