Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Discrete video codes with a codebook-size knob and learned priors reach 0.043–0.064 bpp and beat H.265 on perceptual quality while using 5–7.6× fewer bits.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

MS-VQ-VAE with EMA codebooks and 3D autoregressive priors achieves 0.043–0.064 bpp on UCF101 64×64 video and outperforms H.265 CRF 36 on LPIPS at 5–7.6× lower bitrate.

T0 review reviewed 2026-07-12 challenge →

load-bearing objection Clean K-sweep RD curve for hierarchical VQ video at 0.04–0.06 bpp that beats H.265 CRF 36 on LPIPS; the 5–7.6 imes factor is slightly soft because BPP is prior cross-entropy, not measured file size. the 3 major comments →

arxiv 2607.02562 v1 pith:7A26DKER submitted 2026-06-28 cs.CV

Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression

Manikanta Kotthapalli , Banafsheh Rekabdar This is my paper
classification cs.CV
keywords learned video compressionvector quantizationVQ-VAEautoregressive priorentropy codingultra-low bitraterate-distortionMS-VQ-VAE
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Continuous-latent learned video codecs cannot reliably go below about 0.1 bits per pixel because Lagrangian rate penalties stop working when the rate term is driven toward zero. This paper shows that vector-quantized (discrete) latents solve the problem by construction: the codebook size K hard-caps information at log2 K bits per symbol, and a separate autoregressive prior then exploits the power-law usage of those codes to push the actual bitrate well below that ceiling with no rate-penalty tuning. Sweeping K in {128, 256, 512, 1024} under a fixed training recipe yields a clean rate–distortion ladder at 0.043–0.064 bpp on 64×64 UCF101 clips—3–7× below the practical floors of H.264 and H.265. Every model beats H.265 CRF 36 on the perceptual metric LPIPS; the K=1024 model does so by 0.072 absolute while using 5.1× fewer bits. The authors also show that ordinary gradient codebook updates collapse at small K, while EMA updates plus dead-code restart keep the full vocabulary alive, making the capacity ceiling real rather than fictitious.

Core claim

Codebook size K is an information-theoretic capacity parameter for multi-scale VQ video codecs. Once EMA-stabilised codebooks with dead-code restart prevent collapse, a learned autoregressive prior converts the resulting power-law index sequences into bitrates of 0.043–0.064 bpp that lie far below traditional codec floors while delivering superior LPIPS to H.265 CRF 36 at every tested K.

What carries the argument

The K-controlled discrete bottleneck plus two-stage training: Stage A trains a hierarchical MS-VQ-VAE with EMA codebook updates and dead-code restart; Stage B freezes the indices and trains 3D autoregressive priors whose cross-entropy is exactly the expected arithmetic-code bitrate (Eq. 6).

Load-bearing premise

That the prior’s expected cross-entropy is a faithful stand-in for real arithmetic-coded file size, and that 64×64 UCF101 results with LPIPS as the main score will carry over to the resolutions and applications where the method would actually be used.

What would settle it

Encode the same 500 clips with a practical arithmetic coder under the trained priors and measure whether the true file sizes stay within a few percent of the reported BPP numbers; if the gap is large, or if the LPIPS advantage disappears at 128^{2}/256^{2} or against VVC/DCVC at matched bitrate, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • K becomes a predictable structural rate-control knob that needs no differentiable rate signal and no λ sweep.
  • At the bitrate floor where continuous-latent codecs lose control, discrete hierarchical VQ plus structured priors is a workable alternative.
  • EMA with dead-code restart is required, not optional, for any VQ video codec operating at small K.
  • The same power-law index statistics that make the prior effective also mark the pipeline as a genuine learned entropy coder (70–85 % efficiency).
  • The architecture is deliberately scoped to offline and storage-optimised uses where 5–7× bitrate savings outweigh sequential prior decoding cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the same K-sweep works at 128^{2} or 256^{2}, the first systematic ultra-low-bitrate RD curves at near-SD resolution become available.
  • Because the priors already produce low residual entropy, the same discrete tokens could serve as a substrate for joint compression-and-generation models.
  • A continuous rate token or parallel masked decoder that preserves the top-to-bottom conditioning would remove the two main practical limits (discrete operating points and sequential latency) without abandoning the hard capacity ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that continuous-latent learned video codecs cannot reliably operate below ~0.1 bpp because Lagrangian rate control lacks a hard information ceiling, whereas discrete VQ latents impose a capacity of log2 K bits per symbol that an autoregressive prior can further reduce by exploiting non-uniform (power-law) code usage. Building on MS-VQ-VAE, the authors train four models with K in {128,256,512,1024} under a uniform protocol, stabilize small-K training via EMA codebook updates plus dead-code restart, and report 0.043–0.064 bpp on 500 UCF101 64×64 clips. Every configuration beats H.265 CRF 36 on LPIPS at 5–7.6× lower estimated rate; at K=1024 the LPIPS margin is 0.072 absolute at 5.1× fewer bits. Codebook analysis shows near-full bottom utilization, 70–85% entropy efficiency, and Zipfian index frequencies.

Significance. If the empirical claims hold under practical arithmetic coding and broader evaluation, the work supplies a clean, interpretable alternative to λ-based rate control for the ultra-low-bitrate regime that continuous codecs and block-transform standards struggle to enter at tiny resolutions. The systematic K-sweep, the demonstration that EMA+restart is essential (not optional) at K≤512, and the entropy-efficiency / power-law analysis are concrete, reusable contributions. The paper is also unusually transparent about latency, PSNR-vs-LPIPS trade-offs, and open problems (continuous rate control, parallel decoding, higher resolution, stronger baselines). These strengths make the manuscript a useful reference point even if some quantitative multipliers need tightening.

major comments (3)
  1. [§4.1 Metrics / Eq. (6)] §4.1 and Eq. (6): BPP is the expected cross-entropy under the frozen 3D autoregressive priors, while H.264/H.265 BPP is measured from actual file sizes. The claim that the entropy–bitstream gap is “typically below 1–2% … below 5% even in the worst case” cites only general literature [7] and is not validated on these 4,608-step 3D priors or the 500-clip test set. Because the headline multipliers (3.3–5× vs H.264, 5–7.6× vs H.265, 5.1× at K=1024) rest entirely on this proxy, the paper should report actual arithmetic-coded bitstream sizes (or a measured overhead bound) for at least one K; otherwise the quantitative superiority claim is not fully supported.
  2. [§4.1 Experimental Setup / Table 1] §4.1–4.2 and Table 1: All results are at 64×64. At this resolution traditional codecs incur substantial header and block-structure overhead, so the reported “practical floors” (H.264 ~0.21 bpp, H.265 ~0.33 bpp) partly reflect container/syntax costs rather than pure coding inefficiency. The central claim that discrete models uniquely access a regime “structurally inaccessible” to H.264/H.265 is therefore resolution-dependent. Either a higher-resolution K-sweep (even 128² on a subset) or a quantitative decomposition of header vs. residual bits for the baselines is needed to keep the 5–7.6× statements from over-generalizing.
  3. [§4.2 / Open Problem 4] §2.2, §5.3 Open Problem 4, and Table 1: The only external baselines are libx264 and libx265. No comparison is made to any learned video codec (DCVC, scale-space flow, or other discrete tokenizers such as MAGVIT-style or TVC/GLC) at matched resolution and rate. While the authors correctly list this as future work, the claim that the pipeline is a “principled learned entropy coder” that outperforms modern codecs on perceptual quality at ultra-low rate currently rests only on block-transform baselines. At minimum, a short discussion of why existing learned methods cannot be run (or a single DCVC-style point at the lowest feasible rate) is required so readers can locate the contribution relative to the learned-compression literature.
minor comments (6)
  1. [Title / headers] Title and running headers inconsistently insert a space (“MS-VQ-V AE”). Normalize to “MS-VQ-VAE” throughout.
  2. [Figure 2] Figure 2 caption says “inverted axis, higher = better” for LPIPS while the text correctly treats LPIPS as lower-better; clarify the plotting convention so the figure is self-contained.
  3. [§1 / References] Reference [1] is an anonymous concurrent MS-VQ-VAE paper; once de-anonymized, ensure the architectural delta of the present work (K-sweep, EMA analysis, entropy coding) is stated in one sentence so novelty relative to [1] is unambiguous.
  4. [Table 2] Table 2 reports CPU single-thread latency; a GPU number (or note that prior sampling is the bottleneck even on GPU) would help readers assess the offline-use-case claim.
  5. [Table 1] No error bars or multi-seed statistics accompany Table 1; even a brief statement that metrics are stable across the 500 clips (or std. over clips) would strengthen the monotonicity claim.
  6. [§3.2] Eq. (2) uses both λ_VGG and β; a short ablation or sensitivity note on these two free parameters would help reproducibility, even if only in the appendix.

Circularity Check

1 steps flagged

No load-bearing circularity; K-sweep RD numbers and LPIPS comparisons are independent empirical measurements against external codecs, with only a non-forcing self-reference to the base architecture paper.

specific steps
  1. self citation load bearing [Abstract and Section 1 (Introduction), contribution (1)]
    "Building on the MS-VQ-VAE architecture introduced in [1], we sweep K ∈ {128, 256, 512, 1024} under a uniform training protocol to trace four operating points on the rate-distortion (RD) curve."

    The architecture is taken from the authors' own concurrent paper [1]. This is ordinary self-citation of prior engineering work and is not load-bearing for the new empirical claims (the K-sweep RD table, EMA collapse analysis, or LPIPS-vs-H.265 numbers), which are generated afresh; it therefore contributes only a score-1 minor flag rather than a circular reduction of the headline results.

full rationale

The paper's central claims (monotonic RD curve under K-sweep, 0.043-0.064 bpp operation, every configuration beating H.265 CRF 36 on LPIPS at 5-7.6x lower rate) are obtained by training four models from scratch under a uniform protocol, freezing the autoencoder, training separate autoregressive priors by cross-entropy on the resulting index sequences, and evaluating expected BPP (Eq. 6) plus PSNR/SSIM/LPIPS on the same 500 held-out UCF101 clips used for libx264/libx265 file-size baselines. None of these quantities is defined in terms of the others, fitted to the test metrics, or forced by a uniqueness theorem. The hard ceiling log2 K is an architectural fact of VQ (Eq. 1), not a fitted prediction; entropy efficiency and power-law rank plots are post-hoc measurements of the trained codebooks. The sole self-citation ([1], the concurrent MS-VQ-VAE architecture paper) supplies the two-level 3D encoder-decoder backbone that is then re-trained and re-characterized; it does not supply the RD numbers, the EMA stability result, or the H.265 LPIPS crossovers. Standard learned-compression practice of reporting prior cross-entropy as rate is used, but that is an external methodological convention, not a circular reduction inside the paper's derivation. Hence only a trivial non-load-bearing self-reference exists.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The work rests on standard VQ-VAE machinery, a small set of hand-chosen training hyperparameters, and the modeling choice that codebook size K is a sufficient rate-control knob. No new physical entities are postulated; the main free parameters are the usual deep-learning knobs plus the discrete K values themselves.

free parameters (5)
  • codebook size K = {128,256,512,1024}
    Swept over {128,256,512,1024}; each value defines a separate model and operating point. Chosen by hand as the structural rate dial.
  • λ_VGG = 0.1
    Perceptual loss weight fixed at 0.1; controls the blur-vs-hallucination trade-off that underpins the LPIPS claims.
  • β (commitment loss) = 0.25
    Fixed at 0.25; standard VQ-VAE hyperparameter that affects codebook utilization.
  • EMA decay γ = 0.99
    Set to 0.99; critical for preventing collapse at small K.
  • learning rate / epochs / batch size = 2e-4 / 20+15 / 8
    Adam 2e-4, 20+15 epochs, batch 8; standard but un-tuned publicly.
axioms (4)
  • domain assumption VQ-VAE straight-through estimator plus EMA codebook updates yield usable discrete latents for reconstruction.
    Invoked throughout §3; taken from van den Oord et al. and Razavi et al.
  • domain assumption Cross-entropy of a well-trained autoregressive prior equals practical arithmetic-coded bitrate within a few percent.
    Stated in §4.1 Metrics; standard in learned compression literature (Ballé, Minnen).
  • domain assumption LPIPS (VGG) is the appropriate primary quality metric at ultra-low bitrate where pixel fidelity is unachievable.
    Justified in §2.1 and §4.1 by reference to Mentzer et al. generative compression work.
  • ad hoc to paper 64×64 UCF101 clips are a valid test bed for the ultra-low-bitrate regime where traditional codecs hit a floor.
    Chosen deliberately in §4.1 and Open Problem 3; not a standard resolution for codec benchmarking.

reviewed 2026-07-12 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression." pith.science (2026). https://pith.science/paper/7A26DKER

@misc{pith2026260702562,
  author       = {Pith},
  title        = {Pith review of: Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7A26DKER}},
  note         = {Machine review of arXiv:2607.02562}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Learned video codecs based on continuous latent representations struggle to operate reliably below 0.1 bits per pixel~(bpp): without a differentiable rate signal, Lagrangian optimisation cannot effectively trade reconstruction quality for bitrate at extreme compression ratios. We demonstrate that discrete latent representations sidestep this limitation entirely. In a vector-quantized~(VQ) codec, the codebook size~$K$ imposes a hard information ceiling of $\log_2 K$ bits per symbol; a learned autoregressive prior then exploits the non-uniform distribution of code usage -- which we show follows a power law -- to push actual bitrates well below this ceiling, without any rate-penalty tuning. Building on the MS-VQ-VAE architecture introduced in~\cite{kotthapalli2026msvqvae}, we sweep $K \in \{128, 256, 512, 1024\}$ under a uniform training protocol to trace four operating points on the rate-distortion~(RD) curve. We identify and resolve a critical training instability: gradient-based VQ collapses catastrophically at $K \leq 512$, whereas EMA-stabilised codebook updates with dead-code restart maintain full utilisation across all configurations. On 500 UCF101 test clips ($64\!\times\!64$, 32~frames), our models operate at 0.043-0.064~bpp -- 3.3-5$\times$ below H.264's practical floor and $5$-$7.6\times$ below H.265's floor at this resolution. Every MS-VQ-VAE configuration outperforms H.265 CRF\,36 on perceptual quality (LPIPS) despite using $5$-$7.6\times$ fewer bits. At $K{=}1024$, the model surpasses H.265 CRF\,36 on LPIPS by a margin of 0.072 absolute while using $5.1\times$ fewer bits. Codebook analysis confirms power-law index distributions and 70-85\% entropy efficiency, establishing the pipeline as a principled learned entropy coder.

Figures

Figures reproduced from arXiv: 2607.02562 by Banafsheh Rekabdar, Manikanta Kotthapalli.

Figure 1
Figure 1. Figure 1: MS-VQ-VAE pipeline. The input video is fed into the Bottom-Level Encoder, whose features [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Rate–distortion curves: MS-VQ-VAE vs. H.264 and H.265. Red circles: our model at [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative frame comparison on UCF101 ( [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Codebook analysis across 𝐾 ∈ {128, 256, 512, 1024}. (a) Codebook utilisation: fraction of entries active per clip, for top and bottom codebooks. EMA with dead-code restart maintains near-complete utilisation throughout. (b) Entropy efficiency 𝜂 = 𝐻(z)/log2 𝐾: bottom codebooks achieve 𝜂 ≈ 0.70–0.85, confirming the prior captures 15–30% of maximum entropy as structural redundancy. (c) Rate decomposition: exp… view at source ↗
Figure 5
Figure 5. Figure 5: Bottom codebook embeddings: PCA projection to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Codebook Capacity Governs Perceptual Quality Across Resolutions in Hierarchical Discrete Video Compression

    eess.IV 2026-07 conditional novelty 6.0

    In MS-VQ-VAE video compression on UCF101, LPIPS quality is driven by codebook size K (p<0.001) and statistically indistinguishable from flat across resolutions (p=0.68).

Reference graph

Works this paper leans on

29 extracted references · 2 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Hierarchical vector-quantized latents for perceptual low- resolution video compression

    Anonymous Authors. Hierarchical vector-quantized latents for perceptual low- resolution video compression. InAnonymous Proceedings, 2026. [Omitted for double-blind review]

    2026

  2. [2]

    G. Lu, W. Zhang, W. Ouyang, L. Chen, Z. Gao, and D. Xu. DVC: An end-to-end deep video compression framework. InProc. IEEE/CVF CVPR, 2019

    2019

  3. [3]

    J. Li, B. Li, and Y. Lu. Deep contextual video compression.Advances in NeurIPS, 34, 2021

    2021

  4. [4]

    van den Oord, O

    A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. InAdvances in NeurIPS, vol. 30, 2017

    2017

  5. [5]

    Razavi, A

    A. Razavi, A. van den Oord, and O. Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. InAdvances in NeurIPS, vol. 32, 2019

    2019

  6. [6]

    Ballé, D

    J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston. Variational image compression with a scale hyperprior. InICLR, 2018

    2018

  7. [7]

    Minnen, J

    D. Minnen, J. Ballé, and G. D. Toderici. Joint autoregressive and hierarchical priors for learned image compression. InAdvances in NeurIPS, vol. 31, 2018

    2018

  8. [8]

    Mentzer, G

    F. Mentzer, G. D. Toderici, M. Tschannen, and E. Agustsson. High-fidelity gener- ative image compression. InAdvances in NeurIPS, vol. 33, 2020

    2020

  9. [9]

    Agustsson, D

    E. Agustsson, D. Minnen, N. Johnston, J. Ballé, S. J. Hwang, and G. Toderici. Scale-space flow for end-to-end optimized video compression. InProc. IEEE/CVF CVPR, 2020

    2020

  10. [10]

    M. Huh, B. Cheung, P. Agrawal, and P. Isola. Commitment issues: Robust training of neural discrete representations.arXiv preprint arXiv:2308.04746, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Mentzer, D

    F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen. Finite scalar quantization: VQ-VAE made simple. InICLR, 2024

    2024

  12. [12]

    van den Oord, N

    A. van den Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel recurrent neural networks. InICML, 2016

    2016

  13. [13]

    X. Chen, N. Mishra, M. Rohaninejad, and P. Abbeel. PixelSNAIL: An improved autoregressive generative model. InICML, 2018

    2018

  14. [14]

    Soomro, A

    K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. Technical Report CRCV-TR-12-01, University of Central Florida, 2012

    2012

  15. [15]

    Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assess- ment: From error visibility to structural similarity.IEEE Trans. Image Processing, 13(4):600–612, 2004

    2004

  16. [16]

    Zhang, P

    R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProc. IEEE/CVF CVPR, 2018

    2018

  17. [17]

    W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas. VideoGPT: Video generation using VQ-VAE and transformers.arXiv preprint arXiv:2104.10157, 2021

    Pith/arXiv arXiv 2021

  18. [18]

    L. Yu, Y. Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M.-H. Yang, Y. Hao, I. Essa, and L. Jiang. MAGVIT: Masked generative video transformer. InProc. IEEE/CVF CVPR, 2023

    2023

  19. [19]

    L. Yu, J. Lezama, N. B. Gundavarapu, L. Versari, K. Sohn, D. Minnen, Y. Cheng, A. Gupta, X. Gu, A. G. Hauptmann, B. Gong, M.-H. Yang, I. Essa, D. Ross, and L. Jiang. Language model beats diffusion – tokenizer is key to visual generation. InICLR, 2024

    2024

  20. [20]

    Z. Jia, H. Li, R. Zhao, and S. Ma. Generative latent coding for ultra-low bitrate image compression. InProc. IEEE/CVF CVPR, 2024

    2024

  21. [21]

    Wiegand, G

    T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra. Overview of the H.264/AVC video coding standard.IEEE Trans. Circuits Syst. Video Technol., 13(7):560–576, 2003

    2003

  22. [22]

    G. J. Sullivan, J. Ohm, W.-J. Han, and T. Wiegand. Overview of the high effi- ciency video coding (HEVC) standard.IEEE Trans. Circuits Syst. Video Technol., 22(12):1649–1668, 2012

    2012

  23. [23]

    Wu and K

    Y. Wu and K. He. Group normalization. InProc. European Conf. Computer Vision (ECCV), 2018

    2018

  24. [24]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. InProc. IEEE/CVF CVPR, 2016

    2016

  25. [25]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. InICLR, 2015

    2015

  26. [26]

    Ballé, P

    J. Ballé, P. Chou, D. Minnen, S. Singh, N. Johnston, E. Agustsson, S. J. Hwang, and G. Toderici. Nonlinear transform coding.IEEE J. Sel. Topics Signal Process., 15(2):339–353, 2021

    2021

  27. [27]

    Sheng, S

    T. Sheng, S. Ma, X. Li, Y. Li, and Z. Gao. Temporal context-aware learned video compression.IEEE Trans. Circuits Syst. Video Technol., 32(1):123–145, 2022

    2022

  28. [28]

    T. M. Cover and J. A. Thomas.Elements of Information Theory, 2nd ed. Wiley- Interscience, 2006

    2006

  29. [29]

    K. Liu, Y. Jiang, I. Choi, and J. Gu. TVC: Tokenized video compression with ultra-low bit rate.Visual Intelligence, 2025

    2025

This paper was first reviewed by grok-4.5 on July 12, 2026.