REVIEW 3 major objections 5 minor 34 references
Codebook capacity, not spatial resolution, governs perceptual quality in hierarchical discrete video codecs.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-07-31 23:37 UTC pith:SJNKQU6L
load-bearing objection Useful, transparent empirical study, but the headline 'capacity, not resolution' is confounded by the depth-scaling rule and K-bitrate coupling; worth peer review with major revision. the 3 major comments →
Codebook Capacity Governs Perceptual Quality Across Resolutions in Hierarchical Discrete Video Compression
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
For the MS-VQ-VAE architecture, perceptual quality (LPIPS) is governed almost entirely by codebook capacity rather than by spatial resolution within the tested range. The log-linear model Q(K,r)=αlog2K+βlog2r+γ fit to 12 operating points (K∈{128,256,512,1024}, r∈{64,128,256}) gives α=−0.0094 (t=−6.6, p<0.001) and β=−0.0009 (t=−0.43, p=0.68, not significant), with R²=0.82. Halving K worsens LPIPS by 0.0094, while doubling resolution improves it by only 0.0009. In parallel, bottom-level entropy efficiency η=H(z)/log2K is stable or improves with resolution (84–87% at 64×64, 92–94% at 256×256), indicating that larger spatial grids use the discrete codebook more efficiently rather than less. Acro
What carries the argument
The central mechanism is the discrete codebook as a hard information bottleneck: each latent symbol carries at most log2K bits, independent of spatial resolution. The paper uses this to make K a structural rate-control parameter (since the arg-min quantizer is non-differentiable, standard Lagrangian rate control is infeasible). The log-linear scaling law Q(K,r)=αlog2K+βlog2r+γ quantifies how this bottleneck interacts with resolution, while entropy efficiency η=H(z)/log2K measures whether larger spatial grids dilute or enrich per-symbol informativeness. The architecture is a hierarchical MS-VQ-VAE with EMA codebook updates and dead-code restart, with a depth-scaling rule (two latent levels at
Load-bearing premise
The resolution comparison is confounded: the 64×64 model uses a two-level hierarchy while 128×128 and 256×256 use three levels, and K simultaneously changes bitrate, so the measured zero effect of resolution could be an artifact of architecture depth or bitrate rather than of resolution itself.
What would settle it
Measure LPIPS at a fixed bitrate (e.g., adjust K at each resolution to produce matched BPP) while keeping the hierarchy depth constant across resolutions; if LPIPS then varies with resolution, the claimed dominance of K over r would fail. Alternatively, run the same protocol at 512×512 or 720p and check whether β remains statistically insignificant.
If this is right
- A hierarchical discrete video codec trained at one resolution can be deployed at another resolution with predictable perceptual quality by choosing K to hit the target LPIPS, avoiding resolution-specific retraining or RD recalibration.
- Codebook size K serves as a practical, structural rate-control parameter in VQ-based codecs, offering a discrete alternative to varying the Lagrange multiplier λ.
- Entropy efficiency improves with resolution, suggesting that larger spatial grids are not a liability for discrete tokenizers but instead provide statistical context that autoregressive priors exploit effectively.
- The reported LPIPS gains over H.264 (25–52% at 128×128) and H.265 (21–37% at 256×256) at matched or lower bitrate indicate that hierarchical discrete codecs are competitive perceptual codecs in the ultra-low bitrate regime.
- The scaling law provides a simple predictive tool: a designer can estimate LPIPS at unseen K or resolution within the tested range using Q(K,r)=−0.0094log2K−0.0009log2r+0.2168.
Where Pith is reading between the lines
- If the scaling law generalizes beyond UCF101 and the 64–256 resolution range, it implies a decoupling of perceptual quality from spatial resolution that continuous-latent codecs do not exhibit, potentially reshaping how multi-resolution streaming codecs are designed: the same frozen checkpoint could serve 240p, 360p, and 480p streams with only K adapted per stream.
- The improvement in bottom-level entropy efficiency with resolution suggests a testable corollary for generative video models: a discrete tokenizer trained on low-resolution video may transfer to higher-resolution generation more readily than continuous latent tokenizers, since the per-symbol information content does not degrade with grid size.
- The K=256 top-level collapse at 256×256 hints that per-level codebook tuning (e.g., small top codebook, large bottom codebook) could yield further gains; the paper's own data shows mid and bottom levels compensate for top-level failure, so the hierarchy is robust but not optimal.
- Because bitrate is not held fixed in the scaling fit, the β≈0 result could also be read as a rate–distortion statement: at naturally occurring bitrates, resolution adds nothing beyond what K already determines; testing at matched bitrate would separate the 'capacity' explanation from a 'bitrate' explanation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a controlled empirical scaling study of a hierarchical MS-VQ-VAE video codec across four codebook sizes (K = 128, 256, 512, 1024) and three spatial resolutions (64×64, 128×128, 256×256) on UCF101. The central claim is that perceptual quality, measured by LPIPS, depends strongly on codebook capacity K but negligibly on spatial resolution r. The authors fit the log-linear model Q(K,r) = α log₂K + β log₂r + γ to 12 operating points and report α = -0.0094 (p < 0.001), β = -0.0009 (p = 0.68), R² = 0.82. They interpret this as evidence that K is the dominant design variable, that entropy efficiency improves with resolution (84–87% at 64×64 vs 92–94% at 256×256), and that their models outperform H.264/H.265 on LPIPS at matched or lower bitrate. A hierarchy depth-scaling rule is introduced: two latent levels at 64×64 and three levels at 128×128 and 256×256.
Significance. If the central claim holds cleanly, it would be practically significant: hierarchical discrete codecs could be redeployed across resolutions by adjusting K rather than by resolution-specific retraining or RD recalibration, with implications for adaptive streaming and scalable video tokenizers. The paper is transparent in reporting the fitted coefficients, t-statistics, and R², and it uses a fixed training protocol across all 12 models, which is a strength. The empirical setup is also unusually explicit about architecture configuration and codebook utilization. However, the main inference is currently threatened by two confounds — resolution is varied together with hierarchy depth, and K is the rate-control parameter so capacity and bitrate move together — and by at least one apparent data inconsistency in the entropy-efficiency tables. These issues must be addressed before the scaling law can be taken as established.
major comments (3)
- [§3.1, Table 1, Contribution 4] The comparison across resolutions does not isolate spatial resolution. The 64×64 system uses a two-level hierarchy, while 128×128 and 256×256 use three levels (Table 1), with the depth-scaling rule introduced in Contribution 4. Consequently, the fitted β in Eq. (4) conflates resolution with the addition of a mid-level latent grid; the total number of symbols Ntot also changes (4,096 vs 16,384 vs 32,768 bottom symbols). If the extra mid-level absorbs part of the resolution increase, the flat LPIPS across r could be an artifact of the depth-scaling heuristic rather than a property of spatial resolution per se. The Limitations section does not flag this confound. To support the claim, the authors should either hold architecture depth fixed across resolutions (e.g., a three-level 64×64 or a two-level 128×128 variant) or include depth as an explicit covariate in the model.
- [§3.5, Table 2, Eq. (4)] K is explicitly the rate-control parameter, and Table 2 shows that BPP increases with both K and resolution (e.g., K=128: 0.0427, 0.0507, 0.0542 bpp across resolutions). The regression in Eq. (4) therefore attributes to α an effect that may simply be the bitrate effect, since K and BPP are collinear. The interpretation that "codebook capacity" rather than "total bitrate" governs LPIPS is not actually identified. The authors should include BPP as a covariate or otherwise demonstrate that K matters beyond its role as a rate lever; otherwise the headline result is consistent with the near-tautological statement that lower bitrate gives worse quality.
- [§5.2, §5.3, Table 3] The statistical support for β ≈ 0 is weak beyond the confounds: n=12 with only three distinct resolution values gives low power, and the model has no out-of-sample validation. More seriously, Table 3 reports identical bottom-level efficiency values for 64×64 and 128×128 (87.2, 86.7, 85.7, 84.2 for K=128..1024), which is implausible given the 4× difference in number of bottom symbols (4,096 vs 16,384). This suggests a copy/paste error and undermines the entropy-efficiency improvement claim, which is stated as a main contribution. The authors should verify the data and re-run the entropy analysis; if the numbers are correct, they need to explain why bottom-level entropy is exactly equal at these two resolutions.
minor comments (5)
- [Eq. (2) vs Eq. (4)] The symbol β is used both for the VQ commitment weight in Eq. (2) and for the resolution coefficient in Eq. (4). This is confusing; consider renaming one of them (e.g., β_VQ).
- [§4 Hardware] The hardware paragraph mentions "identical ... batch normalisation" but the architecture uses Group Normalisation (§3.1). Please correct the wording.
- [Figure 2] The Greek letters α and β appear to be missing in the caption text of Figure 2 (the equations show "= 0.0094" etc.). Please fix the rendering.
- [§5.2] The fit reports R² = 0.82 but not adjusted R², which is relevant for n=12 with three parameters. Reporting the standard error of the estimate and a leave-one-out or cross-validated R² would strengthen the claim.
- [§5.1] The comparison to H.264 at 64×64 uses only CRF 36 and 32; the claim that the codec outperforms H.264 at matched or lower bitrate is not demonstrated in that resolution range, since the reported H.264 points are at higher bitrate. Please provide a denser H.264/H.265 sweep or qualify the claim.
Circularity Check
No circularity: Eq. (4) is an empirical fit to 12 new operating points; self-citations supply provenance, not proof.
full rationale
I walked the paper's claimed derivation chain. The central quantitative claim is the log-linear model Q(K,r)=αlog2K+βlog2r+γ (Eq. 4), fitted by OLS to 12 measured LPIPS values. No term in this model is defined in terms of LPIPS or of the fitted coefficients; α and β are free parameters and the data could have yielded a significant β. The '10× more influential' statement is a direct re-expression of the fitted estimates, not a prediction of held-out data or a quantity forced by construction. The entropy-efficiency values and bitrate decompositions are independent measurements. Self-citations to the authors' prior MS-VQ-VAE papers are used for architecture provenance and for one 64×64 H.265 baseline, but the 12 operating points are newly trained under a fixed protocol, so the central claim does not reduce to those citations. The unresolved confound between spatial resolution and hierarchy depth (two latent levels at 64×64 vs. three at 128×128/256×256), and the use of K as a rate-control parameter so that BPP changes with both K and r, are threats to the causal interpretation of β; they are a correctness/validity concern, not circularity, because β is not fixed by the definition of r or K. The comparison against H.264/H.265 provides external grounding. No step in the paper's reasoning equates its conclusion to its inputs by construction.
Axiom & Free-Parameter Ledger
free parameters (3)
- α (alpha) =
-0.0094
- β (beta) =
-0.0009
- γ (gamma) =
0.2168
axioms (4)
- ad hoc to paper Log-linear functional form Q(K,r)=αlog2K+βlog2r+γ
- domain assumption LPIPS (VGG) is treated as the primary and sufficient measure of perceptual quality
- domain assumption UCF101 and a fixed 500-clip test set represent the regime of interest
- domain assumption EMA update with dead-code restart yields converged codebooks at K≤512
Cite this review
Pith. "Pith review of Codebook Capacity Governs Perceptual Quality Across Resolutions in Hierarchical Discrete Video Compression." pith.science (2026). https://pith.science/paper/SJNKQU6L
@misc{pith2026260723366,
author = {Pith},
title = {Pith review of: Codebook Capacity Governs Perceptual Quality Across Resolutions in Hierarchical Discrete Video Compression},
year = {2026},
howpublished = {\url{https://pith.science/paper/SJNKQU6L}},
note = {Machine review of arXiv:2607.23366}
}
read the original abstract
Learned video codecs based on continuous latent representations typically require resolution-specific retraining or rate-distortion (RD) recalibration when scaling to new spatial resolutions, because entropy models and Lagrangian weights are tightly coupled to the operating point. We investigate whether hierarchical discrete latent codecs exhibit the same sensitivity. Using a controlled empirical study of MS-VQ-VAE video compression across codebook sizes $K \in \{128,256,512,1024\}$ and resolutions $64\times64$, $128\times128$, and $256\times256$ on UCF101, we show that perceptual quality (LPIPS) depends strongly on codebook capacity but only negligibly on spatial resolution. Fitting a log-linear model $Q(K,r) = \alpha\log_2 K + \beta\log_2 r + \gamma$ to all 12 operating points yields $\alpha=-0.0094$ ($t=-6.6$, $p<0.001$) and $\beta=-0.0009$ ($t=-0.43$, $p=0.68$, not significant), with $R^2=0.82$. Codebook capacity is therefore roughly $10\times$ more influential than spatial resolution per log-unit increase. In parallel, bottom-level entropy efficiency $\eta=H(z)/\log_2 K$ remains stable or improves with resolution (84-87% at $64\times64$; 92-94% at $256\times256$), confirming that larger spatial grids are utilized more efficiently rather than less. Across all resolutions and codebook sizes, our models outperform H.264 on LPIPS at matched or lower bitrate, with gains of 25-52% at $128\times128$ and 21-37% over H.265 at $256\times256$. These findings suggest that codebook size $K$, not spatial resolution, is the dominant design variable governing perceptual compression quality in hierarchical discrete video codecs -- a property that may simplify multi-resolution deployment and inform the design of scalable discrete tokenizers for generative video models.
Figures
Reference graph
Works this paper leans on
-
[1]
Guyon and A
I. Guyon and A. Elisseeff. An Introduction to Variable and Feature Selection. JMLR
-
[2]
Guyon and C
I. Guyon and C. Aliferis and A. Elisseeff , title =
-
[3]
Advances in Neural Information Processing Systems , volume =
van den Oord, Aaron and Vinyals, Oriol and Kavukcuoglu, Koray , title =. Advances in Neural Information Processing Systems , volume =
-
[4]
Advances in Neural Information Processing Systems , volume =
Razavi, Ali and van den Oord, Aaron and Vinyals, Oriol , title =. Advances in Neural Information Processing Systems , volume =
-
[5]
International Conference on Learning Representations , year =
Mentzer, Fabian and Minnen, David and Agustsson, Eirikur and Tschannen, Michael , title =. International Conference on Learning Representations , year =
-
[6]
arXiv preprint arXiv:2308.04746 , year =
Huh, Minyoung and Cheung, Brian and Agrawal, Pulkit and Isola, Phillip , title =. arXiv preprint arXiv:2308.04746 , year =
-
[7]
Variational Image Compression with a Scale Hyperprior , booktitle =
Ball. Variational Image Compression with a Scale Hyperprior , booktitle =
-
[8]
Joint Autoregressive and Hierarchical Priors for Learned Image Compression , booktitle =
Minnen, David and Ball. Joint Autoregressive and Hierarchical Priors for Learned Image Compression , booktitle =
-
[9]
and Tschannen, Michael and Agustsson, Eirikur , title =
Mentzer, Fabian and Toderici, George D. and Tschannen, Michael and Agustsson, Eirikur , title =. Advances in Neural Information Processing Systems , volume =
-
[10]
Nonlinear Transform Coding , journal =
Ball. Nonlinear Transform Coding , journal =
-
[11]
Proceedings of the
Lu, Guo and Zhang, Wanli and Ouyang, Wanli and Chen, Li and Gao, Zhiyong and Xu, Dong , title =. Proceedings of the
-
[12]
Advances in Neural Information Processing Systems , volume =
Li, Jiahao and Li, Bin and Lu, Yan , title =. Advances in Neural Information Processing Systems , volume =
-
[13]
Proceedings of the
Li, Jiahao and Li, Bin and Lu, Yan , title =. Proceedings of the
-
[14]
Scale-Space Flow for End-to-End Optimized Video Compression , booktitle =
Agustsson, Eirikur and Minnen, David and Johnston, Nick and Ball. Scale-Space Flow for End-to-End Optimized Video Compression , booktitle =
-
[15]
and Bjontegaard, Gisle and Luthra, Ajay , title =
Wiegand, Thomas and Sullivan, Gary J. and Bjontegaard, Gisle and Luthra, Ajay , title =
-
[16]
and Ohm, Jens-Rainer and Han, Woo-Jin and Wiegand, Thomas , title =
Sullivan, Gary J. and Ohm, Jens-Rainer and Han, Woo-Jin and Wiegand, Thomas , title =
-
[17]
Proceedings of the International Conference on Machine Learning , year =
van den Oord, Aaron and Kalchbrenner, Nal and Kavukcuoglu, Koray , title =. Proceedings of the International Conference on Machine Learning , year =
-
[18]
Proceedings of the International Conference on Machine Learning , year =
Chen, Xi and Mishra, Nikhil and Rohaninejad, Mostafa and Abbeel, Pieter , title =. Proceedings of the International Conference on Machine Learning , year =
-
[19]
Proceedings of the
Yu, Lijun and Cheng, Yong and Sohn, Kihyuk and Lezama, Jos. Proceedings of the
-
[20]
Language Model Beats Diffusion---Tokenizer Is Key to Visual Generation , booktitle =
Yu, Lijun and Lezama, Jos. Language Model Beats Diffusion---Tokenizer Is Key to Visual Generation , booktitle =
-
[21]
arXiv preprint arXiv:2312.14125 , year =
Kondratyuk, Danila and others , title =. arXiv preprint arXiv:2312.14125 , year =
-
[22]
arXiv preprint arXiv:2104.10157 , year =
Yan, Wilson and Zhang, Yunzhi and Abbeel, Pieter and Srinivas, Aravind , title =. arXiv preprint arXiv:2104.10157 , year =
-
[23]
and Shechtman, Eli and Wang, Oliver , title =
Zhang, Richard and Isola, Phillip and Efros, Alexei A. and Shechtman, Eli and Wang, Oliver , title =. Proceedings of the
-
[24]
and Sheikh, Hamid R
Wang, Zhou and Bovik, Alan C. and Sheikh, Hamid R. and Simoncelli, Eero P. , title =
-
[25]
Proceedings of the European Conference on Computer Vision , year =
Johnson, Justin and Alahi, Alexandre and Fei-Fei, Li , title =. Proceedings of the European Conference on Computer Vision , year =
-
[26]
Soomro, Khurram and Zamir, Amir Roshan and Shah, Mubarak , title =
-
[27]
Proceedings of the
He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian , title =. Proceedings of the
-
[28]
Proceedings of the European Conference on Computer Vision , year =
Wu, Yuxin and He, Kaiming , title =. Proceedings of the European Conference on Computer Vision , year =
-
[29]
and Ba, Jimmy , title =
Kingma, Diederik P. and Ba, Jimmy , title =. International Conference on Learning Representations , year =
-
[30]
arXiv preprint arXiv:1409.1556 , year =
Simonyan, Karen and Zisserman, Andrew , title =. arXiv preprint arXiv:1409.1556 , year =
-
[31]
arXiv preprint arXiv:2001.08361 , year =
Kaplan, Jared and McCandlish, Sam and Henighan, Tom and others , title =. arXiv preprint arXiv:2001.08361 , year =
Pith/arXiv arXiv 2001
-
[32]
and Thomas, Joy A
Cover, Thomas M. and Thomas, Joy A. , title =
-
[33]
Advances in Visual Computing (ISVC 2025) , series =
Kotthapalli, Manikanta and Rekabdar, Banafsheh , title =. Advances in Visual Computing (ISVC 2025) , series =. 2026 , doi =
2025
-
[34]
arXiv preprint arXiv:2607.02562 , year =
Kotthapalli, Manikanta and Rekabdar, Banafsheh , title =. arXiv preprint arXiv:2607.02562 , year =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.