REVIEW 4 major objections 5 minor 1 cited by
Robust Residual Finite Scalar Quantization for Neural Compression
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Adding LayerNorm between residual FSQ stages beats RVQ on DNSMOS
desk verdict The LayerNorm variant that carries the paper cannot be decoded from the bitstream as described; that flaw sinks the main claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Residual magnitude decay is the obstacle: in naive multi-stage FSQ, ∥r_k∥ ≈ α^k∥z∥ with α < 0.3, so later FSQ layers quantize near-zero signals with boundaries designed for unit-scale inputs. The remedy is a per-stage conditioning layer: learnable scaling, q_k = FSQ_k(α_k r_{k−1}) with r_k = r_{k−1} − q_k/α_k, or invertible LayerNorm, which normalizes r_{k−1} before FSQ and applies LayerNorm^{−1} to the quantized value when forming the residual. The LayerNorm version is the one that carries the reported gains, because it holds the input statistics roughly constant across stages.
What would settle it
Run the decoder without any side information: reconstruct from the transmitted indices alone, following equations (6)-(8). If the decoder cannot produce the same r_k as the encoder because LayerNorm's mean and variance are missing, then the claimed perfect reconstruction fails. A second falsifier: on any held-out audio set, compute the stagewise residual magnitudes under naive RFSQ; if the decay exponent α is not well below 0.3, the paper's motivating problem would not apply.
Extended reading notes
Core claim
RFSQ-LayerNorm is the paper's central construction: quantize the first stage normally, then for each later stage normalize the current residual with LayerNorm, apply FSQ to the normalized residual, and undo the normalization when forming the quantized reconstruction. The paper reports that this inverse normalization prevents residual magnitude collapse—in naive residual FSQ the residual magnitudes decay roughly as α^k with α < 0.3—and that with a 4-stage non-uniform bit allocation (8, 6, 5, 5 bits) the reconstructed audio reaches DNSMOS 3.646, beating RVQ's 3.518 at the same 24 bits/frame. On ImageNet at 40 bits, the same LayerNorm conditioning lowers L1 reconstruction error to 0.102 (from 0
Load-bearing premise
The load-bearing assumption is that the decoder can invert each stage's LayerNorm using only the quantization indices; since the decoder never sees the residual's mean and variance, the paper's claim that normalization is invertible holds only if those statistics are somehow available at decode time.
Editorial extensions
If this is right
- If the claimed numbers hold, codebook-free residual scalar quantization is a viable drop-in for RVQ in neural codecs, removing codebook collapse and codebook lookup at equal or better perceptual quality.
- The 4-stage front-heavy bit allocation (8, 6, 5, 5) suggests a concrete recipe: spend more bits early, where residual energy is high, and normalize later stages to keep them useful.
- The accumulated-error argument of O(√K) implies an optimal number of stages; the paper's empirical 4-stage optimum follows from balancing refinement against error accumulation.
- The same LayerNorm conditioning transfers across modalities: audio and image gains come from the same mechanism, so other compression tasks should inherit it if trained with the same loss families.
Reading between the lines
- The invertibility of LayerNorm is only established at the level of arithmetic on the encoder side; a decoder that receives only indices must somehow know the mean and standard deviation of each residual. If those statistics are not derivable from the indices, a real-world decoder would need side information or a learned estimate—an extension the paper does not describe.
- A falsifiable cross-check: measure the residual magnitude decay exponent α on a new dataset before choosing RFSQ. If α is close to 1, naive residual FSQ already works and conditioning buys little; if α < 0.3, the paper's mechanism predicts conditioning gains should track the normalization's ability to keep stagewise input variance constant.
- The improvement over RVQ plausibly depends on the DNSMOS proxy; retraining the same codec with a human-listening test would tell whether the 3.6% gap is perceptually meaningful outside the metric.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Robust Residual Finite Scalar Quantization (RFSQ), a multi-stage extension of Finite Scalar Quantization (FSQ) intended to overcome residual magnitude decay in successive quantization stages. Two conditioning strategies are introduced: learnable per-stage scaling factors and an 'invertible' LayerNorm applied before each FSQ stage. The authors report audio reconstruction results on Emilia (DNSMOS 3.646 for RFSQ-4S-NU-LN vs. 3.518 for RVQ) and image reconstruction results on ImageNet (0.102 L1 and 0.100 LPIPS at 40 bits), claiming consistent gains from LayerNorm conditioning. The paper also asserts exponential residual decay and an O(√K) total error growth, but provides no derivation or empirical verification for either claim. The central concern is that the LayerNorm decoder is not implementable from the information available at decode time, and the algorithm's accumulator q_total is inconsistent with the inverse recursions.
Significance. If the proposed method were sound and reproducible, it would be a practically useful contribution: it retains FSQ's codebook-free simplicity while adding multi-stage refinement, and it reports gains over a strong RVQ baseline in audio and improved image metrics. The paper provides a useful ablation across conditioning strategies and stage counts. However, the headline method (LayerNorm) is not specified in a way that a decoder could execute, and the reconstruction path in Algorithm 1 is internally inconsistent. The theoretical motivation is asserted rather than demonstrated. As a result, the reported numbers cannot currently be verified or taken as evidence for the central claims. The paper ships no code or proofs in the submission, so the strengths usually expected for such a contribution (reproducibility, machine-checked derivations) are absent.
major comments (4)
- [§3.3, Eq. (8), Algorithm 1] The LayerNorm inverse is not computable at decode time. Eq. (8) defines r_k = r_{k-1} - LayerNorm^{-1}(q_k), which requires the mean and variance of r_{k-1}. The decoder receives only quantization indices and fixed FSQ levels; it does not receive z or any per-stage residual statistics. Unless side information for μ and σ is transmitted (which would consume bits and is not described), the inverse cannot be evaluated. The phrase 'invertible layer normalization' does not resolve this, since standard LayerNorm is input-dependent. This is a load-bearing flaw: the reported DNSMOS and LPIPS results for RFSQ-LN hinge on an unspecified, non-implementable decoder.
- [Algorithm 1, line 17] The accumulator q_total += q_k is inconsistent with the inverse recursions in Eq. (5) and Eq. (8). For the Scale strategy, the correct reconstruction from indices is q_1 + Σ_{k≥2} q_k/α_k, not Σ q_k. For LayerNorm, it would be q_1 + Σ_{k≥2} (q_k σ_{k-1} + μ_{k-1}), which is unavailable. If q_total is intended as the decoder input, then the paper never defines the actual decoded signal; if q_total is not the decoder input, the decoder-side signal formation is missing. Either way, the experimental pipeline cannot be reconstructed from the manuscript.
- [§3.2, §4.1] The paper asserts that residual magnitudes decay exponentially (∥r_k∥ ≈ α^k ∥z∥, α < 0.3) and that the total error grows as O(√K). These statements are central to motivating RFSQ and explaining why 4 stages are optimal, but no derivation, measurement, or plot is provided. Please supply a proof or an empirical measurement (e.g., residual norms by stage) so the claim is testable. As written, these are unsupported assertions.
- [§4.2, Table 2] The image experiments compare only RFSQ variants; there is no external baseline such as single-stage FSQ, RVQ, or a standard learned codec at the same bit rates. The stated improvement 'over unconditioned variants' is an internal ablation, not a demonstration of state-of-the-art performance or generalizability. Without external baselines and error bars, the ImageNet results do not support the paper's broader claims. Add comparable baselines and statistical significance information.
minor comments (5)
- [Abstract/Introduction] Typo: 'VQ-V AE' should be 'VQ-VAE'.
- [Eq. (1)] Please specify the assumed input range for z_i (e.g., [-1,1]) and how values outside that range are clipped. The rounding formula is otherwise ambiguous.
- [Table 2] The configurations 'RFSQ-2×2048' and 'RFSQ-4×1024' do not state the FSQ level counts per dimension or how the bit totals 22.0 and 40.0 are derived. Please clarify for reproducibility.
- [Algorithm 1] Please add a separate decoder algorithm. The current algorithm returns q_total and indices, but the decoder's reconstruction procedure is never specified. This is partly covered by the major comments, but a clear pseudocode summary would help.
- [Figures] Figures 1 and 2 are referenced in the text but appear only as captions in the submitted text. Ensure the final PDF includes the actual plots with readable labels.
Circularity Check
No circular derivation: RFSQ's gains are empirical comparisons against independent baselines; the LayerNorm decoder underspecification is a reproducibility gap, not circularity.
full rationale
The paper's central claims are experimental: RFSQ-LayerNorm achieves DNSMOS 3.646 vs RVQ 3.518 at 24 bits/frame, and LayerNorm conditioning improves ImageNet L1/LPIPS. These are benchmark comparisons, not consequences of a fitted parameter disguised as a prediction. The residual magnitude decay model (∥r_k∥≈α^k∥z∥, α<0.3, §3.2) is presented and used only as an empirical motivation; it is not an output claimed to be derived from first principles. The two conditioning strategies (Eqs. 4-8) are algorithmic proposals, and the reported gains are measured rather than deduced from the equations. There is no self-citation chain: references [1]-[16] are to external prior work (FSQ, RVQ, EnCodec, datasets, metrics), and no uniqueness theorem or ansatz is imported from the authors' own earlier papers. The paper does, however, contain two non-circular internal flaws worth flagging. First, §3.3 Eq. (8) defines r_k = r_{k-1} - LayerNorm^{-1}(q_k) and states LayerNorm is 'invertible' with 'perfect reconstruction', but the decoder receives only quantization indices and fixed FSQ levels; it cannot compute the mean and variance of r_{k-1} needed to evaluate LayerNorm^{-1}(q_k), and the paper gives no running-statistics or side-information mechanism. Second, Algorithm 1 line 17 accumulates q_total += q_k for all stages, whereas for Scale and LayerNorm stages the contribution to the reconstruction is q_k/α_k and LayerNorm^{-1}(q_k), respectively; if q_total is the decoded signal, reconstruction is incorrect, and if it is not, the actual decoder-side signal formation is never specified. These are reproducibility/correctness gaps, not circular derivations: no equation is equivalent to its input by construction, and no fitted quantity is relabeled as a prediction. Hence the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- learnable scaling factors α_k =
initialized to 1.0, learned during training
assumptions (3)
- domain assumption LayerNorm is invertible and the decoder can obtain the normalization statistics of each residual.
- domain assumption Residual magnitudes decay exponentially as ∥r_k∥ ≈ α^k ∥z∥ with α < 0.3.
- domain assumption Total quantization error grows as O(√K) for K stages.
Cite this review
Pith. "Pith review of Robust Residual Finite Scalar Quantization for Neural Compression." pith.science (2026). https://pith.science/paper/2JGWRGTJ
@misc{pith2026250815860,
author = {Pith},
title = {Pith review of: Robust Residual Finite Scalar Quantization for Neural Compression},
year = {2026},
howpublished = {\url{https://pith.science/paper/2JGWRGTJ}},
note = {Machine review of arXiv:2508.15860}
}
read the original abstract
Finite Scalar Quantization (FSQ) offers simplified training but suffers from residual magnitude decay in multi-stage settings, where subsequent stages receive exponentially weaker signals. We propose Robust Residual Finite Scalar Quantization (RFSQ), addressing this fundamental limitation through two novel conditioning strategies: learnable scaling factors and invertible layer normalization. Our experiments across audio and image modalities demonstrate RFSQ's effectiveness and generalizability. In audio reconstruction at 24 bits/frame, RFSQ-LayerNorm achieves 3.646 DNSMOS, a 3.6% improvement over state-of-the-art RVQ (3.518). On ImageNet, RFSQ achieves 0.102 L1 loss and 0.100 perceptual loss, with LayerNorm providing 9.7% L1 improvement and 17.4% perceptual improvement over unconditioned variants. The LayerNorm strategy consistently outperforms alternatives by maintaining normalized input statistics across stages, effectively preventing exponential magnitude decay that limits naive residual approaches. RFSQ combines FSQ's simplicity with multi-stage quantization's representational power, establishing a new standard for neural compression across diverse modalities.
Forward citations
Cited by 1 Pith paper
-
AffectCodec: Emotion-Preserving Neural Speech Codec with Block-Diagonal Residual FSQ
AffectCodec applies block-diagonal projections in residual FSQ to explicitly allocate bits to emotion and acoustic subspaces, combined with emotion conditioning, yielding better emotion preservation at low bitrates wi...
Reference graph
Works this paper leans on
-
[1]
INTRODUCTION Vector quantization has been a cornerstone of neural com- pression since the foundational work of Gray [1], establishing the theoretical framework for discrete representation learn- ing in neural networks. The introduction of VQ-V AE [2] marked a paradigm shift by revolutionizing discrete repre- sentation learning, enabling end-to-end trainin...
-
[2]
We identify and analyze the residual magnitude decay problem in naive residual FSQ implementations, pro- viding theoretical and empirical evidence of its impact
-
[3]
We propose two novel conditioning strategies—learnable scaling and invertible LayerNorm—that robustly ad- dress this problem while maintaining FSQ’s simplicity
-
[4]
We conduct comprehensive experiments on both audio and image reconstruction tasks, demonstrating signifi- cant improvements over strong baselines
-
[5]
We establish RFSQ as a general framework applicable to various architectures and modalities, with consistent performance gains
-
[6]
RELATED WORK Vector quantization [2, 3] established discrete representation learning but suffers from codebook collapse and training in- stability [9]. FSQ [4] eliminates codebook learning by quan- tizing dimensions independently to predefined levels, show- ing success in image generation [10] and speech synthesis [5, 6]. Residual VQ [11, 12] applies hier...
arXiv 2025
-
[7]
METHOD 3.1. Background: Finite Scalar Quantization FSQ quantizes ad-dimensional vectorz∈R d by indepen- dently quantizing each dimension to a finite set of levels. For dimensioniwithL i levels: FSQi(zi) =round zi ·(L i −1) 2 · 2 Li −1 (1) The total codebook size is Qd i=1 Li, with code ratePd i=1 log2(Li)bits per token. 3.2. Naive Residual FSQ and Its Pro...
-
[8]
Code is available at https://github.com/zhuxiaoxuhit/robust_ rfsq
EXPERIMENTS We evaluate RFSQ on two challenging modalities: large-scale audio reconstruction and image compression, demonstrating its effectiveness and generalizability. Code is available at https://github.com/zhuxiaoxuhit/robust_ rfsq. 4.1. Audio Reconstruction Experiments We evaluate RFSQ on a large-scale audio reconstruction task using the Emilia datas...
Show all 26 references
-
[9]
Our experiments demonstrate RFSQ’s Fig
CONCLUSION We propose Robust Residual Finite Scalar Quantization (RFSQ), a novel framework addressing the fundamental residual magnitude decay problem through intelligent con- ditioning strategies. Our experiments demonstrate RFSQ’s Fig. 2: Visual quality comparison. From top:...
-
[10]
The image reconstruc- tion experiments were inspired by duchenzhuang’s FSQ- pytorch project
ACKNOWLEDGEMENTS The audio encoder/decoder architectures are based on the SEANet design from EnCodec [12]. The image reconstruc- tion experiments were inspired by duchenzhuang’s FSQ- pytorch project. We also acknowledge the EnCodec codebase for providing reference implementati...
-
[11]
Vector quantization,
Robert M Gray, “Vector quantization,”IEEE ASSP Magazine, vol. 1, no. 2, pp. 4–29, 1984
1984
-
[12]
Neural discrete representation learning,
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu, “Neural discrete representation learning,” in31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 2017
2017
-
[13]
Generating diverse high-fidelity images with vq-vae- 2,
Ali Razavi, Aaron van den Oord, and Oriol Vinyals, “Generating diverse high-fidelity images with vq-vae- 2,” in33rd Conference on Neural Information Process- ing Systems (NeurIPS 2019), Vancouver, Canada, 2019
2019
-
[14]
Finite scalar quantization: Vq-vae made simple,
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen, “Finite scalar quantization: Vq-vae made simple,”arXiv preprint arXiv:2309.15505, 2023
2023 arXiv
-
[15]
Cosyvoice 2: Scalable streaming speech synthesis with large language mod- els,
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhi- jie Yan, and Jingren Zhou, “Cosyvoice 2: Scalable streaming speech synthes...
2024 arXiv
-
[16]
Indextts: An industrial-level controllable and efficient zero-shot text-to-speech system,
Wei Deng, Siyi Zhou, Jingchen Shu, Jinchao Wang, and Lu Wang, “Indextts: An industrial-level controllable and efficient zero-shot text-to-speech system,”arXiv preprint arXiv:2502.05512, 2025
2025 arXiv
-
[17]
Codec-asr: Training automatic speech recognition on discrete speech representations,
Kunal Dhawan, Nithin Rao Koluguri, Ante Juki ´c, Ryan Langman, Jagadeesh Balam, and Boris Ginsburg, “Codec-asr: Training automatic speech recognition on discrete speech representations,” inProc. Interspeech, 2024
2024
-
[18]
Scaling transformers for low-bitrate high-quality speech cod- ing,
Julian D Parker, Anton Smirnov, Jordi Pons, CJ Carr, Zack Zukowski, Zach Evans, and Xubo Liu, “Scaling transformers for low-bitrate high-quality speech cod- ing,”arXiv preprint arXiv:2411.19842, 2024
2024 arXiv
-
[19]
Robust training of vector quantized bottleneck mod- els,
Adrian Ła ´ncucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, Hans J.G.A. Dolfing, Sameer Khurana, Tanel Alum ¨ae, and Antoine Laurent, “Robust training of vector quantized bottleneck mod- els,” inInternational Joint Conference on Neural Net- works (IJCNN), 2020
2020
-
[20]
Maskgit: Masked generative im- age transformer,
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman, “Maskgit: Masked generative im- age transformer,”Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2022
2022
-
[21]
Soundstream: An end-to-end neural audio codec,
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi, “Soundstream: An end-to-end neural audio codec,”IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, 2021
2021
-
[22]
High fidelity neural audio compression,
Alexandre D ´efossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438, 2022
2022 arXiv
-
[23]
Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,
Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu, “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,”...
2024
-
[24]
An open source im- plementation of itu-t recommendation p.808 with vali- dation,
Babak Naderi and Ross Cutler, “An open source im- plementation of itu-t recommendation p.808 with vali- dation,” inProc. Interspeech, 2020, pp. 2862–2866
2020
-
[25]
Imagenet: A large-scale hierarchical image database,
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,”Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009
2009
-
[26]
The unreasonable effec- tiveness of deep features as a perceptual metric,
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang, “The unreasonable effec- tiveness of deep features as a perceptual metric,”Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018
2018
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.