REVIEW 3 major objections 7 minor 29 references
Efficient Evaluation of Quantization-Effects in Neural Codecs
T0 review · 3 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read A modified straight-through estimator that reconnects quantization noise to the encoder's gradient stabilizes neural-codec training and outperforms the standard estimator on the descript-audio-codec.
desk verdict mSTE is a genuinely new twist on straight-through estimation that stabilizes training without commitment loss, but the paper overclaims DAC improvement based only on training curves. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the modified straight-through estimator (mSTE), which changes only the backward pass of a quantizer: the quantization error $Q_e$ is multiplied by $\sigma_{Q_e}/\mathrm{sg}[\sigma_{Q_e}]$, where $\sigma_{Q_e}$ is the standard deviation of $Q_e$, so the forward pass still feeds the decoder the true quantized embedding $E_q$. The added term connects the noise level to the encoder output through $\partial \sigma_{Q_e}/\partial E$, mirroring the noise-approximation update of Equation (15) and preventing the encoder from escaping to large $E$. The supporting machinery is the proposed evaluation framework: a 30-dimensional Gaussian source quantized to 2 bits per value, rotated by an orthogonal matrix to form the input, with the pre-rotation quantized signal as the target, and a three-layer MLP encoder/decoder trained with MSE loss. This setup reproduces the divergence of STE and stability of noise-approximation training of full codecs while taking under an hour and less than 400 MB of GPU memory.
What would settle it
Train the descript-audio-codec with standard STE and no commitment loss for more than the 3,000 steps reported, and measure the encoder norm $\|E\|_1$ and reconstruction quality; if $E$ settles at a finite value, or if mSTE does not yield lower distortion in a listening or objective test, the paper's central instability claim and mSTE advantage would be falsified.
Extended reading notes
Core claim
The central claim is that the straight-through estimator, used to backpropagate through a nondifferentiable quantizer, is unstable without a commitment loss because the quantization error $Q_e = E_q - E$ is stopped from contributing to the encoder gradient. The encoder therefore receives a gradient that rewards increasing the norm of $E$ to maximize the embedding-to-noise ratio, and since a larger $E$ also produces a larger $Q_e$, the process diverges. The paper's proposed modified straight-through estimator computes $D^{\text{mSTE}}_{in} = E + \mathrm{sg}[Q_e]\cdot \sigma_{Q_e}/\mathrm{sg}[\sigma_{Q_e}]$, which leaves the forward pass unchanged but gives the backward pass a normalized, graph-connected noise term, so the encoder update sees $\partial D^{\text{mSTE}}_{in}/\partial E = 1 + \mathrm{sg}[Q_e/\sigma_{Q_e}]\,\partial \sigma_{Q_e}/\partial E$. In the synthetic setup and in the descript-audio-codec, this stabilizes the encoder embedding norm, removes the need for commitment loss, and yields lower training loss than standard STE over training.
Load-bearing premise
The load-bearing premise is that the 30-dimensional, 2-bit Gaussian testbed with a three-layer MLP reproduces the quantizer-related gradient dynamics of full-size codecs, so that mSTE's stability and STE's divergence observed there transfer to real audio codecs and to final reconstruction quality.
Editorial extensions
If this is right
- Neural codecs can be trained with the straight-through estimator without a commitment loss when the mSTE backward pass is used, avoiding the trivial $E=0$ solution that commitment loss can encourage.
- Using mSTE gives the descript-audio-codec a stable encoder norm and a lower Mel-loss than standard STE, with the gap growing the longer training runs.
- The synthetic evaluation setup reproduces qualitative training behaviors of full codecs, so quantizer and gradient-estimator choices can be screened in under an hour instead of about a week on a GPU.
- The instability of STE without commitment loss is explained by the quantization noise being detached from the computational graph, a mechanism confirmed by the parallel behavior of detached noise approximation in both the synthetic model and an internal audio codec.
Reading between the lines
- The mSTE mechanism is generic to discrete latent-variable models, so it may also stabilize VQ-VAEs or image codecs where commitment loss is used; the paper only validates on audio codecs, so this is an extrapolation.
- Because the synthetic framework isolates the quantizer from data complexity, it could be used to benchmark other gradient estimators, such as ReinMax or SPIGOT, against each other before committing to a full codec training run; the paper lists these as future work rather than testing them.
- The step-function loss observed with mSTE plus commitment loss suggests that at low bitrates the commitment loss pinches embeddings toward quantization levels and slows fine-grained adjustment; a direct comparison of final reconstruction quality would test whether mSTE without commitment loss is strictly preferable.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a low-cost evaluation framework for neural codecs, using synthetic Gaussian data quantized to a fixed number of bits and a small fully connected encoder-quantizer-decoder, and uses it to compare straight-through estimation (STE) with noise approximation (NA). Based on observed encoder-output growth under STE without commitment loss (CL), the authors propose a modified STE (mSTE) in Eq. (10), which multiplies the stopped quantization error by the estimated standard deviation of that error and stops the standard deviation in the forward pass. The paper reports that mSTE stabilizes the encoder output and lowers MSE in the synthetic framework without CL, and that these patterns transfer to an internal speech codec and to descript-audio-codec (DAC), where the Mel-loss training curves for DAC with STE plus CL and with mSTE converge, while DAC with plain STE diverges. The conclusion states that mSTE outperforms STE when training DAC or the proposed framework.
Significance. The mSTE modification is simple, the derivation in Eqs. (10)-(11) is algebraically consistent with the forward pass, and the synthetic framework is genuinely cheap (Table 1), making it a useful vehicle for quantizer studies. The use of an open-source codec (DAC) for validation is a strength. If supported by final-quality measurements, the claim that mSTE stabilizes STE without a commitment loss would be practically valuable for neural codec training. However, the current DAC evidence consists only of training Mel-loss and MA-E curves from single runs, so the paper overstates what is established; the core behavioral observations are credible, but the 'outperforms' claim needs stronger evidence.
major comments (3)
- [§5.2, Fig. 8] The only DAC evidence for the paper's central claim is the training Mel-loss and MA-E over 3,000 steps, with no final checkpoints, no objective reconstruction quality metrics, and no listening tests. Mel-loss is an auxiliary spectral loss and MA-E is an encoder-magnitude diagnostic; neither establishes that the decoded audio is better, and a lower Mel-loss can coincide with over-smoothed output. Please report final-checkpoint comparisons of DAC_CL_STE, DAC_CL_mSTE, and DACmSTE on held-out audio using at least one standard metric such as ViSQOL, PESQ, or SI-SNR, or soften the conclusion to 'mSTE prevents training divergence'.
- [§5.1 and §5.2, Figs. 3-8] All reported curves are single runs, with no seeds, error bars, or checkpoint-selection procedure. Because the improvement claim in Fig. 8 rests on differences between two noisy training curves that both converge, the key comparisons should be repeated with at least three seeds and the spread reported.
- [§5.2, Fig. 7] The internal-codec validation is under-specified: no architecture, dataset, or reconstruction loss is reported, and only MA-E over training is shown, so the consistency claim is based on a single qualitative pattern. Please add these details and at least one reconstruction-quality curve, or explicitly label the internal-codec result as anecdotal support.
minor comments (7)
- [§3.1, Eq. (6)] The rotation matrix Q reuses the symbol of the quantizer module Q from Fig. 1; rename one of them (e.g., R) for clarity.
- [§3.1, Fig. 2] It would help to state explicitly that Y is the network input and Xq is the target in both the equation and the figure; currently the target definition appears only in the following sentence.
- [§4, Eq. (12)] The embedding-to-noise ratio is defined only as 20 log10(α σ_E) ∈ [0,8] dB; please give the exact signal and noise power definitions used.
- [§5.1, Fig. 6 caption] The sentence 'the blue and the red curve overlap in the MA-E' is ambiguous because the figure has four curves; say which conditions are overlapping.
- [§5.2, Fig. 8] The Mel-loss definition is referenced but not described; a one-sentence definition would help readers who are not DAC specialists.
- [§5.1, Fig. 3] The phrase 'both an input frame and the bits of the quantizer are the same (60 Bits)' mixes bits-per-frame and bits-per-value; rephrase to compare 60 bits/frame to 60 bits/frame or 2 bits/value to 2 bits/value.
- [Reproducibility] Consider releasing code and checkpoints for the synthetic framework and the DAC modifications, since the paper is otherwise silent on code availability.
Circularity Check
No significant circularity: the mSTE modification is an independent algorithmic proposal, and the DAC validation is external and not fitted to the framework's outcomes.
full rationale
The derivation chain is self-contained. The proposed mSTE (Eq. 10) is a new gradient estimator defined from first principles, with no free parameters fitted to DAC results. The paper's synthetic framework is used to generate hypotheses about STE instability, which are then tested independently on an external codec (DAC) and an internal codec. The DAC experiments do not fit any parameter from the framework; they simply compare training curves for STE versus mSTE. The central claim that mSTE stabilizes training is supported by the DAC training curves, and while these curves lack final reconstruction-quality or perceptual metrics, that is an evidence-completeness issue, not circularity. No load-bearing self-citations are present, and no result is assumed by construction. The paper also explicitly attributes the instability observation to prior external work (Van Den Oord et al., 2017), and the mSTE connection to the reparameterization trick is acknowledged as a similarity, not presented as a new fundamental theorem. Therefore, the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- Data bit depth and SQ levels =
2 bits per value, levels {-1.5, -0.5, 0.5, 1.5}
- Framework dimensions =
P=30, N=2000
- Embedding-to-noise ratio =
0 to 8 dB (alpha range)
- Commitment loss weight =
0.1
- Training updates =
100 epochs, 2000 updates each
assumptions (4)
- domain assumption The straight-through estimator provides a valid non-zero gradient approximation for quantization, i.e., dD_in/dE = 1 in the backward pass.
- standard math An orthogonal rotation preserves the information content of the quantized input, so Y and Xq carry the same number of bits.
- ad hoc to paper The proposed mSTE's stabilization is due to connecting the quantization noise to the computational graph through sigma_Qe, as hypothesized.
- domain assumption The low-complexity surrogate model emulates the non-linear behavior of larger codecs sufficiently for the studied phenomena.
Cite this review
Pith. "Pith review of Efficient Evaluation of Quantization-Effects in Neural Codecs." pith.science (2026). https://pith.science/paper/QYRWVGCE
@misc{pith2026250204770,
author = {Pith},
title = {Pith review of: Efficient Evaluation of Quantization-Effects in Neural Codecs},
year = {2026},
howpublished = {\url{https://pith.science/paper/QYRWVGCE}},
note = {Machine review of arXiv:2502.04770}
}
read the original abstract
Neural codecs, comprising an encoder, quantizer, and decoder, enable signal transmission at exceptionally low bitrates. Training these systems requires techniques like the straight-through estimator, soft-to-hard annealing, or statistical quantizer emulation to allow a non-zero gradient across the quantizer. Evaluating the effect of quantization in neural codecs, like the influence of gradient passing techniques on the whole system, is often costly and time-consuming due to training demands and the lack of affordable and reliable metrics. This paper proposes an efficient evaluation framework for neural codecs using simulated data with a defined number of bits and low-complexity neural encoders/decoders to emulate the non-linear behavior in larger networks. Our system is highly efficient in terms of training time and computational and hardware requirements, allowing us to uncover distinct behaviors in neural codecs. We propose a modification to stabilize training with the straight-through estimator based on our findings. We validate our findings against an internal neural audio codec and against the state-of-the-art descript-audio-codec.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Soft-to-hard vector quantization for end-to-end learning compress- ible representations
Timofte, R., Benini, L., and V an Gool, L. Soft-to-hard vector quantization for end-to-end learning compress- ible representations. Adv. Neural Inf. Process. Syst. , pp. 1142–1152, 2017
work page 2017
-
[2]
Generative adversarial networks for extreme learned image compression
Agustsson, E., Tschannen, M., Mentzer, F., Timofte, R., and V an Gool, L. Generative adversarial networks for extreme learned image compression. In Proc. IEEE Int. Conf. Comput. Vis., pp. 221–231, 2019
work page 2019
-
[3]
Ai, Y ., Jiang, X. H., Lu, Y . X., Du, H. P ., and Ling, Z. H. APCodec: A neural audio codec with parallel amplitude and phase spectrum encoding and decoding. In IEEE/ACM Trans. Audio Speech Lang. Process. , vol- ume 32, pp. 3256–3269, 2024. Ball´ e, J., Laparra, V ., and Simoncelli, E. P . End-to-end optimized image compression. In Int. Conf. Learn. Rep-...
work page 2024
-
[4]
Agustsson, E., Hwang, S. J., and Toderici, G. Nonlinear transform coding. IEEE J. Sel. T op. Signal Process., 15 (2):339–353, 2021
work page 2021
-
[5]
Estimat- ing or propagating gradients through stochastic neu- rons for conditional computation
Bengio, Y ., L´ eonard, N., and Courville, A. Estimat- ing or propagating gradients through stochastic neu- rons for conditional computation. In arXiv Prepr . arXiv1308.3432, 2013
arXiv 2013
-
[6]
Bhaskaran, V . and Konstantinides, K. Image and Video Compression Standards. Springer, Boston, MA, 1995
work page 1995
-
[7]
Biswas, A. and Jia, D. Audio codec enhancement with generative adversarial networks. In ICASSP , IEEE Int. Conf. Acoust. Speech Signal Process. - Proc. , pp. 356– 360, 2020
work page 2020
-
[8]
Brandenburg, K., Stoll, G., Dehery, Y . F., Johnston, J. D., v.d. Kerkhof, L., and Schroder, E. F. ISO-MPEG-1 Au- dio: A generic standard for coding of high-quality digital audio. AES J. Audio Eng. Soc. , 42(10):780–792, 1994
work page 1994
Show all 29 references
-
[9]
Neural speech coding for real-time communications using constant bitrate scalar quantiza- tion
Brendel, A., Pia, N., Gupta, K., Behringer, L., Fuchs, G., and Multrus, M. Neural speech coding for real-time communications using constant bitrate scalar quantiza- tion. IEEE J. Sel. T op. Signal Process., pp. 1–15, 2024. B¨ uthe, J., Mustafa, A., V alin, J.-M., Helwani, K., and
2024
-
[10]
Goodwin, M. M. NOLACE: Improving low-complexity speech codec enhancement through adaptive temporal shaping. In IEEE Int. Conf. Acoust. Speech Signal Pro- cess., pp. 476–480, 2024. D´ efossez, A., Copet, J., Synnaeve, G., and Adi, Y . High fidelity neural audio compression. Tran...
2024
-
[11]
V ., Li, Y ., Lim, F
Garbacea, C., Den Oord, A. V ., Li, Y ., Lim, F. S., Luebs, A., Vinyals, O., and Walters, T. C. Low bit-rate speech coding with VQ-V AE and a wavenet decoder. InICASSP , IEEE Int. Conf. Acoust. Speech Signal Process. - Proc. , pp. 735–739, 2019
2019
-
[12]
Categorical reparameteri- zation with gumbel-softmax
Jang, E., Gu, S., and Poole, B. Categorical reparameteri- zation with gumbel-softmax. In Int. Conf. Learn. Repre- sent. ICLR, 2017
2017
-
[13]
Kingma, D. P . and Welling, M. Auto-encoding variational bayes. In Int. Conf. Learn. Represent. ICLR , 2014
2014
-
[14]
B., Lim, F
Kleijn, W . B., Lim, F. S., Luebs, A., Skoglund, J., Stimberg, F., Wang, Q., and Walters, T. C. Wavenet based low rate speech coding. In ICASSP , IEEE Int. Conf. Acoust. Speech Signal Process. - Proc. , pp. 676–680, 2018
2018
-
[15]
High-quality speech coding with sample RNN
Klejsa, J., Hedelin, P ., Zhou, C., Fejgin, R., and Villemoes , L. High-quality speech coding with sample RNN. In ICASSP , IEEE Int. Conf. Acoust. Speech Signal Process. - Proc., pp. 7155–7159, 2019
2019
-
[16]
Enhancement of coded speech using a mask-based post-filter
Korse, S., Gupta, K., and Fuchs, G. Enhancement of coded speech using a mask-based post-filter. In ICASSP , IEEE Int. Conf. Acoust. Speech Signal Process. - Proc. , pp. 6764–6768, 2020
2020
-
[17]
PostGAN: A GAN-based post-processor to enhance the quality of coded speech
Korse, S., Pia, N., Gupta, K., and Fuchs, G. PostGAN: A GAN-based post-processor to enhance the quality of coded speech. In ICASSP , IEEE Int. Conf. Acoust. Speech Signal Process. - Proc. , pp. 4153–4157, 2022
2022
-
[18]
Bridging dis- crete and backpropagation: Straight-through and beyond
Liu, L., Dong, C., Liu, X., Y u, B., and Gao, J. Bridging dis- crete and backpropagation: Straight-through and beyond. Adv. Neural Inf. Process. Syst., 36, 2023
2023
-
[19]
J., Mnih, A., and Teh, Y
Maddison, C. J., Mnih, A., and Teh, Y . W . The concrete distribution: A continuous relaxation of discrete random variables. In Int. Conf. Learn. Represent. ICLR , 2017
2017
-
[20]
Finite scalar quantization: VQ-V AE made simple
Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: VQ-V AE made simple. In Int. Conf. Learn. Represent. ICLR, 2024. 8 Efficient Evaluation of Quantization-Effects in Neural Cod ecs
2024
-
[21]
Speech quality evaluation of neural audio codecs
Muller, T., Ragot, S., Gros, L., Philippe, P ., and Scalart, P . Speech quality evaluation of neural audio codecs. In Interspeech, pp. 1760–1764, 2024
2024
-
[22]
A streamwise GAN vocoder for wideband speech coding at very low bit rate
Pia, N. A streamwise GAN vocoder for wideband speech coding at very low bit rate. In IEEE W ork. Appl. Signal Process. to Audio Acoust. , pp. 66–70, 2021. O’Shaughnessy, D. Review of methods for coding of speech signals. Eurasip J. Audio, Speech, Music Pro- cess., (1):8, 2023
2021
-
[23]
Peng, H., Thomson, S., and Smith, N. A. Backpropagat- ing through structured argmax using a SPIGOT. In ACL- Annu. Meet. Assoc. Comput. Linguist. Proc. Conf. (Long Pap., volume 1, pp. 1863–1873. Association for Compu- tational Linguistics (ACL), 2018. V ali, M. H. and Backstrom...
2018
-
[24]
(eds.), Proc
Kacic, Z. (eds.), Proc. Annu. Conf. Int. Speech Commun. Assoc. INTERSPEECH, pp. 3406–3410, 2019. V alin, J. M., Maxwell, G., Terriberry, T. B., and V os, K. High-quality, low-delay music coding in the opus codec. In Audio Eng. Soc. Conv., pp. 73–82. Audio Engineering Society, ...
2019
-
[25]
Wallace, G. K. The JPEG still picture compression stan- dard. Commun. ACM, 34(4):30–44, 1991
1991
-
[26]
J., Bjøntegaard, G., and Luthra, A
Wiegand, T., Sullivan, G. J., Bjøntegaard, G., and Luthra, A. Overview of the H.264/A VC video coding standard. IEEE Trans. Circuits Syst. Video T echnol. , 13(7):560– 576, 2003
2003
-
[27]
SoundStream: An End-to-End Neu- ral Audio Codec
Tagliasacchi, M. SoundStream: An End-to-End Neu- ral Audio Codec. IEEE/ACM Trans. Audio Speech Lang. Process., 30:495–507, 2022
2022
-
[28]
J., Jia, Y ., Chen, Z., and Wu, Y
Zen, H., Dang, V ., Clark, R., Zhang, Y ., Weiss, R. J., Jia, Y ., Chen, Z., and Wu, Y . LibriTTS: A corpus derived from LibriSpeech for text-to-speech. In Interspeech, pp. 1526–1530, 2019
2019
-
[29]
Convolutional neural networks to enhance coded speech
Zhao, Z., Liu, H., and Fingscheidt, T. Convolutional neural networks to enhance coded speech. IEEE/ACM Trans. Audio Speech Lang. Process., 27(4):663–678, 2019. 9
2019
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.