REVIEW 4 major objections 6 minor 36 references
ComplexDec: A Domain-robust High-fidelity Neural Audio Codec with Complex Spectrum Modeling
T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read ComplexDec claims that coding speech as complex spectra without downsampling preserves naturalness on expressive out-of-domain speech at 24 kbps, even when trained on only 30 hours of reading-style data.
desk verdict Useful empirical codec paper with an over-claimed causal story; the OOD robustness is real, but the information-loss mechanism is not isolated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the complex-spectrum RVQAE with no temporal or dimensional compression. Two residual-vector-quantizer autoencoders, one for the real part and one for the imaginary part of the STFT, share the same encoder and decoder but use independent codebooks; the 150 Hz frame rate and 256-dimensional embedding allow the model to keep the 24 kbps bitrate (16 10-bit codebooks, 8 per part) with only a 1.25 compression ratio, versus 5 for AudioDec. A score-based post-filter trained with score matching and reverse-SDE sampling, together with amplitude companding, refines the decoded spectra. The no-downsampling design is what lets the codec carry the long-term harmonic structure that the paper identifies as missing when out-of-domain speech is coded by downsampled baselines.
What would settle it
Train ComplexDec with the same complex-spectrum inputs and losses but insert a single downsampling and upsampling stage to raise the compression ratio to about 5; if out-of-domain MOS remains near 3.9 instead of falling toward 1.5, the paper's attribution of robustness to low compression ratio is falsified.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that easing information loss in the codec bottleneck transfers to out-of-domain robustness without extra training data. ComplexDec encodes and decodes the real and imaginary components of 48 kHz STFT spectra using two residual-vector-quantizer autoencoders that share an encoder and decoder but keep separate codebooks; with 16 10-bit codebooks (8 per component) at 150 Hz, the bitrate is still 24 kbps but the compression ratio is only 1.25, compared with 5 for AudioDec. In listening tests, overall MOS for ComplexDec is 3.93 in-domain and 3.99 out-of-domain, whereas AudioDec falls from 3.06 to 1.49 and ScoreDec from 4.28 to 2.46; the same pattern appears in waveform and perceptual objective metrics. The paper takes this, plus the harmonic structure visible in spectrograms and the fact that whisper, which lacks harmonics, degrades least, as evidence that lost long-term harmonic information, not data scale, is the root of neural codec domain failure.
Load-bearing premise
The load-bearing premise is that the out-of-domain gains come from easing information loss by avoiding temporal and dimensional compression, but ComplexDec also changes the input representation, architecture, and training losses at the same time, so that causal link is not isolated.
Editorial extensions
If this is right
- A codec trained on just 30 hours of reading-style speech can keep natural quality on expressive, loud, and whispered speech from a different recording environment, provided it avoids temporal and dimensional compression.
- Compression ratio, not just bitrate, is the quantity that predicts out-of-domain robustness: at the same 24 kbps, the lower-ratio codec degrades far less.
- A score-based post-filter can sharpen coded spectra but cannot restore harmonics the encoder already lost, so encoder information loss is the bottleneck.
- Low-dimensional 256-dimensional complex-spectrum tokens are easier for regression-based generative models to consume than the 1024-dimensional codes used by some low-ratio baselines.
- Open-source codecs trained on far more data still show large out-of-domain degradation, which implies that simply scaling training data does not solve the robustness problem.
Reading between the lines
- A cleaner causal test would vary only the compression ratio, for example by changing the code dimension at fixed 150 Hz frame rate, while keeping the complex-spectrum representation, architecture, and losses fixed; the paper does not run that ablation.
- The same design rationale predicts that robustness problems will be milder for signals without stable harmonic structure, consistent with the paper's observation that whisper coding degrades least.
- Because the perceptual gap between ComplexDec and DAC appears only at full bandwidth, out-of-domain codec evaluation should include 48 kHz listening tests rather than relying on downsampled objective metrics.
- If the information-loss mechanism is right, one-step or distilled post-filter inference could make ComplexDec streamable without losing its robustness advantage, since the SPF is the only slow non-causal component.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ComplexDec, a full-band 48 kHz neural audio codec that operates in the complex STFT domain without temporal downsampling, using 16 10-bit codebooks (8 for the real and 8 for the imaginary part) at a 24 kbps bitrate together with a score-based post-filter. The model is trained on the 30-hour VCTK clean subset and evaluated in-domain and out-of-domain on expressive EARS speech, with AudioDec, ScoreDec, Encodec, and DAC as baselines. The authors report that ComplexDec maintains similar objective and subjective quality in- and out-of-domain whereas the waveform-based baselines degrade markedly, and they attribute this robustness to reduced information loss from temporal and dimensional compression, quantified by a low compression ratio of 1.25 versus 5 for AudioDec.
Significance. If the empirical robustness results hold, the paper makes a useful contribution: it demonstrates that a codec trained on only 30 hours of reading-style speech can generalize to expressive speech, it provides what appears to be the first dedicated out-of-domain robustness evaluation of neural codecs, and it shows that a low-dimensional (256-dim) discrete representation is feasible for full-band coding, which is favorable for regression-based generative models. The whisper-condition observation (AudioDec/ScoreDec degrade less on unvoiced speech) offers a falsifiable nuance consistent with the harmonic-structure account. However, as detailed below, the causal attribution to compression-ratio information loss is not established by the reported comparisons, and several statistical details of the evaluations need strengthening. The contribution is therefore more secure as an empirical demonstration of a robust codec design than as a demonstration of the proposed mechanism.
major comments (4)
- [§II.A-II.B, Table I] The compression ratio for ComplexDec is computed as 48000/(150×256)=1.25, but ComplexDec codes both the real and imaginary parts of the 256-bin complex spectrum as two separate 256-dimensional streams (Sec. II.B, II.C). Under the definition given in Sec. II.A, the total latent dimensionality is 512, giving 48000/(150×512)=0.625 rather than 1.25. This discrepancy affects the headline quantitative comparison with AudioDec's ratio of 5 and with DAC's 0.3125. The authors should state explicitly whether H denotes complex coefficients or real values and recompute the ratios consistently; the current figure of 1.25 is not internally consistent with the architecture.
- [§II.B, §III.B, §IV] The central causal claim that low information loss (as proxied by compression ratio) causes out-of-domain robustness is not isolated by the experiments. ComplexDec differs from AudioDec and ScoreDec in multiple dimensions simultaneously: input representation (complex STFT versus waveform), encoder/decoder topology (no downsampling/upsampling), quantizer structure (8+8 complex codebooks versus 16 waveform codebooks), training losses (complex MSE/MAE plus multi-resolution mel versus single-resolution mel), and the presence of the score-based post-filter (shared only with ScoreDec). DAC differs further in code dimension, training data scale (thousands of hours versus 30 hours), and output bandwidth (24 kHz versus 48 kHz). Table I and the Discussion interpret the observed ordering as 'high correlation between information loss and domain robustness,' but no experiment varies the compression ratio while holding the other factors fixed. The robust OOD behavior could equally be caused by the complex spectral representation preserving harmonic structure (Fig. 2), the absence of temporal downsampling per se, the higher code capacity, or the SPF. I recommend either softening the causal language to 'consistent with' or adding ablations that isolate the compression ratio, e.g., training a waveform-domain ComplexDec variant with the same no-downsampling architecture and 256-dimensional codes, or varying the hop length/number of codebooks within ComplexDec while keeping representation and losses constant.
- [§III.B, Table I] Objective metrics in Table I are reported as point estimates without error bars or significance tests. Claims that ComplexDec 'achieves similar in-domain and out-of-domain performances' and that baselines 'suffer serious degradation' rely on single numbers, and the metrics are computed after downsampling to 24 kHz (waveform metrics) and 16 kHz (STOI/PESQ), so full-band artifacts are not captured. The authors acknowledge the lack of full-band objective measures, but the downsampled numbers are still used to support the robustness claim. I ask for per-utterance standard deviations, bootstrap confidence intervals, or pairwise significance tests for at least the key comparisons in Table I.
- [§III.C, Table II] The MOS evaluation uses only 10 utterances per system and 12 raters, and 95% confidence intervals are reported only for the Overall row. The text uses 'significantly' without any inferential statistics (e.g., paired tests or CI-based comparisons with multiple-comparison correction). For example, in the out-of-domain column, ComplexDec's Overall MOS of 3.99±0.14 and DAC's 3.70±0.16 have nearly overlapping CIs, and in-domain/out-of-domain differences for ComplexDec are not tested. Please provide per-style CIs, per-listener variance, and a justified significance analysis, or restrict the claims to descriptive trends.
minor comments (6)
- [Abstract] Abstract: 'AuidoDec' should be 'AudioDec'.
- [Table II title] Table II title: 'CONFIDENCE INTERNALS' should be 'CONFIDENCE INTERVALS'.
- [Sec. III.D] Sec. III.D: 'ComplexDex' should be 'ComplexDec' in the RTF sentence.
- [Sec. III.B] Sec. III.B: 'The results demonstrate the high correlation between information loss and domain robustness' implies a quantitative correlation that is never computed; consider replacing with 'are consistent with' or reporting a correlation measure.
- [Sec. II.B] Sec. II.B: The phrase 'low spectral dimension setting (e.g. 256-dim)' should clarify that this is the number of complex frequency bins and that real and imaginary streams together constitute 512 real dimensions, to avoid the confusion noted in Major Comment 1.
- [Sec. III.A] Sec. III.A: 'The total length of the training data' should be 'The total duration of the training data' for grammatical precision.
Circularity Check
No significant circularity: the paper's robustness claim is empirical and independently measured, not derived from its compression-ratio definition by construction.
full rationale
The paper's central claim is that information loss from temporal and dimensional compression degrades out-of-domain robustness, and that ComplexDec mitigates this by coding in the complex spectral domain. This is presented as an architectural argument and then tested with objective and subjective evaluations on an held-out expressive corpus, EARS, after training only on VCTK. The OOD metrics (SI-SDR, STOI, PESQ, MOS) are measured independently of the compression-ratio definition; no equation in the paper makes OOD quality a function of the compression ratio. The compression-ratio calculations in Section II.A are arithmetic identities, but the robustness results in Table I and Table II are empirical outcomes, not consequences of those definitions. Self-citations to AudioDec and ScoreDec appear as baseline systems and as the source of the score-based post-filter component, but they are not invoked as proof of the OOD-robustness claim, and the baselines are externally available codecs whose behavior is measured rather than assumed. The concern that compression ratio is not varied independently of representation, architecture, and training losses is a legitimate experimental-design weakness about causal attribution, but it is not circularity under the requested definition: the paper does not define OOD robustness in terms of compression ratio, nor does it fit a parameter and then rename that fit as a prediction. Therefore no circular step can be exhibited, and the appropriate score is 0.
Assumptions & free parameters
free parameters (4)
- Amplitude companding constant alpha =
0.5
- Amplitude scaling constant beta =
0.15
- Loss weights for RVQAE training =
L_VQ=1.0, L_mel=45, L_MSE=200, L_MAE=200
- Spectral analysis parameters =
hop=320, STFT size=510, Hann window
assumptions (4)
- standard math STFT with the chosen parameters provides an invertible and information-preserving representation of 48 kHz audio.
- domain assumption The score-based post-filter (SPF) trained with score matching improves decoded complex spectra without changing the information-loss characteristics of the codec.
- ad hoc to paper Compression ratio (temporal/dimensional compression) is a valid proxy for information loss.
- domain assumption VCTK reading-style speech and EARS expressive speech represent sufficiently distinct domains to test out-of-domain robustness.
Cite this review
Pith. "Pith review of ComplexDec: A Domain-robust High-fidelity Neural Audio Codec with Complex Spectrum Modeling." pith.science (2026). https://pith.science/paper/NDJS75ZI
@misc{pith2026250202019,
author = {Pith},
title = {Pith review of: ComplexDec: A Domain-robust High-fidelity Neural Audio Codec with Complex Spectrum Modeling},
year = {2026},
howpublished = {\url{https://pith.science/paper/NDJS75ZI}},
note = {Machine review of arXiv:2502.02019}
}
read the original abstract
Neural audio codecs have been widely adopted in audio-generative tasks because their compact and discrete representations are suitable for both large-language-model-style and regression-based generative models. However, most neural codecs struggle to model out-of-domain audio, resulting in error propagations to downstream generative tasks. In this paper, we first argue that information loss from codec compression degrades out-of-domain robustness. Then, we propose full-band 48~kHz ComplexDec with complex spectral input and output to ease the information loss while adopting the same 24~kbps bitrate as the baseline AuidoDec and ScoreDec. Objective and subjective evaluations demonstrate the out-of-domain robustness of ComplexDec trained using only the 30-hour VCTK corpus.
Figures
Reference graph
Works this paper leans on
-
[1]
Linear predictive coding systems,
T. Tremain, “Linear predictive coding systems,” in Proc. ICASSP, 1976, vol. 1, pp. 474–478
work page 1976
-
[2]
Regular-pulse excitation–a novel approach to effective and efficient multipulse coding of speech,
P. Kroon, E. Deprettere, and R. Sluyter, “Regular-pulse excitation–a novel approach to effective and efficient multipulse coding of speech,” IEEE/ACM TASLP, vol. 34, no. 5, pp. 1054–1063, 1986
work page 1986
-
[3]
ISO/MPEG-1 audio: A generic standard for coding of high-quality digital audio,
K. Brandenburg and G. Stoll, “ISO/MPEG-1 audio: A generic standard for coding of high-quality digital audio,” AES, vol. 42, no. 10, pp. 780–792, 1994
work page 1994
-
[4]
A toll quality 8 kb/s speech codec for the personal communications system (pcs),
R. Salami et al., “A toll quality 8 kb/s speech codec for the personal communications system (pcs),” IEEE TVT, vol. 43, no. 3, pp. 808–816, 1994
work page 1994
-
[5]
MPEG-4 ALS: An emerging standard for lossless audio coding,
Tilman Liebchen and Yuriy A Reznik, “MPEG-4 ALS: An emerging standard for lossless audio coding,” in Proc. DCC, 2004, pp. 439–448
work page 2004
-
[6]
Coalson, Free Lossless Audio Codec , Accessed: 2000
J. Coalson, Free Lossless Audio Codec , Accessed: 2000
work page 2000
-
[7]
High-quality, low-delay music coding in the opus codec,
J.-M. Valin, G. Maxwell, T. B. Terriberry, and K. V os, “High-quality, low-delay music coding in the opus codec,” in AESC 135 , 2013
work page 2013
-
[8]
The adaptive multirate wideband speech codec (AMR-WB),
B. Bessette et al., “The adaptive multirate wideband speech codec (AMR-WB),” IEEE TSAP , vol. 10, no. 8, pp. 620–636, 2002
work page 2002
Show all 36 references
-
[9]
Overview of the EVS codec architecture,
M. Dietz et al., “Overview of the EVS codec architecture,” in Proc. ICASSP, 2015, pp. 5698–5702
2015
-
[10]
SoundStream: An end-to-end neural audio codec,
N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “SoundStream: An end-to-end neural audio codec,” IEEE/ACM TASLP, vol. 30, pp. 495–507, 2021
2021
-
[11]
High fidelity neural audio compression,
A. D ´efossez et al., “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438, 2022
2022 arXiv
-
[12]
AudioDec: An open-source streaming high-fidelity neural audio codec,
Y .-C. Wu, I. D. Gebru, D. Markovi ´c, and A. Richard, “AudioDec: An open-source streaming high-fidelity neural audio codec,” in Proc. ICASSP, 2023
2023
-
[13]
A review of vector quantization techniques,
A. Vasuki and P.T. Vanathi, “A review of vector quantization techniques,” IEEE Potentials, vol. 25, no. 4, pp. 39–47, 2006
2006
-
[14]
On Generative Spoken Language Modeling from Raw Audio,
Kushal Lakhotia et al., “On Generative Spoken Language Modeling from Raw Audio,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 1336–1354, 12 2021
2021
-
[15]
Audiolm: a language modeling approach to audio generation,
Zal ´an Borsos et al., “Audiolm: a language modeling approach to audio generation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
2023
-
[16]
Neural codec language models are zero-shot text to speech synthesizers,
Chengyi Wang et al., “Neural codec language models are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111 , 2023
2023 arXiv
-
[17]
Simple and controllable music generation,
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre D´efossez, “Simple and controllable music generation,” Proc. NeurIPS, vol. 36, 2024
2024
-
[18]
Score- Dec: A phase-preserving high-fidelity audio codec with a generalized score-based diffusion post-filter,
Y .-C. Wu, D. Markovi´c, S. Krenn, I. D. Gebru, and A. Richard, “Score- Dec: A phase-preserving high-fidelity audio codec with a generalized score-based diffusion post-filter,” in Proc. ICASSP, 2024
2024
-
[19]
CSTR VCTK corpus: En- glish multi-speaker corpus for CSTR voice cloning toolkit,
C. Veaux, J. Yamagishi, and K. MacDonald, “CSTR VCTK corpus: En- glish multi-speaker corpus for CSTR voice cloning toolkit,” University of Edinburgh. CSTR , 2017
2017
-
[20]
EARS: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation,
J. Richter, Y .-C. Wu, S. Krenn, S. Welker, B. Lay, S. Watanabe, A. Richard, and T. Gerkmann, “EARS: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation,” in Interspeech, 2024
2024
-
[21]
Apcodec: A neural audio codec with parallel amplitude and phase spectrum encoding and decoding,
Yang Ai, Xiao-Hang Jiang, Ye-Xin Lu, Hui-Peng Du, and Zhen-Hua Ling, “Apcodec: A neural audio codec with parallel amplitude and phase spectrum encoding and decoding,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 32, pp. 3256–3269, 2024
2024
-
[22]
Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec,
Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng, “Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 591–595
2024
-
[23]
High-fidelity audio compression with improved rvqgan,
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar, “High-fidelity audio compression with improved rvqgan,” Proc. NeurIPS, vol. 36, 2024
2024
-
[24]
Neural discrete representation learning,
A. Van Den Oord et al., “Neural discrete representation learning,” NIPS, vol. 30, 2017
2017
-
[25]
Estimation of non-normalized statistical models by score matching.,
A. Hyv ¨arinen and P. Dayan, “Estimation of non-normalized statistical models by score matching.,” Journal of Machine Learning Research , vol. 6, no. 4, 2005
2005
-
[26]
Speech enhancement with score-based generative models in the complex STFT domain,
S. Welker, J. Richter, and T. Gerkmann, “Speech enhancement with score-based generative models in the complex STFT domain,” in Proc. Interspeech, 2022, pp. 2928–2932
2022
-
[27]
Speech enhancement and dereverberation with diffusion-based genera- tive models,
J. Richter, S. Welker, J.-M. Lemercier, B. Lay, and T. Gerkmann, “Speech enhancement and dereverberation with diffusion-based genera- tive models,” IEEE/ACM TASLP, vol. 31, pp. 2351–2364, 2023
2023
-
[28]
On the theory of the brownian motion,
G. E. Uhlenbeck and L. S. Ornstein, “On the theory of the brownian motion,” Physical review, vol. 36, no. 5, pp. 823, 1930
1930
-
[29]
Reverse-time diffusion equation models,
B. D. Anderson, “Reverse-time diffusion equation models,” Stochastic Processes and their Applications , vol. 12, no. 3, pp. 313–326, 1982
1982
-
[30]
Score-based generative modeling through stochastic differential equations,
Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” in Proc. ICLR, 2021
2021
-
[31]
Attention is all you need,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
-
[32]
Noisy speech database for training speech enhancement algorithms and TTS models,
C. Valentini-Botinhao, “Noisy speech database for training speech enhancement algorithms and TTS models,” University of Edinburgh. CSTR, 2017
2017
-
[33]
Adam: A method for stochastic optimization,
Diederik P Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014
2014 arXiv
-
[34]
Sdr–half-baked or well done?,
J. L. Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “Sdr–half-baked or well done?,” in Proc. ICASSP, 2019, pp. 626–630
2019
-
[35]
An algorithm for intelligibility prediction of time–frequency weighted noisy speech,
C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An algorithm for intelligibility prediction of time–frequency weighted noisy speech,” IEEE/ACM TASLP, vol. 19, no. 7, pp. 2125–2136, 2011
2011
-
[36]
Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,
A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” in Proc. ICASSP, 2001, vol. 2, pp. 749–752
2001
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.