{"id":"66d489de-5662-4f3d-a3b8-7901c696f8d2","arxiv_id":"2505.07235","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MUFFIN is a neural audio codec that quantizes separate latent frequency bands with dedicated codebooks, claiming SOTA reconstruction quality and a competitive 12.5 Hz ultra-low-rate variant.","lead":"MUFFIN is a neural audio codec that compresses sound by splitting the compressed signal into three frequency bands and assigning a dedicated codebook to each, informed by how human hearing treats different bands. It reports higher reconstructed audio quality than existing codecs at equal or lower bitrates, including a 12.5-token-per-second mode that still stays competitive.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MBS-RVQ's specified 37.5-75 Hz band exceeds the Nyquist frequency of the 75 Hz-sampled latent (24 kHz/320), so the described multi-band mechanism cannot operate as stated.","rationale":"The reader's weakest assumption concerns unablated architectural differences between MUFFIN and the retrained HiFi-Codec baseline. That concern is reasonable, but the ablations in Appendix C (Tables 7 and 8) do directly remove the proposed MBS-RVQ and the modified snake activation within the MUFFIN architecture, so the components' contributions are at least internally tested. The more serious problem is that the MBS-RVQ mechanism as described appears physically impossible: a real latent sampled at 75 Hz cannot contain spectral content above 37.5 Hz, yet the method specifies a 37.5-75 Hz band. This is not a matter of tuning or ablation; it is a mathematical inconsistency in the central construction. If the implementation follows the text, Codebook 3 would be empty or a mirror of lower bands, contradicting the speaker-identity results. If the implementation deviates from the text, the paper's methodology section is misleading and the experiments cannot be reproduced as written. I also note the supporting Lipschitz proof in Appendix B is inaccurate: the derivative of the modified snake is 1 + beta*sin(2*alpha*x), so the Lipschitz constant is at most 1+|beta|, not |beta| as claimed. This does not break the qualitative Lipschitz continuity, but it indicates the theoretical justification is not carefully derived. The 'first NPC' claim also appears overstated given the cited prior work on psychoacoustic neural codecs (e.g., Zhen et al. 2020; Petermann et al. 2023). However, the Nyquist violation is the load-bearing issue: it directly undermines the described mechanism that the empirical results are attributed to. The paper's empirical comparisons may still be valid if the code implements a corrected partition, so a conditional verdict is appropriate pending inspection of the code. I therefore recommend keeping the reader's CONDITIONAL verdict, and did not find grounds to move to accept or reject without the proposed check.","tokens_in":22067,"tokens_out":7723,"duration_ms":72449,"concrete_test":"Inspect the released MUFFIN code (github.com/dianwen-ng/MUFFIN) to determine the exact FFT implementation and band masks in MBS-RVQ. Feed a synthetic latent whose temporal modulation is a 50 Hz sine (above the 37.5 Hz Nyquist for a 75 Hz frame rate) and check whether any energy is assigned to Codebook 3. If the documented masks are used, the 50 Hz component must alias to 25 Hz or be zeroed; a nonzero output in the '37.5-75 Hz' band would prove the implementation uses a different sample rate or band definition than the paper states.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states that the latent representation is produced by compressing 24 kHz audio by a factor of 320, which Table G confirms yields a 75 Hz frame rate. The paper then defines three non-overlapping bands: 0-18.75 Hz, 18.75-37.5 Hz, and 37.5-75 Hz, with scale factors 4:2:1. For a real signal sampled at 75 Hz, the Nyquist limit is 37.5 Hz; the DFT bins above Nyquist are the conjugate mirror of the lower bins and carry no independent information. A '37.5-75 Hz' band therefore either is empty or duplicates the lower frequencies, so Codebook 3 cannot encode the distinct high-frequency cues that Section 4.2 and Appendix E attribute to it. If the released code instead implements a different partition, such as a 150 Hz latent rate or a complex-valued decomposition, then the paper's specification is inaccurate and the experiments are not reproducible from the text. Either way, the central claim that psychoacoustically guided multi-band splitting drives the performance gains is not supported by the described method. This is more fundamental than the ablation gap: even if Table 7 shows MBS-RVQ outperforming vanilla RVQ, the mechanism being ablated is not the one described.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MUFFIN, a fully convolutional neural audio codec. Its central proposal is Multi-Band Spectral Residual Vector Quantization (MBS-RVQ), which applies an FFT-based frequency-band decomposition to the encoder's latent representation before quantization, using hand-selected band boundaries and 4:2:1 scale factors. The paper also proposes a modified snake activation with amplitude and bias terms, a transformer-inspired convolutional backbone, and reports reconstruction results on LibriTTS, IEMOCAP, GTZAN, and BBC, plus a zero-shot TTS evaluation with VALL-E. High-compression variants operating at 25 Hz and 12.5 Hz frame rates are presented, with the 12.5 Hz version claimed as state-of-the-art.","tokens_in":22364,"tokens_out":8527,"duration_ms":77701,"significance":"If the empirical results are reproducible, MUFFIN would be a competitive neural codec with an interesting factorized token representation for downstream generative tasks. The retrained HiFi-Codec baseline, the evaluation across multiple domains, and the downstream TTS experiment are notable strengths. However, the central mechanistic contribution as written is physically incoherent: the described three-band split on a 75 Hz real-valued latent places one band above the Nyquist frequency. This undercuts the reproducibility of the method and the interpretation of the codebook analyses. The theoretical support in Appendix B also contains a concrete error. The empirical findings may still hold, but the manuscript's description of the method needs major correction.","major_comments":[{"comment":"The proposed three-band split of a 75 Hz latent into 0–18.75 Hz, 18.75–37.5 Hz, and 37.5–75 Hz is not realizable for a real-valued signal sampled at 75 Hz. The Nyquist limit is 37.5 Hz, so the third band is either empty or duplicates the lower bands in conjugate form. Consequently, Codebook 3 as described cannot encode the high-frequency/speaker cues that Section 4.2 and Appendix E attribute to it. Please clarify whether the latent representation is complex-valued, whether the code uses a different effective latent rate, or how the 37.5–75 Hz band is populated; as written, the method is physically impossible and not reproducible from the text.","section":"Section 3.1 and Appendix E"},{"comment":"The derivative of the modified snake activation is f'(x)=1+β sin(2αx), but the text writes |f'(x)|=|β sin(2αx)|≤|β|, which is algebraically incorrect. The correct bound is |f'(x)|≤1+|β|, so the activation is (1+|β|)-Lipschitz, not |β|-Lipschitz. In addition, the claim that convolutional and linear layers 'ensure' Lipschitz continuity is vacuous without bounds on the weight matrices; please correct the calculation and either provide norm control for the encoder or soften the claim.","section":"Appendix B"},{"comment":"The theorem is attributed to Cover (1999), but the stated inequality is not a standard theorem from that reference and no proof is supplied. Since the result is invoked to justify the bitrate allocation in MBS-RVQ, please provide a self-contained proof with precise definitions of H(B_k|x(t)) and Δ(B_k,x(t)), or recast the statement as a heuristic motivated by psychoacoustics rather than a theorem with external attribution.","section":"Section 3.2, Theorem 3.1"},{"comment":"The ablation of MBS-RVQ versus vanilla RVQ isolates that module, but the main paper claims that psychoacoustic multi-band coding is responsible for the gains. The comparison with the retrained HiFi-Codec baseline also includes unreported differences: ConvNeXt inverted bottlenecks, grouped convolutions, the 64-bin mel loss, and the number of codebooks. Without ablating these factors, the central attribution to MBS-RVQ is not established; please discuss this limitation and, if feasible, add at least a loss-function and architecture ablation.","section":"Appendix C, Tables 7 and 8"}],"minor_comments":[{"comment":"WER values above 100 (e.g., 154, 191) are unusual; please clarify whether these are percentages, raw edit-distance rates, or device-independent scores, and define the maximum possible value for the Whisper-based evaluation.","section":"Section 4.2, Table 5"},{"comment":"The symbols '▽' and '▲' are not explained in the captions; please add a note that they denote the 25 Hz and 12.5 Hz variants, respectively.","section":"Tables 1-4"},{"comment":"The figure reproduces panels from French & Steinberg (1947); please specify whether the underlying book/copyright terms permit redistribution, or use original plots.","section":"Appendix A, Figure 3"},{"comment":"The column 'Encoding Rate' in the hyperparameter table is not defined; I assume it is the list of strides, but it differs from the 'Downsampling Rate' column. Please clarify.","section":"Section G"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The band-splitting error is not a subtle issue; it is a fundamental physical impossibility in the described algorithm. I recommend requesting the authors provide the exact FFT/band-splitting code or a precise description of the latent representation and its spectral decomposition before considering acceptance. The empirical results may still be valid if the code differs from the text, but the current manuscript is not reproducible as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nHere's my read on MUFFIN (arXiv:2505.07235). The empirical package is the strongest part. They retrained HiFi-Codec under matched conditions, evaluated on speech, music, environmental sounds, plus a zero-shot TTS downstream, and released code and demos. The MBS-RVQ idea—dedicated codebooks for separate frequency bands in the latent—is genuinely new next to SoundStream, Encodec, HiFi-Codec, and DAC. The 12.5 Hz variant results are worth attention. I give them credit for that.\n\nBut there is a load-bearing problem. Section 3.1 says the latent runs at 75 Hz (24 kHz / 320) and defines three bands: 0–18.75, 18.75–37.5, and 37.5–75 Hz. For a real signal sampled at 75 Hz, Nyquist is 37.5 Hz. The third band is empty or a mirror of the lower spectrum. That cannot be what the code does. Appendix E later says the splits follow a logarithmic scale relative to the latent sampling rate, which contradicts the specific numbers in 3.1. Either the text is wrong or the implementation is different. As written, the paper is not reproducible from its own description, and the claim that codebook 3 captures speaker identity from a 37.5–75 Hz band is unsupported.\n\nThe other issues are more standard. Theorem 3.1 is attributed to Cover but isn't in Cover and is unproved. The Lipschitz proof in Appendix B drops the constant 1 in the derivative, so the claimed constant is wrong. The ablations toggle MBS-RVQ and snake, but not the ConvNeXt blocks, grouped convolutions, or the 64-bin mel loss, so the attribution of gains to the proposed mechanism is not isolated. And the 'first NPC' label doesn't survive contact with Zhen et al. and other psychoacoustic neural codecs.\n\nIf I were the editor, I'd send this to review rather than desk-reject, because the empirical content is substantial and a referee can check the released code to see what the band split actually is. But the revision would be major: state the FFT axis precisely, align the text with the code's actual band boundaries, fix or remove the theorem, and correct the Lipschitz constant. As it stands, the central mechanism is incoherent as written.\n\nFor your own work, the comparison tables are useful, but I'd hold off citing until the band-definition issue is resolved. I'd maybe bring it to a reading group—the Nyquist slip is a good teaching moment about writing up spectral methods.\n\nBest.","headline":"MUFFIN has a credible empirical package and a genuinely new multi-band RVQ idea, but the central band-splitting specification is physically impossible as written (37.5–75 Hz on a 75 Hz latent) and needs correction before the mechanism can be taken seriously.","tokens_in":22901,"tokens_out":7405,"would_cite":false,"duration_ms":68430,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MUFFIN claims psychoacoustic band quantization beats full-band neural codecs and reaches 12.5 Hz tokens.","keywords":["neural audio coding","psychoacoustic compression","multi-band spectral residual vector quantization","residual vector quantization","speech codec","zero-shot text-to-speech","snake activation","audio tokenization"],"falsifier":"Retrain the same MUFFIN architecture with vanilla RVQ while holding every non-quantizer component fixed, then also add each non-quantizer change one at a time to the retrained HiFi-Codec baseline; if any single change closes the LibriTTS test-clean gap in PESQ or UTMOS, the paper's central mechanism is not what drives the improvement.","tokens_in":21910,"feed_emoji":"🎧","tokens_out":7674,"duration_ms":69403,"temperature":0.7,"pith_summary":"MUFFIN is a fully convolutional neural codec whose central idea is to perform quantization not on the full latent spectrum at once, but on three frequency bands selected from psychoacoustic knowledge. The paper claims this aligns compression effort with what the ear actually resolves: low bands carry intelligibility, mid bands carry articulation, and high bands carry speaker identity and timbre. On LibriTTS, IEMOCAP, GTZAN, and BBC benchmarks, it reports better reconstruction scores than Encodec, DAC, and a retrained HiFi-Codec at comparable or lower bitrates. A highly compressed variant runs at 12.5 Hz (100 tokens per second, 0.9 kB/s) with perceived quality close to the full-rate model, and MUFFIN tokens improve zero-shot TTS quality in a VALL-E-based system. The paper thus argues that psychoacoustic band structure is a usable inductive bias for neural audio coding, not just a legacy-codec trick.","feed_headline":"Neural audio codec hits 12.5 Hz tokens with top quality","feed_subtitle":"MUFFIN splits latent frequencies by perception to beat HiFi-Codec on speech, music, and zero-shot TTS.","key_machinery":"The load-bearing object is Multi-Band Spectral Residual Vector Quantization (MBS-RVQ), a quantizer that operates in the latent space of a convolutional autoencoder: the latent $z$ is transformed with FFT, filtered into three frequency bands (0–18.75, 18.75–37.5, 37.5–75 Hz) with scale factors 4, 2, and 1, converted back with inverse FFT, and each band is quantized with its own EMA codebook; quantization residuals feed the next band, so each codebook refines what the previous one missed. The second component is the modified snake activation $f(x) = x + \\frac{\\beta}{\\alpha}\\sin^2(\\alpha x) + \\gamma$, whose derivative is bounded by $|\\beta|$, giving Lipschitz stability and, the paper argues, preserving psychoacoustic distances in latent space. The paper also invokes a perceptual-entropy bound $E_p \\ge \\sum_k H(B_k \\mid x(t)) - \\sum_k \\Delta(B_k, x(t))$ to argue that multi-band modeling lowers the minimal bit rate needed for transparent coding.","core_discovery":"The central claim is that a neural audio codec can improve both compression efficiency and perceived fidelity by splitting the quantizer's latent representation into psychoacoustic frequency bands and allocating separate codebooks to each band with a sequence of residual refinements. This is realized by MBS-RVQ: an FFT of the latent, band-pass filtering into 0–18.75, 18.75–37.5, and 37.5–75 Hz, inverse FFT, then sequential quantization with scale factors 4, 2, and 1, where each stage quantizes the residual left by the previous stage. The architectural counterpart is the modified snake activation $f(x) = x + \\frac{\\beta}{\\alpha}\\sin^2(\\alpha x) + \\gamma$, claimed to preserve fine spectral detail while staying Lipschitz continuous. The paper reports that the resulting codebooks spontaneously separate speech content from speaker identity, with the high-band codebook forming well-separated speaker clusters in t-SNE, and that a 1920× downsampled variant reaches a 12.5 Hz token rate with UTMOS scores at or above the 75 Hz model. It also claims these tokens give state-of-the-art zero-shot TTS results when plugged into a VALL-E system.","pith_inferences":["The band-split design is tuned to speech; the paper's own music experiments show instrumental content does not separate cleanly across codebooks 2 and 3. Re-tuning the three-way split to a music-specific perceptual scale, such as ERB bands matched to instrument partials, is a direct testable extension the paper leaves open.","The codebook analysis suggests a label-free route to factorized speech representations, but the paper only shows speaker clustering, not verification accuracy. Computing a speaker equal-error rate from Codebook 3 embeddings alone would quantify how close this comes to supervised factorization.","The VALL-E difficulty at 12.5 Hz hints that ultra-low token rates change the LM's prompt-length balance. A systematic prompt-length sweep for 12.5 Hz tokens is a concrete next experiment, and it is an inference from the paper's reported failure, not a claim it makes.","If MBS-RVQ is the effective mechanism, the first codebook should carry less semantic load than in vanilla RVQ; the WER table supports this and predicts that swapping band ratios, for example 2:1:1 instead of 4:2:1, should visibly shift where intelligibility lives."],"forward_implications":["At 3.0 kB/s-class rates, MUFFIN reports lower STFT and MEL distances and higher PESQ, STOI, UTMOS, and ViSQOL than Encodec, DAC, and a retrained HiFi-Codec on speech, emotional speech, music, and general audio.","The 12.5 Hz variant delivers 100 tokens per second at 0.9 kB/s with UTMOS at or above the default 75 Hz model, and it outperforms Mimi on naturalness and reconstruction fidelity.","MUFFIN tokens improve zero-shot TTS: VALL-E with MUFFIN achieves a WER of 12.20 versus 21.05 with Encodec and 32.35 with HiFi-Codec, with higher MOS and speaker similarity scores.","Codebook 3 (37.5–75 Hz) captures speaker identity without supervision, while codebooks 1 and 2 carry intelligibility and articulation; using only codebooks 1 and 2 yields 19.2 WER on LibriSpeech."],"supporting_citations":[{"why":"Supplies the HiFi-Codec architecture and the retrained baseline that MUFFIN must outperform in reconstruction comparisons.","marker":"(Yang et al., 2023)"},{"why":"Supplies the SoundStream residual vector quantization and multi-scale STFT adversarial training that the quantizer design builds on.","marker":"(Zeghidour et al., 2022)"},{"why":"Provides the EnCodec baseline and the EMA codebook with successive residual refinement used by MBS-RVQ.","marker":"(Défossez et al., 2022)"},{"why":"Supplies the improved RVQGAN and snake-activation inspiration for periodic modeling and high-frequency preservation.","marker":"(Kumar et al., 2024)"},{"why":"Provides the HiFi-GAN multi-receptive-field fusion, multi-period and multi-scale discriminators, and feature-matching losses used in training.","marker":"(Kong et al., 2020)"},{"why":"Supplies VALL-E, the zero-shot TTS framework used to test whether MUFFIN tokens improve downstream generative quality.","marker":"(Wang et al., 2023)"}],"fun_headline_variants":["Audio codec reaches 12.5 Hz token rate with band-wise refinement","MUFFIN codec uses perceptual bands to shrink audio to 12.5 Hz tokens","Psychoacoustic frequency-split codec hits 12.5 Hz with high fidelity","Neural audio codec: multi-band quantization for 12.5 Hz tokens","Perceptual band split enables neural codec to hit 12.5 Hz rate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison rests on the assumption that the reported gains come from the psychoacoustic multi-band quantizer and the modified snake activation, yet the ablation evidence removes those components without isolating the other architectural and training differences from HiFi-Codec, such as the ConvNeXt-style blocks, grouped convolutions, 64-bin mel loss, and added codebooks.","fun_headline_variants_meta":{"raw":{"variants":["Audio codec reaches 12.5 Hz token rate with band-wise refinement","MUFFIN codec uses perceptual bands to shrink audio to 12.5 Hz tokens","Psychoacoustic frequency-split codec hits 12.5 Hz with high fidelity","Neural audio codec: multi-band quantization for 12.5 Hz tokens","Perceptual band split enables neural codec to hit 12.5 Hz rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000627,"raw_usage":{"total_tokens":2920,"prompt_tokens":982,"completion_tokens":1938,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1829}},"tokens_in":598,"tokens_out":1938,"duration_ms":14156,"temperature":1.0,"reasoning_tokens":1829,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:21:27.978977+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same MUFFIN architecture with vanilla RVQ while holding every non-quantizer component fixed, then also add each non-quantizer change one at a time to the retrained HiFi-Codec baseline; if any single change closes the LibriTTS test-clean gap in PESQ or UTMOS, the paper's central mechanism is not what drives the improvement.","supporting_citations":[],"review_version":1}