{"id":"261350ad-f300-4218-ad0d-634949ce636c","arxiv_id":"2507.12090","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A compact Mamba-based predictor using frozen speech embeddings and radial-basis score encoding achieves strong MOS prediction with limited dependence on the audio sampling rate.","lead":"MambaRate is a lightweight model that predicts human speech-quality scores from audio features without needing a large fine-tuned speech model. It beat the challenge baseline by about 14 percent on a low-data track, though it placed fourth out of five systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The sampling-rate robustness claim is undermined because all audio is downsampled to 16 kHz, discarding the very high-frequency content that distinguishes 24/48 kHz recordings.","rationale":"The reader identified the downsampling-to-16 kHz step as the weakest assumption, and this is indeed the most load-bearing issue. The paper's central claim about sampling-rate robustness is directly contradicted by the preprocessing that removes sampling-rate-specific information. The empirical result against the baseline may still be valid, but the interpretation is misleading. I see no other concern that would change the conditional verdict: the paper lacks code and error bars, yet the leaderboard provides some external validation. A per-sampling-rate evaluation is the clearest way to settle whether the claim holds. Therefore the reader's CONDITIONAL verdict is appropriate, so no change is needed.","tokens_in":157,"tokens_out":4305,"duration_ms":60763,"concrete_test":"Obtain the AudioMOS Track 3 test set (or replicate with a held-out split of the training set) and compute T16's system-level SRCC separately for the subsets of utterances originally recorded at 16, 24, and 48 kHz. If the SRCC on the 48 kHz subset is significantly lower than on the 16 kHz subset, or if T16's predictions for the same utterance are identical before and after downsampling from 48 kHz to 16 kHz, then the model is not truly robust to sampling rate and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A states: 'All audios are preprocessed ... and downsampled to 16kHz for compatibility with the pre-trained models in use.' This means MambaRate never sees spectral content above 8 kHz. The title and abstract claim 'limited bias regarding the sampling rate' and 'supporting different sampling rates' (Conclusion), but if the input is always normalized to 16 kHz, the model is sampling-rate-agnostic by construction, not by learned robustness. The central claim would only be meaningful if high-frequency information were irrelevant to human quality judgments in AudioMOS Track 3. No evidence is provided for that premise, and no analysis shows whether the challenge ratings depend on content above 8 kHz. While this preprocessing does not invalidate the empirical comparison to the baseline (B03), which presumably uses the same 16 kHz front-end, it does invalidate the interpretation that MambaRate achieves cross-sampling-rate perception. The claimed 'robustness' could be an artifact of input normalization that hides, rather than handles, sampling-rate differences.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MambaRate, a lightweight MOS prediction model that uses frozen WavLM-Large embeddings fed through Mamba-2 blocks, with target ratings encoded via Gaussian radial basis functions. The system was submitted to Track 3 of the AudioMOS 2025 Challenge, where the T16 version ranked fourth among five systems and is claimed to outperform the challenge baseline (B03) by about 14% in system-level SRCC. The paper also reports utterance-level comparisons against a UTMOS-style baseline on BVCC and challenge data, and ablation studies exploring different WavLM layers and alternative SSL embeddings (Whisper, VGGish). The manuscript emphasizes sampling-rate robustness as a central contribution.","tokens_in":7086,"tokens_out":5930,"duration_ms":66969,"significance":"If the empirical results hold, MambaRate demonstrates that a model with roughly 900k parameters, using offline SSL embeddings and no fine-tuning of the front-end, can compete with a fine-tuned wav2vec-2.0 baseline in a few-shot challenge setting, at a fraction of the training cost. The independent challenge leaderboard provides external validation for the T16 result, which is a clear strength. However, the significance of the paper's headline claim—\"Speech Quality Assessment Across Different Sampling Rates\"—is substantially undermined by the preprocessing choice described in Section III-A, where all audio is downsampled to 16 kHz before the model ever sees it. Also, the ablation comparisons are reported on the validation split used for model selection, and no statistical uncertainty is reported anywhere. As such, the paper's empirical value is real but modest, and its framing overstates what is actually demonstrated.","major_comments":[{"comment":"Section III-A states: \"All audios are preprocessed ... and downsampled to 16kHz for compatibility with the pre-trained models in use.\" This means MambaRate never processes spectral content above 8 kHz, so the abstract's claim of \"limited bias regarding the sampling rate\" and the conclusion's \"supporting different sampling rates\" are not supported by the experiments. The model is sampling-rate-agnostic by input normalization, not by learned robustness against sampling-rate variation. Please reframe these claims to describe a 16 kHz front-end applied to multi-rate recordings, or provide evidence (e.g., per-sampling-rate metrics or an analysis of whether human quality ratings in Track 3 are insensitive to content above 8 kHz) that the downsampling does not hide the very differences the paper claims to address.","section":"III-A and Abstract/Conclusion"},{"comment":"The evaluation protocol for Table II is underspecified: it is unclear whether the metrics are computed on the challenge test set, the 10% held-out split of the challenge training set, or the challenge development set. Since T16's leaderboard result is a system-level SRCC while Table II reports utterance-level metrics, the claimed ~14% improvement over B03 cannot be directly verified from the paper. Please state explicitly which partition is used for each column and provide the system-level SRCC values for T16 and B03 in the text (or in the figure caption).","section":"III-C, Table II"},{"comment":"The ablations in Tables III and IV are trained on a 70-15-15 split, and the reported metrics are computed on the same 15% validation split used for model selection (e.g., choosing the WavLM layer). This selection-on-validation procedure inflates the apparent differences and makes the statement that these ablations \"outperform T16\" unreliable. Please report results on a truly held-out partition, such as the 10% test split used for T16, or use repeated cross-validation with variance estimates.","section":"III-D, Tables III and IV"},{"comment":"No error bars, confidence intervals, or significance tests are provided for any table. Differences between WavLM layers in Table III and between embedding types in Table IV are only a few hundredths in SRCC, which may well be within run-to-run variance for a model of this size. At minimum, report multiple seeded runs with standard deviations for the key comparisons (Tables II and III). Without this, the robustness of the reported improvements cannot be assessed.","section":"All experiments"}],"minor_comments":[{"comment":"The abstract contains two typographical issues: \"Correllation\" should be \"Correlation\" and \"fewshot\" should be \"few-shot.\"","section":"Abstract"},{"comment":"The RBF width sigma in Eq. (1) is never defined; please report its value or describe how it is chosen, since it directly controls the shape of the target encoding.","section":"II, Eq. (1)"},{"comment":"The text says \"an 1D convolutional layer\" and \"a multi-perceptron block\"; these should be \"a 1D convolutional layer\" and \"a multilayer perceptron (MLP) block\" for clarity.","section":"II"},{"comment":"The SV56 amplitude normalization is mentioned without a reference; please cite the corresponding ITU-T recommendation or the implementation used.","section":"III-A"},{"comment":"The column headers (\"BVCC\", \"BVCC+C\", \"C\", \"BVCC+C\") make the comparison difficult to parse; clarify in the caption or table body which column corresponds to the baseline and which to MambaRate, and what each training condition means.","section":"Table II caption"},{"comment":"The decoding from the 16 RBF outputs uses argmax over centers, which quantizes predictions to 16 discrete levels; a weighted average over centers would give a continuous prediction and might improve MSE without affecting the rank-based metrics.","section":"II (decoding)"},{"comment":"The sentence \"MambaRate clearly outperforms both setups of the baseline, although the former is trained only on the training data of the challenge\" is ambiguous: identify whether \"the former\" refers to MambaRate or to one of the baseline setups.","section":"III-C"},{"comment":"The cosine annealing scheduler parameter \"Tmax = 10\" is not specified in units; clarify whether it is 10 epochs or 10 optimizer steps.","section":"II"},{"comment":"Figure 2 is reproduced without axis labels or numeric values; please provide the system-level SRCC values for all challenge systems in the caption or in the text.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper reads like a challenge system description rather than a full research article. The T16 leaderboard result is credible and the computational-efficiency angle is a strength, but the title and abstract promise more than the experiments deliver on sampling-rate robustness. The authors should either substantially rework the framing or add the per-rate analysis and high-frequency-content checks that would support the current claims. Also, the use of validation-set metrics in the ablation tables is a concern that should be addressed during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing here is the held-out result: MambaRate, a 900k-parameter model using frozen WavLM embeddings, Mamba-2 blocks, and Gaussian RBF targets, beat the fine-tuned wav2vec baseline (B03) by ~14% SRCC in the AudioMOS 2025 Track 3 few-shot setting, and ranked fourth of five. The leaderboard is external and blind, so the comparison is honest and not circular. That is worth recording.\n\nWhat's genuinely new is narrow but real: the specific combination of frozen SSL features, selective state space layers, and RBF target encoding for MOS prediction is not in the cited prior work, and the system costs minutes to train rather than hours. The paper also does a decent set of ablations, showing WavLM layer choice matters and that Whisper and VGGish embeddings are viable alternatives.\n\nNow the soft spots. The title and abstract claim 'limited bias regarding the sampling rate' and 'supporting different sampling rates,' but Section III-A says all audio is downsampled to 16 kHz before feature extraction. That means the model never sees spectral content above 8 kHz. It is sampling-rate agnostic by construction, not by learned robustness. If the Track 3 human ratings are influenced by frequencies above 8 kHz, MambaRate is blind to them. The authors offer no analysis of whether the challenge ratings depend on that content. This doesn't invalidate the baseline comparison—B03 presumably uses the same front-end—but it invalidates the cross-sampling-rate interpretation. The claim should be reframed as 'robust to input normalization' or supported with an experiment that preserves the original sampling rate.\n\nSecond, there are no error bars, confidence intervals, or significance tests anywhere. Given the small training set (400 files) and the modest leaderboard differences, I wouldn't accept the fine distinctions between systems without some measure of uncertainty. The layer ablation is also selection-on-validation followed by test reporting, so the best-layer advantage is likely optimistic.\n\nNo code is released, which weakens reproducibility for a methods claim.\n\nOverall, this is a solid workshop-level challenge paper. The empirical result is plausible and externally evaluated; the interpretation overshoots it. I'd send it to peer review with a required revision: fix the sampling-rate claim or provide evidence that 8 kHz cutoff doesn't matter, add uncertainty quantification, and ideally release code. The core comparison is worth preserving.\n\nBest.","headline":"A competent challenge paper with a real held-out result, but the sampling-rate robustness headline is undercut by its own 16 kHz front-end.","tokens_in":7631,"tokens_out":3120,"would_cite":false,"duration_ms":32338,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MambaRate shows that a 900k-parameter model using frozen WavLM embeddings and Mamba-2 layers can match or beat a fine-tuned wav2vec MOS baseline in a few-shot setting, ranking fourth of five in AudioMOS 2025 Track 3.","keywords":["MOS prediction","speech quality assessment","sampling rate robustness","state space models","WavLM embeddings","Gaussian radial basis functions","few-shot learning","AudioMOS challenge"],"falsifier":"Take a set of 24 kHz or 48 kHz utterances, create two versions of each (one full-band and one low-pass filtered at 8 kHz), and collect human MOS for both. If human scores separate the pairs while MambaRate, which sees only 16 kHz audio, gives identical predictions for both, then the model has not actually learned to be sampling-rate invariant; it has learned to ignore the missing band. A simpler check is to compute MambaRate's output before and after downsampling and see whether it changes at all.","tokens_in":6700,"feed_emoji":"🎙️","tokens_out":6295,"duration_ms":61703,"temperature":0.7,"pith_summary":"The paper tries to establish that accurate MOS prediction across different sampling rates does not require a large fine-tuned SSL model. MambaRate uses frozen WavLM embeddings, a small convolutional downsampler, and Mamba-2 state space layers, totaling about 900k trainable parameters. In the AudioMOS 2025 Track 3 few-shot setting, its T16 version beat the fine-tuned wav2vec baseline (B03) by roughly 14% in SRCC without any pretraining on MOS data, and finished fourth of five, about 6% behind the winner. The paper also reports ablations showing that embeddings from other WavLM layers or from Whisper improve on T16. If this holds, lightweight MOS predictors can be trained in minutes on small data while generalizing across 16, 24, and 48 kHz speech.","feed_headline":"A 900k-parameter model beats a fine-tuned wav2vec MOS baseline","feed_subtitle":"MambaRate ranks 4th of 5 in AudioMOS 2025 Track 3 while training in minutes on few-shot data.","key_machinery":"The architecture combines frozen WavLM-Large embeddings (layer 6 for T16) with a 1D convolutional downsampling block, several Mamba-2 blocks (a selective state space model that processes sequences efficiently), and a two-layer MLP head with sigmoid output. The central encoding trick is that each MOS rating is converted into a 16-dimensional vector via Gaussian radial basis functions centered between 1 and 5, and the model is trained with MSE to predict that vector; at inference, the scalar prediction is the center with the maximum value. This avoids treating MOS as a plain regression target and lets the model represent rating uncertainty continuously. The whole model is trained from scratch on the challenge's few hundred clips with AdamW, cosine annealing, and early stopping in well under 30 epochs.","core_discovery":"The paper's central claim is that a compact predictor about 900k parameters in size can predict MOS for speech recorded at 16, 24, and 48 kHz almost as well as or better than a much larger fine-tuned wav2vec 2.0 baseline, provided the input is normalized to 16 kHz and represented by frozen self-supervised embeddings. Specifically, MambaRate's challenged T16 system achieved a system-level SRCC about 14% higher than the baseline in the few-shot Track 3 setting, ranking fourth out of five systems and coming within roughly 6% of the winning system. On the BVCC 16 kHz benchmark, MambaRate matched the baseline in LCC and SRCC while improving MSE and Kendall's tau. Additional experiments with mean ratings and embeddings from later WavLM layers or from Whisper outperform the submitted T16, supporting the paper's conclusion that offline pre-trained representations can be efficiently adopted for few-shot MOS prediction across sampling rates.","pith_inferences":["Because every input is downsampled to 16 kHz before scoring, MambaRate's sampling-rate robustness is really robustness to input normalization; any quality cue that lives only above 8 kHz is invisible to the model, so the claim should be tested on band-limited versus full-band versions of the same recordings.","The RBF target encoding may be the key reason the small model trains so quickly; a testable extension is to compare it against a plain scalar regression head with the same architecture.","Whisper embeddings, trained for ASR, beating WavLM embeddings suggests that quality-relevant information is largely captured by speech content representations, and ensembling multiple frozen layers could improve accuracy further.","A useful follow-up would measure at what training-set size the fine-tuned wav2vec baseline overtakes MambaRate, mapping where the few-shot advantage ends."],"forward_implications":["A roughly 900k-parameter MOS predictor trained for minutes can beat a fine-tuned wav2vec baseline in a few-shot setting, reducing the compute needed for objective speech quality evaluation.","Frozen SSL embeddings from a single layer are sufficient input for competitive MOS prediction, and no listener or system metadata is required.","Pre-training on BVCC before fine-tuning on challenge data improves performance, raising system-level SRCC from 0.879 to 0.930.","Using embeddings from WavLM layer 9 or from Whisper outperforms the submitted layer-6 T16 version, suggesting that input representation choice matters more than added capacity.","The same design should transfer to other small-data MOS tasks, with future work planned on the SOMOS and VCC2018 datasets."],"supporting_citations":[{"why":"WavLM-Large is the frozen SSL backbone whose embeddings MambaRate consumes.","marker":"[8]"},{"why":"Mamba-2 is the selective state space layer that replaces attention for sequence modeling.","marker":"[19]"},{"why":"Describes the fine-tuned wav2vec baseline (B03) that MambaRate must beat, and the sv56 preprocessing it follows.","marker":"[13]"},{"why":"BVCC is the public 16 kHz MOS dataset used for the conventional benchmark and for pre-training variants.","marker":"[14]"},{"why":"Gaussian radial basis function encoding that maps scalar MOS ratings to 16-dimensional targets.","marker":"[21]"},{"why":"Supports the choice of WavLM layer 6 as giving better audio classification than the last layer.","marker":"[16]"},{"why":"Provides the publicly available implementation and weights used to extract offline SSL embeddings.","marker":"[15]"}],"fun_headline_variants":["MambaRate: 900k params beat wav2vec MOS by 14%","900k-param MambaRate beats wav2vec baseline in few-shot MOS","Compact MambaRate beats fine-tuned wav2vec in MOS scoring","900k params: MambaRate tops wav2vec MOS baseline in few-shot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on the assumption that downsampling every input to 16 kHz keeps all the audible information that drives human quality ratings, so any quality differences tied to frequencies above 8 kHz are either absent or not needed in the challenge data.","fun_headline_variants_meta":{"raw":{"variants":["MambaRate: 900k params beat wav2vec MOS by 14%","900k-param MambaRate beats wav2vec baseline in few-shot MOS","Compact MambaRate beats fine-tuned wav2vec in MOS scoring","900k params: MambaRate tops wav2vec MOS baseline in few-shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000599,"raw_usage":{"total_tokens":2784,"prompt_tokens":916,"completion_tokens":1868,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1778}},"tokens_in":532,"tokens_out":1868,"duration_ms":16327,"temperature":1.0,"reasoning_tokens":1778,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:53:36.208732+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of 24 kHz or 48 kHz utterances, create two versions of each (one full-band and one low-pass filtered at 8 kHz), and collect human MOS for both. If human scores separate the pairs while MambaRate, which sees only 16 kHz audio, gives identical predictions for both, then the model has not actually learned to be sampling-rate invariant; it has learned to ignore the missing band. A simpler check is to compute MambaRate's output before and after downsampling and see whether it changes at all.","supporting_citations":[{"cited_title":"Generalization ability of mos prediction networks,","cited_arxiv_id":null,"evidence_quote":"Describes the fine-tuned wav2vec baseline (B03) that MambaRate must beat, and the sv56 preprocessing it follows."},{"cited_title":"How do voices from past speech synthesis challenges compare today?","cited_arxiv_id":null,"evidence_quote":"BVCC is the public 16 kHz MOS dataset used for the conventional benchmark and for pre-training variants."},{"cited_title":"Predicting locations of cryptic pockets from single protein structures using the pocketminer graph neural network,","cited_arxiv_id":null,"evidence_quote":"Gaussian radial basis function encoding that maps scalar MOS ratings to 16-dimensional targets."},{"cited_title":"Towards out-of-distribution detection in vocoder recognition via latent feature reconstruction,","cited_arxiv_id":null,"evidence_quote":"Supports the choice of WavLM layer 6 as giving better audio classification than the last layer."}],"review_version":1}