Pith. sign in

REVIEW 4 major objections 4 minor 23 references

Are Mamba-based Audio Foundation Models the Best Fit for Non-Verbal Emotion Recognition?

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that Mamba-based state-space audio foundation models outperform attention-based audio foundation models on non-verbal emotion recognition, and that fusing one of each with a Renyi-divergence alignment loss sets a new…

desk verdict First Audio-Mamba NVER benchmark with a plausible fusion idea, but the architecture-versus-pretraining confound means the headline causal claim is not established. read the letter →

arxiv 2506.02258 v1 pith:CUWITIEF submitted 2025-06-02 eess.AS cs.SD

Mohd Mujtaba Akhtar , Orchid Chetia Phukan , Girish , Swarup Ranjan Behera , Ananda Chandra Nayak , Sanjib Kumar Nayak , Arun Balaji Buduru , Rajesh Sharma This is my paper · ORCID
classification eess.AScs.SD
keywords non-verbalemotionrecognitionMambaaudiofoundationmodelsstate-spaceRenyidivergencemodelfusionself-attentionvocal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that Mamba-based audio foundation models, which encode sound with state-space recurrences, are better suited to recognizing emotions from non-verbal vocal sounds than attention-based audio foundation models. On three benchmark datasets (ASVP-ESD, JNV, and VIVAE), the authors find that frozen Audio-Mamba embeddings consistently outperform embeddings from WavLM, wav2vec2, UniSpeech-SAT, and HuBERT, and they attribute the gap to Mamba's structured state-space modeling. The paper also introduces RENO, a fusion framework that aligns one Mamba model and one attention model with a Renyi-divergence loss plus self-attention, and reports that this heterogeneous fusion sets a new state of the art. A sympathetic reader would care because it suggests the backbone choice for emotion-sensing audio systems may shift from attention to state-space architectures, and it offers a recipe for combining the two families.

What carries the argument

Audio-Mamba, a selective state-space model that tracks long-range dependencies through a structured recurrence rather than pairwise attention, supplies the representations that carry the argument. RENO aligns pairs of frozen embeddings by minimizing the Renyi divergence (order beta = 2) between their feature distributions and then applies multi-head self-attention before classification.

What would settle it

Train an attention model (for example, a WavLM-sized transformer) on AudioSet with the same self-supervised objective as Audio-Mamba, freeze its last-layer embeddings, and evaluate on ASVP-ESD, JNV, and VIVAE; if the accuracy gap shrinks or reverses, the architectural advantage claimed for Mamba is not the cause.

Watch

Extended reading notes

Core claim

The paper claims that frozen last-layer embeddings from Audio-Mamba, a selective state-space model pretrained on AudioSet, consistently beat embeddings from WavLM, wav2vec2, UniSpeech-SAT, and HuBERT for recognizing emotions in non-verbal vocalizations across the ASVP-ESD, JNV, and VIVAE datasets. It further claims that a heterogeneous fusion of one Audio-Mamba model and one attention model, aligned with a Renyi-divergence loss and refined by self-attention (the RENO framework), outperforms individual models, concatenation baselines, and homogeneous attention-model fusion, setting a new state of the art.

Load-bearing premise

The comparison assumes that differences between Audio-Mamba and the attention models come from their architecture, but the models also differ in pretraining data, objective, and size, so the observed gap might come from those factors instead.

Editorial extensions

If this is right

  • Across all three datasets, the best Audio-Mamba single model beats every attention-based model on both accuracy and macro F1, so the advantage is not dataset-specific.
  • The tiny Audio-Mamba (4.8M parameters) outperforms 94M-parameter attention models, implying the gain comes from the architecture, not model scale.
  • Heterogeneous fusion of an Audio-Mamba and an attention model with RENO outperforms homogeneous attention-only fusion, indicating that the two representation families carry complementary information.
  • RENO's Renyi-divergence alignment improves over simple concatenation for every tested FM pair, showing that the alignment loss is doing real work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the architectural comparison is confounded: the attention models were pretrained on LibriSpeech while Audio-Mamba was pretrained on AudioSet, so a testable extension would be to pretrain or fine-tune all models on the same corpus and repeat the evaluation.
  • If the same-architecture advantage persists under matched pretraining, Mamba-style backbones may become a strong default for paralinguistic tasks beyond NVER, such as speech emotion recognition and audio deepfake detection.
  • A natural next step is to test whether the Renyi-divergence alignment in RENO also beats KL divergence or cosine alignment for other heterogeneous foundation-model fusion settings, since only the order-beta-2 case is studied here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper investigates whether Mamba-based audio foundation models (MAFMs) outperform attention-based audio foundation models (AAFMs) for non-verbal emotion recognition (NVER). It compares frozen last-layer embeddings from three Audio-Mamba variants with WavLM, wav2vec2, UniSpeech-SAT, and HuBERT across three NVER datasets using FCN and CNN downstream classifiers, and proposes RENO, a fusion framework combining self-attention with a Renyi-divergence loss. The authors report that MAFMs consistently outperform AAFMs and that heterogeneous fusion of a MAFM and an AAFM through RENO achieves state-of-the-art results.

Significance. If the claims were properly isolated, the paper would be a useful first benchmark of Mamba-based audio representations for NVER and a plausible contribution to fusion methodology for emotion recognition. The authors provide a reproducible protocol, release code, and examine a genuinely underexplored problem. However, the central architectural claim is currently confounded by pretraining-domain mismatch, and the reported SOTA numbers are affected by selection and lack of statistical grounding. The empirical conclusions are therefore not yet established, although they are worth pursuing with better controls.

major comments (4)
  1. [Section II and Section IV-B] The headline claim that MAFMs outperform AAFMs because of state-space modeling is confounded: every MAFM used is an Audio-Mamba model pretrained on AudioSet, while every AAFM baseline (WavLM, wav2vec2, UniSpeech-SAT, HuBERT) is pretrained on 960 hours of LibriSpeech. The performance gaps in Table I, including the striking 4.8M-parameter result, can therefore be explained by pretraining-domain overlap with non-verbal vocalizations rather than by the Mamba architecture. The causal attribution in Section IV-B and the t-SNE evidence in Figure 2 inherit the same confound. To support the claim, the comparison must include an attention-based audio foundation model pretrained on AudioSet, or a Mamba-based model pretrained on speech, so that architecture and pretraining data are not varied simultaneously.
  2. [Section IV-A and Section IV-B] The hyperparameters beta=2, delta=0.2, and lambda=0.4 are reported as chosen by 'preliminary exploration' on the evaluation datasets, and Section IV-B selects the best fusion pair for each dataset after seeing the results. This constitutes tuning and model selection on the test data. The SOTA claim is therefore based on the maximum over many evaluated configurations, not on a pre-specified hypothesis. The authors should define a validation-based selection protocol (e.g., nested cross-validation or a held-out validation fold) and report the performance of the selected configuration on held-out test data, rather than the best per-dataset pair.
  3. [Tables I and II] All reported numbers are five-fold averages without standard deviations, confidence intervals, or significance tests. Many adjacent differences in Tables I and II are a few percentage points (e.g., 64.29 vs. 62.42 for A(B) vs. W2 on JNV with CNN), so the claimed consistency of MAFM superiority is not statistically demonstrated. In addition, the abstract and Section IV-B claim 'setting SOTA' without comparing against any published NVER results. The authors should include the previous SOTA numbers, and provide variance estimates and appropriate significance tests for the central comparisons.
  4. [Section III-A, Eq. (2)] Renyi divergence is defined for probability distributions, but in Eq. (2) it is applied to raw feature vectors z_x and z_y with no normalization, projection, or non-negativity guarantee. With beta=2, the term (z_{x,j}+delta)^beta (z_{y,j}+delta)^{1-beta} can be negative or undefined if any coordinate plus delta is negative, and the logarithm is then not real-valued. The paper should either justify why this quantity is a valid alignment loss for raw embeddings, or modify the formulation (e.g., through softmax normalization) and verify that the loss is well-defined on the actual feature distributions used.
minor comments (4)
  1. [Abstract and Section I] There are several typos and grammatical errors: 'to to effectively align' in Section I, 'repesentational spacne', 'renyi rivergence loss' in Section IV-B, and 'Q,K,Q stands for query, key, value' in Section III-A (should be Q, K, V).
  2. [Table II] The column header 'Concat RENO' is ambiguous because each cell block contains four numbers per dataset. The reader has to infer that the first two numbers under 'Concat' are accuracy and F1 and the next two are RENO's accuracy and F1. Please label the subcolumns explicitly.
  3. [Figure 2] The t-SNE plots compare only Audio-Mamba (Base) with wav2vec2 and are presented without any quantitative cluster-quality metric; as discussed, this also repeats the pretraining-domain confound.
  4. [Section III-A] The dimensions of the self-attention inputs and the exact form of the flattened features fed into Eq. (2) are not specified. Please clarify whether the Renyi loss operates on the output of the convolutional blocks or directly on the pooled FM embeddings, since this affects reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the paper's claims are empirical comparisons against externally pretrained frozen foundation models, and its self-citations are motivational rather than load-bearing.

full rationale

The paper does not offer a formal derivation chain; its central claims are empirical. Audio-Mamba representations come from an externally released AudioSet-pretrained checkpoint, and the attention-based baselines (WavLM, Wav2vec2, UniSpeech-SAT, HuBERT) come from externally released LibriSpeech-pretrained checkpoints. Downstream classifiers are standard FCN/CNN models, and RENO combines cross-entropy with a Renyi-divergence loss; no equation in the paper reduces a predicted performance number to an input parameter or to the hypothesis statement by construction. The self-citations [8] and [15] are used only to motivate exploring FM fusion in related tasks; they do not carry the argument for MAFM superiority or for RENO's effectiveness. The pretraining-domain confound (MAFMs pretrained on AudioSet versus AAFMs pretrained on LibriSpeech) is a real experimental-control and correctness concern, but it is not circularity: the claimed gaps are empirical observations that could have come out differently, and no fitted quantity is renamed as a prediction. The fixed hyperparameters (beta=2, delta=0.2, lambda=0.4) were selected by preliminary exploration on the target datasets, and the paper reports the best of many fusion pairs as SOTA; this is optimistic selection relative to external generalization, not a logical reduction of the claim to its inputs. Therefore no specific circular step can be quoted and exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several exploration-chosen hyperparameters, a domain assumption about cross-pretraining comparability, and the unstated premise that selecting the best fusion pair per dataset after seeing results is a valid way to establish SOTA. No new physical or conceptual entities are introduced.

free parameters (3)
  • renyi_order_beta = 2
    Controls the divergence order in Eq. (2) and was fixed after preliminary exploration.
  • stability_delta = 0.2
    Added in Eq. (2) for numerical stability; value chosen by exploration.
  • loss_tradeoff_lambda = 0.4
    Weights cross-entropy and Renyi divergence losses in Eq. (3); chosen after preliminary exploration.
assumptions (5)
  • standard math Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V is a valid operation on flattened 1D features.
    Invoked in Eq. (1), Section III-A, without proof.
  • domain assumption Frozen last-hidden-state average-pooled features are sufficient representations for NVER.
    Used throughout Sections II and III; no analysis of alternative pooling or layer choices is provided.
  • domain assumption Audio-Mamba pretrained on AudioSet and attention models pretrained on LibriSpeech can be directly compared as audio foundation models.
    Section II introduces both model families; Section IV-B attributes score differences to state-space versus attention architecture.
  • ad hoc to paper Renyi divergence over raw feature vectors acts as a meaningful alignment loss.
    Eq. (2) applies Renyi divergence to feature values without normalization or a proof that this aligns distributions.
  • domain assumption Five-fold averaged scores without variances are stable enough to rank models.
    All conclusions in Section IV-B rely on the absence of error bars or significance tests.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Mamba-based Audio Foundation Models the Best Fit for Non-Verbal Emotion Recognition?." pith.science (2026). https://pith.science/paper/CUWITIEF

@misc{pith2026250602258,
  author       = {Pith},
  title        = {Pith review of: Are Mamba-based Audio Foundation Models the Best Fit for Non-Verbal Emotion Recognition?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CUWITIEF}},
  note         = {Machine review of arXiv:2506.02258}
}
read the original abstract

In this work, we focus on non-verbal vocal sounds emotion recognition (NVER). We investigate mamba-based audio foundation models (MAFMs) for the first time for NVER and hypothesize that MAFMs will outperform attention-based audio foundation models (AAFMs) for NVER by leveraging its state-space modeling to capture intrinsic emotional structures more effectively. Unlike AAFMs, which may amplify irrelevant patterns due to their attention mechanisms, MAFMs will extract more stable and context-aware representations, enabling better differentiation of subtle non-verbal emotional cues. Our experiments with state-of-the-art (SOTA) AAFMs and MAFMs validates our hypothesis. Further, motivated from related research such as speech emotion recognition, synthetic speech detection, where fusion of foundation models (FMs) have showed improved performance, we also explore fusion of FMs for NVER. To this end, we propose, RENO, that uses renyi-divergence as a novel loss function for effective alignment of the FMs. It also makes use of self-attention for better intra-representation interaction of the FMs. With RENO, through the heterogeneous fusion of MAFMs and AAFMs, we show the topmost performance in comparison to individual FMs, its fusion and also setting SOTA in comparison to previous SOTA work.

Figures

Figures reproduced from arXiv: 2506.02258 by the authors.

Figure 1
Figure 1. Novel Framework: RENO; MSA stands for Multi-head Self-attention A. RENO We propose RENO, a novel framework designed to align feature representations from distinct FMs. The architecture is given in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. t-SNE plots for ASVP-ESD (a) (b) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Confusion Matrix for JNV dataset; Subfigures (a) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 15 canonical work pages

  1. [1]

    Speech emotion recognition based on hmm and svm,

    Y .-L. Lin and G. Wei, “Speech emotion recognition based on hmm and svm,” in2005 international conference on machine learning and cybernetics, vol. 8. IEEE, 2005, pp. 4898–4901

    work page 2005

  2. [2]

    Emotion recognition in speech using mfcc and wavelet features,

    K. K. Kishore and P. K. Satish, “Emotion recognition in speech using mfcc and wavelet features,” in2013 3rd IEEE International Advance Computing Conference (IACC). IEEE, 2013, pp. 842–847

    work page 2013

  3. [3]

    Speech emotion recognition based on feature selection and extreme learning machine decision tree,

    Z.-T. Liu, M. Wu, W.-H. Cao, J.-W. Mao, J.-P. Xu, and G.-Z. Tan, “Speech emotion recognition based on feature selection and extreme learning machine decision tree,”Neurocomputing, vol. 273, pp. 271– 280, 2018

    work page 2018

  4. [4]

    Speech emotion recognition with dual-sequence lstm architecture,

    J. Wang, M. Xue, R. Culhane, E. Diao, J. Ding, and V . Tarokh, “Speech emotion recognition with dual-sequence lstm architecture,” inICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6474–6478

    work page 2020

  5. [5]

    Convolution neural network based automatic speech emotion recognition using mel-frequency cepstrum coefficients,

    M. D. Pawar and R. D. Kokate, “Convolution neural network based automatic speech emotion recognition using mel-frequency cepstrum coefficients,”Multimedia Tools and Applications, vol. 80, pp. 15 563– 15 587, 2021

    work page 2021

  6. [6]

    Ctnet: Conversational transformer network for emotion recognition,

    Z. Lian, B. Liu, and J. Tao, “Ctnet: Conversational transformer network for emotion recognition,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 985–1000, 2021

    work page 2021

  7. [7]

    Emotion recognition from speech using wav2vec 2.0 embeddings,

    L. Pepino, P. E. Riera, and L. Ferrer, “Emotion recognition from speech using wav2vec 2.0 embeddings,”ArXiv, vol. abs/2104.03502, pp. 3400–3404, 2021

    arXiv 2021

  8. [8]

    Transforming the embeddings: A lightweight technique for speech emotion recognition tasks,

    O. C. Phukan, A. B. Buduru, and R. Sharma, “Transforming the embeddings: A lightweight technique for speech emotion recognition tasks,”arXiv preprint arXiv:2305.18640, 2023

    arXiv 2023

Show all 23 references
  1. [9]

    Adapting wavlm for speech emotion recognition,

    D. Diatlova, A. Udalov, V . Shutov, and E. Spirin, “Adapting wavlm for speech emotion recognition,” inProc. odyssey 2024, 2024, pp. 303– 308

    2024

  2. [10]

    Audio mamba: Selective state spaces for self-supervised audio representations,

    S. Yadav and Z.-H. Tan, “Audio mamba: Selective state spaces for self-supervised audio representations,” inInterspeech 2024, 2024, pp. 552–556

    2024

  3. [11]

    Speech emotion recognition considering nonverbal vocalization in affective conver- sations,

    J.-H. Hsu, M.-H. Su, C.-H. Wu, and Y .-H. Chen, “Speech emotion recognition considering nonverbal vocalization in affective conver- sations,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1675–1686, 2021

    2021

  4. [12]

    Jvnv: A corpus of japanese emotional speech with verbal content and nonverbal expressions,

    D. Xin, J. Jiang, S. Takamichi, Y . Saito, A. Aizawa, and H. Saruwatari, “Jvnv: A corpus of japanese emotional speech with verbal content and nonverbal expressions,”IEEE Access, vol. 12, pp. 19 752–19 764, 2024

    2024

  5. [13]

    Large-scale nonverbal vocalization detection using transformers,

    P. Tzirakis, A. Baird, J. Brooks, C. Gagne, L. Kim, M. Opara, C. Gregory, J. Metrick, G. Boseck, V . Tiruvadiet al., “Large-scale nonverbal vocalization detection using transformers,” inICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing ...

    2023

  6. [14]

    Investigation of ensemble of self-supervised models for speech emotion recognition,

    Y . Wu, P. Yue, C. Cheng, and T. Li, “Investigation of ensemble of self-supervised models for speech emotion recognition,” in2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2023, pp. 988–995

    2023

  7. [15]

    Heterogeneity over homogeneity: Investigating multilingual speech pre-trained models for detecting audio deepfake,

    O. Chetia Phukan, G. Kashyap, A. B. Buduru, and R. Sharma, “Heterogeneity over homogeneity: Investigating multilingual speech pre-trained models for detecting audio deepfake,” inFindings of the Association for Computational Linguistics: NAACL 2024, K. Duh, H. Gomez, and S. Bet...

    2024

  8. [16]

    Wavlm: Large-scale self-supervised pre- training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self-supervised pre- training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  9. [17]

    Unispeech-sat: Universal speech representation learning with speaker aware pre-training,

    S. Chen, Y . Wu, C. Wang, Z. Chen, Z. Chen, S. Liu, J. Wu, Y . Qian, F. Wei, J. Li, and X. Yu, “Unispeech-sat: Universal speech representation learning with speaker aware pre-training,”ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing ...

    2022

  10. [18]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020

    2020

  11. [19]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  12. [20]

    R ´enyi divergence and kullback-leibler divergence,

    T. Van Erven and P. Harremos, “R ´enyi divergence and kullback-leibler divergence,”IEEE Transactions on Information Theory, vol. 60, no. 7, pp. 3797–3820, 2014

    2014

  13. [21]

    Asvp-esd: A dataset and its benchmark for emotion recognition using both speech and non-speech utterances,

    D. Landry, Q. He, H. Yan, and Y . Li, “Asvp-esd: A dataset and its benchmark for emotion recognition using both speech and non-speech utterances,”Global Scientific Journals, vol. 8, pp. 1793–1798, 2020

    2020

  14. [22]

    Jnv corpus: A corpus of japanese nonverbal vocalizations with diverse phrases and emotions,

    D. Xin, S. Takamichi, and H. Saruwatari, “Jnv corpus: A corpus of japanese nonverbal vocalizations with diverse phrases and emotions,” Speech Commun., vol. 156, p. 103004, 2023

    2023

  15. [23]

    The variably intense vocalizations of affect and emotion (vivae) corpus prompts new per- spective on nonspeech perception

    N. Holz, P. Larrouy-Maestri, and D. Poeppel, “The variably intense vocalizations of affect and emotion (vivae) corpus prompts new per- spective on nonspeech perception.”Emotion, vol. 22, no. 1, p. 213, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.