Pith. sign in

REVIEW 2 major objections 5 minor 53 references

Generated multilingual transcripts boost speech sentiment analysis, and distillation packs the gains into an audio-only model with no extra inference cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 01:52 UTC pith:QGTHOX7Z

load-bearing objection Solid systems paper: generated multilingual text as privileged info + CCMT + KD gives real ~5-pt multimodal and ~1.5-pt distilled gains on MSP-Podcast polarity, with public code and no load-bearing flaw. the 2 major comments →

arxiv 2607.06611 v1 pith:QGTHOX7Z submitted 2026-07-07 cs.CL cs.AIcs.LGcs.SD

Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts

Andrei-George Durdun , Victor Constantinescu , Radu Tudor Ionescu This is my paper
classification cs.CL cs.AIcs.LGcs.SD
keywords audio sentiment analysisknowledge distillationcross-modal transformerautomatic speech recognitionmachine translationprivileged informationMSP-PodcastWavLM
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Recognizing whether speech is positive or negative requires both how something is said and what is said. Audio foundation models already do well, yet the authors show they still miss useful signal that can be recovered from the words themselves. They automatically transcribe each utterance, machine-translate the transcript into several languages, and fuse the audio with those multilingual text views through a cascade of cross-modal transformers. The resulting multimodal teacher lifts macro-F1 by nearly six points and accuracy by more than five points on a large naturalistic podcast corpus. Knowledge distillation then transfers most of that gain into a pure audio student whose inference cost is identical to the original audio baseline. The practical upshot is a deployable speech-only classifier that has been trained with privileged linguistic information it never needs at test time.

Core claim

Automatically generated English transcripts and their machine translations into other languages supply complementary polarity cues that a cascaded cross-modal transformer can fuse with audio, producing large gains over a strong WavLM baseline; those gains can be distilled into an audio-only student that retains the higher accuracy at the original inference speed.

What carries the argument

Cascaded Cross-Modal Transformer (CCMT) teacher that progressively integrates one modality at a time via cross-attention, followed by temperature-scaled knowledge distillation into a WavLM student under the learning-using-privileged-information setting.

Load-bearing premise

ASR and translation errors still leave enough complementary sentiment signal for the teacher to learn, and that signal can be compressed into the audio student.

What would settle it

Replace the generated transcripts and translations with random or scrambled text of the same length; if the multimodal teacher and the distilled student then lose essentially all of their reported gains over the plain WavLM baseline, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a knowledge-distillation pipeline for speech sentiment polarity classification (negative/neutral/positive) on MSP-Podcast. A multimodal teacher first obtains English ASR transcripts (Faster-Whisper) and automatic translations into Spanish, German and French (NLLB-200), encodes them with language-specific transformers (RoBERTa, RoBERTuito, GBERT, CamemBERT) and the audio with WavLM, then fuses the modalities via a cascaded cross-modal transformer (CCMT). Soft targets from the best teachers are distilled into a WavLM-only student that never sees text at inference. On the official MSP-Podcast splits the best multimodal teachers improve macro-F1 by up to +5.89 % and accuracy by +5.15 % over a strong WavLM baseline; distillation recovers +1.54 % macro-F1 / +0.81 % accuracy for the student at identical inference cost. Ablations examine language combinations and the distillation weight λ; code is released.

Significance. If the reported gains hold, the work supplies a practical LUPI-style recipe that lets practitioners keep the latency of a pure audio foundation model while still exploiting complementary lexical polarity cues that are available only at training time. The combination of large-scale naturalistic data (MSP-Podcast), progressive unimodal-then-multimodal training, language ablations, a λ-sweep, and public code makes the empirical claim reproducible and useful for real-time applications (call centers, driver monitoring, virtual assistants). The absolute KD lift is modest, yet the demonstration that ASR/NMT noise still leaves transferable signal is a concrete, non-trivial contribution to the multimodal-SER literature.

major comments (2)
  1. Table 2 reports point estimates only; no standard deviations, bootstrap intervals or statistical significance tests accompany the claimed +5.89 % / +1.54 % macro-F1 gains. Because the absolute KD improvement is small (+0.81 % accuracy) and only the two best teachers were distilled, it is impossible to judge whether the student gains are reliable or could reverse under re-seeding or different teacher selection. Adding error bars or a paired significance test on the official test-1 split is load-bearing for the central claim that distillation “boosts performance without any computational overhead.”
  2. All quantitative claims rest on a single corpus (MSP-Podcast). Section 4 maps the original emotion labels onto three polarity classes via a valence-based rule that is dataset-specific; no cross-corpus or cross-domain evaluation is provided. While the large official splits mitigate overfitting concerns, the generalizability of both the multimodal gains and the distillable residual signal remains untested. A second naturalistic corpus (or at least an out-of-domain subset) would substantially strengthen the claim that generated multilingual transcripts constitute generally useful privileged information.
minor comments (5)
  1. In Table 1 the KD weight is labeled “α” while Eq. (5) and the surrounding text use λ; unify the notation.
  2. Figure 1 caption and the main text both refer to “privileged information” / LUPI, yet the abstract and introduction never introduce the acronym; a one-sentence definition would help non-specialist readers.
  3. Inference times in Table 2 (up to 76 s for the full five-modality CCMT) are measured on a consumer GPU with batch size 8; a short note on whether the times include ASR/NMT or only the fusion stage would clarify the practical bottleneck.
  4. The mapping from continuous valence to discrete polarity (Section 4) is described only qualitatively; stating the exact valence thresholds used would improve reproducibility.
  5. A few typographical inconsistencies appear (e.g., “Automati( Spee(h” in Figure 1, mixed en/em-dashes). A light copy-edit pass would polish the presentation.

Circularity Check

0 steps flagged

No circularity: purely empirical multimodal + KD pipeline whose claims are measured against held-out labels, not derived by construction from fitted inputs.

full rationale

The paper is an empirical systems paper. Its central claims are performance deltas on the official MSP-Podcast test-1 split (Table 2): multimodal CCMT teachers that fuse WavLM audio with ASR/NMT text reach ~0.68 macro-F1 / ~0.69 accuracy versus a WavLM baseline of 0.6239 / 0.6425, and KD (Eqs. 2–5) transfers a further +1.54 % macro-F1 / +0.81 % accuracy to an audio-only student. The fusion equations (Eq. 1) are standard cross-attention; the KD objective is the ordinary temperature-scaled KL + CE combination. Neither equation, nor the progressive training pipeline (unimodal fine-tuning → cached embeddings → CCMT → distillation), reduces a claimed prediction to a fitted constant by construction. Self-citations to the authors’ earlier CCMT work supply the fusion module architecture but do not define the evaluation metric or force the reported gains; those gains are measured against external ground-truth polarity labels. Ablations (text-only models, progressive language addition, λ-sweep in Figure 2) further test rather than tautologically restate the inputs. Consequently the derivation chain contains no self-definitional step, no fitted-input-called-prediction, and no load-bearing uniqueness theorem imported from the authors. Score 0 is the correct outcome.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The paper is an empirical systems paper. Its claims rest on standard ML practice plus a handful of domain assumptions about the utility of noisy generated text and on a small set of free hyperparameters chosen by validation. No new physical or mathematical entities are postulated.

free parameters (4)
  • distillation weight λ = 0.7
    Balances hard CE loss against soft KL loss; default 0.7 chosen after a sweep (Figure 2); central student gains depend on this value.
  • temperature τ = 2.0
    Softens teacher logits for KD; fixed at 2.0.
  • LoRA ranks and alphas = r=8/16, α=16/32
    r=16/α=32 for text, r=8/α=16 for WavLM; control capacity of adapters.
  • choice of target languages (ES, DE, FR) = ES+DE+FR
    Selected among many possible languages; ablation shows they help, but the set is a design choice.
axioms (3)
  • domain assumption ASR transcripts and NMT translations retain usable sentiment polarity signal despite recognition and translation errors.
    Stated as working hypothesis (i) in the introduction and validated only empirically on MSP-Podcast; if false the multimodal gains disappear.
  • domain assumption Emotion categories and valence scores of MSP-Podcast can be deterministically mapped onto three polarity classes (negative/neutral/positive).
    Described in Section 4; the mapping is a preprocessing choice that defines the task labels.
  • domain assumption Soft teacher logits convey transferable inter-class structure that a unimodal student can exploit (standard KD / LUPI assumption).
    Eqs. (2)–(5); the distillation gains rest on this premise.

pith-pipeline@v1.1.0-grok45 · 17081 in / 2497 out tokens · 33888 ms · 2026-07-11T01:52:22.061126+00:00 · methodology

0 comments
read the original abstract

Automatically recognizing the sentiment, positive or negative, from speech is a challenging task, requiring both the analysis of vocal inflections and the interpretation of uttered words. Recent solutions rely on audio foundation models to solve the task, but it remains unclear if such models can take all aspects into account. To this end, we propose a multimodal solution that integrates audio and text information via cross-modal transformers, where text transcripts are automatically generated via an automatic speech recognition (ASR) tool. Moreover, we create multiple text modalities by automatically translating the transcripts into multiple languages via machine translation tools. Audio and multilingual text features are combined via a cascaded architecture comprising cross-modal transformer blocks that integrate modalities one by one. We further distill knowledge from the multimodal model, called teacher, into a unimodal (audio only) model, called student. We conduct experiments on a large-scale dataset, demonstrating that the automatically generated textual information can bring significant performance boosts in multimodal sentiment polarity classification. Our ablation study confirms that both automatic transcripts and automatic translations are helpful. Moreover, we show that the audio-only model can be enhanced via distillation, boosting performance without any computational overhead during inference. To reproduce the reported results, we publicly release our code at https://github.com/andreidurdun/cross-modal-audio-sentiment.

Figures

Figures reproduced from arXiv: 2607.06611 by Andrei-George Durdun, Radu Tudor Ionescu, Victor Constantinescu.

Figure 1
Figure 1. Figure 1: The proposed pipeline based on learning under privileged information, which distills information from a multimodal (audio-text) teacher [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy rates of the WavLM student across di [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 53 canonical work pages · 7 internal anchors

  1. [1]

    The evolution of sentiment analysis and conversational AI: Techniques applications and future research directions

    Anilsagar, T., Syed, S.S.A., 2025. The evolution of sentiment analysis and conversational AI: Techniques applications and future research directions. Journal of Systems Engineering and Electronics 35, 71–82

    work page 2025

  2. [2]

    Sentiment analysis and emotion recognition from speech using universal speech representations

    Atmaja, B.T., Sasou, A., 2022. Sentiment analysis and emotion recognition from speech using universal speech representations. Sensors 22, 6369

    work page 2022

  3. [3]

    wav2vec 2.0: A framework for self-supervised learning of speech representations, in: Proceedings of NeurIPS, pp

    Baevski, A., Zhou, Y ., Mohamed, A., Auli, M., 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations, in: Proceedings of NeurIPS, pp. 12449–12460

    work page 2020

  4. [4]

    TweetEval: Unified benchmark and comparative evaluation for tweet classification, in: Findings of EMNLP, pp

    Barbieri, F., Camacho-Collados, J., Espinosa Anke, L., Neves, L., 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification, in: Findings of EMNLP, pp. 1644–1650

    work page 2020

  5. [5]

    Sentiment Analysis of Customer Feedback and Reviews in E-Commerce Systems, in: Proceedings of ICTCS, pp

    Bulkrock, O., Qusef, A., BaniMustafa, A., 2025. Sentiment Analysis of Customer Feedback and Reviews in E-Commerce Systems, in: Proceedings of ICTCS, pp. 379–385

    work page 2025

  6. [6]

    IEMOCAP: Interactive emotional dyadic motion capture database

    Busso, C., Bulut, M., Lee, C.C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J.N., Lee, S., Narayanan, S.S., 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation 42, 335–359

    work page 2008

  7. [7]

    The MSP-Podcast Corpus

    Busso, C., Lotfian, R., Sridhar, K., Salman, A.N., Lin, W.C., Goncalves, L., Parthasarathy, S., Naini, A.R., Leem, S.G., Martinez-Lucas, L., et al., 2025. The MSP-Podcast Corpus. arXiv preprint arXiv:2509.09791

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    German’s next language model, in: Proceedings of COLING, pp

    Chan, B., Schweter, S., M ¨oller, T., 2020. German’s next language model, in: Proceedings of COLING, pp. 6788–6796

    work page 2020

  9. [9]

    WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

    Chen, S., Wang, C., Chen, Z., Wu, Y ., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y ., Qian, Y ., Wu, J., Zeng, M., Yu, X., Wei, F., 2022. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. IEEE Journal of Selected Topics in Signal Processing 16, 1505–1518

    work page 2022

  10. [10]

    BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of NAACT-HLT, pp

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2019. BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of NAACT-HLT, pp. 4171–4186

    work page 2019

  11. [11]

    Optimized Sentiment Analysis in Tagalog Speech Using PCA and BRNN on Prosodic Suprasegmental and MFCC Features, in: Proceedings of ICTC, pp

    Garcia, A.B., Gerardo, B.D., Medina, R.P., 2024. Optimized Sentiment Analysis in Tagalog Speech Using PCA and BRNN on Prosodic Suprasegmental and MFCC Features, in: Proceedings of ICTC, pp. 60–65

    work page 2024

  12. [12]

    A review on speech emotion recognition: A survey, recent advances, challenges, and the influence of noise

    George, S.M., Ilyas, P.M., 2024. A review on speech emotion recognition: A survey, recent advances, challenges, and the influence of noise. Neurocomputing 568, 127015

    work page 2024

  13. [13]

    Teacher-Student Training and Triplet Loss for Facial Expression Recognition under Occlusion, in: Proceedings of ICPR, pp

    Georgescu, M.I., Ionescu, R.T., 2020. Teacher-Student Training and Triplet Loss for Facial Expression Recognition under Occlusion, in: Proceedings of ICPR, pp. 2288–2295

    work page 2020

  14. [14]

    AST: Audio Spectrogram Transformer, in: Proceedings of INTERSPEECH, pp

    Gong, Y ., Chung, Y .A., Glass, J., 2021. AST: Audio Spectrogram Transformer, in: Proceedings of INTERSPEECH, pp. 571–575

    work page 2021

  15. [15]

    Distilling the Knowledge in a Neural Network

    Hinton, G., Vinyals, O., Dean, J., 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  16. [16]

    HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

    Hsu, W.N., Bolte, B., Tsai, Y .H.H., Lakhotia, K., Salakhutdinov, R., Mohamed, A., 2021. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. IEEE/ACM Transactions on Audio, Speech and Language Processing 29, 3451–3460

    work page 2021

  17. [17]

    LoRA: Low-Rank Adaptation of Large Language Models, in: Proceedings of ICLR

    Hu, E.J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., Chen, W., 2022. LoRA: Low-Rank Adaptation of Large Language Models, in: Proceedings of ICLR

    work page 2022

  18. [18]

    Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech, in: Pro- ceedings of ICML, pp

    Kim, J., Kong, J., Son, J., 2021. Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech, in: Pro- ceedings of ICML, pp. 5530–5540

    work page 2021

  19. [19]

    faster-whisper: Faster Whisper transcription with CTranslate2.https://github.com/SYSTRAN/faster-whisper

    Klein, G., et al., 2023. faster-whisper: Faster Whisper transcription with CTranslate2.https://github.com/SYSTRAN/faster-whisper

    work page 2023

  20. [20]

    Incorporating end-to-end speech recognition models for sentiment analysis, in: Proceedings of ICRA, pp

    Lakomkin, E., Zamani, M.A., Weber, C., Magg, S., Wermter, S., 2019. Incorporating end-to-end speech recognition models for sentiment analysis, in: Proceedings of ICRA, pp. 7976–7982

    work page 2019

  21. [21]

    Unimodal-driven distillation in multimodal emotion recognition with dynamic fusion, in: Proceedings of ICME, pp

    Li, J., Yu, R., Huang, H., Yan, H., 2025. Unimodal-driven distillation in multimodal emotion recognition with dynamic fusion, in: Proceedings of ICME, pp. 1–6

    work page 2025

  22. [22]

    Speech Emotion Recognition With ASR Transcripts: a Comprehensive Study on Word Error Rate and Fusion Techniques, in: Proceedings of SLT, pp

    Li, Y ., Bell, P., Lai, C., 2024. Speech Emotion Recognition With ASR Transcripts: a Comprehensive Study on Word Error Rate and Fusion Techniques, in: Proceedings of SLT, pp. 518–525

    work page 2024

  23. [23]

    Decoupled multimodal distilling for emotion recognition, in: Proceedings of CVPR, pp

    Li, Y ., Wang, Y ., Cui, Z., 2023. Decoupled multimodal distilling for emotion recognition, in: Proceedings of CVPR, pp. 6631–6640

    work page 2023

  24. [24]

    A survey of deep learning-based multimodal emotion recognition: Speech, text, and face

    Lian, H., Lu, C., Li, S., Zhao, Y ., Tang, C., Zong, Y ., 2023. A survey of deep learning-based multimodal emotion recognition: Speech, text, and face. Entropy 25, 1440

    work page 2023

  25. [25]

    Development of interactive English e-learning video entertainment teaching environment based on virtual reality and game teaching emotion analysis

    Liu, J., 2025. Development of interactive English e-learning video entertainment teaching environment based on virtual reality and game teaching emotion analysis. Entertainment Computing 52, 100884

    work page 2025

  26. [26]

    Unifying distillation and privileged information, in: Proceedings of ICLR

    Lopez-Paz, D., Bottou, L., Sch ¨olkopf, B., Vapnik, V ., 2016. Unifying distillation and privileged information, in: Proceedings of ICLR

    work page 2016

  27. [27]

    ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors

    Luo, H., Ji, L., Huang, Y ., Wang, B., Ji, S., Li, T., 2021. ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors. arXiv preprint arXiv:2112.01368

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  28. [28]

    Audio sentiment analysis by heterogeneous signal features learned from utterance-based parallel neural network, in: Proceedings of AffCon@AAAI, pp

    Luo, Z., Xu, H., Chen, F., 2019. Audio sentiment analysis by heterogeneous signal features learned from utterance-based parallel neural network, in: Proceedings of AffCon@AAAI, pp. 80–87

    work page 2019

  29. [29]

    CamemBERT: a tasty French language model, in: Proceedings of ACL, pp

    Martin, L., Muller, B., Ortiz Su ´arez, P.J., Dupont, Y ., Romary, L., de la Clergerie, ´E., Seddah, D., Sagot, B., 2020. CamemBERT: a tasty French language model, in: Proceedings of ACL, pp. 7203–7219

    work page 2020

  30. [30]

    Learning using generated privileged information by text-to-image diffusion models, in: Proceedings of ICPR, pp

    Menadil, R.E., Georgescu, M.I., Ionescu, R.T., 2024. Learning using generated privileged information by text-to-image diffusion models, in: Proceedings of ICPR, pp. 423–438

    work page 2024

  31. [31]

    Verbal sentiment analysis and detection using recurrent neural network, in: Advanced Data Mining Tools and Methods for Social Computing

    Mohanty, M.D., Mohanty, M.N., 2022. Verbal sentiment analysis and detection using recurrent neural network, in: Advanced Data Mining Tools and Methods for Social Computing. Academic Press, pp. 85–106. 9

    work page 2022

  32. [32]

    Bridging Modalities: Knowledge Distillation and Masked Training for Translating Multi-Modal Emotion Recognition to Uni-Modal, Speech-Only Emotion Recognition

    Muaz, M., Paull, N., Malagavalli, J., 2024. Bridging modalities: Knowledge distillation and masked training for translating multi-modal emotion recognition to uni-modal, speech-only emotion recognition. arXiv preprint arXiv:2401.03000

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  33. [33]

    Naini, A., Goncalves, L., Salman, A., Mote, P., ¨Ulgen, I., Thebaud, T., Moro-Vel´azquez, L., Garcia, L., Dehak, N., Sisman, B., Busso, C.,

    work page

  34. [34]

    4668–4672

    The Interspeech 2025 Challenge on Speech Emotion Recognition in Naturalistic Conditions, in: Proceedings of INTERSPEECH, pp. 4668–4672

    work page 2025

  35. [35]

    No Language Left Behind: Scaling Human-Centered Machine Translation

    NLLB Team, Costa-Juss `a, M.R., Cross, J., C ¸ elebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., et al., 2022. No Language Left Behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  36. [36]

    RoBERTuito: a pre-trained language model for social media text in Spanish, in: Proceedings of LREC, pp

    P ´erez, J.M., Furman, D.A., Alonso Alemany, L., Luque, F.M., 2022. RoBERTuito: a pre-trained language model for social media text in Spanish, in: Proceedings of LREC, pp. 7235–7243

    work page 2022

  37. [37]

    MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations, in: Proceedings of ACL, pp

    Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., Mihalcea, R., 2019. MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations, in: Proceedings of ACL, pp. 527–536

    work page 2019

  38. [38]

    Scaling speech technology to 1,000+languages

    Pratap, V ., Tjandra, A., Shi, B., Tomasello, P., Babu, A., Kundu, S., Elkahky, A., Ni, Z., Vyas, A., Fazel-Zarandi, M., Baevski, A., Adi, Y ., Zhang, X., Hsu, W.N., Conneau, A., Auli, M., 2024. Scaling speech technology to 1,000+languages. Journal of Machine Learning Research 25, 1–52

    work page 2024

  39. [39]

    Robust Speech Recognition via Large-Scale Weak Supervision, in: Proceedings of ICML, pp

    Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I., 2023. Robust Speech Recognition via Large-Scale Weak Supervision, in: Proceedings of ICML, pp. 28492–28518

    work page 2023

  40. [40]

    Cascaded cross-modal transformer for audio-textual classification

    Ristea, N.C., Anghel, A., Ionescu, R.T., 2024. Cascaded cross-modal transformer for audio-textual classification. Artificial Intelligence Review 57, 225

    work page 2024

  41. [41]

    Cascaded cross-modal transformer for request and complaint detection, in: Proceedings of ACMMM, pp

    Ristea, N.C., Ionescu, R.T., 2023. Cascaded cross-modal transformer for request and complaint detection, in: Proceedings of ACMMM, pp. 9467–9471

    work page 2023

  42. [42]

    SepTr: Separable Transformer for Audio Spectrogram Processing, in: Proceedings of INTER- SPEECH, pp

    Ristea, N.C., Ionescu, R.T., Khan, F.S., 2022. SepTr: Separable Transformer for Audio Spectrogram Processing, in: Proceedings of INTER- SPEECH, pp. 4103–4107

    work page 2022

  43. [43]

    An integrated approach for mental health assessment using emotion analysis and scales

    Shanthi, N., Stonier, A.A., Sherine, A., Devaraju, T., Abinash, S., Ajay, R., Arul Prasath, V ., Ganji, V ., 2025. An integrated approach for mental health assessment using emotion analysis and scales. Healthcare Technology Letters 12, e12040

    work page 2025

  44. [44]

    Leveraging pre-trained language model for speech sentiment analysis, in: Proceedings of INTERSPEECH, pp

    Shon, S., Brusco, P., Pan, J., Han, K.J., Watanabe, S., 2021. Leveraging pre-trained language model for speech sentiment analysis, in: Proceedings of INTERSPEECH, pp. 3420–3424

    work page 2021

  45. [45]

    A comparative study on Bengali speech sentiment analysis based on audio data, in: Proceedings of BigComp, pp

    Shruti, A.C., Rifat, R.H., Kamal, M., Alam, M.G.R., 2023. A comparative study on Bengali speech sentiment analysis based on audio data, in: Proceedings of BigComp, pp. 219–226

    work page 2023

  46. [46]

    Multimodal transformer for unaligned multimodal language sequences, in: Proceedings of ACL, pp

    Tsai, Y .H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.P., Salakhutdinov, R., 2019. Multimodal transformer for unaligned multimodal language sequences, in: Proceedings of ACL, pp. 6558–6569

    work page 2019

  47. [47]

    Hierarchical cross-modal attention and dual audio pathways for enhanced multimodal sentiment analysis

    Vamsidhar, D., Desai, P., Shahade, A.K., Patil, S., Deshmukh, P.V ., 2025. Hierarchical cross-modal attention and dual audio pathways for enhanced multimodal sentiment analysis. Scientific Reports 15, 25440

    work page 2025

  48. [48]

    Learning using privileged information: Similarity control and knowledge transfer

    Vapnik, V ., Izmailov, R., 2015. Learning using privileged information: Similarity control and knowledge transfer. Journal of Machine Learning Research 16, 2023–2049

    work page 2015

  49. [49]

    A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding

    Wang, Y ., Boumadane, A., Heba, A., 2021. A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding. arXiv preprint arXiv:2111.02735

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  50. [50]

    Enriching multimodal sentiment analysis through textual emotional descriptions of visual-audio content, in: Proceedings of AAAI, pp

    Wu, S., He, D., Wang, X., Wang, L., Dang, J., 2025. Enriching multimodal sentiment analysis through textual emotional descriptions of visual-audio content, in: Proceedings of AAAI, pp. 1601–1609

    work page 2025

  51. [51]

    A Self-Adjusting Fusion Representation Learning Model for Unaligned Text-Audio Sequences

    Yang, K., Zhang, R., Xu, H., Gao, K., 2022. A self-adjusting fusion representation learning model for unaligned text-audio sequences. arXiv preprint arXiv:2212.11772

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  52. [52]

    Multimodal speech emotion recognition using audio and text, in: Proceedings of SLT, pp

    Yoon, S., Byun, S., Jung, K., 2018. Multimodal speech emotion recognition using audio and text, in: Proceedings of SLT, pp. 112–118

    work page 2018

  53. [53]

    Personality-aware multimodal driver emotion recognition towards intelligent connected vehicles

    Zhang, P., Hu, M., Zhang, H., Wu, C., Yang, Z., 2026. Personality-aware multimodal driver emotion recognition towards intelligent connected vehicles. IEEE Transactions on Affective Computing 17, 801–816. 10

    work page 2026