Pith. sign in

REVIEW 3 major objections 7 minor 47 references

EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EmotionRankCLAP claims that a Rank-N-Contrast loss over valence-arousal labels, instead of symmetric cross-entropy, preserves the ordered structure of speech and text emotion embeddings.

desk verdict A useful, well-controlled empirical paper whose central loss equation appears to be missing a term; as written it cannot reproduce the headline results, but the fix looks easy and the comparison is worth taking seriously. read the letter →

arxiv 2505.23732 v1 pith:ENOUB6WM submitted 2025-05-29 cs.LG

Shreeram Suresh Chandra , Lucas Goncalves , Junchen Lu , Carlos Busso , Berrak Sisman This is my paper
classification cs.LG
keywords speechemotionrecognitionordinalRank-N-Contrastcontrastivelanguage-audiopretrainingvalence-arousalcross-modalretrievalspeakingstyledescriptions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the standard contrastive loss used to align emotional speech with text fails to respect the ordinal nature of emotion, and that replacing it with a supervised Rank-N-Contrast objective over valence-arousal labels produces an embedding space that keeps both modalities ordered by emotional intensity. It proposes EmotionRankCLAP, which pairs a frozen WavLM-based speech-emotion encoder, a frozen DistilRoBERTa text encoder, and LLM-generated speaking-style captions derived from valence-arousal values, then trains the projections with a cross-modal Rank-N-Contrast loss. If the claim is right, speech and text embeddings become closer to each other and better preserve fine-grained emotional ordering, which would improve retrieval, captioning, and other tasks that need continuous emotion understanding. The paper tests this with a cross-modal retrieval test using Kendall's tau for valence and arousal ordinal consistency and reports consistent gains over emotion-CLAP baselines.

What carries the argument

The central object is the cross-modal Rank-N-Contrast loss $L_{RNC-CM}$, a supervised contrastive objective that ranks every possible speech-text pair in a batch by the L2 distance between their valence-arousal labels. For an audio anchor $\hat{E}^a_i$ and a candidate text embedding $\hat{E}^t_j$, the negative set $S_{i,j}$ contains text embeddings whose valence-arousal labels are farther from the anchor than the candidate's; the loss then maximizes the softmax likelihood of closer candidates over farther ones. The paper pairs this loss with LLM-generated speaking-style descriptions whose prompts specify valence and arousal values, so the text side carries continuous ordinal information rather than a categorical emotion name. This combined setup, not the encoders alone, is what the paper argues preserves ordinality and closes the modality gap.

What would settle it

Retrain the same pipeline with a frozen audio encoder that was pretrained without dimensional emotion labels and compare the rank-consistency gap between the Rank-N-Contrast and symmetric cross-entropy losses; if the gap disappears or drops to chance, the ordinal structure comes from the encoder, not from the proposed loss.

Watch

Extended reading notes

Core claim

EmotionRankCLAP's central claim is that the ordinal structure of emotion, such as the fact that disgust and fear are closer to each other than happiness and fear, and that intensity differences within a category matter, should be reflected in the shared audio-text embedding space, and that a cross-modal Rank-N-Contrast loss is the right tool to impose it. Instead of aligning only the matching audio-text pairs as the symmetric cross-entropy loss does, the loss uses every one of the $N \times N$ pairs in a batch, treating a text sample as a positive or negative relative to the audio anchor based on the L2 distance of their valence-arousal labels. This supervision comes from dimensional attributes rather than only categorical labels, with pseudo-captions generated from valence-arousal prompts. The paper's evidence is a retrieval task in which text descriptions at fixed valence or arousal levels retrieve speech utterances, and the retrieved utterances' annotated values are compared to the prompt values with Kendall's tau; EmotionRankCLAP reports higher tau for both valence and arousal, along with lower distributional mismatch between the modalities, than SCE- and SupCon-based baselines.

Load-bearing premise

The load-bearing premise is that the frozen audio encoder already provides embeddings whose geometry reflects valence-arousal ordering; if that encoder carries the ordinal signal, the measured gains may be inherited from pretraining rather than created by the Rank-N-Contrast loss.

Editorial extensions

If this is right

  • Emotion-based CLAP models that keep the symmetric cross-entropy loss are expected to retain a measurable modality gap and weaker ordinal consistency; switching to the Rank-N-Contrast loss narrows that gap.
  • Dimensional-attribute captions carry ordinal signal that categorical captions lack, so future emotion-text alignment systems should condition text generation on valence-arousal values.
  • The learned space should support tasks like emotion retrieval and captioning with finer-than-categorical distinctions, such as separating mild from intense happiness.
  • Because the audio encoder is frozen and only projection layers are trained, the approach can be dropped into existing CLAP pipelines without retraining the backbone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The same rank objective could transfer to other continuous speech or paralinguistic attributes such as dominance, perceived age, or speaking rate, since the loss only needs a scalar or vector label per sample.
  • Editorial inference: The comparison does not isolate the frozen audio encoder's pretraining; if that encoder already orders valence-arousal, the reported ordinality may be largely inherited, and a test with an emotion-agnostic frozen encoder would clarify the loss's own contribution.
  • Editorial inference: A natural next experiment is to check whether the improved ordinal consistency carries over to held-out speakers or unseen description phrasings, rather than only to the prompt template used during evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes EmotionRankCLAP, a supervised cross-modal contrastive learning method that aligns speech and text embeddings while preserving the ordinal structure of valence and arousal. Audio and text are encoded by frozen pretrained encoders (WavLM-based SER model and DistilRoBERTa) and passed through learned projection layers. The training objective is a cross-modal version of Rank-N-Contrast (RNC-CM), in which every audio-text pair is treated as a positive with a rank-dependent set of negatives based on L2 distances in the valence-arousal label plane. The authors generate LLM speaking-style captions from a valence-arousal grid, and they evaluate the method on MSP-Podcast v1.12 using two metrics: cross-modal distribution alignment (MMD and Wasserstein distance) and a cross-modal retrieval test of valence/arousal ordinal consistency (Kendall's tau). They report that EmotionRankCLAP outperforms existing emotion-CLAP baselines, with statistically significant improvements in ordinal consistency.

Significance. If the result holds, the paper offers a reasonable step toward using continuous emotional attributes rather than discrete categories in CLAP-style speech-text alignment. The controlled comparison against CLAP-SCE, which shares the same captions, encoders, and training setup, is a genuine strength and supports the relative claim about the loss function. The proposed evaluation protocol, a text-to-audio retrieval test scored by Kendall's tau, is a useful diagnostic for ordinal structure. However, the central derivation contains an incorrect loss equation, the frozen audio encoder's training data are not disclosed, and the ordinality test uses captions drawn from the same prompt/grid as the training data. These issues jointly limit the reproducibility and the strength of the broader 'bridging' claim, although the relative comparison within the paper remains informative.

major comments (3)
  1. [3.2, Eq. (3)] The denominator on the right-hand side of Eq. (3) sums only over S_i,j, which by definition contains the negatives (text embeddings whose label distance from the audio anchor is strictly greater than that of the positive). The numerator term exp(sim(E_a^i,E_t^j)/tau) is therefore not included in the denominator. As a consequence, P can exceed 1, -log P is negative and unbounded below, and for the farthest text embedding in a batch (for which S_i,j is empty) the loss is a division by zero. The standard Rank-N-Contrast objective (Ref. [16]) includes the positive sample in the denominator and assigns zero loss to the farthest positive pair. Please correct Eq. (3) to include the positive term in the denominator, e.g., exp(sim(a_i,t_j)/tau) / [exp(sim(a_i,t_j)/tau) + sum_{k in S_i,j} exp(sim(a_i,t_k)/tau)], and confirm that the implemented loss matches the corrected form. Without a corrected equation or released code, the reported results in Table 2 are not reproducible from the manuscript as written.
  2. [3.1, 4.1] The frozen audio encoder is described as a 'pre-trained, frozen WavLM-based dimensional SER model' (Ref. [27]), but the paper does not disclose the corpus on which this SER model was trained. If that model was trained on MSP-Podcast or another dimensional-emotion corpus, the audio embedding space already carries valence-arousal ordering before any training of the projection layers. Because CLAP-SCE and the other baselines share the same frozen audio encoder, the relative comparison is still a controlled test of the loss function; however, the paper's broader claim that the proposed method 'bridges' ordinal speech emotion and natural language is overstated without isolating the contribution of the RNC-CM loss from the ordinality inherited from the audio encoder. Please report the SER model's training data and add an ablation with a non-SER audio encoder, or at least discuss this confound explicitly as a limitation.
  3. [4.2.2] The ordinal-consistency test generates the query captions with the same prompt (Figure 2) and the same valence-arousal grid that were used to create the training captions. This makes the retrieval test a same-distribution probe; it does not demonstrate that the embeddings preserve ordinality for captions from other sources, such as human-written descriptions or captions produced by CLAP4emo's pipeline. The relative comparison across models is unaffected, but the absolute VOC/AOC values may be inflated by the model's familiarity with the prompt template. Please evaluate on out-of-distribution captions and clarify whether the 100 lists are per condition (VOC and AOC separately) or total.
minor comments (7)
  1. [4.2.1, Table 1] The text says EmotionRankCLAP 'significantly outperforms' the baselines on MMD and Wasserstein distance, but Table 1 reports no significance indicators or p-values; please add the statistical test used and its results.
  2. [Table 2] CLAP4emo is cited as [10] in Table 2, but the correct reference for CLAP4emo is [13]; reference [10] is a different paper on emotional text-to-speech.
  3. [Figure 1] The figure caption uses 'Positive Pair Corresponding Negative Pair(s)', but in the proposed RNC formulation every text sample is a positive for a given audio anchor, with a rank-dependent negative set; please clarify the terminology to avoid confusion with standard CLAP positive pairs.
  4. [Eq. (3)] The denominator expression contains a formatting error with mismatched parentheses; please ensure all equations have balanced parentheses and consistent notation.
  5. [4.1] Training details do not report random seeds or the number of training runs; since the paper reports mean and standard deviation across evaluation lists, please state whether the model itself was trained once or multiple times.
  6. [4.2.2] The description of the 100 lists is ambiguous: it is unclear whether 100 lists are generated for VOC and another 100 for AOC, or whether the same 100 lists are used for both; please clarify.
  7. [Abstract footnote] The paper states that the generated captions are released, but the only link given is the project page; please include a direct URL or repository for the captions and, if possible, the training code.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: the EmotionRankCLAP vs CLAP-SCE comparison is a controlled ablation; the frozen SER encoder self-citation is an input component, not a circular premise.

full rationale

The paper's central claim is comparative: EmotionRankCLAP (RNC-CM loss) surpasses CLAP-SCE on the same dimensional-attribute captions with the same frozen audio/text encoders and projection architecture. Since the only difference is the loss, the reported MMD/Wasserstein and VOC/AOC differences are not forced by construction. The retrieval test uses 100 newly generated prompt lists and retrieves from the held-out MSP-Podcast test-1 set, with Kendall's tau computed against ground-truth valence/arousal annotations; this is an out-of-sample evaluation of the learned embedding ordering, not a re-statement of the training objective. The only self-citation of note is the frozen WavLM-based dimensional SER model [27] used as the audio encoder. That self-citation is not load-bearing in a circular sense: [27] is a public, code-reproduced challenge baseline, and all baselines in the comparison inherit the same audio geometry, so it cannot manufacture the loss-specific improvement. A caveat for interpretation, not circularity, is that the absolute ordinal structure of audio embeddings is imported from that pretrained SER model and the paper does not disclose its training data; the relative comparison controls for this. Separately, Eq. (3) as printed omits the positive pair from the denominator, making the likelihood unnormalized and the loss undefined for the farthest pair in a batch; this is a correctness/reproducibility defect, not a circularity, and no circularity score is assigned to it.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two fitted or unspecified quantities (temperature, MMD bandwidth) and three domain assumptions about the valence-arousal label space, LLM caption faithfulness, and the pretrained SER audio encoder. No new physical or theoretical entities are introduced.

free parameters (2)
  • Temperature tau = learned (initialized 1.0)
    Learnable scaling in the Rank-N-Contrast loss, trained on validation loss. Its value affects the sharpness of the ranking distribution and the reported retrieval scores.
  • MMD RBF kernel bandwidth = not reported
    Maximum mean discrepancy results depend on the RBF kernel bandwidth; the paper does not state how it was chosen.
assumptions (3)
  • domain assumption Valence-arousal L2 distance is a valid ordinal supervision signal for emotion
    The loss in Eqs. 3-4 ranks all cross-modal pairs by Euclidean distance in the valence-arousal plane; the paper assumes this geometric ordering captures perceived emotion similarity.
  • domain assumption LLM-generated speaking style descriptions track valence-arousal values faithfully
    Section 3.4: captions are generated with OpenAI o1 from a prompt that inserts valence and arousal values; the entire retrieval probe assumes the resulting text semantics vary monotonically with these values.
  • domain assumption Frozen WavLM-based SER encoder embeds speech with meaningful dimensional emotion geometry
    Section 3.1: the audio encoder is taken from a pretrained dimensional SER baseline and never fine-tuned; the method's ordinal retrieval performance depends on this pretrained geometry.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast." pith.science (2026). https://pith.science/paper/ENOUB6WM

@misc{pith2026250523732,
  author       = {Pith},
  title        = {Pith review of: EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ENOUB6WM}},
  note         = {Machine review of arXiv:2505.23732}
}
read the original abstract

Current emotion-based contrastive language-audio pretraining (CLAP) methods typically learn by na\"ively aligning audio samples with corresponding text prompts. Consequently, this approach fails to capture the ordinal nature of emotions, hindering inter-emotion understanding and often resulting in a wide modality gap between the audio and text embeddings due to insufficient alignment. To handle these drawbacks, we introduce EmotionRankCLAP, a supervised contrastive learning approach that uses dimensional attributes of emotional speech and natural language prompts to jointly capture fine-grained emotion variations and improve cross-modal alignment. Our approach utilizes a Rank-N-Contrast objective to learn ordered relationships by contrasting samples based on their rankings in the valence-arousal space. EmotionRankCLAP outperforms existing emotion-CLAP methods in modeling emotion ordinality across modalities, measured via a cross-modal retrieval task.

Figures

Figures reproduced from arXiv: 2505.23732 by the authors.

Figure 2
Figure 2. Prompt used to generate emotional style descriptions based on valence-arousal values. 4. Experiments In this section, we discuss the experimentation settings, the baselines and the evaluations used to probe the properties of the cross-modal embeddings. 4.1. Experimental setup Dataset: We use the MSP-Podcast v1.12 corpus [17] for train￾ing, validation, and testing. Collected from real-world podcasts, it features sign… view at source ↗
Figure 1
Figure 1. Illustration of Rank-N-Contrast in a cross-modal set￾ting. The anchor is boxed in blue. (a) A batch of speech-text pairs along with their valence-arousal labels. (b) Positive and negative pair selection via Rank-N-Contrast criteria. batch to form positive-negative pairs based on a ranking crite￾rion. Each positive pair is assigned corresponding negative pairs according to their similarity ranking, ensuring a structu… view at source ↗
Figure 3
Figure 3. Cross-Modality Emotion Ordinality Test: This figure shows a three-sample example for valence ordinal consistency, while the actual evaluation uses 14 samples per list, repeated across 100 lists for both valence and arousal. value in each list and vary arousal similarly. After generating these lists, we use the trained CLAP model as a retrieval sys￾tem to find the most similar speech utterances for each textual descr… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 37 canonical work pages

  1. [16]

    Don’t classify ratings of affect; rank them!

    H. P. Martinez, G. N. Yannakakis, and J. Hallam, “Don’t classify ratings of affect; rank them!” IEEE transactions on affective com- puting, vol. 5, no. 3, pp. 314–326, 2014

    work page 2014

  2. [27]

    Explaining and mitigat- ing the modality gap in contrastive multimodal learning,

    C. Yaras, S. Chen, P. Wang, and Q. Qu, “Explaining and mitigat- ing the modality gap in contrastive multimodal learning,” arXiv preprint arXiv:2412.07909, 2024

    arXiv 2024

  3. [1]

    EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast

    Introduction The expression and perception of human emotion are inherently continuous in nature [1]. Emotions also possess an ordinal na- ture, as humans are more adept at detecting relative changes in expression rather than identifying absolute emotional states [2]. However, existing paralinguistic models that attempt to capture the ordinality of speech ...

    work page Pith review arXiv 2025

  4. [2]

    Cross-modal contrastive learning Contrastive learning has proven to be an effective approach for aligning multiple modalities in shared representation spaces [5, 14, 18]

    Related work 2.1. Cross-modal contrastive learning Contrastive learning has proven to be an effective approach for aligning multiple modalities in shared representation spaces [5, 14, 18]. While unsupervised contrastive learning relies solely on modality co-occurrence, it can lead to imprecise alignments without capturing task-specific semantic relationsh...

    work page

  5. [3]

    High-valence, high arousal caption

    EmotionRankCLAP We propose EmotionRankCLAP, a supervised cross-modal con- trastive learning framework to align emotional speech with nat- ural language speaking style descriptions in a shared embedding space, leveraging the ordinal nature of speech emotions through a Rank-N-Contrast learning objective. 3.1. Problem Formulation Let {X a i , Xt i } for i ∈ ...

    work page

  6. [4]

    as positive, d(X a 1 , Xt

    work page

  7. [5]

    This makes X t 2 and X t 3 negative samples since d(X a 1 , Xt

    = 0 as both share the same label. This makes X t 2 and X t 3 negative samples since d(X a 1 , Xt

    work page

  8. [6]

    > 0 and d(X a 1 , Xt

    work page

Show all 47 references
  1. [7]

    Similarly, when X t 2 forms a positive pair with X a 1 , X t 3 is a negative sample since d(X a 1 , Xt

    > 0. Similarly, when X t 2 forms a positive pair with X a 1 , X t 3 is a negative sample since d(X a 1 , Xt

  2. [8]

    In this case, X t 1 is not a negative sample since d(X a 1 , Xt

    > d(X a 1 , Xt 2). In this case, X t 1 is not a negative sample since d(X a 1 , Xt

  3. [9]

    < d(X a 1 , Xt 2). Thus, structured relationships emerge: closer positive pairs tend to have more negative samples, reinforcing their closeness, while distant positive pairs have fewer negative samples, reduc- ing their attraction. For a batch of N, we iterate over each X a i ...

  4. [10]

    Experiments In this section, we discuss the experimentation settings, the baselines and the evaluations used to probe the properties of the cross-modal embeddings. 4.1. Experimental setup Dataset: We use the MSP-Podcast v1.12 corpus [17] for train- ing, validation, and testing...

  5. [11]

    We generate natural language speaking style de- scriptions using dimensional attributes of speech emotion and we show that this is crucial in preserving emotion ordinality

    Conclusions This work proposes EmotionRankCLAP, a supervised con- trastive learning approach that leverages the ordinal nature of emotions to learn a cross-modal representation space to align di- mensional speech emotions with corresponding speaking style descriptions. We gene...

  6. [12]

    Acknowledgment This work is supported by NSF CAREER award IIS-2338979

  7. [13]

    Core affect and the psychological construction of emotion

    J. A. Russell, “Core affect and the psychological construction of emotion.” Psychological review, vol. 110, no. 1, p. 145, 2003

    2003

  8. [14]

    The ordinal nature of emotions: An emerging approach,

    G. N. Yannakakis, R. Cowie, and C. Busso, “The ordinal nature of emotions: An emerging approach,” IEEE Transactions on Af- fective Computing, vol. 12, no. 1, pp. 16–35, 2018

    2018

  9. [15]

    Ranking emotional attributes with deep neural networks,

    S. Parthasarathy, R. Lotfian, and C. Busso, “Ranking emotional attributes with deep neural networks,” in2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 4995–4999

    2017

  10. [17]

    Clap learning audio concepts from natural language supervision,

    B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, “Clap learning audio concepts from natural language supervision,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  11. [18]

    Training audio captioning models without audio,

    S. Deshmukh, B. Elizalde, D. Emmanouilidou, B. Raj, R. Singh, and H. Wang, “Training audio captioning models without audio,” in ICASSP 2024 - 2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), 2024, pp. 371–375

    2024

  12. [19]

    Reclap: Improving zero shot audio classifica- tion by describing sounds,

    S. Ghosh, S. Kumar, C. K. R. Evuru, O. Nieto, R. Duraiswami, and D. Manocha, “Reclap: Improving zero shot audio classifica- tion by describing sounds,” CoRR, 2024

    2024

  13. [20]

    Audio retrieval with wavtext5k and clap training,

    S. Deshmukh, B. Elizalde, and H. Wang, “Audio retrieval with wavtext5k and clap training,” in Interspeech 2023 , 2023, pp. 2948–2952

    2023

  14. [21]

    Gemo-clap: Gender-attribute-enhanced contrastive language- audio pretraining for accurate speech emotion recognition,

    Y . Pan, Y . Hu, Y . Yang, W. Fei, J. Yao, H. Lu, L. Ma, and J. Zhao, “Gemo-clap: Gender-attribute-enhanced contrastive language- audio pretraining for accurate speech emotion recognition,” in ICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Pro...

    2024

  15. [22]

    En- hancing emotional text-to-speech controllability with natural lan- guage guidance through contrastive learning and diffusion mod- els,

    X. Jing, K. Zhou, A. Triantafyllopoulos, and B. W. Schuller, “En- hancing emotional text-to-speech controllability with natural lan- guage guidance through contrastive learning and diffusion mod- els,” arXiv preprint arXiv:2409.06451, 2024

    2024 arXiv

  16. [23]

    Prompting audios using acoustic properties for emotion representation,

    H. Dhamyal, B. Elizalde, S. Deshmukh, H. Wang, B. Raj, and R. Singh, “Prompting audios using acoustic properties for emotion representation,” in ICASSP 2024-2024 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11 936–11 940

    2024

  17. [24]

    Paraclap – to- wards a general language-audio model for computational paralin- guistic tasks,

    X. Jing, A. Triantafyllopoulos, and B. Schuller, “Paraclap – to- wards a general language-audio model for computational paralin- guistic tasks,” in Interspeech 2024, 2024, pp. 1155–1159

    2024

  18. [25]

    Clap4emo: Chatgpt-assisted speech emotion retrieval with natural language supervision,

    W.-C. Lin, S. Ghaffarzadegan, L. Bondi, A. Kumar, S. Das, and H.-H. Wu, “Clap4emo: Chatgpt-assisted speech emotion retrieval with natural language supervision,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2024,...

    2024

  19. [26]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763

    2021

  20. [28]

    Rank-n-contrast: learning continuous representations for regression,

    K. Zha, P. Cao, J. Son, Y . Yang, and D. Katabi, “Rank-n-contrast: learning continuous representations for regression,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  21. [29]

    Building naturalistic emotionally bal- anced speech corpus by retrieving emotional speech from existing podcast recordings,

    R. Lotfian and C. Busso, “Building naturalistic emotionally bal- anced speech corpus by retrieving emotional speech from existing podcast recordings,” IEEE Transactions on Affective Computing, vol. 10, no. 4, pp. 471–483, 2017

    2017

  22. [30]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (I...

    2023

  23. [31]

    Supervised contrastive learning,

    P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y . Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” Advances in neural information processing systems , vol. 33, pp. 18 661–18 673, 2020

    2020

  24. [32]

    Emotion- aligned contrastive learning between images and music,

    S. Stewart, K. Avramidis, T. Feng, and S. Narayanan, “Emotion- aligned contrastive learning between images and music,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 8135– 8139

    2024

  25. [33]

    Audiocaps: Generat- ing captions for audios in the wild,

    C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generat- ing captions for audios in the wild,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Paper...

    2019

  26. [34]

    Llm supervised pre-training for mul- timodal emotion recognition in conversations,

    S. Dutta and S. Ganapathy, “Llm supervised pre-training for mul- timodal emotion recognition in conversations,” arXiv preprint arXiv:2501.11468, 2025

    2025 arXiv

  27. [35]

    Empower typed descriptions by large language models for speech emotion recognition,

    H. Wu, H.-C. Chou, K.-W. Chang, L. Goncalves, J. Du, J.-S. R. Jang, C.-C. Lee, and H.-Y . Lee, “Empower typed descriptions by large language models for speech emotion recognition,” in 2024 Asia Pacific Signal and Information Processing Association An- nual Summit and Conferenc...

    2024

  28. [36]

    Secap: Speech emotion captioning with large language model,

    Y . Xu, H. Chen, J. Yu, Q. Huang, Z. Wu, S.-X. Zhang, G. Li, Y . Luo, and R. Gu, “Secap: Speech emotion captioning with large language model,” in Proceedings of the AAAI Conference on Ar- tificial Intelligence, vol. 38, no. 17, 2024, pp. 19 323–19 331

    2024

  29. [37]

    Aligncap: Aligning speech emotion captioning to human preferences,

    Z. Liang, H. Shi, and H. Chen, “Aligncap: Aligning speech emotion captioning to human preferences,” arXiv preprint arXiv:2410.19134, 2024

    2024 arXiv

  30. [38]

    Instructtts: Modelling expressive tts in discrete latent space with natural lan- guage style prompt,

    D. Yang, S. Liu, R. Huang, C. Weng, and H. Meng, “Instructtts: Modelling expressive tts in discrete latent space with natural lan- guage style prompt,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  31. [39]

    Odyssey 2024-speech emotion recognition challenge: Dataset, baseline framework, and results,

    L. Goncalves, A. N. Salman, A. R. Naini, L. M. Velazquez, T. Thebaud, L. P. Garcia, N. Dehak, B. Sisman, and C. Busso, “Odyssey 2024-speech emotion recognition challenge: Dataset, baseline framework, and results,”Development, vol. 10, no. 9,290, pp. 4–54, 2024

    2024

  32. [40]

    Emotion english distilroberta-base,

    J. Hartmann, “Emotion english distilroberta-base,” https:// huggingface.co/j-hartmann/emotion-english-distilroberta-base/, 2022

    2022

  33. [41]

    Iemocap: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, pp. 335–359, 2008

    2008

  34. [42]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. [Online]. Available: https://arxiv. org/abs/2303.08774

    2023 arXiv

  35. [43]

    Crowdsourcing a word- emotion association lexicon,

    S. M. Mohammad and P. D. Turney, “Crowdsourcing a word- emotion association lexicon,”Computational Intelligence, vol. 29, no. 3, pp. 436–465, 2013

    2013

  36. [44]

    Introducing Parsel- mouth: A Python interface to Praat,

    Y . Jadoul, B. Thompson, and B. de Boer, “Introducing Parsel- mouth: A Python interface to Praat,” Journal of Phonetics , vol. 71, pp. 1–15, 2018

    2018

  37. [45]

    A kernel two-sample test,

    A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Sch ¨olkopf, and A. Smola, “A kernel two-sample test,”Journal of Machine Learn- ing Research, vol. 13, no. Mar, pp. 723–773, 2012

    2012

  38. [46]

    Computational optimal transport,

    G. Peyr ´e and M. Cuturi, “Computational optimal transport,”Foun- dations and Trends in Machine Learning , vol. 11, no. 5-6, pp. 355–607, 2019

    2019

  39. [47]

    A new measure of rank correlation,

    M. G. Kendall, “A new measure of rank correlation,” Biometrika, vol. 30, no. 1/2, pp. 81–93, 1938. [Online]. Available: https://doi.org/10.2307/2332226

    1938 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.