Enhancing Speech Emotion Recognition Leveraging Aligning Timestamps of ASR Transcripts and Speaker Diarization

Berlin Chen; Hsuan-Yu Wang; Pei-Ying Lee

REVIEW 3 major objections 5 minor 20 references

Enhancing Speech Emotion Recognition Leveraging Aligning Timestamps of ASR Transcripts and Speaker Diarization

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Aligning ASR word timestamps with speaker diarization lifts speech emotion accuracy from 56.8% to 66.8% on the four-class IEMOCAP benchmark.

desk verdict Sensible, well-described alignment pipeline, but the headline claim is confounded: the only 'without alignment' number equals the with-alignment frozen run, so fine-tuning alone could explain the gain. read the letter →

arxiv 2507.19356 v1 pith:DCCAVS6A submitted 2025-07-25 cs.CL cs.SDeess.AS

Hsuan-Yu Wang , Pei-Ying Lee , Berlin Chen This is my paper · ORCID

classification cs.CLcs.SDeess.AS

keywords speechemotionrecognitiontimestampalignmentspeakerdiarizationautomaticmultimodalfusioncross-attentionconversationalturnreconstructionIEMOCAP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that synchronizing automatic speech recognition transcripts (what was said) with speaker diarization (who spoke when), before emotion classification, is a decisive factor in conversational speech emotion recognition. It builds a preprocessing pipeline that flattens word-level ASR output into a single stream, merges consecutive words into turns when the speaker label stays the same and the pause stays under 1.5 seconds, and attributes each reconstructed turn to one speaker. With aligned turns as input to a text-audio cross-attention model, accuracy rises from 56.82% to 66.81% and macro F1 from 47.10% to 66.48% on the four-class IEMOCAP benchmark. The authors read this as evidence that upstream temporal alignment matters as much as the fusion architecture, because fragmented input hides emotions that unfold over longer stretches of speech, most visibly sadness.

What carries the argument

The load-bearing mechanism is a three-stage timestamp alignment block. It first flattens word-level ASR outputs that carry start and end timestamps into a single chronological word stream. It then groups consecutive words into a conversational turn whenever two conditions hold: the diarization speaker label is unchanged and the pause between words is no more than 1.5 seconds; a speaker change or a longer pause starts a new turn. Finally, each turn is packaged as one segment whose timestamp spans the first word's start to the last word's end, with its speaker label attached. A second mechanism, the cross-attention gated fusion block, lets the text and audio segment embeddings attend to each other in both directions, then a forget gate blends each attended representation with its original before a linear classifier; this part is held fixed across the aligned and unaligned conditions, so the reported gains isolate the alignment step.

What would settle it

Measure the precision and recall of the pipeline's reconstructed turns against IEMOCAP's manual turn boundaries, and re-run the model with pause thresholds of 0.5s, 1.0s, 1.5s, 2.0s, and 3.0s; poor turn-reconstruction quality or large accuracy swings across thresholds would show the reported gain is not due to faithful turn recovery.

Watch

Extended reading notes

Core claim

The central claim is that in automatic multimodal speech emotion recognition, the step that aligns ASR transcripts with speaker diarization timestamps is not a minor preprocessing detail but a primary determinant of accuracy. The paper argues that when word-level ASR output is flattened and regrouped into speaker-consistent turns, the resulting segments give the downstream text-audio fusion model the contextual completeness it needs. The evidence is the comparison on the four-class IEMOCAP task: the aligned pipeline reaches 66.81% accuracy and 66.48% macro F1, versus 56.82% and 47.10% for the same model fed with a coarser segment-overlap association. The paper also claims the gain is concentrated in emotions requiring longer temporal context, with sadness F1 rising from 0.26 to 0.67. Finally, with oracle segmentation boundaries, its aligned pipeline's time-weighted emotion error rate falls to 53.90%, below the 66.03% reported for the earlier integrated pipeline, which the paper reads as showing that voice-activity detection is the remaining bottleneck.

Load-bearing premise

The entire improvement rests on the assumption that the pre-trained models' word timestamps and speaker labels are accurate enough that grouping same-speaker words separated by less than 1.5 seconds of silence reconstructs true conversational turns.

Editorial extensions

If this is right

Conversational SER systems should treat turn reconstruction as a core pipeline stage rather than optional cleanup: alignment alone moves macro F1 from 47.10 to 66.48 on the paper's four-class setup.
Emotions expressed over long durations, especially sadness, benefit most: sadness F1 rises from 0.26 to 0.67 when fragmented words are merged into complete turns.
Voice-activity detection errors, not the fusion model, are the dominant error source in the automatic pipeline: oracle segment boundaries lower TEER to 53.90%, below the 66.03% reported for the earlier integrated baseline.
Fine-tuning the pretrained embedding extractors is required to realize the alignment gain, since freezing them keeps accuracy at 56.82%, the same value as the unaligned baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not measure whether its reconstructed turns actually coincide with true conversational turns; a necessary validation is to score turn-boundary precision and recall against manual annotations before attributing the accuracy gain to faithful turn recovery.
The 1.5-second pause threshold is asserted rather than tuned, so a threshold sweep is a natural robustness test: if performance varies sharply across 0.5s to 3.0s thresholds, the pipeline's benefit is threshold-specific rather than structural.
Because the alignment is isolated with a fixed fusion architecture, the same preprocessing gain should be tested with simpler fusion (e.g., concatenation) to see whether the benefit is architecture-independent.
The turn-reconstruction logic may transfer to other turn-level language tasks such as conversational sentiment or dialogue summarization, but that is an extension beyond the paper's evidence.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the quick take. The paper builds a sensible preprocessing pipeline—WhisperX word timestamps, Pyannote speaker segments, a pause-threshold grouping into speaker-consistent turns—and feeds the resulting segments into a gated cross-attention fusion of wav2vec 2.0 and RoBERTa for IEMOCAP emotion classification. The writeup is clear, the method is modular, and they ship code. That part is genuinely useful.

What it does well: the alignment procedure is described at a level that can be reimplemented, and the paper is honest about prior work showing misalignment hurts SER. The per-emotion gain for 'sad' is plausible if alignment restores longer context.

But the central claim doesn't survive contact with the tables. Table I's 'Without Alignment' row is 56.82 accuracy and 47.10 macro-F1. Table IV's 'With Alignment, Frozen Embeddings' row is exactly 56.82 and 47.10. Their proposed approach (66.81/66.48) is exactly the fine-tuned embeddings row. So either alignment provides zero benefit in the frozen regime, or the 'without alignment' baseline was actually an alignment+frozen run that was relabeled. Section IV.B says both frozen and fine-tuned experiments were run for both the with- and without-alignment setups, but the without-alignment fine-tuned cell is missing. That's the decisive confound: fine-tuning the extractors alone can explain the entire reported gain.

The VAD-Oracle comparison in Table III is also not a fair comparison—it uses ground-truth boundaries, so it's an upper bound, and it's compared against Wu et al. on a different number of emotion categories. No error bars or significance tests anywhere. The 1.5-second pause threshold is a free parameter with no sensitivity analysis, which is minor given the confound.

I don't think the authors are being deceptive; the pipeline itself is reasonable and they point to the right related work. But as reported, the headline claim—that timestamp alignment improves accuracy—is unidentified. The fix is simple on paper: run the missing cell, add error bars, and compare under matched conditions.

Who's this for? People building automatic conversational SER front-ends. It's a useful shred of engineering insight if the ablation gets fixed. Right now, I'd send it to peer review because the question matters and the code is there, but I'd expect major revision before acceptance.

Referee Report

3 major / 5 minor

Summary. The paper proposes a preprocessing alignment pipeline that combines WhisperX word-level ASR timestamps with Pyannote speaker diarization segments, grouping words into speaker-attributed turns using speaker consistency and a 1.5-second pause threshold. It then trains a multimodal SER model with RoBERTa text embeddings and wav2vec 2.0 audio embeddings fused by cross-attention and a forget gate, followed by a linear classifier. Experiments on the IEMOCAP 4-class subset (happy, sad, angry, neutral) report 66.81% accuracy and 66.48% macro F1 for the proposed aligned pipeline versus 56.82% accuracy and 47.10% macro F1 for a without-alignment baseline, along with TEER and sTEER comparisons. The authors conclude that precise timestamp alignment significantly improves SER accuracy.

Significance. The contribution is a modular, task-agnostic alignment front end that could be plugged into any downstream SER model, and the authors state that code is available. The paper also engages with duration-aware metrics (TEER, sTEER), which is appropriate for automatically segmented conversational speech. However, the central empirical claim is not cleanly supported by the experiments as reported: the key comparison varies alignment and fine-tuning simultaneously, and the baseline row in Table I is numerically identical to the with-alignment frozen-embedding row in Table IV. If the missing no-alignment fine-tuned result is supplied and the comparison is unconfounded, the finding would be useful; as it stands, the claimed causal effect of alignment is unidentified. I therefore cannot recommend acceptance on the current evidence.

major comments (3)

[Table I vs. Table IV; Section IV.B] The central claim that alignment improves SER is not supported because the 'Without Alignment' baseline in Table I has exactly the same accuracy (56.82%) and macro F1 (47.10%) as the 'With Alignment, Frozen Embeddings' row in Table IV. Section IV.B says frozen and fine-tuned experiments were run for both with-alignment and without-alignment setups, but no without-alignment fine-tuned result is presented. The 2x2 design (alignment x extractor training) is therefore incomplete, and the 10-point gain attributed to alignment could be entirely due to fine-tuning wav2vec 2.0 and RoBERTa on IEMOCAP. If the baseline is truly without-alignment and frozen, then alignment yields no benefit in the frozen regime; if the baseline is actually a relabeled with-alignment frozen run, then no valid without-alignment condition exists. The authors must report all four cells, with multiple seeds and error bars or significance tests, before the alignment effect can be claimed.
[Table III; Section V] The TEER/sTEER analysis does not isolate the alignment contribution. Table III compares a 'Standard Pipeline' with a 'VAD-Oracle Pipeline' in which ground-truth boundaries replace internal VAD, but no TEER/sTEER values are given for the 'Without Alignment' baseline or for the 'Proposed Approach' from Table I. The large drop in TEER when VAD is replaced shows the effect of segmentation quality, not the effect of the proposed timestamp alignment. In addition, the comparison to Wu et al. is not direct because that baseline was evaluated on six emotion categories whereas this paper uses four; the authors should provide an equivalent-condition comparison or explicitly state the limitation.
[Section III.A, Figure 2, Table II] The mechanistic claim that turn reconstruction drives the improvement is not validated. The grouping step relies on a hand-set 1.5-second pause threshold and on the accuracy of WhisperX word timestamps and Pyannote speaker segments, yet the paper reports no measurement of alignment accuracy, turn-boundary precision/recall, or sensitivity to the pause threshold. Without such diagnostics, the large per-emotion gains in Table II (e.g., sad F1 from 0.26 to 0.67) cannot be attributed to turn coherence rather than to other confounding factors. A threshold sweep and a comparison against oracle turn boundaries would make the mechanism credible.

minor comments (5)

[Abstract and Section III.B] The label 'Wav2Vec' in the abstract should be made consistent with 'wav2vec 2.0' used in Section III.B.
[Table I vs. Table IV] The proposed approach's weighted F1 is listed as 66.88% in Table I but the corresponding fine-tuned row in Table IV lists 66.81%; the discrepancy should be reconciled.
[Equations (1) and (2)] TEER and sTEER are defined only symbolically; the manuscript should specify how MS, FA, CONFemo, CONFemo+spk, and TOTAL are computed and how these metrics are evaluated on the 4-class subset.
[Section III.B] The phrase 'which is implemented as a Transformer-based architecture' appears twice in consecutive sentences; the redundancy should be removed.
[References] Several references cite arXiv preprints without venue information (e.g., [4], [10]); please ensure bibliographic entries are complete and consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the proposed pipeline is evaluated on an external benchmark, and the main weakness is a confounded baseline comparison rather than a derivation that reduces to its own inputs.

full rationale

The paper's contribution is an engineering pipeline (WhisperX word timestamps plus Pyannote diarization, grouped by speaker and pause into turns, followed by RoBERTa and wav2vec 2.0 embeddings with cross-attention fusion), and there is no place where a derived quantity is defined in terms of the thing it is supposed to predict. The 1.5-second pause threshold is a hand-set hyperparameter, not fitted to the emotion labels or to the reported SER numbers, and the system is evaluated on the external IEMOCAP benchmark rather than on its own training objective. No load-bearing claim rests on a citation to the authors' own prior work; all citations are to external methods or openly available tools. The most serious weakness is that Table I's 'Without Alignment' numbers coincide exactly with Table IV's 'With Alignment + Frozen Embeddings' numbers, while no without-alignment fine-tuned cell is reported; this is a confounded experimental comparison (alignment and fine-tuning vary together) and a threat to the causal attribution, but it is not a circular derivation because the reported improvement is not equivalent to the input by construction. The VAD-oracle comparison is likewise a benchmarking choice, not a circular step. Therefore no circularity is identified.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper introduces one hand-set free parameter (pause threshold) and relies on several domain assumptions about the accuracy of upstream ASR, diarization, and embedding models. No new entities such as particles, forces, or dimensions are introduced. The main unstated load is that the upstream tool outputs and the heuristic grouping recover true conversational turns, which is never validated.

free parameters (1)

pause threshold for turn grouping = 1.5 seconds
Introduced in Section III.A as the predefined pause threshold that controls when words are merged into turns. It is hand-selected, not swept or justified, and directly affects segment length and thus SER input.

assumptions (5)

domain assumption IEMOCAP emotion labels, speaker IDs, and turn boundaries are reliable ground truth.
Section IV.A uses IEMOCAP annotations as the evaluation reference without questioning their reliability; this underlies all reported accuracies and F1 scores.
domain assumption WhisperX word-level timestamps are accurate enough for the alignment pipeline.
Section III.A relies on word timestamps (e.g., [6.92s to 7.16s]) to reconstruct turns; no timestamp error analysis is provided.
domain assumption Pyannote speaker diarization labels are accurate enough to assign speakers to words.
Section III.A merges words based on consistent speaker labels; the paper does not measure diarization error or its effect on turn reconstruction.
domain assumption Mean pooling of wav2vec 2.0 and RoBERTa output embeddings preserves enough information for emotion classification.
Section III.B pools all audio and text embeddings into a single vector before fusion; the paper provides no evidence that this compression retains emotion-relevant details.
domain assumption The cross-modality gated attention fusion of [4], adapted from three modalities to two, is effective for SER.
Section III.B borrows the fusion mechanism as-is; no comparison to simpler fusion baselines (e.g., concatenation) is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing Speech Emotion Recognition Leveraging Aligning Timestamps of ASR Transcripts and Speaker Diarization." pith.science (2026). https://pith.science/paper/DCCAVS6A

@misc{pith2026250719356,
  author       = {Pith},
  title        = {Pith review of: Enhancing Speech Emotion Recognition Leveraging Aligning Timestamps of ASR Transcripts and Speaker Diarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DCCAVS6A}},
  note         = {Machine review of arXiv:2507.19356}
}

read the original abstract

In this paper, we investigate the impact of incorporating timestamp-based alignment between Automatic Speech Recognition (ASR) transcripts and Speaker Diarization (SD) outputs on Speech Emotion Recognition (SER) accuracy. Misalignment between these two modalities often reduces the reliability of multimodal emotion recognition systems, particularly in conversational contexts. To address this issue, we introduce an alignment pipeline utilizing pre-trained ASR and speaker diarization models, systematically synchronizing timestamps to generate accurately labeled speaker segments. Our multimodal approach combines textual embeddings extracted via RoBERTa with audio embeddings from Wav2Vec, leveraging cross-attention fusion enhanced by a gating mechanism. Experimental evaluations on the IEMOCAP benchmark dataset demonstrate that precise timestamp alignment improves SER accuracy, outperforming baseline methods that lack synchronization. The results highlight the critical importance of temporal alignment, demonstrating its effectiveness in enhancing overall emotion recognition accuracy and providing a foundation for robust multimodal emotion analysis.

Figures

Figures reproduced from arXiv: 2507.19356 by the authors.

**Figure 1.** Proposed Model ASR model, while another approach [15] uses encoderdecoder attractors to jointly infer transcripts and speaker turns. [16] formally introduced the Speech Emotion Diarization (SED) task and proposed the EDER metric to assess timesensitive accuracy. [17] demonstrated that syntactic alignment cues further improve speaker attribution in realistic meetings. While effective, these methods are often tight… view at source ↗

**Figure 2.** Timestamp Alignment Block • Flatten Word Stream: The process begins by integrating the output from the upstream ASR and diarization stage. All speaker-attributed words, complete with their individual timestamps, are extracted from their initial segments and organized into a single, chronologically ordered stream. This step effectively dissolves the preliminary, and often fragmented, ASR segment boundaries to create… view at source ↗

**Figure 3.** Transcript Comparison: without vs. with Alignment [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 13 canonical work pages

[1]

Integrating emotion recognition with speech recognition and speaker diarisation for conversations,

W. Wu, C. Zhang, and P. C. Woodland, “Integrating emotion recognition with speech recognition and speaker diarisation for conversations,” in Proc. Interspeech, pp. 3607–3611, Aug. 2023

work page 2023
[2]

RoBERTa: A robustly optimized BERT pretraining approach,

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692, July 2019

arXiv 1907
[3]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” arXiv preprint arXiv:2006.11477, Oct. 2020

arXiv 2006
[4]

Cross-Modality Gated Attention Fusion for Multimodal Sentiment Analysis

M. Jiang and S. Ji, “Cross-modality gated attention fusion for multi- modal sentiment analysis,” arXiv preprint arXiv:2208.11893, Aug. 2022

work page Pith review arXiv 2022
[5]

IEMOCAP: In- teractive emotional dyadic motion capture database,

C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Provost, S. Kim, J. Chang, S. Lee, and S. Narayanan, “IEMOCAP: In- teractive emotional dyadic motion capture database,” Language Resources and Evaluation, vol. 42, pp. 335–359, 2008

work page 2008
[6]

WhisperX: Time accurate speech transcription of long form audio,

M. Bain, J. Huh, T. Han, and A. Zisserman, “WhisperX: Time accurate speech transcription of long form audio,” in Proc. Interspeech, pp. 4489– 4493, Aug. 2023, doi:10.21437/Interspeech.2023-78

doi:10.21437/interspeech.2023-78 2023
[7]

Speech emotion recognition with ASR transcripts: A comprehensive study on word error rate and fusion techniques,

Y . Li, P. Bell, and C. Lai, “Speech emotion recognition with ASR transcripts: A comprehensive study on word error rate and fusion techniques,” in Proc. SLT , pp. 518–525, Macao, 2024, doi:10.1109/SLT61566.2024.10832143

arXiv 2024
[8]

emoDARTS: Joint Optimisation of CNN & Sequential Neural Network Architectures for Superior Speech Emotion Recognition

Thejan Rajapakshe, Rajib Rana, Sara Khalifa, Berrak Sisman, Bjorn W. Schuller, Carlos Busso, “emoDARTS: Joint optimisation of CNN and sequential neural network architectures for superior speech emotion recognition,” arXiv preprint arXiv:2403.14083 , Mar. 2024. 1https://github.com/M4RC034/SER-Align.git

work page Pith review arXiv 2024

Show all 20 references

[9]

Speech sentiment analysis via pre- trained features from end-to-end ASR models,

W. Lu, Y . Chen, and X. Chen, “Speech sentiment analysis via pre- trained features from end-to-end ASR models,” in Proc. ICASSP , pp. 7174–7178, 2020

2020
[10]

Learning discriminative features from spectrograms using center loss for speech emotion recognition,

Z. Li, Y . Wang, and T. Chen, “Learning discriminative features from spectrograms using center loss for speech emotion recognition,” arXiv preprint arXiv:2501.01103, Jan. 2025

2025 arXiv
[11]

Multimodal multi-loss fusion network for sentiment analysis,

Z. Wu, Z. Gong, J. Koo, and J. Hirschberg, “Multimodal multi-loss fusion network for sentiment analysis,” in Proc. NAACL HLT , vol. 1, pp. 3588–3602, June 2024

2024
[12]

Speech emotion recognition in dyadic dialogues with attentive interaction modeling,

J. Zhao, S. Chen, J. Liang, and Q. Jin, “Speech emotion recognition in dyadic dialogues with attentive interaction modeling,” in Proc. Inter- speech, pp. 1671–1675, 2019

2019
[13]

Temporal context in speech emotion recognition,

Yangyang Xia, Li-Wei Chen, Alexander I. Rudnicky, Richard M Stern et al. , “Temporal context in speech emotion recognition,” in Proc. Interspeech, pp. 3705-3709, 2021

2021
[14]

Transcribe-to-diarize: Neural speaker diarization for unlimited number of speakers using end-to-end speaker-attributed ASR,

Naoyuki Kanda, Xiong Xiao, Yashesh Gaur, Xiaofei Wang, Zhong Meng, Zhuo Chen, Takuya Yoshioka, “Transcribe-to-diarize: Neural speaker diarization for unlimited number of speakers using end-to-end speaker-attributed ASR,” in Proc. ICASSP, pp. 7193–7197, May 2022

2022
[15]

Streaming speaker-attributed ASR with token-level speaker embeddings

Kanda, N., Wu, J., Wu, Y ., Xiao, X., Meng, Z., Wang, X., Gaur, Y ., Chen, Z., Li, J., & Yoshioka, T. Streaming speaker-attributed ASR with token-level speaker embeddings. arXiv preprint arXiv:2203.16685. 2022

2022 arXiv
[16]

Speech emotion diarization: Which emotion appears when?,

Y . Wang, M. Ravanelli, and A. Yacoubi, “Speech emotion diarization: Which emotion appears when?,” in Proc. ASRU, pp. 125–132, 2023

2023
[17]

Meeting recognition with continuous speech separation and transcription-supported diarization,

Thilo von Neumann, Christoph Boeddeker, Tobias Cord-Landwehr, Marc Delcroix, Reinhold Haeb-Umbach, “Meeting recognition with continuous speech separation and transcription-supported diarization,” in Proc. HSCMA Workshop at ICASSP , May 2024

2024
[18]

pyannote.audio: neural building blocks for speaker diarization,

Herv ´e Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, Marie-Philippe Gill, “pyannote.audio: neural building blocks for speaker diarization,” in ICASSP, 2020

2020
[19]

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,

N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,”arXiv preprint arXiv:1908.10084, Aug

1908 arXiv
[2019]

2019, Accepted at EMNLP-IJCNLP 2019]

[Revised: Nov. 2019, Accepted at EMNLP-IJCNLP 2019]

2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Integrating emotion recognition with speech recognition and speaker diarisation for conversations,

W. Wu, C. Zhang, and P. C. Woodland, “Integrating emotion recognition with speech recognition and speaker diarisation for conversations,” in Proc. Interspeech, pp. 3607–3611, Aug. 2023

work page 2023

[2] [2]

RoBERTa: A robustly optimized BERT pretraining approach,

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692, July 2019

arXiv 1907

[3] [3]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” arXiv preprint arXiv:2006.11477, Oct. 2020

arXiv 2006

[4] [4]

Cross-Modality Gated Attention Fusion for Multimodal Sentiment Analysis

M. Jiang and S. Ji, “Cross-modality gated attention fusion for multi- modal sentiment analysis,” arXiv preprint arXiv:2208.11893, Aug. 2022

work page Pith review arXiv 2022

[5] [5]

IEMOCAP: In- teractive emotional dyadic motion capture database,

C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Provost, S. Kim, J. Chang, S. Lee, and S. Narayanan, “IEMOCAP: In- teractive emotional dyadic motion capture database,” Language Resources and Evaluation, vol. 42, pp. 335–359, 2008

work page 2008

[6] [6]

WhisperX: Time accurate speech transcription of long form audio,

M. Bain, J. Huh, T. Han, and A. Zisserman, “WhisperX: Time accurate speech transcription of long form audio,” in Proc. Interspeech, pp. 4489– 4493, Aug. 2023, doi:10.21437/Interspeech.2023-78

doi:10.21437/interspeech.2023-78 2023

[7] [7]

Speech emotion recognition with ASR transcripts: A comprehensive study on word error rate and fusion techniques,

Y . Li, P. Bell, and C. Lai, “Speech emotion recognition with ASR transcripts: A comprehensive study on word error rate and fusion techniques,” in Proc. SLT , pp. 518–525, Macao, 2024, doi:10.1109/SLT61566.2024.10832143

arXiv 2024

[8] [8]

emoDARTS: Joint Optimisation of CNN & Sequential Neural Network Architectures for Superior Speech Emotion Recognition

Thejan Rajapakshe, Rajib Rana, Sara Khalifa, Berrak Sisman, Bjorn W. Schuller, Carlos Busso, “emoDARTS: Joint optimisation of CNN and sequential neural network architectures for superior speech emotion recognition,” arXiv preprint arXiv:2403.14083 , Mar. 2024. 1https://github.com/M4RC034/SER-Align.git

work page Pith review arXiv 2024