REVIEW 6 major objections 8 minor 30 references
SpikEmo: Enhancing Emotion Recognition With Spiking Temporal Dynamics in Conversations
T0 review · 6 major / 8 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Spiking timing lifts conversation emotion recognition by 1.5 points
desk verdict Useful incremental ERC paper with a plausible spiking-temporal architecture, but the SOTA claim rests on baseline comparability and single runs that the paper doesn't yet nail down. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is the Dynamic Spiking Weight Adaptation (DSWA) module, whose core is spiking self-attention: $SSA(Q_s,K_s,V_s)=S(\tau Q_sK_s^\top)V_s$, where $S$ is a step function that turns continuous query-key products into sparse spike signals. This converts each modality's feature sequence into spike trains over $T$ time steps, letting the network represent when an emotional cue matters rather than only what it is; softmax-weighted features are then added back through a residual connection. The other supporting mechanism is the combined loss $L = L_{CE} + \lambda_1 L_{DSC} + \lambda_2 L_{corr}$, where the DSC term down-weights easy samples to counter long-tail class imbalance and the correlation term encourages mutually aligned multimodal representations without collapsing their covariance.
What would settle it
Run SpikEmo and the Table II baselines with one shared feature-extraction pipeline and identical training splits; if the weighted-F1 differences shrink to within random-seed noise on either dataset, the claim that spiking temporal modeling causes the improvement is not supported.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that replacing purely continuous sequence modeling with discrete spike-based temporal modeling is enough to push multimodal conversation emotion recognition past prior art without scaling up parameters. SpikEmo extracts per-modality semantic representations, passes each through a spiking Transformer whose self-attention is gated by a step function over scaled query-key products, and re-weights the original features by softmax attention to highlight emotionally salient moments. The final classification fuses the three spiking-modulated modality streams with cross-modal attention and optimizes a combined objective: cross-entropy, a class-balancing dice-style loss, and a correlation loss that aligns modality pairs. Across MELD and IEMOCAP, the paper reports the highest weighted F1 among the discriminative models it compares against, with the largest per-class improvements in tail emotions.
Load-bearing premise
The 1.5-point lead assumes the baseline results in Table II were obtained under the same features, preprocessing, splits, and training protocol as SpikEmo, but the paper does not say so.
Editorial extensions
If this is right
- If the reported gains hold, a 31-million-parameter discriminative model can compete with billion-parameter generative LLM baselines on conversation emotion recognition, making low-resource deployment realistic.
- The spiking temporal module contributes more than fusing extra features: removing it drops weighted F1 by 3.48 points on MELD and 2.47 on IEMOCAP in the paper's ablation.
- The class-balancing loss shifts performance toward tail emotions, with fear on MELD and happiness on IEMOCAP improving beyond the previous best models.
- Trimodal input remains the best configuration, but text alone outperforms audio or video alone, and any combination containing text stays competitive.
Reading between the lines
- A controlled head-to-head with identical feature encoders and preprocessing would be needed to confirm that the gain comes from spike timing rather than from the particular text, audio, and visual features SpikEmo uses.
- The same two-stage semantic-plus-spiking-temporal recipe could be tried on finer-grained affective tasks such as emotion intensity tracking or sentiment shift within a single utterance, where timing matters at a smaller scale.
- Because performance kept rising as the time step grew from 2 to 64, the reported $T=32$ choice may understate what the model can do; larger or adaptive time horizons are a natural next test.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SpikEmo, a discriminative multimodal framework for Emotion Recognition in Conversations. The architecture couples modality-level encoders (RoBERTa for text; OpenSMILE + DialogueRNN for audio; VisExtNet + DialogueRNN for video) with a spiking Transformer module (DSWA/SSA) and a MultiAttn fusion layer, and trains with a composite loss consisting of cross-entropy, a DSC loss, and a correlation loss. On MELD and IEMOCAP, the authors report weighted F1 scores of 65.92 and 71.50, respectively, representing 1.49% and 1.50% improvements over the best discriminative baselines they tabulate; they also emphasize that the 31M-parameter model approaches LLM-based results with far fewer parameters. The paper includes ablations showing that removing the proposed losses or the dynamic spiking module lowers performance, and a modality study showing the benefit of trimodal input.
Significance. If the reported gains are reproducible under matched conditions, SpikEmo would be a useful contribution: it suggests that a compact spiking architecture can be competitive on standard ERC benchmarks and considerably cheaper than LLM alternatives. The paper provides code and a clear ablation structure, and I see no circular derivation; the free parameters are standard validation-set choices. However, as written the empirical evidence does not yet establish the headline claim. The 1.5% margins over heterogeneous baselines are within typical seed-to-seed variation, the baseline rows lack a stated common evaluation protocol, and the spiking contribution is never isolated from the attention architecture itself. These are fixable with additional experiments and reporting, so the work is a suitable candidate for major revision.
major comments (6)
- [Section IV-A.4, Table II] Table II does not state whether the 12 discriminative baselines were run under the same feature encoders, preprocessing, tokenization, and train/validation/test splits as SpikEmo. Since SpikEmo uses RoBERTa text features, OpenSMILE + DialogueRNN audio features, and VisExtNet + DialogueRNN video features, while several listed baselines were published with different encoders, the reported 1.49% and 1.50% margins may be attributable to input features rather than to the spiking temporal model. Please report the provenance of each baseline row and either re-run baselines under the authors' uniform protocol or justify why the original numbers are directly comparable.
- [Section IV-B, Abstract] The paper uses "significantly outperforms" and claims improvements of 1.49% on MELD and 1.50% on IEMOCAP without any repeated runs, error bars, confidence intervals, or significance tests. Weighted-F1 differences of this size on these benchmarks are commonly within run-to-run variability; without at least several seeds per configuration, the quantitative central claim is not statistically supported. Add multi-seed results (mean and standard deviation) and, where feasible, paired tests for the main comparison and the ablations.
- [Section IV-C, Table IV] The ablation labeled "w/oDSWA" removes the entire dynamic contextualized modeling module, so it does not establish that spiking dynamics, as opposed to additional attention parameters or the Transformer structure, cause the improvement. To support the central attribution to "spiking temporal dynamics", the authors should replace SSA with a continuous (non-spiking) self-attention layer of matched capacity and report that comparison; otherwise the measured gain could be due to any temporal attention enhancement.
- [Section IV-A.4, Eqs. (8)-(11)] The loss specification is internally inconsistent. The implementation details list "λ1, λ2, and λ3" as 0.3, 1, 0.4 for MELD and 0.4, 0.6, 1 for IEMOCAP, but Eq. (11) defines only λ1 and λ2; Eq. (8) instead uses α1, α2, α3 for Lcorr. The same paragraph refers to "α and γ for the Lcorr loss" with values 1.5 and 0.5, although α and γ appear in the DSC loss (Eq. (9)), not in Lcorr. These inconsistencies make the reported configuration unreproducible from the text, and they matter because Section IV-C credits the loss combination with part of the performance gain.
- [Section IV-D, Fig. 4] The hyperparameter study reports a clear upward trend in performance as the time step T grows from 2 to 64, yet the main experiments fix T=32. The paper gives no criterion (e.g., a plateau, compute budget, or validation-based selection) for stopping at 32. If T=64 is indeed better, the main result is understated and the chosen configuration is unexplained; please report the T=64 numbers or justify the cutoff.
- [Section I and Section V, Table III] The introduction and conclusion state that SpikEmo surpasses existing state-of-the-art methods "including those based on large language models (LLMs)", but this is contradicted by Table III: on MELD, LoRA + InstructERC with Llama2 achieves 69.15 weighted F1, while SpikEmo achieves 65.92. Please restrict the claim to discriminative models, or frame the LLM comparison strictly in terms of competitive performance at far smaller parameter count.
minor comments (8)
- [Table I] Table I lists MELD as having 6 classes and IEMOCAP as having 7, while Table II shows 7 MELD columns and 6 IEMOCAP columns and the text states MELD has seven emotion categories; please correct the class counts.
- [Eq. (1)] Equation (1) has unbalanced parentheses: the displayed expression "SSA(...)=S(τ · Q_s^m (K_s^m)^T )V_s^m )" should be rewritten for clarity.
- [Section III-B and Eq. (12)] The notation uses \hat{F}^c_i in Eq. (12) for what appears to be the visual modality, which is denoted with superscript v elsewhere; please use consistent modality superscripts throughout the fusion description.
- [Section IV-A.5] Please correct the typo "Metrix" to "Metrics" in the evaluation subsection.
- [Table III] Table III is difficult to read because model names and parameter counts are merged with the "LoRA + Backbone" and "≥ 6B" labels; separate columns and a note on inference time or energy, in addition to parameter count, would support the efficiency claim.
- [References] Reference [29] duplicates reference [2] (DialogueRNN); please use a single citation for this work.
- [Eq. (9)] The DSC loss formula is ambiguous and appears to differ from the standard Dice loss cited as [22]; please clarify the exact expression and the roles of α and γ in both numerator and denominator.
- [Section III-A.2 and Section IV-D] Please clarify how the internal simulation time step T relates to the conversation time axis; as written, T appears to be a neuron time step, which is conceptually different from the dialogue context length that the paper claims to model.
Circularity Check
No significant circularity: SpikEmo's central claim is an empirical benchmark result, not a reduction to fitted constants or self-referential derivation.
full rationale
The paper's central claim is that SpikEmo improves weighted F1 by 1.49% on MELD and 1.50% on IEMOCAP (Section IV-B, Table II). This is an experimentally measured comparison against external baselines, not a quantity derived from the model's own definitions. The method consists of standard feature encoders (RoBERTa, OpenSMILE, VisExtNet), a spiking self-attention module from Spikformer, a MultiAttn fusion module, and losses including DSC loss, each specified by explicit equations (Eqs. 1-13). No parameter is fitted to a target result and then renamed as a prediction; the reported gains are observed outcomes under a fixed training protocol. The ablation study (Table IV) provides an internal control showing that removing LDSC, Lcorr, or DSWA lowers performance, so the architecture's components are not vacuous. Hyperparameter choices such as T=32 and loss weights are tuned on validation sets, which is standard practice and does not constitute circularity. The only self-citation, reference [31], appears in a generic related-work sentence about recent large language models and plays no load-bearing role in the method, evaluation, or conclusions. The baseline comparability concern raised in the reader's take is a legitimate correctness or fairness risk about whether Table II numbers were produced under identical feature extractors and splits, but that is not a circularity in the paper's derivation chain; it is a potential confound in empirical comparison. Under the rule that circularity must be exhibited by a specific reduction or fabricating a fitted input as a prediction, no such step exists here.
Assumptions & free parameters
free parameters (6)
- DSC loss alpha =
1.5
- DSC loss gamma =
0.5
- lambda_1 (LDSC weight) =
0.3 (MELD), 0.4 (IEMOCAP)
- lambda_2 (Lcorr weight) =
1 (MELD), 0.6 (IEMOCAP)
- alpha_1, alpha_2, alpha_3 (Lcorr weights) =
not specified
- time step T =
32
assumptions (3)
- domain assumption The benchmark datasets (MELD, IEMOCAP) and their ground-truth labels are correctly used, including the standard train/validation/test splits.
- domain assumption The spiking neuron model and the SSA layer operate as described in Spikformer [16], and all borrowed components (RoBERTa, OpenSMILE, DialogueRNN, MultiAttn, VisExtNet, DSC loss) behave as in their original publications.
- domain assumption Weighted-F1 is the appropriate evaluation metric and all baselines are measured under the same metric.
Cite this review
Pith. "Pith review of SpikEmo: Enhancing Emotion Recognition With Spiking Temporal Dynamics in Conversations." pith.science (2026). https://pith.science/paper/APOJJZ4Z
@misc{pith2026241113917,
author = {Pith},
title = {Pith review of: SpikEmo: Enhancing Emotion Recognition With Spiking Temporal Dynamics in Conversations},
year = {2026},
howpublished = {\url{https://pith.science/paper/APOJJZ4Z}},
note = {Machine review of arXiv:2411.13917}
}
read the original abstract
In affective computing, the task of Emotion Recognition in Conversations (ERC) has emerged as a focal area of research. The primary objective of this task is to predict emotional states within conversations by analyzing multimodal data including text, audio, and video. While existing studies have progressed in extracting and fusing representations from multimodal data, they often overlook the temporal dynamics in the data during conversations. To address this challenge, we have developed the SpikEmo framework, which is based on spiking neurons and employs a Semantic & Dynamic Two-stage Modeling approach to more precisely capture the complex temporal features of multimodal emotional data. Additionally, to tackle the class imbalance and emotional semantic similarity problems in the ERC tasks, we have devised an innovative combination of loss functions that significantly enhances the model's performance when dealing with ERC data characterized by long-tail distributions. Extensive experiments conducted on multiple ERC benchmark datasets demonstrate that SpikEmo significantly outperforms existing state-of-the-art methods in ERC tasks. Our code is available at https://github.com/Yu-xm/SpikEmo.git.
Figures
Reference graph
Works this paper leans on
-
[1]
Poria S, Cambria E, Hazarika D, et al. Context-dependent sentiment analysis in user-generated videos[C]//Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers). 2017: 873-883
work page 2017
-
[3]
Dialoguegcn: A graph convolu- tional neural network for emotion recognition in conversation[J]
Ghosal D, Majumder N, Poria S, et al. Dialoguegcn: A graph convolu- tional neural network for emotion recognition in conversation[J]. arXiv preprint arXiv:1908.11540, 2019
arXiv 1908
-
[4]
Lu X, Zhao Y , Wu Y , et al. An iterative emotion interaction network for emotion recognition in conversations[C]//Proceedings of the 28th international conference on computational linguistics. 2020: 4078-4088
work page 2020
-
[5]
Wang Y , Zhang J, Ma J, et al. Contextualized emotion recognition in conversation as sequence tagging[C]//Proceedings of the 21th annual meeting of the special interest group on discourse and dialogue. 2020: 186-195
work page 2020
-
[6]
Ishiwatari T, Yasuda Y , Miyazaki T, et al. Relation-aware graph attention networks with relational position encodings for emotion recognition in conversations[C]//Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). 2020: 7360-7370
work page 2020
-
[7]
Li Q, Gkoumas D, Sordoni A, et al. Quantum-inspired neural network for conversational emotion recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(15): 13270-13278
work page 2021
-
[8]
Hu J, Liu Y , Zhao J, et al. MMGCN: Multimodal fusion via deep graph convolution network for emotion recognition in conversation[J]. arXiv preprint arXiv:2107.06779, 2021
arXiv 2021
-
[9]
A multi-view network for real-time emotion recognition in conversations[J]
Ma H, Wang J, Lin H, et al. A multi-view network for real-time emotion recognition in conversations[J]. Knowledge-Based Systems, 2022, 236: 107751
work page 2022
Show all 30 references
-
[10]
Hu D, Hou X, Wei L, et al. MM-DFN: Multimodal dynamic fusion network for emotion recognition in conversations[C]//ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 7037-7041
2022
-
[11]
Shi T, Huang S L. MultiEMO: An attention-based correlation-aware multimodal fusion framework for emotion recognition in conversa- tions[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers). 2023: 14752- 14766
2023
-
[12]
GA2MIF: graph and attention based two-stage multi-source information fusion for conversational emotion detection[J]
Li J, Wang X, Lv G, et al. GA2MIF: graph and attention based two-stage multi-source information fusion for conversational emotion detection[J]. IEEE Transactions on affective computing, 2023, 15(1): 130-143
2023
-
[13]
Dynamic emotion modeling with learn- able graphs and graph inception network[J]
Shirian A, Tripathi S, Guha T. Dynamic emotion modeling with learn- able graphs and graph inception network[J]. IEEE Transactions on Multimedia, 2021, 24: 780-790
2021
-
[14]
DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation[J]
Kim J, Cho J, Park J, et al. DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation[J]. arXiv preprint arXiv:2408.06010, 2024
2024 arXiv
-
[15]
Spiking neural networks[J]
Ghosh-Dastidar S, Adeli H. Spiking neural networks[J]. International journal of neural systems, 2009, 19(04): 295-308
2009
-
[16]
Spikformer: When spiking neural network meets transformer[J]
Zhou Z, Zhu Y , He C, et al. Spikformer: When spiking neural network meets transformer[J]. arXiv preprint arXiv:2209.15425, 2022
2022 arXiv
-
[17]
Spikformer v2: Join the high accuracy club on imagenet with an snn ticket[J]
Zhou Z, Che K, Fang W, et al. Spikformer v2: Join the high accuracy club on imagenet with an snn ticket[J]. arXiv preprint arXiv:2401.02020, 2024
2024 arXiv
-
[18]
Meld: A multimodal multi- party dataset for emotion recognition in conversations[J]
Poria S, Hazarika D, Majumder N, et al. Meld: A multimodal multi- party dataset for emotion recognition in conversations[J]. arXiv preprint arXiv:1810.02508, 2018
2018 arXiv
-
[19]
IEMOCAP: Interactive emotional dyadic motion capture database[J]
Busso C, Bulut M, Lee C C, et al. IEMOCAP: Interactive emotional dyadic motion capture database[J]. Language resources and evaluation, 2008, 42: 335-359
2008
-
[20]
An efficient approach to informative feature extraction from multimodal data[C]//Proceedings of the AAAI Conference on Artificial Intelligence
Wang L, Wu J, Huang S L, et al. An efficient approach to informative feature extraction from multimodal data[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2019, 33(01): 5281-5288
2019
-
[21]
An efficient approach for audio-visual emo- tion recognition with missing labels and missing modalities[C]//2021 IEEE international conference on multimedia and Expo (ICME)
Ma F, Huang S L, Zhang L. An efficient approach for audio-visual emo- tion recognition with missing labels and missing modalities[C]//2021 IEEE international conference on multimedia and Expo (ICME). IEEE, 2021: 1-6
2021
-
[22]
Dice loss for data-imbalanced NLP tasks[J]
Li X, Sun X, Meng Y , et al. Dice loss for data-imbalanced NLP tasks[J]. arXiv preprint arXiv:1911.02855, 2019
1911 arXiv
-
[23]
Focal loss for dense object detec- tion[C]//Proceedings of the IEEE international conference on computer vision
Lin T Y , Goyal P, Girshick R, et al. Focal loss for dense object detec- tion[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2980-2988
2017
-
[24]
Deep long-tailed learning: A survey[J]
Zhang Y , Kang B, Hooi B, et al. Deep long-tailed learning: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(9): 10795-10816
2023
-
[25]
Distribution alignment: A unified frame- work for long-tail visual recognition[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Zhang S, Li Z, Yan S, et al. Distribution alignment: A unified frame- work for long-tail visual recognition[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 2361-2370
2021
-
[26]
Balancing methods for multi- label text classification with long-tailed class distribution[J]
Huang Y , Giledereli B, K ¨oksal A, et al. Balancing methods for multi- label text classification with long-tailed class distribution[J]. arXiv preprint arXiv:2109.04712, 2021
2021 arXiv
-
[27]
Roberta: A robustly optimized bert pretraining approach[J]
Liu Y . Roberta: A robustly optimized bert pretraining approach[J]. arXiv preprint arXiv:1907.11692, 2019
1907 arXiv
-
[28]
Opensmile: the munich versatile and fast open-source audio feature extractor[C]//Proceedings of the 18th ACM international conference on Multimedia
Eyben F, W ¨ollmer M, Schuller B. Opensmile: the munich versatile and fast open-source audio feature extractor[C]//Proceedings of the 18th ACM international conference on Multimedia. 2010: 1459-1462
2010
-
[29]
Dialoguernn: An attentive rnn for emotion detection in conversations[C]//Proceedings of the AAAI conference on artificial intelligence
Majumder N, Poria S, Hazarika D, et al. Dialoguernn: An attentive rnn for emotion detection in conversations[C]//Proceedings of the AAAI conference on artificial intelligence. 2019, 33(01): 6818-6825
2019
-
[30]
Instructerc: Reforming emotion recogni- tion in conversation with a retrieval multi-task llms framework[J]
Lei S, Dong G, Wang X, et al. Instructerc: Reforming emotion recogni- tion in conversation with a retrieval multi-task llms framework[J]. arXiv preprint arXiv:2309.11911, 2023
2023 arXiv
-
[31]
Fake artificial intelligence generated contents (faigc): A survey of theories, detection methods, and oppor- tunities[J]
Yu X, Wang Y , Chen Y , et al. Fake artificial intelligence generated contents (faigc): A survey of theories, detection methods, and oppor- tunities[J]. arXiv preprint arXiv:2405.00711, 2024
2024 arXiv
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.