Pith. sign in

REVIEW 4 major objections 4 minor 65 references

Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Deep canonical correlation analysis fuses EEG and eye-movement signals into a coordinated hyperspace and, with weighted-sum fusion and an SVM classifier, reports state-of-the-art emotion recognition accuracy on all five benchmark datasets…

desk verdict A sincere DCCA-for-emotion-fusion paper whose two new headline numbers are biased by grid-search on the evaluation folds; the rest is solid enough to referee, but the SOTA claim needs re-validation. read the letter →

arxiv 1908.05349 v1 pith:S5WM7HZV submitted 2019-08-13 cs.LG cs.CV

Wei Liu , Jie-Lin Qiu , Wei-Long Zheng , Bao-Liang Lu This is my paper · ORCID
classification cs.LGcs.CV
keywords deepcanonicalcorrelationanalysismultimodalemotionrecognitionEEGeyemovementfusionaffectivecomputingnoiserobustnessmutualinformationneuralestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that deep canonical correlation analysis (DCCA) is a better way to fuse physiological modalities for emotion recognition than concatenation or autoencoder-based fusion. Rather than feeding EEG and eye-movement features into a single joint model, DCCA learns a separate neural transformation for each modality and coordinates the two outputs under a canonical-correlation constraint, forming a shared hyperspace. The paper reports the highest recognition accuracies on all five benchmark datasets it evaluates, including 94.58% on SEED and 83.08% on SEED-V, and shows that the method remains accurate when noise is added to the input features. If the results hold, coordinated representation learning gives affective computing a practical fusion front-end that keeps per-modality representations explicit while concentrating shared, emotion-relevant information.

What carries the argument

The load-bearing object is deep canonical correlation analysis (DCCA), a coordinated-representation architecture that sends each modality through its own multilayer network and then scores the pair of outputs with the classical canonical-correlation objective. Concretely, after centering, the outputs $O_1$ and $O_2$ are scored by the total correlation $\operatorname{corr}(O_1,O_2)=(\operatorname{tr}(T'T))^{1/2}$, where $T=\hat\Sigma_{11}^{-1/2}\hat\Sigma_{12}\hat\Sigma_{22}^{-1/2}$, and the gradient for backpropagation is obtained from the singular-value decomposition of $T$. This constraint creates the shared hyperspace $S$, and the paper fuses the aligned outputs as $O=\alpha_1 O_1+\alpha_2 O_2$ with $\alpha_1+\alpha_2=1$ before training an SVM classifier on $O$.

What would settle it

Re-run the five-dataset evaluation with the grid search over output dimension and fusion weight restricted to training folds or a separate validation split, then measure accuracy on untouched test folds; if the reported margins over the comparison methods shrink or reverse, the state-of-the-art claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that applying deep canonical correlation analysis (DCCA) to multimodal emotion recognition achieves state-of-the-art accuracy on all five evaluated datasets: 94.58% on SEED, 87.45% on SEED-IV, 83.08% on SEED-V, 84.33% and 85.62% on the two binary DEAP tasks and 88.51% on the four-class DEAP task, and 88.99%, 90.57%, and 90.67% on the three binary DREAMER tasks. The proposed mechanism is a coordinated hyperspace: two deep networks transform EEG and eye-movement (or ECG) features separately, the canonical-correlation objective aligns the two output spaces, and a weighted sum of the transformed features is classified by an SVM. The paper also claims that the transformed features are more homogeneous across modalities and more discriminative across emotions, supported by t-SNE visualizations and by higher mutual information estimated with the MINE algorithm, and that DCCA degrades more gracefully than the compared baselines when Gaussian noise is added to or replaces parts of the SEED-V features.

Load-bearing premise

The load-bearing premise is that the reported accuracies honestly measure generalization; but the output dimension and fusion weight were selected by grid search on the same cross-validation folds that produced the reported numbers, so those numbers could be optimistic if the model choices were not locked to a separate validation split.

Editorial extensions

If this is right

  • Emotion recognition systems can use DCCA as a general fusion front-end in place of feature concatenation, while keeping the per-modality transformed features available for inspection.
  • The fusion weight $\alpha_1$ gives a practical control knob: raising the EEG weight improves clean accuracy but lowers noise robustness, so the operating point can be chosen from the expected noise level.
  • Because the transformed features separate emotions more cleanly and align the two modalities, a simple downstream classifier (an SVM in this paper) can achieve the reported accuracies without task-specific architecture engineering.
  • The method transfers across five different dataset designs and modality pairs (EEG plus eye movement, EEG plus peripheral signals, EEG plus ECG), suggesting the coordinating mechanism, not a particular sensor, drives the gain.
  • On noisy SEED-V data, DCCA maintains higher accuracy than concatenation, fuzzy integral, and BDAE under most tested noise levels, indicating it can serve as a preprocessing step in noisy recording environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the benefit truly comes from maximizing shared information between modalities, then replacing the linear CCA objective with a nonlinear or contrastive alignment loss could yield similar or larger gains; the paper's own mutual-information analysis makes this a testable prediction.
  • The paper's high-noise results show fuzzy integral fusion overtaking DCCA once Gaussian noise reaches $N(0,3)$, so a hybrid that keeps DCCA features for the classifier but falls back to decision-level fusion under strong noise might be more robust than either method alone.
  • The method is demonstrated only for two modalities; extending it to three or more signals would require a generalized or pairwise CCA constraint, which the paper does not address.
  • Because the output dimension and fusion weight were selected per dataset by grid search, learning those weights automatically during training would make the approach easier to deploy on a new emotion-recognition dataset.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript introduces deep canonical correlation analysis (DCCA) to multimodal emotion recognition. Two deep networks map EEG and eye-movement/ECG/peripheral features into a common d-dimensional space under a canonical-correlation objective; the transformed features are fused by weighted sum with weights alpha_1 and alpha_2 and classified by an SVM. The authors report accuracy rates on SEED, SEED-IV, SEED-V, DEAP, and DREAMER and claim state-of-the-art results on all five datasets. Additional experiments on SEED-V add Gaussian and replacement noise to study robustness, and t-SNE and MINE analyses are used to argue that DCCA yields homogeneous and discriminative representations. The central claim is the accuracy superiority on all five benchmarks.

Significance. If the reported accuracies are unbiased estimates of generalization, the paper would provide a useful application of an established representation-learning method to affective computing, with a clear comparison against concatenation, MAX, fuzzy integral, BDAE, and DGCNN baselines. The DCCA gradient formulas follow the original DCCA derivation, the robustness study is a useful addition, and the MINE analysis code is linked. However, the significance is conditional on the evaluation protocol being unbiased, because the headline margins over the best baselines are modest (e.g., 83.08 vs. 79.70 on SEED-V; 90.57 vs. 86.64 on DREAMER valence) and the hyperparameter selection procedure can inflate those margins.

major comments (4)
  1. [V-B and Table II] For SEED-V, the reported 83.08% accuracy is obtained after a grid search over output dimensions 5-50 and alpha_1 in {0, 0.1, ..., 1.0} (Section V-B), and the architecture in Table II is described as "searching for the best numbers between 50 and 200". The text does not state that these choices were made on a held-out validation split, and Figure 2 appears to show the grid-search accuracies themselves. If the same three-fold splits are used for model selection and for the final accuracy, the reported number is the maximum over the grid rather than an unbiased estimate of generalization. Because the improvement over BDAE (83.08 vs. 79.70) is small, this selection bias could change the conclusion. Please re-run with a nested split (selection inside each training fold, evaluation on the held-out fold) or report unbiased estimates for a configuration fixed on training folds only.
  2. [V-D and Table IX] For DREAMER, the output dimension is selected from {5, 10, 15, 20, 25, 30} and alpha_1 from {0, 0.1, ..., 1.0} for each of the three classification tasks (Section V-D), using the same leave-one-out cross-validation that produces the reported accuracies 88.99%, 90.57%, and 90.67%. The comparison in Table IX also tunes BDAE's output dimension on the same folds. Selecting the best configuration per task on the evaluation folds makes each reported number an order statistic, and the reported margins over BDAE and DGCNN are only a few points. This does not support the state-of-the-art claim. Please report results for a configuration fixed on training folds only, with the selection rule specified before evaluation.
  3. [III-B, Eq. (7)] The stated objective in Eq. (7), corr(O1,O2) = (tr(T'T))^{1/2}, is the Frobenius norm of T, not the total canonical correlation (sum of the top singular values, i.e., tr((T'T)^{1/2})). The gradient formulas in Eqs. (8)-(10) are those derived in [28] for the trace-norm objective. Because these two objectives have different derivatives, the method description is internally inconsistent. Please correct Eq. (7) to the trace-norm objective or derive the correct gradients for the Frobenius objective.
  4. [V-B, Figures 4-6] The claim that DCCA "preserves emotion-related information" is not supported by the presented evidence. The MINE estimates of mutual information between transformed modalities primarily reflect the fact that DCCA is trained to maximize cross-modal correlation, and the t-SNE plots are produced after hyperparameters were selected to maximize emotion-classification accuracy. Neither analysis conditions on the emotion labels. To support the interpretative claim, the authors could compare mutual information between the transformed features and the emotion labels before and after transformation, or evaluate emotion classification of the transformed features on held-out data.
minor comments (4)
  1. [Throughout] There are several typos: "from multiple perspective" should be "from multiple perspectives" (Section I), "Transfromed" should be "transformed" (Figure 5 caption), "attemps" should be "attempts" (Section V-C), and "The accuracies drops" should be "The accuracies drop" (Figure 8 caption).
  2. [Section V-B and Table VII] The text states that MAX fusion and fuzzy integral fusion yielded mean accuracies of 73.14% and 73.62%, but Table VII reports 73.17% and 73.24% for the same methods; these numbers should be harmonized.
  3. [References] References [16] and [17] appear to be the same paper by Soleymani et al.; they should be consolidated to avoid duplicate citation.
  4. [Figure 2] The heatmap in Figure 2 reports accuracy values rounded to integers, so the exact grid-search results cannot be reconstructed; please provide the full grid table or make the code available for reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

Benchmark accuracy claims are empirical and not circular; the interpretative claim that DCCA preserves emotion-related information is partially circular because the supporting mutual-information increase is a direct consequence of the CCA training objective.

  1. self definitional [Section V-B, final MI paragraph (after Fig. 6); objective Eq. (6)]
    "The transformed features have more mutual information than the original features, indicating that EEG and eye movement features in the coordinated hyperspace provide more shared emotion-related information"

    DCCA is trained by Eq. (6) to maximize corr(f1(X1;W1), f2(X2;W2)); the CCA constraint that defines the coordinated hyperspace is itself a maximization of cross-modal dependence. Observing more mutual information between the two transformed outputs after training is therefore a restatement of the training objective, not an independent measurement that the preserved shared information is emotion-related. The t-SNE 'homogeneous distributions' observation is likewise the definition of the coordinated hyperspace. Because the MI comparison uses no emotion labels and the homogeneity is built into the objective, the paper's conclusion that DCCA preserves emotion-related information is circular for the homogeneity claim and unsupported for the emotion-content claim.

full rationale

The main result, state-of-the-art recognition accuracy on the five datasets, is an empirical benchmark claim compared against external labels and published baselines; it is not derived from the model definition or from a fitted parameter renamed as a prediction, so it is not circular. The self-citation to the authors' previous work [29] for the SEED, SEED-IV, and DEAP numbers is load-bearing for those three datasets, but it transfers empirical results on public data rather than a theorem or a uniqueness argument, so under the hard rules it does not by itself constitute circularity. The grid-search selection of output dimensions and fusion weights on the same evaluation folds for SEED-V and DREAMER is a real selection-bias / external-validity concern, but it is not a circular derivation: the reported accuracy is not equal by construction to the fitted hyperparameter. The one genuinely circular piece is the interpretative mutual-information claim: since DCCA's objective is to maximize cross-modal correlation, the later observation that transformed modalities share more mutual information is a direct consequence of the training target rather than independent evidence that the preserved information is emotion-related. This affects a secondary claim in the abstract and conclusions, while the empirical accuracy claims remain self-contained, giving a partial-circularity score of 4.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central accuracy claim rests on several upstream choices: Gaussian-based DE feature extraction, invertibility of covariance matrices in CCA, per-dataset network and fusion hyperparameters selected on test folds, and the interpretive link between correlation and emotion-relevant content. These are mostly standard or acknowledged assumptions, except for the test-data-dependent hyperparameter selection and the unsupported correlation-to-emotion interpretation. No new physical entities are introduced.

free parameters (4)
  • DCCA output dimension d per dataset = SEED: 20, SEED-IV: 20, SEED-V: 12, DEAP: 20, DREAMER: 5
    Chosen by grid search on the evaluation folds for SEED-V (Section V-B) and DREAMER (Section V-D); fixed from prior work [29] for the other datasets. The output dimension controls the capacity of the coordinated space and affects classification accuracy.
  • Fusion weight alpha_1 for EEG modality (with alpha_2 = 1 - alpha_1) = SEED-V: 0.7; DREAMER arousal: 0.9; DREAMER valence: 0.8; DREAMER dominance: 0.9
    Selected via grid search on test folds (Sections V-B and V-D). The reported accuracies are produced after this selection, so the weights are effectively fitted to the test data.
  • DCCA network architecture (hidden layers and units) = Table II: e.g., SEED 6 layers with units 400, 200, 150, 120, 60, 20; SEED-V and DREAMER searched between 50-200 and…
    Per-dataset architecture chosen by hand or search with no stated selection criterion; architecture directly shapes the learned representations.
  • Regularization constant r1, learning rate, and batch size = r1 = 1e-8, learning rate = 0.001, batch size = 100
    Fixed globally (Section IV-C) without sensitivity analysis; these values affect the DCCA training outcome and downstream accuracy.
assumptions (4)
  • domain assumption The covariance matrices Sigma_11 and Sigma_22 used in CCA are nonsingular.
    The DCCA training equations (Section III-B) require inverting these covariance estimates; the paper states this assumption is 'usually satisfied in practice' (Section III-A).
  • domain assumption EEG signals in short time windows follow a Gaussian distribution, so differential entropy reduces to 0.5*log(2*pi*e*sigma^2).
    Used to compute DE features (Section IV-B1) and justified by reference to Shi et al. [54] rather than re-tested here.
  • ad hoc to paper High canonical correlation between transformed modality outputs corresponds to shared emotion-relevant information.
    The paper assumes that maximizing CCA correlation preserves emotion-related information and discards irrelevant information (Sections I and V-B). This link is asserted, not independently demonstrated.
  • standard math MINE provides reliable mutual information estimates for the feature distributions.
    The MI comparison in Section V-B relies on the MINE estimator from Belghazi et al. [50] and its consistency properties.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis." pith.science (2026). https://pith.science/paper/S5WM7HZV

@misc{pith2026190805349,
  author       = {Pith},
  title        = {Pith review of: Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S5WM7HZV}},
  note         = {Machine review of arXiv:1908.05349}
}
read the original abstract

Multimodal signals are more powerful than unimodal data for emotion recognition since they can represent emotions more comprehensively. In this paper, we introduce deep canonical correlation analysis (DCCA) to multimodal emotion recognition. The basic idea behind DCCA is to transform each modality separately and coordinate different modalities into a hyperspace by using specified canonical correlation analysis constraints. We evaluate the performance of DCCA on five multimodal datasets: the SEED, SEED-IV, SEED-V, DEAP, and DREAMER datasets. Our experimental results demonstrate that DCCA achieves state-of-the-art recognition accuracy rates on all five datasets: 94.58% on the SEED dataset, 87.45% on the SEED-IV dataset, 84.33% and 85.62% for two binary classification tasks and 88.51% for a four-category classification task on the DEAP dataset, 83.08% on the SEED-V dataset, and 88.99%, 90.57%, and 90.67% for three binary classification tasks on the DREAMER dataset. We also compare the noise robustness of DCCA with that of existing methods when adding various amounts of noise to the SEED-V dataset. The experimental results indicate that DCCA has greater robustness. By visualizing feature distributions with t-SNE and calculating the mutual information between different modalities before and after using DCCA, we find that the features transformed by DCCA from different modalities are more homogeneous and discriminative across emotions.

Figures

Figures reproduced from arXiv: 1908.05349 by the authors.

Figure 1
Figure 1. The structure of DCCA. Different modalities are transformed by [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Selection of the best output dimension and EEG weight of DCCA on [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the confusion matrices of different methods on the [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Feature distribution visualization by the t-SNE algorithm. The original features, transformed features, and fused features from the three subjects are [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Distributions of EEG and eye movement features for the sad emotion. The transformed features have more compact distributions from both inter [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Mutual information (MI) estimation with MINE. The green curve shows the estimated MI for the original EEG features and eye movement features. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Performance of DCCA with different weight combinations when [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Model performances after adding Gaussian noise of different variances. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Performance of DCCA with different weight combinations after [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: The trends of the average recognition accuracies of different noise [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Selecting the best output dimension and weight combinations of [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 58 canonical work pages

  1. [28]

    Deep canonical correlation analysis,

    G. Andrew, R. Arora, J. Bilmes, and K. Livescu, “Deep canonical correlation analysis,” in International Conference on Machine Learning, 2013, pp. 1247–1255

    work page 2013

  2. [1]

    R. W. Picard, Affective computing. MIT press, 2000

    2000

  3. [2]

    Toward machine emotional in- telligence: Analysis of affective physiological state,

    R. W. Picard, E. Vyzas, and J. Healey, “Toward machine emotional in- telligence: Analysis of affective physiological state,” IEEE Transactions on Pattern Analysis & Machine Intelligence , no. 10, pp. 1175–1191, 2001

    work page 2001

  4. [3]

    Survey on speech emotion recognition: Features, classification schemes, and databases,

    M. El Ayadi, M. S. Kamel, and F. Karray, “Survey on speech emotion recognition: Features, classification schemes, and databases,” Pattern Recognition, vol. 44, no. 3, pp. 572–587, 2011

    work page 2011

  5. [4]

    A brief review of facial emotion recognition based on visual information,

    B. Ko, “A brief review of facial emotion recognition based on visual information,” Sensors, vol. 18, no. 2, p. 401, 2018

    2018

  6. [5]

    Current state of text sentiment analysis from opinion to emotion mining,

    A. Yadollahi, A. G. Shahraki, and O. R. Zaiane, “Current state of text sentiment analysis from opinion to emotion mining,” ACM Computing Surveys (CSUR), vol. 50, no. 2, p. 25, 2017

    work page 2017

  7. [6]

    Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks,

    W.-L. Zheng and B.-L. Lu, “Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks,” IEEE Transactions on Autonomous Mental Development , vol. 7, no. 3, pp. 162–175, 2015

    work page 2015

  8. [7]

    Identifying stable patterns over time for emotion recognition from EEG,

    W.-L. Zheng, J.-Y . Zhu, and B.-L. Lu, “Identifying stable patterns over time for emotion recognition from EEG,”IEEE Transactions on Affective Computing, doi: 10.1109/TAFFC.2017.2712143

    arXiv 2017

Show all 65 references
  1. [8]

    EEG-based emotion recognition using hierarchical network with subnetwork nodes,

    Y . Yang, Q. J. Wu, W.-L. Zheng, and B.-L. Lu, “EEG-based emotion recognition using hierarchical network with subnetwork nodes,” IEEE Transactions on Cognitive and Developmental Systems , vol. 10, no. 2, pp. 408–419, 2018

    2018

  2. [9]

    Cross-subject EEG feature selection for emotion recognition using transfer recursive feature elimination,

    Z. Yin, Y . Wang, L. Liu, W. Zhang, and J. Zhang, “Cross-subject EEG feature selection for emotion recognition using transfer recursive feature elimination,” Frontiers in Neurorobotics, vol. 11, p. 19, 2017

    2017

  3. [10]

    Gen- eralizations of the subject-independent feature set for music-induced emotion recognition,

    Y .-P. Lin, J.-H. Chen, J.-R. Duann, C.-T. Lin, and T.-P. Jung, “Gen- eralizations of the subject-independent feature set for music-induced emotion recognition,” in 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society . IEEE, 2011, pp. 6092–6095

    2011

  4. [11]

    Emotional state classification from eeg data using machine learning approach,

    X.-W. Wang, D. Nie, and B.-L. Lu, “Emotional state classification from eeg data using machine learning approach,” Neurocomputing, vol. 129, pp. 94–106, 2014

    2014

  5. [12]

    Emotion recognition based on physiological changes in music listening,

    J. Kim and E. Andr ´e, “Emotion recognition based on physiological changes in music listening,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, pp. 2067–2083, 2008

    2008

  6. [13]

    The coupling of emotion and cognition in the eye: Introducing the pupil old/new effect,

    M. L.-H. V ˜o, A. M. Jacobs, L. Kuchinke, M. Hofmann, M. Conrad, A. Schacht, and F. Hutzler, “The coupling of emotion and cognition in the eye: Introducing the pupil old/new effect,”Psychophysiology, vol. 45, no. 1, pp. 130–140, 2008

    2008

  7. [14]

    A review of affective computing: from unimodal analysis to multimodal fusion,

    S. Poria, E. Cambria, R. Bajpai, and A. Hussain, “A review of affective computing: from unimodal analysis to multimodal fusion,” Information Fusion, vol. 37, pp. 98–125, 2017

    2017

  8. [15]

    Emotion- meter: A multimodal framework for recognizing human emotions,

    W.-L. Zheng, W. Liu, Y .-F. Lu, B.-L. Lu, and A. Cichocki, “Emotion- meter: A multimodal framework for recognizing human emotions,”IEEE Transactions on Cybernetics, vol. 49, no. 3, pp. 1110–1122, March 2019

    2019

  9. [16]

    Multimodal emotion recognition in response to videos,

    M. Soleymani, M. Pantic, and T. Pun, “Multimodal emotion recognition in response to videos,” IEEE Transactions on Affective Computing , vol. 3, no. 2, pp. 211–223, 2012

    2012

  10. [17]

    Multimodal emotion recognition in response to videos,

    M. Soleymani, M. Pantic, and T. Pun, “Multimodal emotion recognition in response to videos,” IEEE Transactions on Affective Computing , vol. 3, no. 2, pp. 211–223, April 2012. 14

    2012

  11. [18]

    Affectnet: A database for facial expression, valence, and arousal computing in the wild,

    A. Mollahosseini, B. Hasani, and M. H. Mahoor, “Affectnet: A database for facial expression, valence, and arousal computing in the wild,” IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 18–31, Jan 2019

    2019

  12. [19]

    Analysis of eeg signals and facial expressions for continuous emotion detection,

    M. Soleymani, S. Asghari-Esfeden, Y . Fu, and M. Pantic, “Analysis of eeg signals and facial expressions for continuous emotion detection,” IEEE Transactions on Affective Computing , vol. 7, no. 1, pp. 17–28, Jan 2016

    2016

  13. [20]

    Combining eye movements and EEG to enhance emotion recognition,

    Y .-F. Lu, W.-L. Zheng, B.-B. Li, and B.-L. Lu, “Combining eye movements and EEG to enhance emotion recognition,” in Twenty-Fourth International Joint Conference on Artificial Intelligence , 2015

    2015

  14. [21]

    DEAP: A database for emotion analysis; using physiological signals,

    S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, “DEAP: A database for emotion analysis; using physiological signals,” IEEE Transactions on Affective Computing, vol. 3, no. 1, pp. 18–31, 2012

    2012

  15. [22]

    Combining feature-level and decision-level fusion in a hierarchical classifier for emotion recognition in the wild,

    B. Sun, L. Li, X. Wu, T. Zuo, Y . Chen, G. Zhou, J. He, and X. Zhu, “Combining feature-level and decision-level fusion in a hierarchical classifier for emotion recognition in the wild,” Journal on Multimodal User Interfaces, vol. 10, no. 2, pp. 125–137, 2016

    2016

  16. [23]

    Multimodal machine learning: A survey and taxonomy,

    T. Baltruaitis, C. Ahuja, and L. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 41, no. 2, pp. 423–443, 2017

    2017

  17. [24]

    Emotion recognition using multimodal deep learning,

    W. Liu, W.-L. Zheng, and B.-L. Lu, “Emotion recognition using multimodal deep learning,” in International Conference on Neural Information Processing. Springer, 2016, pp. 521–529

    2016

  18. [25]

    Multimodal emotion recognition using deep neural networks,

    H. Tang, W. Liu, W.-L. Zheng, and B.-L. Lu, “Multimodal emotion recognition using deep neural networks,” in International Conference on Neural Information Processing . Springer, 2017, pp. 811–819

    2017

  19. [26]

    Emotion recogni- tion from multi-channel EEG data through convolutional recurrent neural network,

    X. Li, D. Song, P. Zhang, G. Yu, Y . Hou, and B. Hu, “Emotion recogni- tion from multi-channel EEG data through convolutional recurrent neural network,” in 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2016, pp. 352–359

    2016

  20. [27]

    Recognition of emotions using multimodal physiological signals and an ensemble deep learning model,

    Z. Yin, M. Zhao, Y . Wang, J. Yang, and J. Zhang, “Recognition of emotions using multimodal physiological signals and an ensemble deep learning model,” Computer Methods and Programs in Biomedicine , vol. 140, pp. 93–110, 2017

    2017

  21. [29]

    Multi-view emotion recognition using deep canonical correlation analysis,

    J.-L. Qiu, W. Liu, and B.-L. Lu, “Multi-view emotion recognition using deep canonical correlation analysis,” in International Conference on Neural Information Processing . Springer, 2018, pp. 221–231

    2018

  22. [30]

    Multimodal data fusion: an overview of methods, challenges, and prospects,

    D. Lahat, T. Adali, and C. Jutten, “Multimodal data fusion: an overview of methods, challenges, and prospects,” Proceedings of the IEEE , vol. 103, no. 9, pp. 1449–1477, 2015

    2015

  23. [31]

    A review and meta-analysis of multimodal affect detection systems,

    S. K. D’Mello and J. Kory, “A review and meta-analysis of multimodal affect detection systems,” ACM Computing Surveys , vol. 47, no. 3, pp. 1–36, 2015

    2015

  24. [32]

    Self- attentive feature-level fusion for multimodal emotion detection,

    D. Hazarika, S. Gorantla, S. Poria, and R. Zimmermann, “Self- attentive feature-level fusion for multimodal emotion detection,” in 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). IEEE, 2018, pp. 196–201

    2018

  25. [33]

    Multimodal deep learning,

    J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y . Ng, “Multimodal deep learning,” in International Conference on Machine Learning, 2011, pp. 689–696

    2011

  26. [34]

    Classification of affects using head movement, skin color features and physiological signals,

    H. Monkaresi, M. Sazzad, and R. A. Calvo, “Classification of affects using head movement, skin color features and physiological signals,” in IEEE International Conference on Systems , 2012

    2012

  27. [35]

    A hybrid fuzzy cognitive map/support vector machine approach for EEG- based emotion classification using compressed sensing,

    K. Guo, R. Chai, H. Candra, Y . Guo, R. Song, H. Nguyen, and S. Su, “A hybrid fuzzy cognitive map/support vector machine approach for EEG- based emotion classification using compressed sensing,” International Journal of Fuzzy Systems , vol. 21, pp. 263–273, 2019

    2019

  28. [36]

    Unsupervised alignment of natural language instructions with video segments,

    I. Naim, Y . C. Song, Q. Liu, H. Kautz, J. Luo, and D. Gildea, “Unsupervised alignment of natural language instructions with video segments,” in Twenty-Eighth AAAI Conference on Artificial Intelligence. AAAI Press, 2014, pp. 1558–1564

    2014

  29. [37]

    Devise: A deep visual-semantic embedding model,

    A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, T. Mikolov et al., “Devise: A deep visual-semantic embedding model,” in Advances in Neural Information Processing Systems , 2013, pp. 2121–2129

    2013

  30. [38]

    Relations between two sets of variates,

    H. Hotelling, “Relations between two sets of variates,” in Breakthroughs in Statistics. Springer, 1992, pp. 162–190

    1992

  31. [39]

    Canonical correlation analysis: An overview with application to learning methods,

    D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, “Canonical correlation analysis: An overview with application to learning methods,” Neural Computation, vol. 16, no. 12, pp. 2639–2664, 2004

    2004

  32. [40]

    Kernel and nonlinear canonical correlation analysis,

    P. L. Lai and C. Fyfe, “Kernel and nonlinear canonical correlation analysis,” International Journal of Neural Systems , vol. 10, no. 05, pp. 365–377, 2000

    2000

  33. [41]

    Probabilistic approach to detecting dependen- cies between data sets,

    A. Klami and S. Kaski, “Probabilistic approach to detecting dependen- cies between data sets,” Neurocomputing, vol. 72, no. 1, pp. 39–46, 2008

    2008

  34. [42]

    Bayesian canonical correlation analysis,

    A. Klami, S. Virtanen, and S. Kaski, “Bayesian canonical correlation analysis,” Journal of Machine Learning Research , vol. 14, no. Apr, pp. 965–1003, 2013

    2013

  35. [43]

    Tensor canonical correlation analysis for action classification,

    T.-K. Kim, S.-F. Wong, and R. Cipolla, “Tensor canonical correlation analysis for action classification,” in 2007 IEEE Conference on Com- puter Vision and Pattern Recognition . IEEE, 2007, pp. 1–8

    2007

  36. [44]

    Sparse canonical correlation analysis,

    D. R. Hardoon and J. Shawe-Taylor, “Sparse canonical correlation analysis,” Machine Learning, vol. 83, no. 3, pp. 331–353, 2011

    2011

  37. [45]

    Cluster canonical correlation analysis,

    N. Rasiwasia, D. Mahajan, V . Mahadevan, and G. Aggarwal, “Cluster canonical correlation analysis,” in Artificial Intelligence and Statistics , 2014, pp. 823–831

    2014

  38. [46]

    Application of the choquet integral in multicriteria decision making,

    M. Grabisch and M. Roubens, “Application of the choquet integral in multicriteria decision making,” Fuzzy Measures & Integrals , pp. 348– 374, 2000

    2000

  39. [47]

    Gender classification by combin- ing clothing, hair and facial component classifiers,

    B. Li, X.-C. Lian, and B.-L. Lu, “Gender classification by combin- ing clothing, hair and facial component classifiers,” Neurocomputing, vol. 76, no. 1, pp. 18–27, 2012

    2012

  40. [48]

    A study on subjective evaluations of printed color images,

    K. Tanaka and M. Sugeno, “A study on subjective evaluations of printed color images,” International Journal of Approximate Reasoning , vol. 5, no. 5, pp. 213–222, 1991

    1991

  41. [49]

    Scikit-learn: Machine learning in Python,

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vander- plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay, “Scikit-learn: Machine learning in Python,” Journal of Machine ...

    2011

  42. [50]

    Mine: mutual information neural estimation,

    M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y . Bengio, A. Courville, and R. D. Hjelm, “Mine: mutual information neural estimation,” arXiv preprint arXiv:1801.04062 , 2018

    2018 arXiv

  43. [51]

    Classification of five emotions from eeg and eye movement signals: Discrimination ability and stability over time,

    T.-H. Li, W. Liu, W.-L. Zheng, and B.-L. Lu, “Classification of five emotions from eeg and eye movement signals: Discrimination ability and stability over time,” in 9th International IEEE/EMBS Conference on Neural Engineering (NER) . IEEE, 2019, pp. 607–610

    2019

  44. [52]

    DREAMER: A database for emotion recognition through eeg and ecg signals from wireless low-cost off- the-shelf devices,

    S. Katsigiannis and N. Ramzan, “DREAMER: A database for emotion recognition through eeg and ecg signals from wireless low-cost off- the-shelf devices,” IEEE Journal of Biomedical and Health Informatics , vol. 22, no. 1, pp. 98–107, 2017

    2017

  45. [53]

    Differential entropy fea- ture for EEG-based emotion classification,

    R.-N. Duan, J.-Y . Zhu, and B.-L. Lu, “Differential entropy fea- ture for EEG-based emotion classification,” in 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER) . IEEE, 2013, pp. 81–84

    2013

  46. [54]

    Differential entropy feature for EEG-based vigilance estimation,

    L.-C. Shi, Y .-Y . Jiao, and B.-L. Lu, “Differential entropy feature for EEG-based vigilance estimation,” in 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 2013, pp. 6627–6630

    2013

  47. [55]

    Off-line and on-line vigilance estimation based on linear dynamical system and manifold learning,

    L.-C. Shi and B.-L. Lu, “Off-line and on-line vigilance estimation based on linear dynamical system and manifold learning,” in 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology. IEEE, 2010, pp. 6587–6590

    2010

  48. [56]

    Automatic ecg-based emotion recognition in music listening,

    Y . Hsu, J. Wang, W. Chiang, and C. Hung, “Automatic ecg-based emotion recognition in music listening,” IEEE Transactions on Affective Computing, pp. 1–16, 2018

    2018

  49. [57]

    Emotion recognition using wireless signals,

    M. Zhao, F. Adib, and D. Katabi, “Emotion recognition using wireless signals,” in Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking . ACM, 2016, pp. 95–108

    2016

  50. [58]

    Frequency analysis of the electro- cardiogram,

    A. M. SCHER and A. C. YOUNG, “Frequency analysis of the electro- cardiogram,” Circulation Research, vol. 8, no. 2, pp. 344–346, 1960

    1960

  51. [59]

    Ecg signals classification based on dis- crete wavelet transform, time domain and frequency domain features,

    S. A. Shufni and M. Y . Mashor, “Ecg signals classification based on dis- crete wavelet transform, time domain and frequency domain features,” in 2015 2nd International Conference on Biomedical Engineering (ICoBE). IEEE, 2015, pp. 1–6

    2015

  52. [60]

    Frequency content and characteristics of ventricular conduction,

    L. G. Tereshchenko and M. E. Josephson, “Frequency content and characteristics of ventricular conduction,” Journal of Electrocardiology, vol. 48, no. 6, pp. 933–937, 2015

    2015

  53. [61]

    Classification of five emotions from eeg and eye movement signals: Complementary representation properties,

    L.-M. Zhao, R. Li, W.-L. Zheng, and B.-L. Lu, “Classification of five emotions from eeg and eye movement signals: Complementary representation properties,” in 9th International IEEE/EMBS Conference on Neural Engineering (NER) . IEEE, 2019, pp. 611–614

    2019

  54. [62]

    EEG emotion recognition using dynamical graph convolutional neural networks,

    T. Song, W. Zheng, P. Song, and Z. Cui, “EEG emotion recognition using dynamical graph convolutional neural networks,” IEEE Transactions on Affective Computing, 2018

    2018

  55. [63]

    A three-stage decision framework for multi-subject emotion recognition using physi- ological signals,

    J. Chen, B. Hu, Y . Wang, Y . Dai, Y . Yao, and S. Zhao, “A three-stage decision framework for multi-subject emotion recognition using physi- ological signals,” in IEEE International Conference on Bioinformatics & Biomedicine, 2017. 15

    2017

  56. [64]

    Robust kernel density estimation,

    J. Kim and C. D. Scott, “Robust kernel density estimation,” Journal of Machine Learning Research , vol. 13, no. Sep, pp. 2529–2565, 2012. Wei Liu received his bachelor’s degree in Automa- tion Science from the School of Advanced Engineer- ing, Beihang University, Beijing, Chin...

    2012

  57. [2018]

    Bao-Liang Lu (M’94–SM’10) received his B.S

    His research focuses on affective computing, brain-computer interaction, machine learning and clinical healthcare. Bao-Liang Lu (M’94–SM’10) received his B.S. degree in Instrument and Control Engineering from the Qingdao University of Science and Technology, Qingdao, China, in...

    1982

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.