Pith. sign in

REVIEW 4 major objections 6 minor 103 references

Multimodal Functional Maximum Correlation for Emotion Recognition

T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Emotion recognition improves when self-supervised training maximizes joint dependence among three physiological signals, not pairwise alignments.

desk verdict The empirical recipe is worth a look, but the 'DTC-grounded' theory is doing more work than the math can support. read the letter →

arxiv 2512.23076 v2 pith:FSSI543T submitted 2025-12-28 cs.LG cs.AIcs.HC

Deyang Zheng , Tianyi Zhang , Wenming Zheng , Shujian Yu This is my paper · ORCID
classification cs.LGcs.AIcs.HC
keywords multimodalemotionrecognitionself-supervisedlearningdualtotalcorrelationfunctionalmaximumanalysisphysiologicalsignalshigher-orderdependencetraceobjectivesubject-independentgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that self-supervised learning for physiological emotion recognition has been limited by pairwise alignment objectives, which capture only two-way dependencies and miss the coordinated joint activity of brain and autonomic signals. To fix this, the authors propose MFMC, a framework that maximizes the Dual Total Correlation (DTC) among three modalities—an information-theoretic measure of joint dependence that includes conditional, higher-order interactions. Because DTC is hard to estimate directly, they prove a sandwich bound linking it to a cyclic sum of mutual information terms, then optimize those terms with a trace-based surrogate derived from functional maximum correlation analysis (FMCA), avoiding the numerical instability of eigenvalue methods. On three public affective-computing benchmarks, MFMC matches or beats supervised baselines under subject-dependent evaluation and surpasses them in most subject-independent settings, suggesting that jointly modeling triple-wise physiological dependence yields more transferable representations.

What carries the argument

The central object is the Dual Total Correlation (DTC), an information-theoretic measure of joint dependence that counts shared information without the double-counting of total correlation. The argument is carried by a sandwich bound—(1/3)Σ_cyc I(pair;third) ≤ DTC ≤ (2/3)Σ_cyc I(pair;third)—which reduces DTC to three joint mutual information terms. Each term is optimized via a trace objective tr(R_X⁻¹ R_XY R_Y⁻¹ R_XYᵀ), the sum of squared canonical correlations from functional maximum correlation analysis (FMCA). This trace objective acts as a stable surrogate for the mutual information terms, bypassing eigenvalue decomposition and the numerical fragility of log-determinant objectives. The t

What would settle it

Estimate the true DTC and the cyclic sum of mutual information terms I(pair;third) on real physiological windows using a non-parametric entropy estimator, and compare them with the trace objective's value across training. If the trace objective does not increase monotonically with the estimated DTC, the surrogate is not tracking the claimed target.

Watch

Extended reading notes

Core claim

The central claim is that the dual total correlation (DTC) of three physiological modalities—such as EEG, EOG, and skin temperature—is the right self-supervised training signal for emotion recognition, and that a tractable surrogate can reach it. The authors show that DTC is sandwiched between one-third and two-thirds of the cyclic sum of joint mutual information terms I(pair; third); maximizing those terms therefore maximizes DTC. Each term is estimated by fusing the embeddings of two modalities and computing a trace objective tr(R₁₂⁻¹ P₁₂,₃ R₃⁻¹ P₁₂,₃ᵀ), the sum of squared canonical correlations, which is a first-order approximation of the log-determinant FMCA objective. After pretraining,

Load-bearing premise

The load-bearing premise is that the trace objective—the sum of squared canonical correlations between the fused pair and the third modality—faithfully represents the joint mutual information terms I(pair; third) in the DTC sandwich bound; if that mapping fails, the theoretical grounding for maximizing DTC reduces to an ad hoc correlation objective.

Editorial extensions

If this is right

  • Self-supervised pretraining for physiological signals no longer requires negative samples, positive pairs, or handcrafted augmentations; the joint dependence structure of the modalities themselves provides the learning signal.
  • A single modality encoder pretrained in a tri-modal DTC objective carries cross-modal information, so emotion recognition can run with one sensor at test time without a fusion network.
  • Modeling higher-order dependence is especially valuable for subject-independent generalization: MFMC outperforms supervised baselines on most held-out-subject splits, implying that DTC-based objectives learn representations less tied to subject identity.
  • The cyclic objective extends from three to any number of modalities via Theorem 2, so the framework can scale to richer multimodal settings beyond the three physiological channels tested here.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension would vary the number of modalities (four or five signals) to see whether subject-independent gains continue or saturate; that would reveal how much of the benefit comes from DTC itself versus merely having more input views.
  • Because the trace surrogate is a sum of canonical correlations rather than true mutual information, an ablation that swaps in a different first-order dependence measure (e.g., Hilbert-Schmidt norm) would clarify whether the specific functional form matters or the key is just avoiding pairwise-only objectives.
  • MFMC's augmentation-light, negative-free recipe should transfer to other label-scarce physiological tasks such as stress detection or sleep staging, where the same central–peripheral coordination structure appears.
  • The Gaussian simulation in the appendix gives a calibration target: computing the actual bound ratio on real data would show how much of the DTC the cyclic objective truly covers, and whether the sandwich proof leaves room for a tighter surrogate.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MFMC, a self-supervised pretraining objective for multimodal physiological emotion recognition that aims to maximize dual total correlation (DTC) over three modalities. The method replaces the intractable DTC with a sandwich bound involving cyclic joint mutual information terms (Theorem 1) and estimates each term with a trace-based 'functional maximum correlation' surrogate (Eq. 22). The learned encoders are then evaluated on 4-class valence/arousal classification on DEAP, CEAP-360VR, and MAHNOB-HCI under subject-dependent and subject-independent protocols. The central claims are that MFMC is the first higher-order multimodal SSL framework for physiological affective computing and that the trace surrogate is a stable, DTC-grounded objective.

Significance. If the theoretical link were established, MFMC would be a useful contribution: it avoids pairwise contrastive objectives, extends naturally to more than two modalities via DTC, and comes with reproducible code and notebooks. The empirical study is broad: three public datasets, two evaluation protocols, multiple baseline families, a shared backbone, and an ablation that substitutes the loss function. The release of executable notebooks for reproducing the DEAP results is a concrete strength. However, the central DTC grounding currently rests on identifying a canonical-correlation trace with mutual information; this identification is not proven or empirically validated, so the theoretical significance as stated is not yet supported.

major comments (4)
  1. [§III-C, Eqs. (22)–(23)] The load-bearing link between the DTC sandwich bound and the actual objective is not established. Each trace term in Eq. (23) is a sum of squared canonical correlations between the fused embedding e12 and e3, not an estimator of the joint mutual information I((X1,X2);X3). Lemma 2 only justifies the trace as a first-order approximation of the log-determinant TSD -Σ log(1-σ_i), and TSD is not mutual information. No theorem or experiment shows that maximizing this trace increases I(pair;third) or DTC. Appendix B validates the tightness of the Theorem 1 bound on simulated Gaussian and functional data, but it does not validate the fidelity of Eq. (22) to the MI terms it is meant to replace. Consequently, the statement that MFMC 'maximizes DTC' is currently unsupported.
  2. [§III-C, Fig. 2, Eq. (23)] Even if the trace objective were a faithful MI estimator, the objective operates on learned embeddings e12 = F(f(X1),g(X2)) and e3 = h(X3). By the data processing inequality, I(e12;e3) ≤ I((X1,X2);X3), so MFMC at best maximizes a lower bound of the desired joint MI terms. This is the same lower-bound status that the paper criticizes in InfoNCE ([39], Section III-A). If the contribution is a lower-bound surrogate, the authors should state this explicitly and compare its properties with other lower-bound estimators rather than presenting it as a 'direct' estimate of I(pair;third).
  3. [§IV-B4, Tables I–III] The empirical state-of-the-art claim is weakened by missing baselines that the paper itself discusses. PhysioSync [37] and the cross-modal ECG–EEG alignment method [49] are cited as recent physiological SSL approaches, but they are not included in any comparison table. GANSER [63] is likewise discussed but not benchmarked. Given the paper positions MFMC against these methods, their absence makes the 'consistently achieves state-of-the-art' claim difficult to evaluate.
  4. [§IV-B5, Table IV] The phrase 'statistically tied best' is used without significance testing. Overlapping mean±std intervals across five folds are not a statistical tie. The authors should report paired significance tests or confidence intervals for the main comparisons, particularly in the subject-independent results where several differences are within one standard deviation.
minor comments (6)
  1. [§IV-B1 vs. Appendix D] The windowing protocol is inconsistent. Main text says for windows >5s the stride is 0.6×window length (6s for 10s windows), while Appendix D says '0.4s stride' and reports 20,097 DEAP windows. Please correct the stride description and verify the sample counts.
  2. [§IV-B5] The subject split for DEAP is given as '15/4 train/test subjects', but DEAP has 32 participants. This appears to be a typo; the split should sum to 32 and match the 5-fold leave-group-out description.
  3. [Tables I–III] The supervised HyperFuseNet baseline is reported with identical accuracy for both modality columns (e.g., 0.995 for EEG and EOG in Table I). Since HyperFuseNet is a multimodal fusion model, the protocol for obtaining per-modality numbers needs clarification: is the classifier retrained per modality, or is a single multimodal model evaluated on each modality separately?
  4. [Appendix A] Theorems 1 and 2 are presented as new results, but their proofs are direct applications of Lemmas 3 and 4 from [76]. Please state explicitly which parts are borrowed and what the new contribution is; the current text cites 'complete proofs in Appendix A' without identifying the level of novelty.
  5. [References] References [63] and [66] refer to the same GANSER paper with different years; please deduplicate.
  6. [Abstract and §VI] The claim 'first SSL framework for physiological emotion recognition that explicitly models higher-order multimodal dependence' should be qualified with respect to the cited multiway SSL methods (SymILE, ImageBind, Gramian-based), which already model more than two modalities in other domains; as written, the novelty claim exceeds what the experiments can establish.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: MFMC's trace objective is unsupervised and externally benchmarked; sole self-citation is in an appendix validation and non-load-bearing.

full rationale

I walked the derivation chain. The DTC decomposition (Eq. 14) and the sandwich bound (Theorem 1, Theorem 2) are proved in Appendix A using Lemmas 4.5 and 4.13 from Austin [76], an external reference; this is independent mathematical support, not a self-citation. The proposed objective (Eqs. 21-23) is an unsupervised trace-based FMCA surrogate, tr(R_X^{-1}P R_Y^{-1}P^T), equal to a sum of squared canonical correlations; it is not fitted to downstream labels, to test accuracies, or to the DTC values it is said to approximate. The gap between this trace surrogate and true mutual information I(pair;third) is an unproved approximation and a correctness risk, but it is not circular: no equation in the paper defines the trace objective in terms of DTC, MI, or the reported emotion-recognition results. The only self-citation is [51] (Yu et al., matrix-based entropy functional), used in Appendix B to estimate DTC in synthetic-data validation of the sandwich bound. That validation is not load-bearing for the main method or for the empirical claims, which are benchmarked on held-out subject splits and against external baselines. Thus no step reduces by construction to its inputs, and the paper does not exhibit fitted-input-called-prediction, self-definitional, or self-citation-loaded circularity. Score 2 reflects the presence of one minor self-citation that is not load-bearing; the 'DTC-grounded' theoretical framing is unsupported but not circular.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The empirical results rest on data-specific selections, while the theoretical claim rests on the DTC/MI chain from cited literature plus the unproven trace-surrogate bridge.

free parameters (3)
  • window_length = 10 s
    Chosen from a validation sweep on DEAP (Fig. 3) and applied to all datasets; accuracy peaks at 10 s, so empirical results are partly tuned to this value.
  • embedding_dim_K = 128
    Shared projection dimension for all modality encoders and fusion network; adopted by hand and affects the rank of the trace surrogate.
  • modality_configs = DEAP: EEG+EOG+SKT; CEAP: EDA+BVP+SKT; MAHNOB: EEG+ECG+EDA
    Selected by validation attention gating (Sec. IV-B2); the trimodal input sets are data-dependent choices.
assumptions (5)
  • standard math Density ratio p(X,Y)/(p(X)p(Y)) admits an orthonormal spectral decomposition with eigenvalues sigma_i in [0,1) (Eq. 5).
    Invoked in Sec. III-A; standard L2 spectral theory for square-integrable density ratios.
  • ad hoc to paper The trace surrogate tr(R_X^{-1} P R_Y^{-1} P^T) approximates the joint mutual information I(pair;third) (Eqs. 22-23).
    Unproven; FMCA measures a correlation/TSD functional, not MI. This is the load-bearing gap between Theorem 1 and the optimized objective.
  • domain assumption Fusing embeddings e1,e2 via a lightweight MLP preserves enough joint information to estimate I((X1,X2);X3).
    Needed for cyclic pair-third estimates; no information-theoretic guarantee for the fusion network.
  • domain assumption DTC is the right objective for multimodal affective alignment and its sandwich bound makes the cyclic sum a reliable surrogate.
    Framing in Sec. III-B; follows from [76] and domain arguments about synergy/redundancy, not proven for physiological data.
  • domain assumption Window-level 10 s segments are i.i.d. enough that SSL pretraining on them transfers to unseen subjects.
    Required for the subject-independent evaluation protocol; physiological segments within a trial are temporally correlated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Functional Maximum Correlation for Emotion Recognition." pith.science (2026). https://pith.science/paper/FSSI543T

@misc{pith2026251223076,
  author       = {Pith},
  title        = {Pith review of: Multimodal Functional Maximum Correlation for Emotion Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FSSI543T}},
  note         = {Machine review of arXiv:2512.23076}
}
read the original abstract

Emotional states manifest as coordinated yet heterogeneous physiological responses across central and autonomic systems, posing a fundamental challenge for multimodal representation learning in affective computing. Learning such joint dynamics is further complicated by the scarcity and subjectivity of affective annotations, which motivates the use of self-supervised learning (SSL). However, most existing SSL approaches rely on pairwise alignment objectives, which are insufficient to characterize dependencies among more than two modalities and fail to capture higher-order interactions arising from coordinated brain and autonomic responses. To address this limitation, we propose Multimodal Functional Maximum Correlation (MFMC), a principled SSL framework that maximizes higher-order multimodal dependence through a Dual Total Correlation (DTC) objective. By deriving a tight sandwich bound and optimizing it using a functional maximum correlation analysis (FMCA) based trace surrogate, MFMC captures joint multimodal interactions directly, without relying on pairwise contrastive losses. Experiments on three public affective computing benchmarks demonstrate that MFMC consistently achieves state-of-the-art or competitive performance under both subject-dependent and subject-independent evaluation protocols, highlighting its robustness to inter-subject variability. In particular, MFMC improves subject-dependent accuracy on CEAP-360VR from 78.9% to 86.8%, and subject-independent accuracy from 27.5% to 33.1% using the EDA signal alone. Moreover, MFMC remains within 0.8 percentage points of the best-performing method on the most challenging EEG subject-independent split of MAHNOB-HCI. Our code is available at https://github.com/DY9910/MFMC.

Figures

Figures reproduced from arXiv: 2512.23076 by the authors.

Figure 1
Figure 1. TC and DTC on three variables X1, X2, X3. In each case, the quantity is represented by the total number of block areas of the diagram. TC counts the triple-overlapped shaded area (a.k.a., interaction information [70]) twice, whereas DTC counts this area just once. and X[M]\{i} to denote all variables except Xi . Then, TC and DTC are defined as: TC(X[M]) = X M i=1 H(Xi) − H(X[M]), (12) DTC(X[M]) = H(X[M]) − X M i=1 H… view at source ↗
Figure 2
Figure 2. Diagram illustrating the maximization total dependence between [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Emotion-discrimination performance versus window [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Learnable attention weights for physiological signal [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Validation curves on the DEAP subject-dependent split. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Simulation results. 5) Method: Using the 10 s window length identified earlier, we trained the same encoder as in all main experiments, but appended a learnable modality-attention mask. Concretely, after global spatial-temporal pooling the model produces a d-dimensiona…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

103 extracted references · 4 linked inside Pith

  1. [39]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,”arXiv preprint arXiv:1807.03748, 2018

    arXiv 2018

  2. [37]

    Physiosync: Temporal and cross-modal contrastive learning inspired by physiological synchronization for eeg-based emotion recognition,

    K. Cui, J. Li, Y . Liu, X. Zhang, Z. Hu, and M. Wang, “Physiosync: Temporal and cross-modal contrastive learning inspired by physiological synchronization for eeg-based emotion recognition,”arXiv preprint arXiv:2504.17163, 2025

    arXiv 2025

  3. [49]

    Cross-modal contrastive learning for emotion recognition: Aligning ecg with eeg-derived features,

    Y . Wu, H. Khalid, Q. Deng, T. Yu, Z. Li, and R. Vezzani, “Cross-modal contrastive learning for emotion recognition: Aligning ecg with eeg-derived features,” inMedical Image Computing and Computer-Assisted Intervention – MICCAI 2025, 2025. [Online]. Available: https://papers.miccai.org/miccai-2025/paper/4191 paper.pdf

    2025

  4. [63]

    Ganser: A self-supervised data augmentation framework for eeg-based emotion recognition,

    Z. Zhang, Y . Liu, and S.-h. Zhong, “Ganser: A self-supervised data augmentation framework for eeg-based emotion recognition,”IEEE Transactions on Affective Computing, vol. 14, no. 3, pp. 2048–2063, 2022

  5. [1]

    Eeg emotion recognition using dynamical graph convolutional neural networks,

    T. Song, W. Zheng, P. Song, and Z. Cui, “Eeg emotion recognition using dynamical graph convolutional neural networks,”IEEE Transactions on Affective Computing, vol. 11, no. 3, pp. 532–541, 2018

    2018

  6. [2]

    The effi- cacy of emotion recognition rehabilitation for people with alzheimer’s disease,

    J. A. Garc ´ıa-Casal, M. Go ˜ni-Imizcoz, M. V . Perea-Bartolom´e, F. Soto- P´erez, S. J. Smith, S. Calvo-Simal, and M. Franco-Mart ´ın, “The effi- cacy of emotion recognition rehabilitation for people with alzheimer’s disease,”Journal of Alzheimer’s Disease, vol. 57, no. 3, pp. 937–951, 2017

    2017

  7. [3]

    Emotion recognition using eeg and physiological data for robot-assisted rehabilitation systems,

    E. G ¨um¨usl¨u, D. Erol Barkana, and H. K ¨ose, “Emotion recognition using eeg and physiological data for robot-assisted rehabilitation systems,” inCompanion publication of the 2020 international conference on multimodal interaction, 2020, pp. 379–387

    2020

  8. [4]

    Divergent spatiotemporal signatures characterize impaired facial emotional recognition in major depressive disorder: An event-related microstate study,

    Y . Liu, K. Dong, and L. Sun, “Divergent spatiotemporal signatures characterize impaired facial emotional recognition in major depressive disorder: An event-related microstate study,”Journal of Affective Disor- ders, vol. 381, pp. 281–290, 2025

    2025

Show all 103 references
  1. [5]

    A real-time automated system for the recognition of human facial expressions,

    K. Anderson and P. W. McOwan, “A real-time automated system for the recognition of human facial expressions,”IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 36, no. 1, pp. 96–105, Feb. 2006

    2006

  2. [6]

    Sponta- neous facial micro-expression analysis using spatiotemporal completed local quantized patterns,

    X. Huang, G. Zhao, X. Hong, W. Zheng, and M. Pietik ¨ainen, “Sponta- neous facial micro-expression analysis using spatiotemporal completed local quantized patterns,”Neurocomputing, vol. 175, pp. 564–578, 2016

    2016

  3. [7]

    Facial micro- expression recognition using spatiotemporal local binary pattern with integral projection,

    X. Huang, S.-J. Wang, G. Zhao, and M. Pietik ¨ainen, “Facial micro- expression recognition using spatiotemporal local binary pattern with integral projection,” inProc. IEEE Int. Conf. Computer Vision Work- shops (ICCVW), 2015, pp. 1–9

    2015

  4. [8]

    Discriminative spatiotemporal local binary pattern with revisited in- tegral projection for spontaneous facial micro-expression recognition,

    X. Huang, S.-J. Wang, X. Liu, G. Zhao, X. Feng, and M. Pietik ¨ainen, “Discriminative spatiotemporal local binary pattern with revisited in- tegral projection for spontaneous facial micro-expression recognition,” IEEE Transactions on Affective Computing, 2017, early access

    2017

  5. [9]

    A main directional mean optical flow feature for spontaneous micro- expression recognition,

    Y .-J. Liu, J.-K. Zhang, W.-J. Yan, S.-J. Wang, G. Zhao, and X. Fu, “A main directional mean optical flow feature for spontaneous micro- expression recognition,”IEEE Transactions on Affective Computing, vol. 7, no. 4, pp. 299–310, 2016

    2016

  6. [10]

    Integrating facial expression and body gesture in videos for emotion recognition,

    J. Yan, W. Zheng, M. Xin, and J. Yan, “Integrating facial expression and body gesture in videos for emotion recognition,”IEICE Transactions on Information and Systems, vol. E97-D, no. 3, pp. 610–613, 2014

    2014

  7. [11]

    Prosody- based automatic detection of annoyance and frustration in human- computer dialog,

    J. Ang, R. Dhillon, A. Krupski, E. Shriberg, and A. Stolcke, “Prosody- based automatic detection of annoyance and frustration in human- computer dialog,” inProc. Int. Conf. Spoken Language Processing (ICSLP), 2002, pp. 2037–2040

    2002

  8. [12]

    Multichannel EEG-based emotion recognition via group sparse canonical correlation analysis,

    W.-L. Zheng, “Multichannel EEG-based emotion recognition via group sparse canonical correlation analysis,”IEEE Transactions on Cognitive and Developmental Systems, vol. 9, no. 3, pp. 281–290, Sep. 2017

    2017

  9. [13]

    Emotion recognition from surface EMG signal using wavelet transform and neural network,

    B. Cheng and G. Liu, “Emotion recognition from surface EMG signal using wavelet transform and neural network,” inProc. 2nd Int. Conf. Bioinformatics and Biomedical Engineering (iCBBE), 2008, pp. 1363– 1366

    2008

  10. [14]

    ECG pattern analysis for emotion detection,

    F. Agrafioti, D. Hatzinakos, and A. K. Anderson, “ECG pattern analysis for emotion detection,”IEEE Transactions on Affective Computing, vol. 3, no. 1, pp. 102–115, Jan. 2012

    2012

  11. [15]

    Hyper- complex multimodal emotion recognition from eeg and peripheral phys- iological signals,

    E. Lopez, E. Chiarantano, E. Grassucci, and D. Comminiello, “Hyper- complex multimodal emotion recognition from eeg and peripheral phys- iological signals,” in2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW). IEEE, 2023, pp. 1–5

    2023

  12. [16]

    Measuring emotion: The self-assessment manikin and the semantic differential,

    M. M. Bradley and P. J. Lang, “Measuring emotion: The self-assessment manikin and the semantic differential,”Journal of Behavior Therapy and Experimental Psychiatry, vol. 25, no. 1, pp. 49–59, 1994

    1994

  13. [17]

    A circumplex model of affect,

    J. A. Russell, “A circumplex model of affect,”Journal of Personality and Social Psychology, vol. 39, no. 6, pp. 1161–1178, 1980

    1980

  14. [18]

    The circumplex model of affect: An integrative approach to affective neuroscience, cognitive de- velopment, and psychopathology,

    J. Posner, J. A. Russell, and B. S. Peterson, “The circumplex model of affect: An integrative approach to affective neuroscience, cognitive de- velopment, and psychopathology,”Development and Psychopathology, vol. 17, no. 3, pp. 715–734, 2005

    2005

  15. [19]

    Pleasure–arousal–dominance: A general framework for describing and measuring individual differences in temperament,

    A. Mehrabian, “Pleasure–arousal–dominance: A general framework for describing and measuring individual differences in temperament,”Cur- rent Psychology, vol. 14, no. 4, pp. 261–292, 1996

    1996

  16. [20]

    Affect detection: An interdisciplinary review of models, methods, and their applications,

    R. A. Calvo and S. K. D’Mello, “Affect detection: An interdisciplinary review of models, methods, and their applications,”IEEE Transactions on Affective Computing, vol. 1, no. 1, pp. 18–37, 2010

    2010

  17. [21]

    Deap: A database for emotion analysis; using physiological signals,

    S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, “Deap: A database for emotion analysis; using physiological signals,”IEEE transactions on affective computing, vol. 3, no. 1, pp. 18–31, 2011

    2011

  18. [22]

    Ceap-360vr: A continuous physiological and behavioral emotion annotation dataset for 360◦ vr videos,

    T. Xue, A. El Ali, T. Zhang, G. Ding, and P. Cesar, “Ceap-360vr: A continuous physiological and behavioral emotion annotation dataset for 360◦ vr videos,”IEEE Transactions on Multimedia, vol. 25, pp. 243– 255, 2021

    2021

  19. [23]

    Amigos: A dataset for affect, personality and mood research on individuals and groups,

    J. A. Miranda-Correa, M. Khomami Abadi, N. Sebe, and I. Patras, “Amigos: A dataset for affect, personality and mood research on individuals and groups,”IEEE Transactions on Affective Computing, vol. 12, no. 2, pp. 479–493, 2018

    2018

  20. [24]

    Emotion recognition for everyday life using physiological signals from wearables: A systematic literature review,

    S. Saganowski, P. Bodera, M. Białowaset al., “Emotion recognition for everyday life using physiological signals from wearables: A systematic literature review,”IEEE Transactions on Affective Computing, 2022

    2022

  21. [25]

    Toward cross-subject and cross-session general- ization in eeg-based emotion recognition: Systematic review, taxonomy, and methods,

    A. Apicella, P. Arpaia, G. D’Errico, D. Marocco, G. Mastrati, N. Moc- caldi, and R. Prevete, “Toward cross-subject and cross-session general- ization in eeg-based emotion recognition: Systematic review, taxonomy, and methods,”Neurocomputing, vol. 604, p. 128354, 2024

    2024

  22. [26]

    A comprehensive re- view of deep learning in eeg-based emotion recognition: classifications, trends, and practical implications,

    W. Ma, Y . Zheng, T. Li, Z. Li, Y . Li, and L. Wang, “A comprehensive re- view of deep learning in eeg-based emotion recognition: classifications, trends, and practical implications,”PeerJ Computer Science, vol. 10, p. e2065, 2024

    2024

  23. [27]

    Autonomic nervous system activity in emotion: A review,

    S. D. Kreibig, “Autonomic nervous system activity in emotion: A review,”Biological Psychology, vol. 84, no. 3, pp. 394–421, 2010

    2010

  24. [28]

    Eeg-based multimodal emotion recognition: A machine learning per- spective,

    H. Liu, T. Lou, Y . Zhang, Y . Wu, Y . Xiao, C. S. Jensen, and D. Zhang, “Eeg-based multimodal emotion recognition: A machine learning per- spective,”IEEE Transactions on Instrumentation and Measurement, vol. 73, pp. 1–29, 2024

    2024

  25. [29]

    Multi- modal machine learning approach for emotion recognition using phys- iological signals,

    M. A. Ramadan, N. M. Salem, L. N. Mahmoud, and I. Sadek, “Multi- modal machine learning approach for emotion recognition using phys- iological signals,”Biomedical Signal Processing and Control, vol. 96, p. 106553, 2024

    2024

  26. [30]

    Advances in multimodal emotion recognition based on brain-computer interfaces,

    Z. He, Z. Li, F. Yang, L. Wang, J. Li, C. Zhou, and J. Pan, “Advances in multimodal emotion recognition based on brain-computer interfaces,” Brain sciences, vol. 10, no. 10, p. 687, 2020

    2020

  27. [31]

    Hierarchical hypercomplex network for multimodal emotion recognition,

    E. Lopez, A. Uncini, and D. Comminiello, “Hierarchical hypercomplex network for multimodal emotion recognition,” in2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2024, pp. 1–6

    2024

  28. [32]

    Corrfeat: correlation-based feature extraction algorithm using skin conductance and pupil diameter for emotion recognition,

    T. Zhang, A. El Ali, C. Wang, X. Zhu, and P. Cesar, “Corrfeat: correlation-based feature extraction algorithm using skin conductance and pupil diameter for emotion recognition,” in2019 International Conference on Multimodal Interaction, 2019, pp. 404–408

    2019

  29. [33]

    Behavioral and physiological signals- based deep multimodal approach for mobile emotion recognition,

    K. Yang, C. Wang, Y . Gu, Z. Sarsenbayeva, B. Tag, T. Dingler, G. Wadley, and J. Goncalves, “Behavioral and physiological signals- based deep multimodal approach for mobile emotion recognition,”IEEE 13 Transactions on Affective Computing, vol. 14, no. 2, pp. 1082–1097, 2021

    2021

  30. [34]

    Multimodal learning for eeg-based emotion recognition,

    R. Pillalamarri and U. Shanmugam, “Multimodal learning for eeg-based emotion recognition,”Artificial Intelligence Review, vol. 58, no. 3, pp. 459–478, 2025

    2025

  31. [35]

    Barlow twins: Self-supervised learning via redundancy reduction,

    J. Zbontar, L. Jing, I. Misra, Y . LeCun, and S. Deny, “Barlow twins: Self-supervised learning via redundancy reduction,” inInternational conference on machine learning, 2021, pp. 12 310–12 320

    2021

  32. [36]

    VICReg: Variance-invariance- covariance regularization for self-supervised learning,

    A. Bardes, J. Ponce, and Y . LeCun, “VICReg: Variance-invariance- covariance regularization for self-supervised learning,” inInternational Conference on Learning Representations, 2022

    2022

  33. [38]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” inInternational conference on machine learning, 2020, pp. 1597–1607

    2020

  34. [40]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  35. [41]

    Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text,

    H. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F. Chang, Y . Cui, and B. Gong, “Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text,”Advances in neural information processing systems, vol. 34, pp. 24 206–24 221, 2021

    2021

  36. [42]

    Imagebind: One embedding space to bind them all,

    R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V . Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 15 180–15 190

    2023

  37. [43]

    Contrasting with symile: Simple model-agnostic representation learning for unlim- ited modalities,

    A. Saporta, A. M. Puli, M. Goldstein, and R. Ranganath, “Contrasting with symile: Simple model-agnostic representation learning for unlim- ited modalities,”Advances in Neural Information Processing Systems, vol. 37, pp. 56 919–56 957, 2024

    2024

  38. [44]

    Gramian multimodal representation learning and alignment,

    G. Cicchetti, E. Grassucci, L. Sigillo, and D. Comminiello, “Gramian multimodal representation learning and alignment,” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  39. [45]

    Applications of self-supervised learning to biomedical signals: A survey,

    F. Del Pup and M. Atzori, “Applications of self-supervised learning to biomedical signals: A survey,”IEEE Access, vol. 11, pp. 144 180– 144 203, 2023

    2023

  40. [46]

    Self- supervised eeg emotion recognition models based on cnn,

    X. Wang, Y . Ma, J. Cammon, F. Fang, Y . Gao, and Y . Zhang, “Self- supervised eeg emotion recognition models based on cnn,”IEEE Trans- actions on Neural Systems and Rehabilitation Engineering, vol. 31, pp. 1952–1962, 2023

    1952

  41. [47]

    Self-supervised elec- troencephalogram representation learning for automatic sleep staging: model development and evaluation study,

    C. Yang, C. Xiao, M. B. Westover, and J. Sun, “Self-supervised elec- troencephalogram representation learning for automatic sleep staging: model development and evaluation study,”JMIR AI, vol. 2, no. 1, p. e46769, 2023

    2023

  42. [48]

    Self supervised learning based emotion recog- nition using physiological signals,

    M. Zhang and Y . Cui, “Self supervised learning based emotion recog- nition using physiological signals,”Frontiers in Human Neuroscience, vol. 18, p. 1334721, 2024

    2024

  43. [50]

    Linear dependence structure of the entropy space,

    T. Sun, “Linear dependence structure of the entropy space,”Inf. Control, vol. 29, no. 4, pp. 337–368, 1975

    1975

  44. [51]

    Measuring dependence with matrix-based entropy functional,

    S. Yu, F. Alesiani, X. Yu, R. Jenssen, and J. Principe, “Measuring dependence with matrix-based entropy functional,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 12, 2021, pp. 10 781–10 789

    2021

  45. [52]

    The normalized cross density functional: A framework to quantify statistical dependence for random processes,

    B. Hu and J. C. Principe, “The normalized cross density functional: A framework to quantify statistical dependence for random processes,” arXiv preprint arXiv:2212.04631, 2022

    2022 arXiv

  46. [53]

    Eeg analysis based on time domain properties,

    B. Hjorth, “Eeg analysis based on time domain properties,”Electroen- cephalography and Clinical Neurophysiology, vol. 29, no. 3, pp. 306– 310, 1970

    1970

  47. [54]

    Emotion recognition from eeg using higher order crossings,

    P. C. Petrantonakis and L. J. Hadjileontiadis, “Emotion recognition from eeg using higher order crossings,”IEEE Transactions on Information Technology in Biomedicine, vol. 14, no. 2, pp. 186–197, 2010

    2010

  48. [55]

    Differential entropy feature for eeg-based vigilance estimation,

    L.-C. Shi, Y .-Y . Jiao, and B.-L. Lu, “Differential entropy feature for eeg-based vigilance estimation,” inProceedings of the 35th Annual International Conference of IEEE Engineering in Medicine and Biology Society, 2013, pp. 6627–6630

    2013

  49. [56]

    Emotion classification based on gamma-band eeg,

    M. Li and B.-L. Lu, “Emotion classification based on gamma-band eeg,”Proceedings of the 31st Annual International Conference of IEEE Engineering in Medicine and Biology Society, pp. 1223–1226, 2009

    2009

  50. [57]

    What does the prefrontal cortex “do

    R. J. Davidson, “What does the prefrontal cortex “do” in affect: Perspectives on frontal eeg asymmetry research,”Biological Psychology, vol. 67, no. 1, pp. 219–234, 2004

    2004

  51. [58]

    A comparative study of window sizes for cnn-based eeg emotion recognition,

    P. Keelawat, W. Sattayaprasert, and T. Wilaiprasitporn, “A comparative study of window sizes for cnn-based eeg emotion recognition,”IEEE Access, vol. 9, pp. 152 340–152 352, 2021

    2021

  52. [59]

    An empirical survey of data augmentation for time series classification with neural networks,

    B. K. Iwana and S. Uchida, “An empirical survey of data augmentation for time series classification with neural networks,”Plos one, vol. 16, no. 7, p. e0254841, 2021

    2021

  53. [60]

    Eeg-based emotion recognition using multi-scale dynamic cnn and gated transformer,

    Z. Cheng, X. Bu, Q. Wang, T. Yang, and J. Tu, “Eeg-based emotion recognition using multi-scale dynamic cnn and gated transformer,” Scientific Reports, vol. 14, no. 1, p. 31319, Dec. 2024. [Online]. Available: https://www.nature.com/articles/s41598-024-82705-z

    2024

  54. [61]

    A survey on contrastive self-supervised learning,

    A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F. Makedon, “A survey on contrastive self-supervised learning,”Technologies, vol. 9, no. 1, p. 2, 2020

    2020

  55. [62]

    Contrastive self-supervised learning for neurodegenerative disorder classification,

    V . Gryshchuk, D. Singh, S. Teipel, M. Dyrba, and F. s. g. ADNI, AIBL, “Contrastive self-supervised learning for neurodegenerative disorder classification,”Frontiers in Neuroinformatics, vol. 19, p. 1527582, 2025

    2025

  56. [64]

    Self-supervised learning via vicreg enables training of emg pattern recognition us- ing continuous data with unclear labels,

    S. T. P. Raghu, D. T. MacIsaac, and E. J. Scheme, “Self-supervised learning via vicreg enables training of emg pattern recognition us- ing continuous data with unclear labels,”Computers in Biology and Medicine, vol. 185, p. 109479, 2025

    2025

  57. [65]

    Dreamer: A database for emotion recognition through eeg and ecg signals from wireless low-cost off- the-shelf devices,

    S. Katsigiannis and N. Ramzan, “Dreamer: A database for emotion recognition through eeg and ecg signals from wireless low-cost off- the-shelf devices,”IEEE journal of biomedical and health informatics, vol. 22, no. 1, pp. 98–107, 2017

    2017

  58. [66]

    GANSER: A Self-Supervised Data Augmentation Framework for EEG-Based Emotion Recognition,

    Z. Zhang, Y . Liu, and S. hua Zhong, “GANSER: A Self-Supervised Data Augmentation Framework for EEG-Based Emotion Recognition,”IEEE Transactions on Affective Computing, vol. 14, no. 3, pp. 2048–2063, 2023

  59. [67]

    Emotion recognition through combining eeg and eog over relevant channels with optimal windowing,

    H. Cai, X. Liu, R. Ni, S. Song, and A. Cangelosi, “Emotion recognition through combining eeg and eog over relevant channels with optimal windowing,”IEEE Transactions on Human-Machine Systems, vol. 53, no. 4, pp. 697–706, 2023

    2023

  60. [68]

    Multi-to-single: Reducing multimodal dependency in emotion recognition through contrastive learning,

    Y .-K. Liu, J. Cai, B.-L. Lu, and W.-L. Zheng, “Multi-to-single: Reducing multimodal dependency in emotion recognition through contrastive learning,” inProceedings of the AAAI Conference on Artificial Intel- ligence, vol. 39, no. 2, 2025, pp. 1438–1446

    2025

  61. [69]

    Accommo- dating audio modality in clip for multimodal processing,

    L. Ruan, A. Hu, Y . Song, L. Zhang, S. Zheng, and Q. Jin, “Accommo- dating audio modality in clip for multimodal processing,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 8, 2023, pp. 9641–9649

    2023

  62. [70]

    Multivariate information transmission,

    W. McGill, “Multivariate information transmission,”Transactions of the IRE Professional Group on Information Theory, vol. 4, no. 4, pp. 93– 111, 1954

    1954

  63. [71]

    Information theoretical analysis of multivariate correla- tion,

    S. Watanabe, “Information theoretical analysis of multivariate correla- tion,”IBM Journal of research and development, vol. 4, no. 1, pp. 66–82, 1960

    1960

  64. [72]

    Synergy, re- dundancy, and multivariate information measures: an experimentalist’s perspective,

    N. Timme, W. Alford, B. Flecker, and J. M. Beggs, “Synergy, re- dundancy, and multivariate information measures: an experimentalist’s perspective,”Journal of computational neuroscience, vol. 36, no. 2, pp. 119–140, 2014

    2014

  65. [73]

    What to align in multimodal contrastive learning?

    B. Dufumier, J. C. Navarro, D. Tuia, and J.-P. Thiran, “What to align in multimodal contrastive learning?” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  66. [74]

    Formal limitations on the measurement of mutual information,

    D. McAllester and K. Stratos, “Formal limitations on the measurement of mutual information,” inInternational Conference on Artificial Intel- ligence and Statistics, 2020, pp. 875–884

    2020

  67. [75]

    Universal features for high-dimensional learning and inference,

    S.-L. Huang, A. Makur, G. W. Wornell, and L. Zheng, “Universal features for high-dimensional learning and inference,”Foundations and Trends® in Communications and Information Theory, vol. 21, no. 1-2, pp. 1–299, 2024

    2024

  68. [76]

    Multi-variate correlation and mixtures of product measures,

    T. Austin, “Multi-variate correlation and mixtures of product measures,” arXiv preprint arXiv:1809.10272, 2018

    2018 arXiv

  69. [77]

    Investigating critical frequency bands and channels for eeg-based emotion recognition with deep neural networks,

    W.-L. Zheng and B.-L. Lu, “Investigating critical frequency bands and channels for eeg-based emotion recognition with deep neural networks,” IEEE Transactions on autonomous mental development, vol. 7, no. 3, pp. 162–175, 2015. 14

    2015

  70. [78]

    A Multimodal Database for Affect Recognition and Implicit Tagging (MAHNOB- HCI),

    M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A Multimodal Database for Affect Recognition and Implicit Tagging (MAHNOB- HCI),”IEEE Transactions on Affective Computing, vol. 3, no. 1, pp. 42–55, 2012. [Online]. Available: https://mahnob-db.eu/hci-tagging/

    2012

  71. [79]

    Multimodal emotion detection via attention-based fusion of facial and speech features,

    D. Mamieva, A. B. Abdusalomov, A. Kutlimuratov, B. Muminov, and T.- K. Whangbo, “Multimodal emotion detection via attention-based fusion of facial and speech features,”Sensors, vol. 23, no. 12, p. 5475, 2023

    2023

  72. [80]

    Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces,

    V . J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance, “Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces,”Journal of neural engineering, vol. 15, no. 5, p. 056013, 2018

    2018

  73. [81]

    Pytorch: An imperative style, high- performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high- p...

    2019

  74. [82]

    Measures of entropy from data using infinitely divisible kernels,

    L. G. S. Giraldo, M. Rao, and J. C. Principe, “Measures of entropy from data using infinitely divisible kernels,”IEEE Transactions on Information Theory, vol. 61, no. 1, pp. 535–548, 2014

    2014

  75. [83]

    Emotion recog- nition for everyday life using physiological signals from wearables: A systematic literature review,

    S. Saganowski, B. Perz, A. G. Polak, and P. Kazienko, “Emotion recog- nition for everyday life using physiological signals from wearables: A systematic literature review,”IEEE Transactions on Affective Computing, vol. 14, no. 3, pp. 1876–1897, 2022. 15 APPENDIXA PROOFS A. Proo...

    2022

  76. [84]

    Dataset:We used the officialData_preprocessed_python.ziprelease of DEAP, which already contains the 40 synchronous channels (32 EEG, 2 EOG, 2 EMG, 1 GSR, 1 respiration belt, 1 plethysmograph, 1 skin temperature) down- sampled to 128 Hz, band-pass filtered to 4-45 Hz, and basel...

  77. [85]

    Protocol:Each trial was further segmented into windows of{0.5,1,2, . . . ,20}s. To keep the number of training samples comparable across scales we used non-overlapping windows whenL≤5s and an adaptive overlap whenL >5s. For every window length we trained exactly the same encod...

  78. [86]

    Very short windows (<3s) yield a drop of≥8pp in macro-F 1, indicating insufficient temporal context

    Results:Both metrics rise steeply up to 10 s, where they attain their global maximum, plateau between 10-14 s with no significant difference (p >0.05), and then decline. Very short windows (<3s) yield a drop of≥8pp in macro-F 1, indicating insufficient temporal context. Figure...

  79. [87]

    The three benchmarks supply more physiological signals than are strictly necessary for emotion recognition

    Adopted setting:We therefore fix the window length to10 sin all subsequent experiments—a choice that yields the best empirical performance on DEAP and lies comfortably within the range recommended by prior work. The three benchmarks supply more physiological signals than are s...

  80. [88]

    Method:Using the 10 s window length identified earlier, we trained the same encoder as in all main experiments, but appended a learnable modality-attention mask. Concretely, after global spatial-temporal pooling the model produces a d-dimensional embedding for every modalitym;...

  81. [89]

    The mask assigns the largest weights to all 32 EEG leads, both horizontal and vertical EOG, and the skin-temperature trace, resulting in 35 retained inputs

    DEAP [21]:Seven signals (32-channel EEG, horizontal and vertical EOG, Zygomaticus and Trapezius EMG, GSR, respiration belt, plethysmograph, skin temperature) are available. The mask assigns the largest weights to all 32 EEG leads, both horizontal and vertical EOG, and the skin...

  82. [90]

    The attention mask ranks BVP, EDA and SKT highest; those three signals are therefore kept for downstream experiments

    CEAP-360VR [22]:This corpus contains only peripheral physiology sampled at approximately 30 Hz: three-axis accelerometer, BVP, EDA, SKT, heart rate (HR) and inter-beat interval (IBI). The attention mask ranks BVP, EDA and SKT highest; those three signals are therefore kept for...

  83. [91]

    Opt.z-score

    MAHNOB-HCI [78]:MAHNOB logs 32-channel EEG plus four peripheral modalities: ECG, GSR, respiration amplitude and skin temperature. The mask highlights EEG, GSR and ECG, so we restrict the input to these three. This automatic gating reduces the input dimensionality by 12-60 (dep...

  84. [92]

    •VICReg.Identical pipeline to SimCLR (noise0.05, shift±10, dropout0.1)

    CEAP-360VR: •SimCLR.Physio Augmentation = Gaussian noise (σ= 0.05)→random shift (±10samples)→channel dropout (p= 0.1) [1]. •VICReg.Identical pipeline to SimCLR (noise0.05, shift±10, dropout0.1). •Barlow Twins.Noise with higher magnitude (σ= 0.1) + shift (±10); no channel dropo...

  85. [93]

    •VICReg.Matches SimCLR exactly (noise0.05, shift±10, dropout0.1)

    MAHNOB-HCI: •SimCLR.EEG Augmentation: noise (σ= 0.05), shift (±10), dropout (p= 0.1). •VICReg.Matches SimCLR exactly (noise0.05, shift±10, dropout0.1). •Barlow Twins.Noise (σ= 0.1) and shift (±10) only; channel dropout omitted. Optional clippedz-score normalisation (off by default)

  86. [94]

    •VICReg.Same settings as SimCLR (noise0.05, shift±10, dropout0.1)

    DEAP: •SimCLR.EEG Augmentation: Gaussian noise (σ= 0.05), shift (±10), dropout (p= 0.1). •VICReg.Same settings as SimCLR (noise0.05, shift±10, dropout0.1). •Barlow Twins.Noise (σ= 0.1) + shift (±10); no dropout. Optional per-windowz-score with±3σclipping (disabled). APPENDIXD ...

  87. [95]

    Why two notebooks?:We provideonenotebook per experimentMFMC_DEAP_main1_subject_dep.ipynbfor thesubject-dependentsplit andMFMC_DEAP_main2_subject_indep.ipynbfor thesubject-independentsplit. Each notebook contains the fullfive-foldcross-validation loop: the code trains folds 1-5...

  88. [96]

    Run the following once: •Unzip raw data:Ensure thats01.dat–s32.datreside in<BASE_PATH>/DEAP/

    Prerequisite and preprocessing:The notebooks expect NumPy tensors produced from the official data_preprocessed_python.zip. Run the following once: •Unzip raw data:Ensure thats01.dat–s32.datreside in<BASE_PATH>/DEAP/. •Activate the environment:(see next sub-section) and runpyth...

  89. [97]

    •Core scientific libraries:numpy 1.21.6,scipy 1.9.3,scikit-learn 1.1.3,matplotlib 3.5.3,joblib 1.2.0

    Software setup:Create a Conda environment with Python 3.9 (any≥3.8 works) and install: •PyTorch stack(CUDA 11.3):torch 1.12.1,torchvision 0.13.1. •Core scientific libraries:numpy 1.21.6,scipy 1.9.3,scikit-learn 1.1.3,matplotlib 3.5.3,joblib 1.2.0. 20 B. Data and Model Pipeline

  90. [98]

    HereN= 20 097, reflecting10s windows (1 280 samples at 128 Hz) extracted with a0.4s stride

    Data flow:All experiments begin with five NumPy tensors generated in preprocessing:eeg_data.npy(N,32,1280), eog_data.npy(N,2,1280),temp_data.npy(N,1,1280),emotion_labels.npy(N), andsubject.npy(N) stored in<BASE_PATH>/Data_processed/. HereN= 20 097, reflecting10s windows (1 280...

  91. [99]

    •Temporal networkShared four-block Conv→Batch Norm→ReLU→Max Pool backbone with filter widths 1→32 →64→128→256, kernel size 11, pooling factor 4 at each block

    Model architecture:Each modality encoder has two stages: a temporal network that processes each sensor channel separately, followed by a channel network that fuses the channels into a 128-D vector. •Temporal networkShared four-block Conv→Batch Norm→ReLU→Max Pool backbone with ...

  92. [100]

    Loop schedule:Each fold is trained for20 000 iterations. Every iteration runs two optimisers back-to-back: •Unsupervised step:Draw a mini-batch (B= 200) of EEG, EOG, and temperature windows, compute the MFMC loss, and updateall_feature_paramswith Adam (lr= 3×10 −4,β 1 = 0.5,β ...

  93. [101]

    Each fold produces a JSON log and a loss-accuracy curve figure inresults/fold_k/

    Cross-validation outputs:Five subject-balanced folds are trained sequentially. Each fold produces a JSON log and a loss-accuracy curve figure inresults/fold_k/. The final notebook cell merges these files intocv_summary.csv, a combined learning-curve plot, and a confusion-matrix image

  94. [102]

    Subject- independent folds reach34.6%±3.7best accuracy and26.9 %±3.2final accuracy

    Accuracy:Subject-dependent folds achieve98.3%±1.4best accuracy and96.4 %±1.8final accuracy. Subject- independent folds reach34.6%±3.7best accuracy and26.9 %±3.2final accuracy. Most remaining errors involve high- versus low-arousal quadrants; temperature contributes the least d...

  95. [103]

    Runtime:One fold trains in roughly one hour on a single NVIDIA L40s GPU (48 GB) with 16 logical CPU cores, so the full five-fold run finishes in about five hours wall-clock

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.