Pith. sign in

REVIEW 4 major objections 6 minor 71 references

Camera-based implicit mind reading by capturing higher-order semantic dynamics of human gaze within environmental context

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the order in which a person's gaze visits meaningful objects in a scene, captured with ordinary cameras, reveals emotional state at roughly EEG-level accuracy (88.74% vs 88.94%) and beats EEG on concealed emotions.

desk verdict A substantial pipeline and new datasets are undermined by a central claim that is not actually tested: the near-EEG accuracy comes from eye-tracker input, not from the camera system. read the letter →

arxiv 2507.12889 v1 pith:3RK323WY submitted 2025-07-17 cs.CV

Mengke Song , Yuge Xie , Qi Cui , Luming Li , Xinyu Liu , Guotao Wang , Chenglizhao Chen , Shanchen Pang This is my paper
classification cs.CV
keywords gaze-basedemotionrecognitionenvironmentalcontextSemanticInteractiveOrdersuser-unawaremonitoringgazescanpathdeceptivedetectioncamera-basedtrackingemotion-attentioninteraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a person's emotional state can be read unobtrusively from the way their gaze moves through the meaningful contents of a scene. Instead of treating gaze as a static fixation point, it models the sequence of semantic objects a person looks at and in what order — the Semantic Interactive Orders — and claims this sequence reveals which of six basic emotions the person is in. On its own collected data the method reaches 88.74% overall accuracy, close to the 88.94% of an EEG-based system, and it does better than EEG (89.82% versus 87.13%) when people try to conceal their emotion. Because everything is captured by standard HD cameras with no wearable devices and no active user task, the authors argue, continuous real-time emotion monitoring becomes practical in classrooms, cockpits, and vehicles. The broader claim is that emotions are not purely internal physiological states but products of human-environment interaction, and that this interaction can substitute for physiological sensing.

What carries the argument

The load-bearing object is the Semantic Interactive Orders (SIO) representation: gaze fixation points predicted from eye appearance and head movement are mapped onto object-level regions of a panoramic scene, and the regions are arranged in viewing order, so each emotion is encoded as an ordered patch sequence over the scene's semantics. That sequence drives EmoGazeNet, a generative-adversarial classifier whose generator applies spatial-temporal positional encoding (object position plus viewing time) inside a Transformer encoder, while its discriminator enforces separation between ordinary appearance features and scanpath-derived temporal-spatial features through an adversarial reverse-suppression loss and an auxiliary scanpath-prediction task. Two supporting mechanisms carry the system: a third-person multi-camera pipeline (eight HD cameras, super-resolution, 3D reconstruction of eye appearances, and online personalized calibration fusing saliency-based 'objective' fixations with user-specific 'subjective' fixations) that produces gaze trajectories without wearables, and the cawF1 evaluation metric ($\mathrm{cawF1}=\sum_i \mathrm{FCC}_i\,\mathrm{bF1}_i/\sum_i \mathrm{FCC}_i$, with FCC a fixation-context consistency score) that holds the model to predicting where people look, not only the emotion label.

What would settle it

Record gaze and scene data from people experiencing genuine, un-instructed emotions — for example during real events where the felt emotion is confirmed at the moment it happens by independent self-report or physiological recording — and run EmoGazeNet trained only on instructed emotions; if accuracy on genuinely felt emotions falls far below the reported 88.74% (or toward chance), the claim that the method reads internal states rather than acted behavior would be refuted.

Watch

Extended reading notes

Core claim

The paper's core claim is that emotion is legible in how visual attention travels through a scene's semantics over time, not just where the gaze rests. Concretely, predicted gaze fixation points are mapped onto object-level regions of a panoramic reconstruction of the environment, these regions are ordered by viewing sequence, and the ordered patch sequence is fed into EmoGazeNet, a GAN-based classifier with a Transformer encoder, scanpath-guided and auxiliary classification branches, and adversarial reverse suppression that keeps appearance features and scanpath features distinct. On the 2D screen dataset the method reports 89.82% accuracy on deceptive emotions and 87.65% on real emotions (88.74% overall), against 88.94% overall for the EEG baseline; on the 360-degree dataset its average cawF1 of 78.14% trails EEG by 1.26% while surpassing facial and gaze-only baselines, and in campus and driving-simulator field experiments it outperforms both facial-expression and physiological-signal methods (81.45% and 83.85% accuracy). The authors interpret these results as showing that gaze-environment interaction dynamics can replace physiological sensors, and that this 'user-unaware' monitoring — no wearables, no active participation — brings continuous emotion recognition into real-world settings.

Load-bearing premise

All emotion labels come from instructed induction or self-simulation — participants watched emotion-triggering videos or were asked to simulate an emotional state — and the paper never verifies that anyone actually felt the labeled emotion, so the system may be learning posed gaze patterns rather than internal states.

Editorial extensions

If this is right

  • If the accuracy numbers hold, continuous emotion monitoring becomes deployable with off-the-shelf HD cameras at EEG-level accuracy (88.74% versus 88.94%) in classrooms, cockpits, and driver monitoring, without wearable sensors or active user participation.
  • Concealed emotion, which defeats facial-expression systems (43.91% deceptive accuracy), is recognized at 89.82% accuracy — above the EEG baseline of 87.13% — so gaze-environment dynamics leak information even when facial behavior is controlled.
  • Field results in a campus square and a driving simulator (81.45% and 83.85% accuracy, with significant ANOVA contrasts against the facial and physiological baselines) support generalization beyond the laboratory, and 17-day monitoring stays stable near 71.5%–72% cawF1.
  • Adopting the proposed cawF1 metric raises the field's evaluation bar: a model must predict both the emotion and the attended regions of the scene, and under that stricter standard the complete system scores 72.22% on real-scene data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because all labels come from instructed induction or self-simulation, the reported accuracy may measure recognition of posed gaze behavior; a test with naturally occurring emotions and independently confirmed labels is the untaken step that would decide whether the 'mind reading' framing is justified.
  • The SIO hypothesis implies a falsifiable regularity: the same person in the same scene should produce statistically distinct object-visitation orders under different emotions, and if scanpath orders do not separate by emotion, the information channel the model relies on would be empty.
  • The current pipeline requires a pre-modeled panoramic reconstruction and fixed camera positions, so moving to unbounded or moving environments would need real-time semantic segmentation and view synthesis — a natural next step the paper only gestures at.
  • Because the system is designed to be 'user-unaware', deployment raises a consent tension: continuous emotion reading without the person's knowledge would require institutional oversight and opt-in policies before classroom or cockpit use becomes ethical.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a camera-based, user-unaware emotion recognition pipeline that estimates gaze from multi-view HD camera images of the eye and head, maps gaze onto semantic objects in a reconstructed panoramic environment, and feeds the resulting scanpath, represented as Semantic Interactive Orders (SIO), into a GAN-based model called EmoGazeNet. The authors introduce three datasets (EmoGaze2D-50, EmoGaze360-1K, Real360), a new metric cawF1, an online calibration method, and a third-person panoramic modeling approach. They report that the context gaze-based method reaches 88.74% overall accuracy on the 2D screen dataset, close to an EEG-based method (88.94%), and that it outperforms gaze-only and facial-expression baselines in real-scene field experiments. The central claim is that ordinary HD cameras can achieve near-EEG-level emotion recognition without wearable sensors, in a user-unaware manner.

Significance. If substantiated, the claim that a camera-only gaze pipeline can match EEG-based emotion recognition would be an important practical result for affective computing, enabling unobtrusive, scalable monitoring in education, driving, and public safety. The paper contains several commendable elements: it attempts real-world deployment in campus and driving-simulator settings, includes longitudinal stability experiments over 17 days, provides component ablations for EmoGazeNet, and proposes a richer representation of gaze as a sequence of semantically meaningful fixations rather than raw coordinates. The scanpath visualizations by gender and emotion are also a useful descriptive contribution. However, the headline quantitative claim is not actually tested end-to-end, and the evaluation metrics and ground-truth labeling have self-referential elements, so the significance of the reported numbers is currently uncertain.

major comments (4)
  1. [Results, 'Performance comparison between proposed deep model EmoGazeNet with existing emotion recognition methods on…] The headline result of 88.74% overall accuracy, which is the basis for the claim of near-EEG performance, is computed on EmoGaze2D-50 using gaze data collected with a standard eye tracker, not with the proposed eight-HD-camera gaze pipeline. The supplementary states that EmoGaze2D-50 shares the collection setup of EmoGaze360-1K, and that EmoGaze360-1K used the WinDB HMD-free approach with a standard eye tracker. The camera-based gaze acquisition method is validated separately on Real360 and the two field experiments, where accuracy ranges from 80.22% to 83.85%. The paper never reports end-to-end performance of the camera-based gaze pipeline on EmoGaze2D-50 or EmoGaze360-1K. Consequently, the central claim that a camera-only system matches EEG-level emotion recognition is not supported by the reported experiments; the 88.74% figure reflects a system with laboratory-grade eye-tracking input. This is a load-bearing mismatch between the claim and the measurement and must be addressed, either by reporting end-to-end camera results on the benchmark datasets or by substantially revising the claim.
  2. [Methods, 'Proposed evaluation metric', Eqs. (4)-(5)] The cawF1 metric is proposed in this paper and is used to rank all methods on EmoGaze360-1K and in the field experiments, but it is not computable in a meaningful way for baselines that do not produce gaze or fixation predictions. Facial-expression and EEG baselines have no fixation-context consistency term, so it is unclear whether FCC is set to a constant, omitted, or computed from some proxy; without this definition, the cawF1 comparisons in Figure 4-b and Figures 6-d/6-e are not interpretable. Additionally, the weights alpha and beta in Eq. (5) are never reported. The paper should state the values of alpha and beta, describe how cawF1 is applied to each baseline, and provide a version of the comparison using standard metrics only, or justify why cawF1 is appropriate for all methods.
  3. [Methods, 'Collection setting'; 'Field experiment to evaluate the practical application of our eye gaze collection…] The ground-truth emotion labels are based on instructed emotion induction or self-simulation, and the paper does not verify that participants actually experienced the target emotion. The 'Collection setting' section states that participants underwent emotion induction through video and image stimuli, while the campus field experiment explicitly asked participants to 'simulate six different emotional states'. No manipulation check, self-report rating, or physiological verification is reported. If participants only acted the emotion, the system may be learning to classify posed gaze behavior rather than internal emotional states, which would invalidate the 'mind reading' framing and the claim of recognizing 'real emotions' in the abstract. The authors should add a manipulation check or clearly restrict their claims to acted/induced emotional behavior.
  4. [Results, 'Performance comparison between proposed deep model EmoGazeNet with existing emotion recognition methods on…] The baselines ACTNN, Toisoul, and CCER are mentioned by name but no implementation details are given for how they were adapted to the new datasets, what input modalities they received, whether they were retrained, or how their cawF1 scores were obtained. The statistical tests are also under-specified: the two-sample t-test and one-way ANOVA p-values are reported without describing the number of subjects, the folds, whether the tests are paired, or the exact comparisons being made. Without this information, the claimed statistical superiority over the EEG-based method on deceptive emotions cannot be assessed. Please provide a clear evaluation protocol, including dataset splits, subject independence, and baseline configurations.
minor comments (6)
  1. [Methods, 'Camera-based gaze tracking method'] There is a typo: 'Furthre' should be 'Further'. Similar typographical issues appear elsewhere, including 'ANOV A' instead of 'ANOVA' and 'fiaxtion' instead of 'fixation' in the supplementary architecture description.
  2. [Figure 5-b and Supplementary 'Detailed explanations of six indicators...'] The six indicators (emotion sensitivity, stability, etc.) are described as quantified on a 1-10 scale, but the method for assigning these ratings is not given; it is unclear whether they come from the model, from human annotators, or from the authors. Please specify the source of these ratings and any inter-rater reliability if humans were involved.
  3. [References] References 52 and 66 are duplicates of the same paper (ShanghaiTechGaze); one of them should be removed and the citation list renumbered.
  4. [Methods, 'Online personalized calibration'] The calibration method depends on several thresholds and windows (e.g., head movement yaw/pitch thresholds, the 200-300 ms online fine-tuning window) that are described qualitatively but never given concrete values. Please provide the actual thresholds used in the experiments.
  5. [Supplementary, 'EmoGaze360-1K and EmoGaze2D-50 datasets construction'] The dataset construction section says EmoGaze360-1K contains 1,000 panoramic images but later mentions '500 emotion-inducing images and 50 emotion-inducing videos... resulting in a total of 2,500 images and 250 videos for each emotional state'; the relationship between these stimulus pools and the 1,000 annotated panoramas should be clarified.
  6. [Results, 'Robustness evaluation'] The long-term stability experiment is reported only as cawF1 fluctuations around 71-73%, but no statistical test or confidence interval is provided; adding a trend test or at least a variance estimate would strengthen the claim of stability.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the emotion predictions are learned outputs, not restatements of the inputs; the main caveats are an eye-tracker/camera mismatch and a self-defined evaluation metric, which are validity concerns rather than circularity.

full rationale

The paper's derivation chain is a supervised pipeline: eight HD cameras estimate gaze scanpaths, EmoGazeNet consumes ERP images plus scanpaths and outputs an emotion distribution, and the six emotion labels come from an emotion-induction protocol. No equation in the paper defines an emotion label as a deterministic function of the gaze/context features, so the predictions are not equivalent to the inputs by construction. The proposed cawF1 metric (Eqs. 4-5) is an evaluation weighting, not a generative model; using it to rank methods may be questionable because alpha and beta are unreported and fixation-context consistency is not defined for EEG- or facial-based baselines, but that is metric validity, not circularity. The most serious weakness is an internal mismatch: the headline 88.74% result is obtained on EmoGaze2D-50/EmoGaze360-1K, whose supplementary Methods state the gaze data were collected with 'a standard eye tracker' under the WinDB approach, while the camera-based gaze pipeline is validated separately on Real360 and field experiments at roughly 80-84% accuracy. This means the central 'camera-based' claim is not tested end-to-end, but it is an unsupported extrapolation rather than a circular reduction. The paper also cites prior work by its own authors (e.g., WinDB, ref. 56) for fixation collection and backbone choices, but these citations are not load-bearing uniqueness arguments and do not by themselves force the reported accuracy values. Verdict: no significant circularity; score 2 reflects the minor self-citation and the self-defined metric, not a circular derivation.

Assumptions & free parameters 7 free parameters · 5 assumptions · 6 invented entities

The method has no derivation; it is an empirical pipeline. The central claim rests on assumptions about gaze-emotion coupling, instructed labels, object-box gaze accuracy, and fair use of the new cawF1 metric. All comparisons use self-built datasets, and key weights and thresholds are unreported, so the number of assumptions the reader must grant is high.

free parameters (7)
  • cawF1 weights alpha, beta = not reported
    Eq. 5 defines FCC = alpha*Sim(local) + beta*Sim(global) with alpha+beta=1. These weights control the metric used for all headline comparisons, but their values are not given.
  • loss balance lambda in Laux_cls = not reported
    Eq. 10 uses lambda to weight the adversarial suppression term; no value or tuning procedure is given.
  • loss balance beta in Ltraj = not reported
    Eq. 11 uses beta to weight the reverse suppression term; value unreported.
  • scanpath reconstruction loss weights alpha, beta = not reported
    Eq. 14 combines MSE and DTW with coefficients alpha and beta; values unreported.
  • projection interval T = 0.1 s
    Chosen as optimal from Supplementary Figure 5; this is a performance-tuned acquisition parameter on the authors' data.
  • number of HD cameras = 8
    Selected as the 'optimal balance' in Figure 7-c using the authors' data.
  • online calibration window = 200-300 ms
    Hand-chosen window for recalibration during scene transitions (Methods, Online Personalized Calibration).
assumptions (5)
  • domain assumption Gaze behavior is an overt and reliable indicator of emotional state through attention allocation.
    The entire framework rests on this link; cited neuroscience (refs 43-46) is suggestive, not a validated mapping for six-class recognition.
  • domain assumption Instructed emotion induction and self-simulation produce valid ground-truth emotional states.
    All labels come from watching emotion-inducing stimuli or instructed simulation; actual internal state is never measured. This is the load-bearing premise for every accuracy number.
  • domain assumption Object-level semantic mapping from gaze coordinates accurately represents what the user attends to.
    Gaze accuracy is measured by whether predicted coordinates fall in any object's bounding box (Eqs. 1-3), which conflates looking near an object with attending to that object.
  • ad hoc to paper cawF1 can be computed fairly for baselines that do not produce gaze or fixation predictions.
    Eqs. 4-5 require fixation context features; the paper does not describe what FCC values are assigned to EEG and facial-expression baselines in Figure 4-b.
  • standard math Standard mathematical tools: cosine similarity, KL divergence, mutual information approximation, DTW, and GAN training are valid for this setting.
    Used in Eqs. 5, 7-16; no novel mathematical claims are made.
invented entities (6)
  • Semantic Interactive Orders (SIO)
    purpose: Represent gaze as an ordered sequence of semantically labeled object patches for emotion classification.
    Introduced in this paper; no external benchmark validates SIO as an emotion representation.
  • cawF1 metric
    purpose: Evaluation metric combining classification with fixation-context consistency.
    Defined in Eqs. 4-5; its validity is asserted, not independently established, and baseline treatment is unspecified.
  • EmoGazeNet model
    purpose: GAN-based classifier mapping SIO sequences to emotion distributions.
    Evaluated only on the authors' datasets; no external replication.
  • Online personalized calibration
    purpose: Adapt gaze estimation to individual eye appearance and head movement.
    Described through a teacher-student mechanism; no external benchmark against head-mounted eye trackers other than one comparison in Figure 7-b.
  • EmoGaze2D-50, EmoGaze360-1K, and Real360 datasets
    purpose: Training and evaluation data for the method.
    Not released; labels are induction-based and cannot be independently checked.
  • Third-person multi-camera panoramic modeling
    purpose: Reconstruct the user's first-person field of view without wearable cameras.
    No direct validation against true first-person views; only indirect emotion recognition performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Camera-based implicit mind reading by capturing higher-order semantic dynamics of human gaze within environmental context." pith.science (2026). https://pith.science/paper/3RK323WY

@misc{pith2026250712889,
  author       = {Pith},
  title        = {Pith review of: Camera-based implicit mind reading by capturing higher-order semantic dynamics of human gaze within environmental context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3RK323WY}},
  note         = {Machine review of arXiv:2507.12889}
}
read the original abstract

Emotion recognition,as a step toward mind reading,seeks to infer internal states from external cues.Most existing methods rely on explicit signals-such as facial expressions,speech,or gestures-that reflect only bodily responses and overlook the influence of environmental context.These cues are often voluntary,easy to mask,and insufficient for capturing deeper,implicit emotions. Physiological signal-based approaches offer more direct access to internal states but require complex sensors that compromise natural behavior and limit scalability.Gaze-based methods typically rely on static fixation analysis and fail to capture the rich,dynamic interactions between gaze and the environment,and thus cannot uncover the deep connection between emotion and implicit behavior.To address these limitations,we propose a novel camera-based,user-unaware emotion recognition approach that integrates gaze fixation patterns with environmental semantics and temporal dynamics.Leveraging standard HD cameras,our method unobtrusively captures users'eye appearance and head movements in natural settings-without the need for specialized hardware or active user participation.From these visual cues,the system estimates gaze trajectories over time and space, providing the basis for modeling the spatial, semantic,and temporal dimensions of gaze behavior. This allows us to capture the dynamic interplay between visual attention and the surrounding environment,revealing that emotions are not merely physiological responses but complex outcomes of human-environment interactions.The proposed approach enables user-unaware,real-time,and continuous emotion recognition,offering high generalizability and low deployment cost.

Figures

Figures reproduced from arXiv: 2507.12889 by the authors.

Figure 1
Figure 1. Comparison of existing emotion recognition methods. a Comparing traditional explicit-centric and novel interaction-aware emotion recognition methods using gaze and environmental dynamics. b Based on the depth of emotional understanding they provide and the complexity of their setup, emotion recognition methods are divided into four methods: Method 1 (facial-based) has deceptive facial and low accuracy; Method 2 phys… view at source ↗
Figure 2
Figure 2. Human-environment interaction for contextual gaze-based emotion recognition. (a) illustrates the concept of leveraging human-environment interaction to infer emotions. (b) introduces a novel contextual gaze-based approach that combines fixation scanpaths with semantic understanding for deeper emotional insights. (c) A multi-camera system captures eye appearances and head movements to enable user-unaware, real-world … view at source ↗
Figure 3
Figure 3. Calibration and semantic-aware modeling for improved contextual gaze-based emotion recognition. a shows an online calibration method that combines subjective (user-specific) and objective (scene-based) fixations to dynamically adapt gaze tracking for personalized emotion recognition. b compares traditional gaze-coordinate methods with the proposed Semantic Interactive Orders (SIO) framework, which integrates coordin… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Experimental validation of our proposed method against other methods and its performance on screen scenes. Quantitative comparisons between our context gaze-based method and physiological signal (such as EEG)-based (ACTNN), facial expression-based (Toisoul), gaze (such…
Figure 5
Figure 5. Figure 5: Experimental validation of our proposed method on different settings and its performance on real scenes. a For real scenes (four conditions), our method achieves the best performance in high-light scene and the worst performance in low-light scene. b This radar chart i…
Figure 6
Figure 6. Figure 6: Robustness validation of our proposed method and component evaluation. a Long-term stability monitoring experiment indicates that during the long-term monitoring period, the proposed method can continuously provide a relatively consistent level of accuracy in emotional…
Figure 7
Figure 7. Figure 7: Ablation studies on Real360 dataset. a When collecting users’ gaze points in the real-world scene of the Real360 dataset, while 360-degree videos generate better user responses than 360-degree images, the overall differences between 2D images, 360-degree images, and 36…
Figure 8
Figure 8. Figure 8: Pipeline of the proposed EmoGazeNet model. The proposed EmoGazeNet model consists of a Generator and a Discriminator. This approach leverages the varying gaze patterns towards different viewing object orders under different emotional states to enhance emotion recogniti…
Figure 9
Figure 9. Figure 9: Performance comparison of different scanpath prediction methods (HAT57, IndivScan58, ScanGan36060, and ScanDMM61) in the Generator of EmoGazeNet model. 27/32 [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]
Figure 10
Figure 10. Figure 10: Performance comparison of different choices of base encoders (GNN62, CNN63, LSTM64, and Transformer65) in the Generator of EmoGazeNet model. 28/32 [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]
Figure 11
Figure 11. Figure 11: Performance comparison of different gaze point prediction methods (ShanghaiTechGaze66, L2CS-Net67, GazeTR68, and AFF-Net69.) 29/32 [PITH_FULL_IMAGE:figures/full_fig_p029_11.png]
Figure 12
Figure 12. Figure 12: Performance comparison of different projection intervals (T=0.1, T=0.2, T=0.3) during eye appearance acquisition and gaze point generation. 30/32 [PITH_FULL_IMAGE:figures/full_fig_p030_12.png]
Figure 13
Figure 13. Figure 13: Performance comparison of different number of EmoGaze360-1K dataset in training EmoGazeNet model. 31/32 [PITH_FULL_IMAGE:figures/full_fig_p031_13.png]
Figure 14
Figure 14. Figure 14: Scanpath visualization of real 360-degree static (A) and dynamic (B) scenes in different genders and emotion states. “Positive Emotion”: happy; “Negative Emotion”: fear, sad, disgust and angry. Positive Emotion Negative Emotion Male Female Male Female A B [PITH_FULL_…
Figure 15
Figure 15. Figure 15: Scanpath visualization of real 360-degree high-light (A) and low-light (B) scenes in different genders and emotion states. “Positive Emotion”: happy; “Negative Emotion”: fear, sad, disgust and angry. 32/32 [PITH_FULL_IMAGE:figures/full_fig_p032_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

71 extracted references · 69 canonical work pages

  1. [1]

    P., Muttenthaler, L., Güçlü, U

    Mahner, F. P., Muttenthaler, L., Güçlü, U. & Hebart, M. N. Dimensions underlying the representational align- ment of deep neural networks with humans. arXiv preprint arXiv:2406.19087 (2024)

    arXiv 2024

  2. [2]

    W., Miner, A

    Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C. & Althoff, T. Human–ai collaboration enables more empathic conversa- tions in text-based peer-to-peer mental health support. Nat. Mach. Intell. 5, 46–57 (2023)

    work page 2023

  3. [3]

    & Damasio, A

    Man, K. & Damasio, A. Homeostasis and soft robotics in the design of feeling machines. Nat. Mach. Intell. 1, 446–452 (2019)

    work page 2019

  4. [4]

    & Nichele, S

    Zhang, J., Yin, Z., Chen, P. & Nichele, S. Emotion recogni- tion using multi-modal data and machine learning techniques: A tutorial and review. Inf. Fusion 59, 103–126 (2020)

    work page 2020

  5. [5]

    Awais, M. et al. Lstm-based emotion detection using phys- iological signals: Iot framework for healthcare and distance learning in covid-19. IEEE Internet Things J. 8, 16863–16871 (2020)

    work page 2020

  6. [6]

    Vine, V ., Boyd, R. L. & Pennebaker, J. W. Natural emotion vocabularies as windows on distress and well-being. Nat. Commun. 11, 4525 (2020)

    work page 2020

  7. [7]

    Schaare, H. L. et al. Associations between mental health, blood pressure and the development of hypertension. Nat. Commun. 14, 1953 (2023)

    work page 2023

  8. [8]

    Jiang, R. et al. The brain structure, inflammatory, and genetic mechanisms mediate the association between physical frailty and depression. Nat. Commun. 15, 4411 (2024)

    work page 2024

Show all 71 references
  1. [9]

    Zhao, S. et al. Curriculum cyclegan for textual sentiment domain adaptation with multiple sources. In Proceedings of the Web Conference 2021, 541–552 (2021)

    2021

  2. [10]

    Zhao, S. et al. An end-to-end visual-audio attention network for emotion recognition in user-generated videos. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol. 34, 303–311 (2020)

    2020

  3. [11]

    Jia, G. et al. Modulating emotional states of rats through a rat-like robot with learned interaction patterns. Nat. Mach. Intell. 6, 1580–1593 (2024)

    2024

  4. [12]

    & FeldmanHall, O

    Heffner, J. & FeldmanHall, O. A probabilistic map of emo- tional experiences during competitive social interactions. Nat. communications 13, 1718 (2022)

    2022

  5. [13]

    & Hung, C.-H

    Hsu, Y .-L., Wang, J.-S., Chiang, W.-C. & Hung, C.-H. Auto- matic ecg-based emotion recognition in music listening.IEEE Trans. Affect. Comput. 11, 85–99 (2020)

    2020

  6. [14]

    esee-d: Emotional state estimation based on eye-tracking dataset

    Skaramagkas, V .et al. esee-d: Emotional state estimation based on eye-tracking dataset. Brain Sci. 13, 589 (2023). 15/32

    2023

  7. [15]

    & Ding, G

    Zhao, S., Hong, X., Yang, J., Zhao, Y . & Ding, G. Toward label-efficient emotion and sentiment analysis. Proc. IEEE 111, 1159–1197 (2023)

    2023

  8. [16]

    Pekrun, R., V ogl, E., Muis, K. R. & Sinatra, G. M. Measuring emotions during epistemic activities: the epistemically-related emotion scales. Cogn. Emot. 31, 1268–1276 (2017)

    2017

  9. [17]

    Russell, J. A. Measures of emotion. In The Measurement of Emotions, 83–111 (Elsevier, 1989)

    1989

  10. [18]

    M., Rae, G

    Nelis, S. M., Rae, G. & Liddell, C. The level of expressed emotion scale: A useful measure of expressed emotion in adolescents? J. Adolesc. 34, 311–318 (2011)

    2011

  11. [19]

    Lee, J. P. et al. Encoding of multi-modal emotional informa- tion via personalized skin-integrated wireless facial interface. Nat. Commun. 15, 530 (2024)

    2024

  12. [20]

    & Xiao, X

    Zhu, T., Li, L., Yang, J., Zhao, S. & Xiao, X. Multimodal emotion classification with multi-level semantic reasoning network. IEEE Trans. Multimed. 25, 6868–6880 (2022)

    2022

  13. [21]

    & Gao, X

    Yang, J., Li, J., Li, L., Wang, X. & Gao, X. A circular- structured representation for visual emotion distribution learn- ing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) , 4237–4246 (2021)

    2021

  14. [22]

    & Wang, S

    Xu, Z. & Wang, S. Emotional attention detection and cor- relation exploration for image emotion distribution learning. IEEE Trans. Affect. Comput. 14, 357–369 (2023)

    2023

  15. [23]

    & Wang, S

    Pan, J. & Wang, S. Progressive visual content understanding network for image emotion classification. In Proceedings of the 31st ACM International Conference on Multimedia , 6034–6044 (2023)

    2023

  16. [24]

    & Yang, J

    Zhang, Z., Wang, L. & Yang, J. Weakly supervised video emotion detection and prediction via cross-modal temporal erasing network. In 2023 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 18888–18897 (2023)

    2023

  17. [25]

    & Zhao, G

    Chen, H., Shi, H., Liu, X., Li, X. & Zhao, G. Smg: A micro-gesture dataset towards spontaneous body gestures for emotional stress state analysis. Int. J. Comput. Vis.131, 1346– 1366 (2023)

    2023

  18. [26]

    Tlaie, A. et al. Inferring internal states across mice and mon- keys using facial features. Nat. Commun. 16, 1–16 (2025)

    2025

  19. [27]

    Le Mau, T. et al. Professional actors demonstrate variabil- ity, not stereotypical expressions, when portraying emotional states in photographs. Nat. Commun. 12, 5037 (2021)

    2021

  20. [28]

    Liu, D. et al. Brain-machine coupled learning method for facial emotion recognition. IEEE Trans. Pattern Anal. Mach. Intell. 45, 10703–10717 (2023)

    2023

  21. [29]

    & Jin, Y

    Xu, L., Gan, Y . & Jin, Y . Class activation regularization-based facial emotion recognition network and its application in stu- dents’ emotional engagement assessment. IEEE Trans. Affect. Comput. (2024)

    2024

  22. [30]

    Wang, H. et al. Rethinking the learning paradigm for dy- namic facial expression recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, 17958–17968 (2023)

    2023

  23. [31]

    Wagner, J. et al. Dawn of the transformer era in speech emo- tion recognition: closing the valence gap. IEEE Trans. Pattern Anal. Mach. Intell. 45, 10745–10759 (2023)

    2023

  24. [32]

    & Busso, C

    Martinez-Lucas, L., Lin, W.-C. & Busso, C. Analyzing continuous-time and sentence-level annotations for speech emotion recognition. IEEE Trans. Affect. Comput. (2024)

    2024

  25. [33]

    & Shams, S

    Kim, M., Jiang, X., Lauter, K., Ismayilzada, E. & Shams, S. Secure human action recognition by encrypted neural network inference. Nat. communications 13, 4799 (2022)

    2022

  26. [34]

    L., de Lima Costa, W., Martínez, E

    Lima, M. L., de Lima Costa, W., Martínez, E. T. & Teichrieb, V . St-gait++: Leveraging spatio-temporal convolutions for gait-based emotion recognition on videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 302–310 (2024)

    2024

  27. [35]

    Lu, H., Hu, X. & Hu, B. See your emotion from gait using un- labeled skeleton data. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, 1826–1834 (2023)

    2023

  28. [36]

    & Cui, Z

    Song, T., Zheng, W., Song, P. & Cui, Z. Eeg emotion recog- nition using dynamical graph convolutional neural networks. IEEE Trans. Affect. Comput. 11, 532–541 (2020)

    2020

  29. [37]

    Li, X. et al. Exploring eeg features in cross-subject emotion recognition. Front. Neurosci.12 (2018)

    2018

  30. [38]

    Huang, W., Wang, W., Li, Y . & Wu, W. Fbstcnet: A spatio- temporal convolutional network integrating power and con- nectivity features for eeg-based emotion decoding. IEEE Trans. Affect. Comput. (2024)

    2024

  31. [39]

    P., Wu, X

    Liu, B., Guo, J., Chen, C. P., Wu, X. & Zhang, T. Fine-grained interpretability for eeg emotion recognition: Concat-aided grad-cam and systematic brain functional network. IEEE Trans. Affect. Comput. 15, 671–684 (2023)

    2023

  32. [40]

    & Liu, Y

    Zhang, Z., Zhong, S. & Liu, Y . Beyond mimicking under- represented emotions: deep data augmentation with emotional subspace constraints for eeg-based emotion recognition. In Proceedings of the AAAI conference on artificial intelligence, vol. 38, 10252–10260 (2024)

    2024

  33. [41]

    Tabbaa, L. et al. Vreed: Virtual reality emotion recognition dataset using eye tracking & physiological measures. In Pro- ceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 5, 1–20 (2021)

    2021

  34. [42]

    Sharma, P. et al. Student engagement detection using emo- tion analysis, eye tracking and head movement with machine learning. In International Conference on Technology and In- novation in Learning, Teaching and Education, 52–68 (2022)

    2022

  35. [43]

    Lin, J.-S. C. & Liang, H.-Y . The influence of service environ- ments on customer emotion and service outcomes. Manag. Serv. Qual. An Int. J. 21, 350–372 (2011)

    2011

  36. [44]

    Farshchi, M. A. & Fisher, N. Emotion and the environment: the forgotten dimension. In Creating the Productive Work- place, 73–92 (CRC Press, 1999)

    1999

  37. [45]

    B., V olkmann, J

    Kashdan, T. B., V olkmann, J. R., Breen, W. E. & Han, S. Social anxiety and romantic relationships: The costs and ben- efits of negative emotion expression are context-dependent. J. Anxiety Disord. 21, 475–492 (2007)

    2007

  38. [46]

    P., Strange, B

    Richardson, M. P., Strange, B. A. & Dolan, R. J. Encoding of emotional memories depends on amygdala and hippocampus and their interactions. Nat. Neurosci. 7, 278–285 (2004)

    2004

  39. [47]

    & Lu, B.-L

    Zheng, W.-L. & Lu, B.-L. Investigating critical frequency bands and channels for eeg-based emotion recognition with deep neural networks. IEEE Trans. Auton. Ment. Dev. 7, 162–175 (2015)

    2015

  40. [48]

    & Chen, W

    Gong, L., Li, M., Zhang, T. & Chen, W. Eeg emotion recog- nition using attention-based convolutional transformer neural network. Biomed. Signal Process. Control. 84 (2023)

    2023

  41. [49]

    & Pan- tic, M

    Toisoul, A., Kossaifi, J., Bulat, A., Tzimiropoulos, G. & Pan- tic, M. Estimation of continuous valence and arousal levels from faces in naturalistic conditions. Nat. Mach. Intell. 3, 42–50 (2021)

    2021

  42. [50]

    Gong, X., Chen, C. P. & Zhang, T. Cross-cultural emotion recognition with eeg and eye movement signals based on mul- tiple stacked broad learning system. IEEE Trans. Comput. Soc. Syst. (2023)

    2023

  43. [51]

    & Friesen, W

    Ekman, P. & Friesen, W. V . Constants across cultures in the face and emotion. J. Pers. Soc. Psychol.17, 124 (1971). 16/32

    1971

  44. [52]

    Lian, D. et al. Multiview multitask gaze estimation with deep convolutional neural networks. IEEE TNNLS 30, 3010–3023 (2018)

    2018

  45. [53]

    Awh, E., Belopolsky, A. V . & Theeuwes, J. Top-down versus bottom-up attentional control: A failed theoretical dichotomy. Trends Cogn. Sci. 16, 437–443 (2012)

    2012

  46. [54]

    Top–down and bottom–up control of visual selection

    Theeuwes, J. Top–down and bottom–up control of visual selection. Acta Psychol 135, 77–99 (2010)

    2010

  47. [55]

    Xu, H. et al. Pandora: A panoramic detection dataset for ob- ject with orientation. In European Conference on Computer Vision, 237–252 (2022)

    2022

  48. [56]

    & Fan, D.-P

    Wang, G., Chen, C., Hao, A., Qin, H. & Fan, D.-P. Windb: Hmd-free and distortion-free panoptic video fixation learning (2023)

    2023

  49. [57]

    Unifying top-down and bottom-up scanpath pre- diction using transformers

    Yang, Z.et al. Unifying top-down and bottom-up scanpath pre- diction using transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 1683–1693 (2024)

    2024

  50. [58]

    & Zhao, Q

    Chen, X., Jiang, M. & Zhao, Q. Beyond average: Indi- vidualized visual scanpath prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 25420–25431 (2024)

    2024

  51. [59]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    2020 arXiv

  52. [60]

    W., Wetzstein, G

    Martin, D., Serrano, A., Bergman, A. W., Wetzstein, G. & Masia, B. Scangan360: A generative model of realistic scan- paths for 360 images. IEEE Trans. Vis. Comput. Graph.28, 2003–2013 (2022)

    2022

  53. [61]

    & Wang, Z

    Sui, X., Fang, Y ., Zhu, H., Wang, S. & Wang, Z. Scandmm: A deep markov model of scanpath prediction for 360deg images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6989–6999 (2023)

    2023

  54. [62]

    & Chen, C

    Song, M., Song, W., Yang, G. & Chen, C. Improving rgb-d salient object detection via modality-aware decoder. IEEE Trans. Image Process.31, 6124–6138 (2022)

    2022

  55. [63]

    & Sun, J

    He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, 770–778 (2016)

    2016

  56. [64]

    & Ding, Z

    Wang, Z., Su, X. & Ding, Z. Long-term traffic prediction based on lstm encoder-decoder architecture. IEEE Trans. Intell. Transp. Syst. 22, 6561–6571 (2020)

    2020

  57. [65]

    & Chen, C

    Song, M., Li, L., Wu, D., Song, W. & Chen, C. Rethink- ing object saliency ranking: A novel whole-flow processing paradigm. IEEE Trans. Image Process.33, 338–353 (2024)

    2024

  58. [66]

    Lian, D. et al. Multiview multitask gaze estimation with deep convolutional neural networks. IEEE Trans. Neural Netw. Learn. Syst. 30, 3010–3023 (2018)

    2018

  59. [67]

    & Al-Hamadi, A

    Abdelrahman, A., Hempel, T., Khalifa, A. & Al-Hamadi, A. L2cs-net: Fine-grained gaze estimation in unconstrained en- vironments. arxiv 2022. arXiv preprint arXiv:2203.03339 (2022)

    2022 arXiv

  60. [68]

    Cheng, Y . & Lu, F. Gaze estimation using transformer. In 2022 26th International Conference on Pattern Recognition (ICPR), 3341–3347 (2022)

    2022

  61. [69]

    common alignment

    Bao, Y ., Cheng, Y ., Liu, Y . & Lu, F. Adaptive feature fu- sion network for gaze tracking in mobile tablets. In 2020 25th International Conference on Pattern Recognition (ICPR), 9936–9943 (2021). Competing interests The authors declare no competing interests. 17/32 Supplemen...

    2021

  62. [70]

    This comprehensive approach allows for a deeper understanding of visual attention in relation to environmental context, resulting in more accurate emotion recognition

    Comprehensive Integration of Fixation Trajectories: Unlike existing datasets that focus on specific eye movement signals like pupil size or diameter, EmoGaze360-1K includes fixation trajectories that reflect interaction with the environment. This comprehensive approach allows ...

  63. [71]

    Positive Emotion

    Non-Intrusive and Cost-Effective Data Collection: EmoGaze360-1K adopts an HMD-free approach, reducing the discomfort often associated with traditional methods like EEG and surface sensors. This design makes data collection more user-friendly and applicable in real-world scenar...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.