Pith. sign in

REVIEW 4 major objections 6 minor 88 references

OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OpenFace 3.0 shows that one compact multitask model can perform four facial behavior analysis tasks in real time on a CPU while matching specialized models.

desk verdict A genuinely useful lightweight multi-task OpenFace successor, but the SOTA claims are overplayed and the key MPIIGaze number needs protocol verification. read the letter →

arxiv 2506.02891 v1 pith:TE4HEU2G submitted 2025-06-03 cs.CV

Jiewen Hu , Leena Mathur , Paul Pu Liang , Louis-Philippe Morency This is my paper
classification cs.CV
keywords facialbehavioranalysismulti-tasklearninglandmarkdetectionactionuniteyegazeestimationemotionrecognitionlightweightmodelreal-timetoolkit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OpenFace 3.0 sets out to show that one compact multitask model can do the work of several specialized facial-analysis systems: detect facial landmarks, recognize facial action units, estimate eye gaze, and classify eight emotions. The paper claims this 29.4M-parameter model runs in real time on a CPU, uses less memory than earlier toolkits, and matches or beats specialized state-of-the-art models on landmarks and gaze while remaining competitive on action units and emotion. If the claim holds, researchers get a single open-source tool that replaces multiple heavyweight pipelines and works in-the-wild, including on angled faces. The paper also reports that multitask training with uncertainty-based loss weighting improves gaze and action-unit performance and keeps emotion recognition stable on non-frontal faces.

What carries the argument

The load-bearing mechanism is the unified facial representation: landmark heatmap coordinates from four stacked Hourglass networks are concatenated with EfficientNet's final-layer contextual features, pretrained on VGGFace2, to make one shared input. Three heads consume it: an action-unit detector builds a per-face cosine-similarity graph over AU features and updates it with a graph convolutional layer, a gaze head predicts yaw and pitch for each eye through separate fully connected layers, and an emotion head classifies eight classes. Training proceeds in three stages (landmark-only, then classifiers with a frozen backbone, then full fine-tuning), with a homoscedastic-uncertainty loss weighting that rebalances tasks of different units and dataset sizes. This mechanism is what lets 29.4M parameters cover all four tasks and keeps inference at 38 ms per frame on a CPU.

What would settle it

Run OpenFace 3.0 and each named baseline (SPIGA, SLPT, ME-GraphAU, L2CS-Net, MCGaze, DDAMFN, EfficientFace) on the same held-out splits of 300W, WFLW, MPIIGaze, Gaze360, DISFA, BP4D, and AffectNet, with identical normalization and scoring; if OpenFace 3.0 no longer matches or beats the published numbers, the central claim is false.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that sharing one facial representation across four tasks is not a compromise: it is a performance lever. OpenFace 3.0 concatenates landmark coordinates from stacked Hourglass networks with contextual features from an EfficientNet backbone pretrained on VGGFace2, then feeds this unified representation to three lightweight heads. Across its single-task, multitask, and uncertainty-weighted multitask variants, the system reports NME of 2.87 on 300W and 4.02 on WFLW for landmarks, angular errors of 2.56 on MPIIGaze and 10.6 on Gaze360, F1 scores of 60 on DISFA and 62 on BP4D for action units, and 0.60 accuracy on AffectNet's eight emotion classes. Table III states that across all tasks the approach either exceeds or performs comparably to specialized state-of-the-art models, state-of-the-art small models, and other toolkits, while Table IV shows 29.4M parameters and 38 ms per frame on CPU. The paper further finds that multitask training with uncertainty weighting improves gaze substantially and stabilizes emotion recognition on faces tilted beyond 45 degrees.

Load-bearing premise

The headline comparison rests on treating previously published state-of-the-art numbers from other papers as directly comparable to OpenFace 3.0, even though the training data, model sizes, validation splits, and metric conventions behind those numbers may differ.

Editorial extensions

If this is right

  • A single 29.4M-parameter model can replace separate toolkits for landmarks, action units, gaze, and emotion, cutting deployment cost and memory footprint.
  • Real-time CPU inference makes facial behavior analysis practical on edge devices, robotics, and webcam-based interfaces without GPU hardware.
  • Shared multitask features improve robustness on non-frontal faces, so in-the-wild videos with head movement should see smaller accuracy drops than specialized emotion models.
  • Uncertainty-weighted multitask training improves gaze estimation substantially, suggesting task weighting is a key tuning lever for future toolkits.
  • Open-source release of training code lets the community add tasks or datasets and retrain the same architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the angle-based evaluation suggests gaze data acts as a regularizer for emotion recognition on non-frontal faces; a direct test would be to train the emotion head with and without the gaze task while holding all other components fixed.
  • Inference: because the shared representation is task-agnostic, adding additional output heads such as valence/arousal, head pose, or identity should cost only a few million parameters, making OpenFace 3.0 a natural seed for a broader face-behavior foundation model.
  • Inference: for downstream users, the fair comparison is end-to-end behavior on their own video, including face detection, alignment, and task head, since detection failures propagate to all four outputs; per-task benchmark numbers alone may overstate or understate real-world performance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents OpenFace 3.0, an open-source toolkit for facial landmark detection, action unit detection, gaze estimation, and emotion recognition using a single lightweight multi-task model. The architecture combines a stacked-hourglass landmark module with an EfficientNet backbone, followed by task-specific heads (a dynamic graph AU module, per-eye FC gaze heads, and an emotion classifier). Training uses a three-stage procedure with homoscedastic uncertainty-based loss weighting. Experiments benchmark the system against SOTA, SOTA-small, and prior toolkits on 300W, WFLW, DISFA, BP4D, MPIIGaze, Gaze360, and AffectNet, and include efficiency comparisons and a small user study.

Significance. If the reported performance is reliable, this is a valuable contribution: a single 29.4M-parameter model that runs four facial analysis tasks in real time on a CPU, with open-source code and an intuitive Python API. The paper also demonstrates the benefit of a three-stage multitask training scheme and uncertainty weighting in an ablation. The system has the potential to be a widely used research tool if the evaluation claims are made rigorous. However, the headline claim of matching or exceeding state-of-the-art specialized models is not currently supported by the evidence presented, and the evaluation protocol is underspecified.

major comments (4)
  1. [Section IV-B, Table III] The caption of Table III states that 'Across all tasks, our approach either exceeds or performs comparably to SOTA models,' but the reported numbers contradict this. For DISFA and BP4D, the best OpenFace 3.0 F1 scores are 59/60 and 59/62, respectively, which are below the listed SOTA (66/66) and below or at the SOTA-small values (61/62). AffectNet accuracy of 0.60 is below the listed SOTA of 0.65. Only the landmark results (300W 2.87, WFLW 4.02) and MPIIGaze (2.56) beat the cited SOTA-small values. The text in Section IV-B must be revised to describe the actual results rather than a blanket claim, and the caption should not overstate the findings.
  2. [Section IV-A, Table III baselines] The SOTA and SOTA-small baseline numbers are not protocol-matched to OpenFace 3.0. The paper does not specify which exact model variant or checkpoint produced each baseline value, which train/test splits were used for 300W, WFLW, MPIIGaze, Gaze360, DISFA, BP4D, or AffectNet, or whether the evaluation protocols (e.g., subject-exclusive vs. random split, metric normalization) coincide. The most striking case is MPIIGaze: OpenFace 3.0 MTL w/ unc. reports 2.56° angular error, below the cited SOTA of 3.14° and well below typical subject-exclusive results from published gaze-estimation models. This strongly suggests a protocol mismatch, such as evaluating in a person-dependent manner or on training data. The authors should report their evaluation protocol in full and rerun the baselines under the same protocol, or substantially temper the 'rivals SOTA' claim.
  3. [Section IV-A.1, metric definition] The paper states that landmark accuracy is measured by NME normalized by inter-ocular distance for both 300W and WFLW. However, the standard WFLW benchmark measures NME normalized by face size (inter-pupil distance or bounding box size), and the cited SOTA baselines on WFLW (4.00 for SPIGA, 4.14 for SLPT) were reported in their original papers with face-size normalization. Comparing inter-ocular NME against face-size-normalized NME is not a valid comparison. The authors should cite the exact normalization used for each baseline and report results in the metric that matches the published literature, or clearly state that a non-standard normalization is used and provide the standard metric as well.
  4. [Section IV-B and IV-C, Table III] All results in Table III and the ablation study are point estimates from what appears to be a single training run. No confidence intervals, standard deviations, or number of random seeds are reported. Consequently, differences such as the MPIIGaze improvement from 4.25° to 2.56° with uncertainty weighting, or the DISFA F1 change from 60 to 59, cannot be assessed for statistical significance. The authors should provide at least three independent runs (or resampling-based uncertainty estimates) and report mean ± standard deviation, so that the claimed benefits of multitasking and uncertainty weighting can be evaluated.
minor comments (6)
  1. [Section III-D] The text says 'Training details for our landmark detection module are in Sections III-D and III-D' and later references 'Section' without a number; these cross-references are broken and should be fixed to point to the correct subsections.
  2. [Section IV-B] In the gaze estimation paragraph, 'trained specifically for this tak' contains a typo; it should read 'task'.
  3. [Section V] The reference to 'Appendix VII-.0.b' is malformed and does not match the actual appendix structure; the user study participant details should be referenced correctly.
  4. [Section VII] The ethics statement contains the duplicated phrase 'for for' in 'training models for for facial analysis tasks.'
  5. [Table V] The face-orientation bins (Easy: 0°–15°, Medium: 15°–45°, Hard: >45°) are not defined in terms of which rotation axis (yaw, pitch, or roll) is used, nor is it explained how the orientation is computed from the model outputs. Clarify this for reproducibility.
  6. [Section V and Appendix] The pilot user study has only 8 participants, and the UMUX-LITE and NASA-TLX scores are reported as averages without variance or individual-level data. Since the study is described as a pilot, this limitation should be stated explicitly, and the scores should be presented with ranges or distributions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: OpenFace 3.0 is an empirical systems paper whose claims rest on external benchmarks and in-paper ablations, not on a derivation that reduces to its own inputs.

full rationale

OpenFace 3.0 does not perform a mathematical derivation whose conclusion is equivalent to its premises. The central claims are empirical: a multi-task model improves speed, memory, and task performance relative to toolkits and specialized models. These claims are supported by measurements on public datasets (300W, WFLW, DISFA, BP4D, MPIIGaze, Gaze360, AffectNet) and by within-paper ablations comparing single-task, multi-task, and uncertainty-weighted multi-task models. The uncertainty weighting is adopted from Kendall et al. [36], an external source, and the loss equations are presented as standard constructions. No parameter is fitted to the reported test sets and then renamed as a prediction. The comparison against OpenFace 2.0 is a same-lab baseline, but it is a direct empirical measurement with a released system, not an imported assumption that forces the result. The Table III 'SOTA' baselines may suffer from protocol mismatch (a correctness and reproducibility concern), but that is not circularity: the paper's own model numbers are held out from training and are not definitionally equal to the baseline numbers. There is no uniqueness theorem, no ansatz smuggled in via citation, and no self-definitional reduction. Therefore the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. OpenFace 3.0 is a software artifact, not a postulated entity. The load-bearing external inputs are public dataset labels and published baseline numbers, neither of which the paper independently verifies.

free parameters (2)
  • Uncertainty weights sigma2_AU, sigma2_gaze, sigma2_emotion = not reported
    Learned task-balancing parameters in the homoscedastic uncertainty loss in Section III-C; no final values or sensitivity analysis are reported.
  • Label smoothing factor alpha = not reported
    Used in the emotion-recognition cross-entropy loss in Section III-D; chosen by hand and its value is not stated.
assumptions (3)
  • domain assumption Labels in 300W, WFLW, DISFA, BP4D, MPIIGaze, Gaze360, and AffectNet are accurate and consistent across tasks.
    Entire multi-task training and evaluation relies on the quality and consistency of these public annotations, used in Section IV-A.
  • domain assumption Published SOTA baseline numbers are directly comparable despite different training sets, architectures, and protocols.
    Table III contrasts OpenFace 3.0 with 'SOTA' figures without specifying the model and protocol per number; this is load-bearing for the headline claim.
  • domain assumption A concatenation of landmark coordinates and EfficientNet features is a sufficient unified representation for all four downstream tasks.
    The architecture in Section III-A assumes no information is lost by compressing the face into this combined vector before the task heads.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis." pith.science (2026). https://pith.science/paper/TE4HEU2G

@misc{pith2026250602891,
  author       = {Pith},
  title        = {Pith review of: OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TE4HEU2G}},
  note         = {Machine review of arXiv:2506.02891}
}
read the original abstract

In recent years, there has been increasing interest in automatic facial behavior analysis systems from computing communities such as vision, multimodal interaction, robotics, and affective computing. Building upon the widespread utility of prior open-source facial analysis systems, we introduce OpenFace 3.0, an open-source toolkit capable of facial landmark detection, facial action unit detection, eye-gaze estimation, and facial emotion recognition. OpenFace 3.0 contributes a lightweight unified model for facial analysis, trained with a multi-task architecture across diverse populations, head poses, lighting conditions, video resolutions, and facial analysis tasks. By leveraging the benefits of parameter sharing through a unified model and training paradigm, OpenFace 3.0 exhibits improvements in prediction performance, inference speed, and memory efficiency over similar toolkits and rivals state-of-the-art models. OpenFace 3.0 can be installed and run with a single line of code and operate in real-time without specialized hardware. OpenFace 3.0 code for training models and running the system is freely available for research purposes and supports contributions from the community.

Figures

Figures reproduced from arXiv: 2506.02891 by the authors.

Figure 1
Figure 1. Visualization of the OPENFACE 3.0 system, a lightweight multi-task modeling approach, trained for efficient facial landmark detection, AU detection, gaze estimation, and emotion recognition. During inference, the system first utilizes a face detector and feature extractor to obtain a unified facial representation that combines contextual facial information with precise facial landmark data. This unified representati… view at source ↗
Figure 2
Figure 2. Samples of task performance between OPENFACE 3.0 and OPENFACE 2.0. OPENFACE 2.0 does not support emotion recognition. vision (beginner, intermediate, advanced) to set up and test OPENFACE 3.0 from scratch (details in Appendix VII-.0.b). Participants were asked to perform several standard tasks with our system, divided into installation and setup, image￾based, and video-based evaluations: • Installation and Setup: In… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

88 extracted references · 62 canonical work pages

  1. [1]

    A. A. Abdelrahman, T. Hempel, A. Khalifa, A. Al-Hamadi, and L. Dinges. L2cs-net: Fine-grained gaze estimation in unconstrained environments. In2023 8th International Conference on Frontiers of Signal Processing (ICFSP), pages 98–102. IEEE, 2023

    2023

  2. [2]

    B. Amos, B. Ludwiczuk, and M. Satyanarayanan. Openface: A general-purpose face recognition library with mobile applications. Technical report, CMU-CS-16-118, CMU School of Computer Sci- ence, 2016

    2016

  3. [3]

    Baltru ˇsaitis, M

    T. Baltru ˇsaitis, M. Mahmoud, and P. Robinson. Cross-dataset learning and person-specific normalisation for automatic action unit detection. In2015 11th IEEE international conference and workshops on auto- matic face and gesture recognition (FG), volume 6, pages 1–6. IEEE, 2015

    2015

  4. [4]

    Baltrusaitis, P

    T. Baltrusaitis, P. Robinson, and L.-P. Morency. Constrained local neural fields for robust facial landmark detection in the wild. In Proceedings of the IEEE international conference on computer vision workshops, pages 354–361, 2013

    2013

  5. [5]

    Baltru ˇsaitis, P

    T. Baltru ˇsaitis, P. Robinson, and L.-P. Morency. Openface: an open source facial behavior analysis toolkit. In2016 IEEE winter conference on applications of computer vision (WACV), pages 1–10. IEEE, 2016

    2016

  6. [6]

    Baltrusaitis, A

    T. Baltrusaitis, A. Zadeh, Y . C. Lim, and L.-P. Morency. Openface 2.0: Facial behavior analysis toolkit. In2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 59–66, 2018

    2018

  7. [7]

    Bhattacharjee and H

    D. Bhattacharjee and H. Roy. Pattern of local gravitational force (plgf): A novel local image descriptor.IEEE transactions on pattern analysis and machine intelligence, 43(2):595–607, 2019

    2019

  8. [8]

    Bishay, K

    M. Bishay, K. Preston, M. Strafuss, G. Page, J. Turcot, and M. Mava- dati. Affdex 2.0: a real-time facial expression analysis toolkit.arXiv preprint arXiv:2202.12059, 2022

    arXiv 2022

Show all 88 references
  1. [9]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    2021 arXiv

  2. [10]

    F. Z. Canal, T. R. M ¨uller, J. C. Matias, G. G. Scotton, A. R. de Sa Junior, E. Pozzebon, and A. C. Sobieranski. A survey on facial emotion recognition techniques: A state-of-the-art literature review. Information Sciences, 582:593–617, 2022

    2022

  3. [11]

    Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman. Vggface2: A dataset for recognising faces across pose and age. In2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 67–74. IEEE, 2018

    2018

  4. [12]

    R. Caruana. Multitask learning.Machine learning, 28(1):41–75, 1997

    1997

  5. [13]

    Chang, Y

    D. Chang, Y . Yin, Z. Li, M. Tran, and M. Soleymani. Libreface: An open-source toolkit for deep facial expression analysis. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 8205–8215, January 2024

    2024

  6. [14]

    Cheng and F

    Y . Cheng and F. Lu. Gaze estimation using transformer. In2022 26th International Conference on Pattern Recognition (ICPR), pages 3341–3347. IEEE, 2022

    2022

  7. [15]

    J. H. Cheong, E. Jolly, T. Xie, S. Byrne, M. Kenney, and L. J. Chang. Py-feat: Python facial expression analysis toolbox.Affective Science, 4(4):781–796, 2023

    2023

  8. [16]

    V . R. R. Chirra, S. R. Uyyala, and V . K. K. Kolli. Virtual facial expression recognition using deep cnn with ensemble learning.Journal of Ambient Intelligence and Humanized Computing, 12(12):10581– 10599, 2021

    2021

  9. [17]

    Corneanu, M

    C. Corneanu, M. Madadi, and S. Escalera. Deep structure inference network for facial action unit recognition. InProceedings of the european conference on computer vision (ECCV), pages 298–313, 2018

    2018

  10. [18]

    J. Deng, J. Guo, E. Ververas, I. Kotsia, and S. Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5203–5212, 2020

    2020

  11. [19]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), 2019

    2019

  12. [20]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.ICLR, 2021

    2021

  13. [21]

    Ekman and W

    P. Ekman and W. V . Friesen. Facial action coding system.Environ- mental Psychology & Nonverbal Behavior, 1978

    1978

  14. [22]

    N. J. Emery. The eyes have it: the neuroethology, function and evolution of social gaze.Neuroscience & biobehavioral reviews, 24(6):581–604, 2000

    2000

  15. [23]

    Eyben, M

    F. Eyben, M. W ¨ollmer, T. Poitschke, B. Schuller, C. Blaschke, B. F ¨arber, and N. Nguyen-Thien. Emotion on the road—necessity, acceptance, and feasibility of affective computing in the car.Advances in Human-Computer Interaction, 2010(1):263593, 2010

    2010

  16. [24]

    Fischer, H

    T. Fischer, H. J. Chang, and Y . Demiris. Rt-gene: Real-time eye gaze estimation in natural environments. InProceedings of the European conference on computer vision (ECCV), pages 334–352, 2018

    2018

  17. [25]

    C. Frith. Role of facial expressions in social interactions.Philo- sophical Transactions of the Royal Society B: Biological Sciences, 364(1535):3453–3458, 2009

    2009

  18. [26]

    Y . Guan, Z. Chen, W. Zeng, Z. Cao, and Y . Xiao. End-to-end video gaze estimation via capturing head-face-eye spatial-temporal interaction context.IEEE Signal Processing Letters, 30:1687–1691, 2023

    2023

  19. [27]

    A. Gudi, H. E. Tasli, T. M. Den Uyl, and A. Maroulis. Deep learning based facs action unit occurrence and intensity estimation. In2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), volume 6, pages 1–5. IEEE, 2015

    2015

  20. [28]

    S. G. Hart. Nasa-task load index (nasa-tlx); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, volume 50, pages 904–908. Sage publications Sage CA: Los Angeles, CA, 2006

    2006

  21. [29]

    Haugh and F

    M. Haugh and F. Bargiela-Chiappini. Face and interaction.Face, communication and social interaction, pages 1–30, 2009

    2009

  22. [30]

    A. G. Howard. Mobilenets: Efficient convolutional neural networks for mobile vision applications.arXiv preprint arXiv:1704.04861, 2017

    2017 arXiv

  23. [31]

    T. Hu, S. Jha, and C. Busso. Robust driver head pose estimation in naturalistic conditions from point-cloud data. In2020 IEEE Intelligent Vehicles Symposium (IV), pages 1176–1182. Ieee, 2020

    2020

  24. [32]

    Jocher, A

    G. Jocher, A. Chaurasia, and J. Qiu. Ultralytics YOLO, Jan. 2023

    2023

  25. [33]

    Kar and P

    A. Kar and P. Corcoran. A review and analysis of eye-gaze estimation systems, algorithms and performance evaluation methods in consumer platforms.IEEE Access, 5:16495–16519, 2017

    2017

  26. [34]

    P. Kar, V . Chudasama, N. Onoe, P. Wasnik, and V . Balasubramanian. Fiducial focus augmentation for facial landmark detection.arXiv preprint arXiv:2402.15044, 2024

    2024 arXiv

  27. [35]

    Kellnhofer, A

    P. Kellnhofer, A. Recasens, S. Stent, W. Matusik, and A. Torralba. Gaze360: Physically unconstrained gaze estimation in the wild. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6912–6921, 2019

    2019

  28. [36]

    Kendall, Y

    A. Kendall, Y . Gal, and R. Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482–7491, 2018

    2018

  29. [37]

    Krafka, A

    K. Krafka, A. Khosla, P. Kellnhofer, H. Kannan, S. Bhandarkar, W. Matusik, and A. Torralba. Eye tracking for everyone. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2176–2184, 2016

    2016

  30. [38]

    X. Lan, Q. Hu, Q. Chen, J. Xue, and J. Cheng. Hih: Towards more accurate face alignment via heatmap in heatmap, 2022

    2022

  31. [39]

    J. R. Lewis, B. S. Utesch, and D. E. Maher. Umux-lite: when there’s no time for the sus. InProceedings of the SIGCHI conference on human factors in computing systems, pages 2099–2102, 2013

    2013

  32. [40]

    L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang. Visualbert: A simple and performant baseline for vision and language.arXiv preprint arXiv:1908.03557, 2019

    1908 arXiv

  33. [41]

    P. P. Liang, Y . Lyu, X. Fan, J. Tsaw, Y . Liu, S. Mo, D. Yogatama, L.- P. Morency, and R. Salakhutdinov. High-modality multimodal trans- former: Quantifying modality & interaction heterogeneity for high- modality representation learning.Transactions on Machine Learning Research, 2022

    2022

  34. [42]

    H. Liu, R. An, Z. Zhang, B. Ma, W. Zhang, Y . Song, Y . Hu, W. Chen, and Y . Ding. Norface: Improving facial expression analysis by identity normalization. InEuropean Conference on Computer Vision, pages 293–314. Springer, 2024

    2024

  35. [43]

    J. Liu, H. Wang, and Y . Feng. An end-to-end deep model with discriminative facial features for facial expression recognition.IEEE Access, 9:12158–12166, 2021

    2021

  36. [44]

    P. Liu, S. Han, Z. Meng, and Y . Tong. Facial expression recognition via a boosted deep belief network. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1805–1812, 2014

    2014

  37. [45]

    S. Liu, K. Koch, Z. Zhou, S. F ¨oll, X. He, T. Menke, E. Fleisch, and F. Wortmann. The empathetic car: Exploring emotion inference via driver behaviour and traffic context.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(3):1– 34, 2021

    2021

  38. [46]

    J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: pretraining task- agnostic visiolinguistic representations for vision-and-language tasks. InProceedings of the 33rd International Conference on Neural Infor- mation Processing Systems, pages 13–23, 2019

    2019

  39. [47]

    C. Luo, S. Song, W. Xie, L. Shen, and H. Gunes. Learning multi- dimensional edge feature-based au relation graph for facial action unit recognition. InProceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-2022. International Joint Con...

    2022

  40. [48]

    Mathur, P

    L. Mathur, P. P. Liang, and L.-P. Morency. Advancing social intel- ligence in ai agents: Technical challenges and open questions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20541–20560, 2024

    2024

  41. [49]

    S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh, and J. F. Cohn. Disfa: A spontaneous facial action intensity database.IEEE Transactions on Affective Computing, 4(2):151–160, 2013

    2013

  42. [50]

    Mollahosseini, B

    A. Mollahosseini, B. Hasani, and M. H. Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild.IEEE Transactions on Affective Computing, 10(1):18–31, 2017

    2017

  43. [51]

    Narayan, V

    K. Narayan, V . VS, R. Chellappa, and V . M. Patel. Facexformer: A uni- fied transformer for facial analysis.arXiv preprint arXiv:2403.12960, 2024

    2024 arXiv

  44. [52]

    H. W. Park, I. Grover, S. Spaulding, L. Gomez, and C. Breazeal. A model-free affective reinforcement learning approach to personal- ization of an autonomous social robot companion for early literacy education. InProceedings of the AAAI conference on artificial intelligence, vo...

    2019

  45. [53]

    Prados-Torreblanca, J

    A. Prados-Torreblanca, J. M. Buenaposada, and L. Baumela. Shape preserving facial landmarks with graph attention networks.arXiv preprint arXiv:2210.07233, 2022

    2022 arXiv

  46. [54]

    L. Qin, M. Wang, C. Deng, K. Wang, X. Chen, J. Hu, and W. Deng. Swinface: a multi-task transformer for face recognition, expression recognition, age estimation and attribute estimation.IEEE Transac- tions on Circuits and Systems for Video Technology, 2023

    2023

  47. [55]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019

    2019

  48. [56]

    Raghav, M

    A. Raghav, M. Gupta, et al. Ensemble learning for facial expression recognition.Full Length Article, 2(1):31–1, 2021

    2021

  49. [57]

    Ringeval, B

    F. Ringeval, B. Schuller, M. Valstar, N. Cummins, R. Cowie, L. Tavabi, M. Schmitt, S. Alisamir, S. Amiriparian, E.-M. Messner, et al. Avec 2019 workshop and challenge: state-of-mind, detecting depression with ai, and cross-cultural affect recognition. InProceedings of the 9th ...

    2019

  50. [58]

    Roy and D

    H. Roy and D. Bhattacharjee. Local-gravity-face (lg-face) for illumination-invariant and heterogeneous face recognition.IEEE Transactions on Information Forensics and Security, 11(7):1412–1424, 2016

    2016

  51. [59]

    Sagonas, E

    C. Sagonas, E. Antonakos, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic. 300 faces in-the-wild challenge: Database and results. Image and vision computing, 47:3–18, 2016

    2016

  52. [60]

    A. V . Savchenko, L. V . Savchenko, and I. Makarov. Classifying emotions and engagement in online learning based on a single facial expression recognition neural network.IEEE Transactions on Affective Computing, 13(4):2132–2143, 2022

    2022

  53. [61]

    Sikka, K

    K. Sikka, K. Dykstra, S. Sathyanarayana, G. Littlewort, and M. Bartlett. Multiple kernel learning for emotion recognition in the wild. InProceedings of the 15th ACM on International conference on multimodal interaction, pages 517–524, 2013

    2013

  54. [62]

    W. Su, X. Zhu, Y . Cao, B. Li, L. Lu, F. Wei, and J. Dai. Vl-bert: Pre- training of generic visual-linguistic representations. InInternational Conference on Learning Representations, 2020

    2020

  55. [63]

    K.-H. Tan, D. J. Kriegman, and N. Ahuja. Appearance-based eye gaze estimation. InSixth IEEE Workshop on Applications of Computer Vision, 2002.(WACV 2002). Proceedings., pages 191–195. IEEE, 2002

    2002

  56. [64]

    Tan and Q

    M. Tan and Q. V . Le. Efficientnet: Rethinking model scaling for convolutional neural networks, 2020

    2020

  57. [65]

    Tavabi, A

    L. Tavabi, A. Poon, A. S. Rizzo, and M. Soleymani. Computer-based ptsd assessment in vr exposure therapy. InHCI International 2020– Late Breaking Papers: Virtual and Augmented Reality: 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19– 24, 2020, Procee...

    2020

  58. [66]

    Y .-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. InACL, pages 6558–6569, 2019

    2019

  59. [67]

    Z. Wang, S. Song, C. Luo, S. Deng, W. Xie, and L. Shen. Multi- scale dynamic and hierarchical relationship modeling for facial action units recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1270–1280, 2024

    2024

  60. [68]

    W. Wu, C. Qian, S. Yang, Q. Wang, Y . Cai, and Q. Zhou. Look at boundary: A boundary-aware face alignment algorithm. InCVPR, 2018

    2018

  61. [69]

    Y . Wu, T. Hassner, K. Kim, G. Medioni, and P. Natarajan. Fa- cial landmark detection with tweaked convolutional neural networks. IEEE transactions on pattern analysis and machine intelligence, 40(12):3067–3074, 2017

    2017

  62. [70]

    Wu and Q

    Y . Wu and Q. Ji. Facial landmark detection: A literature survey. International Journal of Computer Vision, 127(2):115–142, 2019

    2019

  63. [71]

    J. Xia, W. Qu, W. Huang, J. Zhang, X. Wang, and M. Xu. Sparse local patch transformer for robust face alignment and landmarks inherent relation learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4052–4061, 2022

    2022

  64. [72]

    S. Xie, H. Hu, and Y . Chen. Facial expression recognition with two- branch disentangled generative adversarial network.IEEE Transac- tions on Circuits and Systems for Video Technology, 31(6):2359–2371, 2020

    2020

  65. [73]

    Xu and W

    T. Xu and W. Takano. Graph stacked hourglass networks for 3d human pose estimation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16105–16114, 2021

    2021

  66. [74]

    F. Xue, Q. Wang, and G. Guo. Transfer: Learning relation-aware facial expression representations with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3601–3610, 2021

    2021

  67. [75]

    J. Yang, Q. Liu, and K. Zhang. Stacked hourglass network for robust facial landmark localisation. InProceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 79–87, 2017

    2017

  68. [76]

    K. Yuan, Z. Yu, X. Liu, W. Xie, H. Yue, and J. Yang. Auformer: Vision transformers are parameter-efficient facial action unit detectors. In European Conference on Computer Vision, pages 427–445. Springer, 2024

    2024

  69. [77]

    Y ¨uce, H

    A. Y ¨uce, H. Gao, and J.-P. Thiran. Discriminant multi-label manifold embedding for facial action unit detection. In2015 11th IEEE Inter- national Conference and Workshops on Automatic face and gesture recognition (FG), volume 6, pages 1–6. IEEE, 2015

    2015

  70. [78]

    Zhang, Y

    S. Zhang, Y . Zhang, Y . Zhang, Y . Wang, and Z. Song. A dual-direction attention mixed feature network for facial expression recognition. Electronics, 12(17):3595, 2023

    2023

  71. [79]

    Zhang, Y

    X. Zhang, Y . Sugano, M. Fritz, and A. Bulling. Appearance-based gaze estimation in the wild. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4511–4520, 2015

    2015

  72. [80]

    Zhang, Y

    X. Zhang, Y . Sugano, M. Fritz, and A. Bulling. It’s written all over your face: Full-face appearance-based gaze estimation. InProceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 51–60, 2017

    2017

  73. [81]

    Zhang, L

    X. Zhang, L. Yin, J. F. Cohn, S. Canavan, M. Reale, A. Horowitz, P. Liu, and J. M. Girard. Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database.Image and Vision Computing, 32(10):692–706, 2014

    2014

  74. [82]

    Z. Zhao, Q. Liu, and F. Zhou. Robust lightweight facial expression recognition network with label distribution training. InProceedings of the AAAI conference on artificial intelligence, volume 35, pages 3510–3519, 2021

    2021

  75. [83]

    Zheng, M

    C. Zheng, M. Mendieta, and C. Chen. Poster: A pyramid cross-fusion transformer network for facial expression recognition. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3146–3155, 2023

    2023

  76. [84]

    R. Zhi, M. Liu, and D. Zhang. A comprehensive survey on automatic facial action unit analysis.The Visual Computer, 36(5):1067–1093, 2020

    2020

  77. [85]

    Y . Zhou, J. Pi, and B. E. Shi. Pose-independent facial action unit intensity regression based on multi-task deep transfer learning. In2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pages 872–877. IEEE, 2017

    2017

  78. [86]

    Z. Zhou, H. Li, H. Liu, N. Wang, G. Yu, and R. Ji. Star loss: Reducing semantic ambiguity in facial landmark detection, 2023

    2023

  79. [87]

    A. Zhu, K. Li, T. Wu, P. Zhao, W. Zhou, and B. Hong. Cross- task multi-branch vision transformer for facial expression and mask wearing classification.arXiv preprint arXiv:2404.14606, 2024. APPENDIX For our pilot study of OPENFACE3.0 usability, we tested the system with 8 grad...

    2024 arXiv

  80. [88]

    Appendix Table VII presents the breakdown of UMUX- LITE usability and NASA-TLX workload scores by par- ticipant experience levels

    studiesperceived workloadacross dimensions, including mental demand (mental activity required), physical demand (physical activity/strain required), temporal demand (time pressure felt while using the system), performance (success and satisfaction in performing a task), effort...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.