Pith. sign in

REVIEW 3 major objections 5 minor 34 references

Emotion estimation from video footage with LSTM

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A four-layer LSTM over 27 MediaPipe blendshape scores classifies facial expressions as happy, sad, or unknown with 71% accuracy and 62% F1-score, matching the FER2013 benchmark at much lower computational cost.

desk verdict A lightweight proof-of-concept emotion estimator with real engineering effort, but the headline benchmark claim is invalid: 71% on a three-class happy/sad/unknown subset is not the seven-class FER2013 benchmark, and the test set is an unexplained, likely biased subsample. read the letter →

arxiv 2501.13432 v3 pith:D3JZCZZI submitted 2025-01-23 cs.CV cs.LGcs.RO

Samer Attrah This is my paper · ORCID
classification cs.CVcs.LGcs.RO
keywords emotionestimationfacialexpressionrecognitionLSTMMediaPipeblendshapesFER2013real-timevideoclassificationsocialroboticsembeddedsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that emotion recognition from a live camera stream can be made cheap enough for a social robot: a four-layer LSTM that reads 27 MediaPipe blendshape scores per face frame classifies the expression as happy, sad, or unknown and reaches 71% accuracy with a 62% F1-score on FER2013. The point of the work is a proof of concept for embedded systems, where a small model that tracks expressions over time could let a robot adapt its conversation or actions in real time. The authors report that the result matches the FER2013 accuracy benchmark without extra training data and with much lower computation cost than transformer-based or CNN-heavy alternatives. Because the task is deliberately narrowed to three classes, the benchmark comparison applies only to that reduced three-class setting.

What carries the argument

The load-bearing mechanism is a four-layer LSTM network that consumes a sequence of MediaPipe blendshape scores extracted from each detected face. Blendshapes are a compact semantic parametrization of facial expression: probability scores, one per expression component, produced alongside the 468 face landmarks; the paper prunes the 52 scores to 27 by discarding channels whose values almost never rise above 0.4. The LSTM layer, a gated recurrent network that keeps long-term dependencies, turns the frame-by-frame blendshape stream into a happy, sad, or unknown classification, and a final Dense layer with Softmax produces the class scores. This combination is what lets the system classify expressions from video rather than from isolated still images, and it is what keeps the computation small enough for embedded robotics.

What would settle it

Run the trained model on the complete official FER2013 private test split, collapsing original labels into happy, sad, and unknown, and compare total accuracy and F1 to the reported 0.72 and 0.63; a large drop would show the current evaluation sample is not representative.

Watch

Extended reading notes

Core claim

The paper's central claim is that a four-layer LSTM network fed with temporal sequences of 27 MediaPipe blendshape scores is sufficient to estimate the main emotion of a face in a video stream. The model is trained on FER2013 images that MediaPipe can detect, after collapsing the seven original labels into happy, sad, and unknown, augmenting the training set, and dropping blendshape channels that almost never activate. On the test set, it records 0.72 categorical accuracy and 0.63 F1-score, which the paper equates with the dataset's benchmark accuracy, and it produces stable happy, sad, or unknown outputs in a live camera demonstration. The paper also reports that reducing the 52 MediaPipe blendshapes to 27 keeps the same metrics while shrinking the model.

Load-bearing premise

The reported accuracy assumes the specific test images used for evaluation are a fair, unbiased sample of the FER2013 test distribution, but the paper does not describe how that test subset was chosen.

Editorial extensions

If this is right

  • A social robot can run emotion feedback on laptop-class hardware while streaming from a camera, because inference needs only 27 numbers per frame rather than a full image.
  • A model trained on static FER2013 images transfers to live video without retraining, keeping its accuracy through the MediaPipe feature-extraction and LSTM classification pipeline.
  • Cutting 52 blendshapes to 27 reduces model size and latency without changing accuracy or F1, which is the basis for the claimed cost savings.
  • The three-class output (happy, sad, unknown) is enough to drive a simple conversational feedback loop: the robot can detect a pleased or displeased reaction and adapt its topic or action.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's benchmark comparison is not apples-to-apples as stated: 71% on three classes is a different task from the seven-class FER2013 leaderboard, so a fair comparison would either evaluate this model on all seven original labels or collapse an existing seven-class model down to the same three labels.
  • Because 'unknown' absorbs six of the seven FER2013 emotions, the practical vocabulary is roughly 'happy / sad / everything else'; a social robot would need other modalities or more classes to tell anger, surprise, fear, and disgust apart.
  • The paper's test-set indexing trick could grow into a systematic error-analysis tool: by keeping the FER2013 row identifier with each blendshape vector, a follow-up study could plot mistakes against occlusion, head pose, and blendshape activation to learn what the model is missing.
  • Categorical focal loss and class weights, both mentioned in the paper but not tested, may reduce the oscillation the authors observe between happy and sad classification in the confusion matrix.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a lightweight LSTM model for real-time facial emotion estimation. Inputs are MediaPipe ARKit blendshape features extracted from FER2013 images; the seven FER2013 emotion classes are collapsed into three (happy, sad, unknown). The model is trained on a subset of FER2013 augmented via flips and rotations, and evaluated on an unexplained test partition. The authors report 71.99% categorical accuracy and 0.6298 F1-score and claim this meets the FER2013 benchmark with reduced computation cost. The paper also includes ablations on blendshape selection, model architecture, and loss function.

Significance. If the evaluation were sound, the system would be a useful proof-of-concept for a low-computation, real-time emotion estimator for social robotics, and the authors contribute public code and a demo video, which supports reproducibility. However, the central benchmark claim is not supported by the reported three-class task, and the test-set composition contradicts the official FER2013 private-test counts. These issues are load-bearing for the stated contribution and must be resolved before the work can be assessed.

major comments (3)
  1. [Abstract and §5] The reported 71.99% accuracy and 0.6298 F1-score are computed on a three-class task after §3.2.1 collapses the seven FER2013 classes into happy, sad, and unknown. The FER2013 benchmark referenced in §5 is a seven-class classification problem, so a three-class accuracy number is not comparable to the leaderboard. The claim that the model "meets the accuracy benchmark of the FER2013 dataset" is therefore unsupported as stated.
  2. [Table 4 and §3.2.2] The confusion matrix in Table 4 has row sums of 291 happy, 1110 unknown, and 245 sad, totaling 1646 images, whereas the official private test set under the paper's own mapping contains 879 happy, 2116 unknown, and 594 sad (3589 images). Section 3.2.2 explicitly describes filtering only the training set with MediaPipe, so the construction of this 1646-image test subset is unexplained. If the test set was subsampled or filtered, the reported accuracy and F1-score are computed on an unrepresentative distribution; the paper provides no description of this subset, making the reported results irreproducible.
  3. [§5] On the reported test subset, always predicting the majority class "unknown" yields 1110/1646 = 67.4% accuracy, so the reported 71.99% is only about 4.6 percentage points above a trivial constant predictor. The paper does not report per-class precision/recall or compare against this majority-class baseline, and the reported F1-score of 0.6298 is not enough to establish practical utility without such context.
minor comments (5)
  1. [§3.2.1, Table 2] Table 2 lists the disgust training count as "Maximum possible" without giving the actual number used; the text's "around 400 images" should be replaced with the exact count.
  2. [§3.2.4] The augmentation description "Random Rotation by 0.2 x 180 degrees" is ambiguous; the rotation range should be specified directly, e.g., uniform in [-36°, +36°].
  3. [§3.3] The paper states the model was trained for 5000 epochs with early stopping, but it does not report the actual number of epochs used or which checkpoint was selected for the final evaluation.
  4. [§5] The paper does not define how the F1-score is computed; it should state whether the reported value is macro-averaged, weighted, or per-class.
  5. [Throughout] The manuscript contains numerous grammatical and typographical errors (e.g., "in this paper" in the abstract, "the face look changes" in §5, "proved that" in §6) that should be corrected in a revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the benchmark-comparison concerns are evaluation-validity issues, not circular reasoning.

full rationale

The paper presents an empirical machine-learning pipeline: MediaPipe blendshapes are extracted from FER2013 images, an LSTM is trained on those features, and metrics are reported on a test set. None of the reported quantities (categorical accuracy 0.7199, F1-score 0.6298, confusion matrix counts) are defined in terms of the model's outputs or fitted parameters in a way that would make them true by construction. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. The abstract's claim that the 71% accuracy 'meets the accuracy benchmark of the FER2013 dataset' is a comparison to an external leaderboard, and that comparison is undermined by the paper's own reduction to three classes in Section 1 and Section 3.2.1, as well as by the unexplained test subset evidenced by Table 4's row sums (291 happy, 1110 unknown, 245 sad) differing from the official private-test counts in Table 1. Those are serious validity concerns about benchmarking and test-set representativeness, not circularity: the derivation does not reduce to its own inputs. Accordingly, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central performance claim rests on several untested choices: the adequacy of MediaPipe blendshapes, the 3-class redefinition, and an unexplained test subset. The listed free parameters and axioms show the most important input choices that are not independently justified.

free parameters (4)
  • Blendshape score threshold = 0.4
    Blendshapes above this threshold are counted as informative; chosen after experiments on the dataset (Section 4.1).
  • High-count threshold = 100
    Minimum number of images where a blendshape must exceed the threshold to be kept; tuned experimentally (Section 4.1).
  • Per-class training counts = 4000 happy, 4000 sad, 1500 for other classes
    Selected to balance the training set (Section 3.2.1).
  • LSTM learning rate = 1.09e-06
    Set by Keras tuner; influences convergence and final performance (Section 3.3).
assumptions (4)
  • standard math Categorical cross-entropy is an appropriate loss for this classification task
    The paper uses CCE; no issue with this assumption.
  • domain assumption MediaPipe blendshapes capture enough information to distinguish happy, sad, and other expressions
    The entire pipeline depends on this; prior work supports it but this paper does not independently validate it.
  • ad hoc to paper Merging five FER2013 classes into one 'unknown' class preserves the meaning of the benchmark comparison
    This redefinition is introduced only for this paper and is not a standard evaluation protocol.
  • domain assumption An LSTM trained on static images can generalize to video streams
    No temporal sequence is constructed in training; the paper does not explain how the LSTM sees multiple frames.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Emotion estimation from video footage with LSTM." pith.science (2026). https://pith.science/paper/D3JZCZZI

@misc{pith2026250113432,
  author       = {Pith},
  title        = {Pith review of: Emotion estimation from video footage with LSTM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D3JZCZZI}},
  note         = {Machine review of arXiv:2501.13432}
}
read the original abstract

Emotion estimation in general is a field that has been studied for a long time, and several approaches exist using machine learning. in this paper, we present an LSTM model, that processes the blend-shapes produced by the library MediaPipe, for a face detected in a live stream of a camera, to estimate the main emotion from the facial expressions, this model is trained on the FER2013 dataset and delivers a result of 71% accuracy and 62% f1-score which meets the accuracy benchmark of the FER2013 dataset, with significantly reduced computation costs. https://github.com/Samir-atra/Emotion_estimation_from_video_footage_with_LSTM_ML_algorithm

Figures

Figures reproduced from arXiv: 2501.13432 by the authors.

Figure 1
Figure 1. Image annotated with MediaPipe Another work is[12], where it compares MediaPipe to OpenFace, so when it comes to the landmark detection problem OpenFace[13] uses OpenCV and detects 68 facial landmarks, while MediaPipe uses TensorFlow[14], to detect 468 landmarks which are arranged in fixed quads and represented by their coordinates (x,y,z). Figure1 shows an image annotated with the landmarks used to detect the expre… view at source ↗
Figure 2
Figure 2. Histogram for the blendshpes of the image in figure1 by MediaPipe [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Images undetectable by MediaPipe sample 3.2.3 Indexing the test set This step came after a few experiments resulted in finding an unexplainable high error rate in the model performance, and as a tracking method added a column to the test set which is represented in a .csv file, that assigns a number for each image in the FER2013 dataset, and that number stays with the image through all the steps of data processing, … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Model structure For the loss function the categorical crossentropy was used, and for that encoded the labels of the images as one-hot vectors, and the metrics for evaluation were the loss, categorical crossentropy, categorical accuracy, and F1-score. 3.4 Model evaluati…
Figure 5
Figure 5. Figure 5: Blendshapes scatter plot for a sample image [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 22 canonical work pages

  1. [11]

    Emotion recognition at a distance: The robustness of machine learning based on hand-crafted facial features vs deep learning models

    Carmen Bisogni, Lucia Cimmino, Maria De Marsico, Fei Hao, and Fabio Narducci. Emotion recognition at a distance: The robustness of machine learning based on hand-crafted facial features vs deep learning models. Image and Vision Computing, 136:104724, 2023

    work page 2023

  2. [12]

    Comparison of facial landmark detection methods for micro-expressions analysis

    Alexander V Savin, Victoria A Sablina, and Michael B Nikiforov. Comparison of facial landmark detection methods for micro-expressions analysis. In 2021 10th Mediterranean Conference on Embedded Computing (MECO), pages 1–4. IEEE, 2021

    work page 2021

  3. [1]

    Paul Ekman and Wallace V . Friesen. Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues , volume 10. Ishk, 2003

    work page 2003

  4. [2]

    Challenges in representation learning: A report on three machine learning contests

    Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in representation learning: A report on three machine learning contests. In Neural information processing: 20th international conference, ICONIP 2013, daegu, korea, november 3-7, 2013. ...

    work page 2013

  5. [3]

    Mediapipe: A framework for perceiving and processing reality

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Yong, Juhyun Lee, et al. Mediapipe: A framework for perceiving and processing reality. In Third workshop on computer vision for AR/VR at IEEE computer vision and pattern recognition (CVPR) , volume 2019, 2019

    work page 2019

  6. [4]

    Facial emotion recognition using handcrafted features and cnn.Procedia Computer Science, 218:1295–1303, 2023

    Chahak Gautam and KR Seeja. Facial emotion recognition using handcrafted features and cnn.Procedia Computer Science, 218:1295–1303, 2023. 6https://youtu.be/RdcjA9ScBmI 10 A PREPRINT - APRIL 3, 2025

    work page 2023

  7. [5]

    A real-time robust facial expression recognition system using hog features

    Pranav Kumar, SL Happy, and Aurobinda Routray. A real-time robust facial expression recognition system using hog features. In 2016 International Conference on Computing, Analytics and Security Trends (CAST) , pages 289–293. IEEE, 2016

    work page 2016

  8. [6]

    Deep facial expression recognition: A survey

    Shan Li and Weihong Deng. Deep facial expression recognition: A survey. IEEE transactions on affective computing, 13(3):1195–1215, 2020

    work page 2020

Show all 34 references
  1. [7]

    Facial expression and body gesture emotion recognition: A systematic review on the use of visual data in affective computing

    Sze Chit Leong, Yuk Ming Tang, Chung Hin Lai, and CKM Lee. Facial expression and body gesture emotion recognition: A systematic review on the use of visual data in affective computing. Computer Science Review, 48:100545, 2023

    2023

  2. [8]

    Image processing with neural networks—a review

    Michael Egmont-Petersen, Dick de Ridder, and Heinz Handels. Image processing with neural networks—a review. Pattern recognition, 35(10):2279–2301, 2002

    2002

  3. [9]

    Development of real-time landmark-based emotion recognition cnn for masked faces

    Akhmedov Farkhod, Akmalbek Bobomirzaevich Abdusalomov, Mukhriddin Mukhiddinov, and Young-Im Cho. Development of real-time landmark-based emotion recognition cnn for masked faces. Sensors, 22(22):8704, 2022

    2022

  4. [10]

    Mediapipe: A framework for building perception pipelines

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172, 2019

    1906 arXiv

  5. [13]

    Openface: An open source facial behavior analysis toolkit

    Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. Openface: An open source facial behavior analysis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 1–10, 2016

    2016

  6. [14]

    Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng

    Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Wa...

    2016

  7. [15]

    Practice and theory of blendshape facial models

    John P Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frederic H Pighin, and Zhigang Deng. Practice and theory of blendshape facial models. Eurographics (State of the Art Reports) , 1(8):2, 2014

    2014

  8. [16]

    What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (F ACS)

    Paul Ekman and Erika L Rosenberg. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (F ACS). Oxford University Press, USA, 1997

    1997

  9. [17]

    Facial expression and emotion

    Paul Ekman. Facial expression and emotion. American psychologist, 48(4):384, 1993

    1993

  10. [18]

    Presentation and validation of the radboud faces database

    Oliver Langner, Ron Dotsch, Gijsbert Bijlstra, Daniel HJ Wigboldus, Skyler T Hawk, and AD Van Knippenberg. Presentation and validation of the radboud faces database. Cognition and emotion, 24(8):1377–1388, 2010

    2010

  11. [19]

    Long short-term memory

    Neural Computation. Long short-term memory. Neural Comput, 9:1735–1780, 2016

    2016

  12. [20]

    Deep learning, 2016

    Ian Goodfellow. Deep learning, 2016

    2016

  13. [21]

    Hybrid deep neural networks for face emotion recognition

    Neha Jain, Shishir Kumar, Amit Kumar, Pourya Shamsolmoali, and Masoumeh Zareapoor. Hybrid deep neural networks for face emotion recognition. Pattern Recognition Letters, 115:101–106, 2018

    2018

  14. [22]

    Web-based database for facial expression analysis

    Maja Pantic, Michel Valstar, Ron Rademaker, and Ludo Maat. Web-based database for facial expression analysis. In 2005 IEEE international conference on multimedia and Expo , pages 5–pp. IEEE, 2005

    2005

  15. [23]

    Induced disgust, happiness and surprise: an addition to the mmi facial expression database

    Michel Valstar, Maja Pantic, et al. Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In Proc. 3rd Intern. Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect, volume 10, page 65. Paris, France., 2010

    2010

  16. [24]

    From individual to group-level emotion recognition: Emotiw 5.0

    Abhinav Dhall, Roland Goecke, Shreya Ghosh, Jyoti Joshi, Jesse Hoey, and Tom Gedeon. From individual to group-level emotion recognition: Emotiw 5.0. In Proceedings of the 19th ACM international conference on multimodal interaction, pages 524–528, 2017

    2017

  17. [25]

    Multi-pie

    Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker. Multi-pie. Image and vision computing, 28(5):807–813, 2010

    2010

  18. [26]

    Affectnet: A database for facial expression, valence, and arousal computing in the wild

    Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1):18–31, 2017

    2017

  19. [27]

    Survey of resampling techniques for improving classification performance in unbalanced datasets

    Ajinkya More. Survey of resampling techniques for improving classification performance in unbalanced datasets. arXiv preprint arXiv:1608.06048, 2016. 11 A PREPRINT - APRIL 3, 2025

    2016 arXiv

  20. [28]

    Empirical evaluation of gated recurrent neural networks on sequence modeling

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014

    2014 arXiv

  21. [29]

    Recurrent neural networks

    Larry R Medsker, Lakhmi Jain, et al. Recurrent neural networks. Design and Applications, 5(64-67):2, 2001

    2001

  22. [30]

    Kerastuner

    Tom O’Malley, Elie Bursztein, James Long, François Chollet, Haifeng Jin, Luca Invernizzi, et al. Kerastuner. https://github.com/keras-team/keras-tuner , 2019

    2019

  23. [31]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  24. [32]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  25. [33]

    Springer New York, New York, NY , 2008

    Mean Squared Error, pages 337–339. Springer New York, New York, NY , 2008

    2008

  26. [34]

    Focal loss for dense object detection

    T Lin. Focal loss for dense object detection. arXiv preprint arXiv:1708.02002, 2017. 12

    2017 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.