{"id":"0010949b-9f05-4b48-b06f-69afe1b3bac7","arxiv_id":"2501.13432","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An LSTM trained on MediaPipe blendshapes reports 71% accuracy on a three-class version of FER2013, but the claimed match to the seven-class benchmark is unsupported.","lead":"This paper describes a lightweight emotion recognition system that uses a webcam, MediaPipe's facial movement scores, and a recurrent neural network to label faces as happy, sad, or unknown. The authors claim it matches a well-known facial expression benchmark with less computation, but the benchmark comparison is not apples-to-apples because the problem was reduced from seven classes to three.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark claim is invalid: 71% accuracy on a 3-class happy/sad/unknown task is not comparable to the 7-class FER2013 leaderboard, and the reported test row sums (291/1110/245) do not match the official private test counts, so the evaluation is neither comparable nor fully described.","rationale":"Good-faith reading: the paper is a proof-of-concept and is candid about the three-class reduction and MediaPipe detection failures. The LSTM architecture and blendshape feature pipeline are described in enough detail to be reproducible, and the ablation of blendshape count and loss function is a genuine attempt at systematic design. However, the abstract's benchmark claim is the central assertion, and it has two independent defects. First, a 3-class accuracy number is not comparable to the 7-class FER2013 leaderboard; this is not a matter of consensus but of task definition. Second, the evaluation appears to use a subset of the official test set. The discrepancy in row sums is large: 1,646 vs 3,589 images, with the unknown class more heavily represented. The manuscript never states that the test set was filtered after §3.2.2's cleaning step, and §3.2.3's indexing step is described as a tracking aid, not an exclusion criterion. If the model was evaluated only on images MediaPipe could process, the results may still be useful for the embedded-system use case, but they cannot be quoted as 'meeting the FER2013 benchmark.' The reader's rejection is therefore appropriate. We do not see a need to change the verdict, only to sharpen the reason: the central issue is the invalid benchmark comparison and the undocumented test subset, not any internal inconsistency in the training procedure.","tokens_in":160,"tokens_out":4532,"duration_ms":48230,"concrete_test":"Check the GitHub repository's evaluation script/data: rerun the MediaPipe feature extraction on every official private-test image, map the seven labels to happy/sad/unknown as in §3.2.1, and record how many images per class are processed successfully. If the totals are not 291/1110/245, the confusion matrix was computed on a non-representative, undocumented subset. Additionally, apply the released model to the full official private test and recompute accuracy/F1; if the full-test numbers are materially below 71%/62%, the reported benchmark claim is an artifact of subset selection. Finally, report the majority-class baseline on the exact test distribution used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 reports 71.99% categorical accuracy and 0.6298 F1 on the 'test set' and calls this 'close to the classification benchmark of the dataset.' But the model was trained and evaluated on only three labels — happy, sad, and unknown — after §3.2.1 collapsed the seven FER2013 classes. The linked FER2013 benchmark is a seven-class classification problem, so a three-class accuracy number is not comparable to it. The reported confusion matrix (Table 4) also indicates that the test set is not the full official private test. Row sums are 291 happy, 1110 unknown, and 245 sad, totaling 1,646 images. Using the paper's own mapping, the official private test contains 879 happy, 2,116 unknown, and 594 sad (3,589 images). §3.2.2 describes filtering only the training set for MediaPipe detection, so the 1,646-image test partition is unexplained. If it is a detectable subset, the class priors shift from 59.0% unknown to 67.5% unknown, and accuracy/F1 are computed on a biased sample. Even on the reported subset, always predicting 'unknown' would give 67.4% accuracy, so the headline 71% is only a few points above a trivial rule. The central claim is therefore unsupported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a lightweight LSTM model for real-time facial emotion estimation. Inputs are MediaPipe ARKit blendshape features extracted from FER2013 images; the seven FER2013 emotion classes are collapsed into three (happy, sad, unknown). The model is trained on a subset of FER2013 augmented via flips and rotations, and evaluated on an unexplained test partition. The authors report 71.99% categorical accuracy and 0.6298 F1-score and claim this meets the FER2013 benchmark with reduced computation cost. The paper also includes ablations on blendshape selection, model architecture, and loss function.","tokens_in":9643,"tokens_out":3743,"duration_ms":31925,"significance":"If the evaluation were sound, the system would be a useful proof-of-concept for a low-computation, real-time emotion estimator for social robotics, and the authors contribute public code and a demo video, which supports reproducibility. However, the central benchmark claim is not supported by the reported three-class task, and the test-set composition contradicts the official FER2013 private-test counts. These issues are load-bearing for the stated contribution and must be resolved before the work can be assessed.","major_comments":[{"comment":"The reported 71.99% accuracy and 0.6298 F1-score are computed on a three-class task after §3.2.1 collapses the seven FER2013 classes into happy, sad, and unknown. The FER2013 benchmark referenced in §5 is a seven-class classification problem, so a three-class accuracy number is not comparable to the leaderboard. The claim that the model \"meets the accuracy benchmark of the FER2013 dataset\" is therefore unsupported as stated.","section":"Abstract and §5"},{"comment":"The confusion matrix in Table 4 has row sums of 291 happy, 1110 unknown, and 245 sad, totaling 1646 images, whereas the official private test set under the paper's own mapping contains 879 happy, 2116 unknown, and 594 sad (3589 images). Section 3.2.2 explicitly describes filtering only the training set with MediaPipe, so the construction of this 1646-image test subset is unexplained. If the test set was subsampled or filtered, the reported accuracy and F1-score are computed on an unrepresentative distribution; the paper provides no description of this subset, making the reported results irreproducible.","section":"Table 4 and §3.2.2"},{"comment":"On the reported test subset, always predicting the majority class \"unknown\" yields 1110/1646 = 67.4% accuracy, so the reported 71.99% is only about 4.6 percentage points above a trivial constant predictor. The paper does not report per-class precision/recall or compare against this majority-class baseline, and the reported F1-score of 0.6298 is not enough to establish practical utility without such context.","section":"§5"}],"minor_comments":[{"comment":"Table 2 lists the disgust training count as \"Maximum possible\" without giving the actual number used; the text's \"around 400 images\" should be replaced with the exact count.","section":"§3.2.1, Table 2"},{"comment":"The augmentation description \"Random Rotation by 0.2 x 180 degrees\" is ambiguous; the rotation range should be specified directly, e.g., uniform in [-36°, +36°].","section":"§3.2.4"},{"comment":"The paper states the model was trained for 5000 epochs with early stopping, but it does not report the actual number of epochs used or which checkpoint was selected for the final evaluation.","section":"§3.3"},{"comment":"The paper does not define how the F1-score is computed; it should state whether the reported value is macro-averaged, weighted, or per-class.","section":"§5"},{"comment":"The manuscript contains numerous grammatical and typographical errors (e.g., \"in this paper\" in the abstract, \"the face look changes\" in §5, \"proved that\" in §6) that should be corrected in a revision.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The central claim of the abstract — that the model meets the FER2013 benchmark — is not defensible because the evaluation is on a reduced three-class task and the test-set composition is unexplained. The mismatch between Table 4 row sums and the official private-test counts is a serious red flag that should be investigated; if the reported results were obtained on an ad hoc subset, the evaluation is not reproducible. This is a load-bearing error that cannot be fixed by local edits; the work would need to be substantially rewritten and re-evaluated before it could be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the headline claim does not survive contact with the paper. The abstract says the model 'meets the accuracy benchmark of the FER2013 dataset' at 71% accuracy and 62% F1, but the task was reduced to three classes (happy, sad, unknown) in Section 3.2.1. The FER2013 leaderboard is a seven-class problem, so a three-class accuracy number is not comparable. The confusion matrix in Table 4 implies a test set of 1,646 images, while the official private test has 3,589 images under the paper's own mapping. The paper describes filtering only the training set for MediaPipe detection (Section 3.2.2), so the test subset is unexplained. On that subset, always predicting the majority class 'unknown' would give 67.4% accuracy, so the reported 71.99% is only a few points above a trivial rule.\n\nThat said, the paper does some things well. It is a clear, readable proof of concept for a lightweight real-time emotion estimator for social robotics. The authors are upfront about the POC scope and limited compute. They ran ablations on blendshape selection (27 of 52), compared Dense vs LSTM, and switched from MSE to categorical cross-entropy. They provide code and a demo video, and the integration with the Gaze project is concrete. That engineering effort is real.\n\nThe novelty is modest. Using MediaPipe blendshapes as features for emotion recognition appears in the paper's own cited literature ([11], [12]), and LSTM classification on such features is also discussed in [11]. Thresholding to 27 blendshapes and merging to three classes are incremental variations. The related work section effectively undercuts the novelty claim.\n\nThe test-set problem is the main issue. Where does the 1,646-image test set come from? The row sums don't match the official counts, and the class priors shift from 59% to 67.5% unknown. This needs a clear explanation before any metric can be trusted. The overclaim about matching the benchmark should be removed.\n\nWho is this for? A reader working on embedded or social robotics might find the pipeline a useful starting point, and the paper is honest about its constraints. But as a research contribution, the central claim is unsupported. I would not send it to peer review as is; with a corrected evaluation and honest framing, it could become a workshop-level paper.","headline":"A lightweight proof-of-concept emotion estimator with real engineering effort, but the headline benchmark claim is invalid: 71% on a three-class happy/sad/unknown subset is not the seven-class FER2013 benchmark, and the test set is an unexplained, likely biased subsample.","tokens_in":10119,"tokens_out":3089,"would_cite":false,"duration_ms":26206,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A four-layer LSTM over 27 MediaPipe blendshape scores classifies facial expressions as happy, sad, or unknown with 71% accuracy and 62% F1-score, matching the FER2013 benchmark at much lower computational cost.","keywords":["emotion estimation","facial expression recognition","LSTM","MediaPipe blendshapes","FER2013","real-time video classification","social robotics","embedded systems"],"falsifier":"Run the trained model on the complete official FER2013 private test split, collapsing original labels into happy, sad, and unknown, and compare total accuracy and F1 to the reported 0.72 and 0.63; a large drop would show the current evaluation sample is not representative.","tokens_in":9115,"feed_emoji":"😊","tokens_out":12000,"duration_ms":91984,"temperature":0.7,"pith_summary":"This paper tries to establish that emotion recognition from a live camera stream can be made cheap enough for a social robot: a four-layer LSTM that reads 27 MediaPipe blendshape scores per face frame classifies the expression as happy, sad, or unknown and reaches 71% accuracy with a 62% F1-score on FER2013. The point of the work is a proof of concept for embedded systems, where a small model that tracks expressions over time could let a robot adapt its conversation or actions in real time. The authors report that the result matches the FER2013 accuracy benchmark without extra training data and with much lower computation cost than transformer-based or CNN-heavy alternatives. Because the task is deliberately narrowed to three classes, the benchmark comparison applies only to that reduced three-class setting.","feed_headline":"Small neural network reads faces as happy, sad, or unknown at 71%","feed_subtitle":"A 27-number-per-frame model trained on FER2013 matches the dataset benchmark at a fraction of the compute.","key_machinery":"The load-bearing mechanism is a four-layer LSTM network that consumes a sequence of MediaPipe blendshape scores extracted from each detected face. Blendshapes are a compact semantic parametrization of facial expression: probability scores, one per expression component, produced alongside the 468 face landmarks; the paper prunes the 52 scores to 27 by discarding channels whose values almost never rise above 0.4. The LSTM layer, a gated recurrent network that keeps long-term dependencies, turns the frame-by-frame blendshape stream into a happy, sad, or unknown classification, and a final Dense layer with Softmax produces the class scores. This combination is what lets the system classify expressions from video rather than from isolated still images, and it is what keeps the computation small enough for embedded robotics.","core_discovery":"The paper's central claim is that a four-layer LSTM network fed with temporal sequences of 27 MediaPipe blendshape scores is sufficient to estimate the main emotion of a face in a video stream. The model is trained on FER2013 images that MediaPipe can detect, after collapsing the seven original labels into happy, sad, and unknown, augmenting the training set, and dropping blendshape channels that almost never activate. On the test set, it records 0.72 categorical accuracy and 0.63 F1-score, which the paper equates with the dataset's benchmark accuracy, and it produces stable happy, sad, or unknown outputs in a live camera demonstration. The paper also reports that reducing the 52 MediaPipe blendshapes to 27 keeps the same metrics while shrinking the model.","pith_inferences":["The paper's benchmark comparison is not apples-to-apples as stated: 71% on three classes is a different task from the seven-class FER2013 leaderboard, so a fair comparison would either evaluate this model on all seven original labels or collapse an existing seven-class model down to the same three labels.","Because 'unknown' absorbs six of the seven FER2013 emotions, the practical vocabulary is roughly 'happy / sad / everything else'; a social robot would need other modalities or more classes to tell anger, surprise, fear, and disgust apart.","The paper's test-set indexing trick could grow into a systematic error-analysis tool: by keeping the FER2013 row identifier with each blendshape vector, a follow-up study could plot mistakes against occlusion, head pose, and blendshape activation to learn what the model is missing.","Categorical focal loss and class weights, both mentioned in the paper but not tested, may reduce the oscillation the authors observe between happy and sad classification in the confusion matrix."],"forward_implications":["A social robot can run emotion feedback on laptop-class hardware while streaming from a camera, because inference needs only 27 numbers per frame rather than a full image.","A model trained on static FER2013 images transfers to live video without retraining, keeping its accuracy through the MediaPipe feature-extraction and LSTM classification pipeline.","Cutting 52 blendshapes to 27 reduces model size and latency without changing accuracy or F1, which is the basis for the claimed cost savings.","The three-class output (happy, sad, unknown) is enough to drive a simple conversational feedback loop: the robot can detect a pleased or displeased reaction and adapt its topic or action."],"supporting_citations":[{"why":"Defines the FER2013 dataset, its seven expression classes, and the train/public-test/private-test split used for training and evaluation.","marker":"[2]"},{"why":"Describes MediaPipe, the face-detection and landmark framework whose blendshape output provides the model's input features.","marker":"[3]"},{"why":"Provides the MediaPipe pipeline description used for the feature-extraction stage alongside [3].","marker":"[10]"},{"why":"Defines blendshape facial models, the semantic parametrization whose 52 scores are pruned to 27 in this system.","marker":"[15]"},{"why":"Defines the LSTM architecture used as the core of the classifier, chosen for handling temporal dependencies in video frames.","marker":"[19]"},{"why":"Surveys FER datasets and deep-learning approaches, motivating the use of recurrent networks for video-based expression classification.","marker":"[6]"},{"why":"Shows that CNN-RNN hybrids outperform CNN-only classifiers, supporting the choice of an LSTM for expression classification.","marker":"[21]"}],"fun_headline_variants":["LSTM reads faces from 27 blendshapes, hits 71% accuracy","Tiny LSTM matches face-emotion benchmark at lower cost","Face emotion in real time: small LSTM, 71% on FER2013","Video emotion model uses 27 face landmarks, matches FER2013","Happy, sad, or unknown: LSTM on video is 71% accurate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy assumes the specific test images used for evaluation are a fair, unbiased sample of the FER2013 test distribution, but the paper does not describe how that test subset was chosen.","fun_headline_variants_meta":{"raw":{"variants":["LSTM reads faces from 27 blendshapes, hits 71% accuracy","Tiny LSTM matches face-emotion benchmark at lower cost","Face emotion in real time: small LSTM, 71% on FER2013","Video emotion model uses 27 face landmarks, matches FER2013","Happy, sad, or unknown: LSTM on video is 71% accurate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000287,"raw_usage":{"total_tokens":1624,"prompt_tokens":823,"completion_tokens":801,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":699}},"tokens_in":439,"tokens_out":801,"duration_ms":687169,"temperature":1.0,"reasoning_tokens":699,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:56:56.476723+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained model on the complete official FER2013 private test split, collapsing original labels into happy, sad, and unknown, and compare total accuracy and F1 to the reported 0.72 and 0.63; a large drop would show the current evaluation sample is not representative.","supporting_citations":[{"cited_title":"Challenges in representation learning: A report on three machine learning contests","cited_arxiv_id":null,"evidence_quote":"Defines the FER2013 dataset, its seven expression classes, and the train/public-test/private-test split used for training and evaluation."},{"cited_title":"Mediapipe: A framework for perceiving and processing reality","cited_arxiv_id":null,"evidence_quote":"Describes MediaPipe, the face-detection and landmark framework whose blendshape output provides the model's input features."},{"cited_title":"Practice and theory of blendshape facial models","cited_arxiv_id":null,"evidence_quote":"Defines blendshape facial models, the semantic parametrization whose 52 scores are pruned to 27 in this system."},{"cited_title":"Long short-term memory","cited_arxiv_id":null,"evidence_quote":"Defines the LSTM architecture used as the core of the classifier, chosen for handling temporal dependencies in video frames."},{"cited_title":"Deep facial expression recognition: A survey","cited_arxiv_id":null,"evidence_quote":"Surveys FER datasets and deep-learning approaches, motivating the use of recurrent networks for video-based expression classification."},{"cited_title":"Hybrid deep neural networks for face emotion recognition","cited_arxiv_id":null,"evidence_quote":"Shows that CNN-RNN hybrids outperform CNN-only classifiers, supporting the choice of an LSTM for expression classification."}],"review_version":1}