Pith. sign in

REVIEW 4 major objections 6 minor 28 references

Improvement in Facial Emotion Recognition using Synthetic Data Generated by Diffusion Model

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read Adding synthetic faces generated by Stable Diffusion to the training set raises facial emotion recognition accuracy from 79.79% to 96.47% on FER2013 and from 94.76% to 99.23% on RAF-DB.

desk verdict The paper's diffusion-augmentation recipe is a reasonable thing to try and the gains are big, but the evaluation protocol is so under-specified that the headline numbers can't be taken at face value. read the letter →

arxiv 2411.10863 v1 pith:NO742EB5 submitted 2024-11-16 cs.CV cs.HCeess.IV

classification cs.CVcs.HCeess.IV
keywords facialemotionrecognitionsyntheticdataaugmentationdiffusionmodelsStableclassimbalanceResEmoteNetFER2013RAF-DB
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the class imbalance in facial emotion recognition can be corrected by adding synthetic face images produced by diffusion models to the training set. Using ResEmoteNet, a convolutional network built from squeeze-and-excitation blocks and residual connections, the authors generate images with Stable Diffusion 2 and Stable Diffusion 3 Medium from text prompts and balance every emotion class to 15,000 samples. They report that this raises accuracy from 79.79% to 96.47% on FER2013 and from 94.76% to 99.23% on RAF-DB, an absolute gain of 16.68 and 4.47 percentage points respectively. The significance is that unbalanced emotion datasets could be repaired synthetically rather than by collecting more labels, and that the reported numbers exceed the comparison methods in the paper's table.

What carries the argument

The load-bearing mechanism is ResEmoteNet, a convolutional network that combines squeeze-and-excitation blocks with residual skip connections, trained on diffusion-augmented data. The squeeze-and-excitation blocks recalibrate channel-wise feature responses, the residual blocks preserve the input signal to ease training, and the augmentation step, using Stable Diffusion 2 and Stable Diffusion 3 Medium, generates new facial images from text prompts. The operation that carries the argument is the systematic balancing of class sizes: FER2013 and RAF-DB are augmented in stages up to 15,000 images per class, and accuracy rises monotonically with each stage.

What would settle it

Retrain ResEmoteNet under the same augmentation protocol but with the validation set taken only from the original training split and test accuracy computed on the official FER2013 private test set and the RAF-DB test set; if the gap over the unaugmented baseline shrinks materially below 16.68 and 4.47 percentage points, the central claim would need revision.

Watch

Extended reading notes

Core claim

The paper's claim is that diffusion-based synthetic augmentation, not a new architecture, is what drives the accuracy jump. The pipeline generates synthetic facial expressions from text prompts, resizes the generated images to the model input size, and merges them into the existing training distributions of FER2013 and RAF-DB so each of the seven emotion classes has up to 15,000 samples. On FER2013 the reported test accuracy rises from 79.79% to 96.47%, with the Fear class improving from 70% to 95% and Happy reaching 100%; on RAF-DB accuracy rises from 94.76% to 99.23%, with four classes at 100%. The authors interpret these results as evidence that balancing the class distribution with synthetic data makes the model generalize better across all emotions.

Load-bearing premise

The reported gains assume that the validation set used for early stopping is drawn only from the training portion and that the test accuracy is measured on the same FER2013 split used by the comparison methods; the paper does not specify either choice.

Editorial extensions

If this is right

  • On FER2013, augmenting each emotion class to 15,000 samples yields 96.47% accuracy, with Happy at 100% and Fear up from 70% to 95%.
  • On RAF-DB, the same augmentation reaches 99.23% accuracy, with Happy, Neutral, Sad, and Surprise all at 100%.
  • The augmented ResEmoteNet outperforms the comparison methods listed in the paper's Table IV on both datasets.
  • The reported gains are largest for the classes that were least represented originally, indicating that balancing alone accounts for much of the improvement.
  • Training remains practical: roughly 6.5 hours for FER2013 and 5 hours for RAF-DB on a Tesla P100, with inference under 5 ms per image.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the claim would be to repeat the augmentation on a different backbone, such as a standard ResNet, to see whether the accuracy jump is specific to ResEmoteNet or transfers to other architectures.
  • Because several classes reach 100% accuracy after augmentation, an independent check on a truly held-out test set would reveal whether this reflects genuine generalization or a distributional overlap between synthetic training images and test images.
  • Comparing 48x48 resizing against the 64x64 resizing used here would clarify whether the improvement comes from class balancing or from a shift in input resolution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a data-augmentation method for facial emotion recognition (FER) that uses Stable Diffusion 2 and Stable Diffusion 3 Medium to generate synthetic per-class training images, which are added to the FER2013 and RAF-DB training sets. The authors then train ResEmoteNet, their own previously proposed architecture, on the augmented data and report accuracies of 96.47% on FER2013 and 99.23% on RAF-DB, compared with 79.79% and 94.76% without augmentation. They compare these results with a set of prior methods in Table IV and conclude that diffusion-based synthetic augmentation substantially improves FER accuracy.

Significance. If the reported numbers are correct, the paper would demonstrate a striking benefit of text-to-image diffusion models for FER, with gains of roughly 17% and 5% on two standard benchmarks. The confusion matrices in Figure 4 are internally consistent, and the idea of balancing classes with synthetic data is well motivated. However, the paper provides no code for the augmentation pipeline, no seeds or error bars, no verification of synthetic label quality, and no control that separates the augmentation effect from the choice of ResEmoteNet. The significance of the empirical claim is therefore currently limited by reproducibility and evaluation concerns.

major comments (4)
  1. [Section IV (Training Configuration) and Section III (Diffusion Based Data Augmentation)] The evaluation protocol is not defined. Section IV says the model is trained for up to 80 epochs with early stopping on validation loss (patience 5), but never states how the validation set is formed from the 28,709 FER2013 training images or the 12,271 RAF-DB training images. Section III's sentence 'The validation and testing sets remained unchanged' is uninterpretable without an initial partition. The paper also does not state whether the FER2013 test accuracy is on the public or the private test split. As a result, the central claim of a 16.68% absolute improvement on FER2013 is not verifiable. The internal inconsistency in Table IV, where the unaugmented ResEmoteNet baseline (79.79%) already exceeds all listed SOTA methods (max 77.85%), strengthens the concern that the baseline and the comparison methods use different evaluation protocols.
  2. [Section V (Results and Discussion)] All results in Table III and Figure 4 are single-run point estimates. Because both the diffusion sampling and the network initialization are stochastic, the reported accuracies are meaningless without a measure of variance. The authors should provide the mean and standard deviation over at least three independent runs, and state the random seeds used for synthetic-data generation and training. This is necessary to determine whether the differences between Aug. 1 and Aug. 4 (e.g., 84.81% versus 96.47% on FER2013) are significant or within run-to-run noise.
  3. [Section III (Diffusion Based Data Augmentation) and Figure 3] The paper provides no verification that the synthetic images actually express the intended emotion. The prompts listed in Table I are generic and the text does not describe how many prompts were used, how many images were generated per prompt, or how the 15,000-images-per-class target was reached without excessive duplication. Without a manual or automated check of a sample of generated images, and without an analysis of the synthetic distribution's similarity to the real distribution, the accuracy gains in Table III could be attributed to mislabeled synthetic examples or to overfitting to synthetic artifacts rather than to corrected class balance.
  4. [Section V.A (Comparison with previous study) and Table IV] The experimental design cannot separate the effect of diffusion-based augmentation from the choice of ResEmoteNet as the backbone, because all experiments use ResEmoteNet only. Since ResEmoteNet is the authors' own prior architecture, a control experiment with a standard FER network (e.g., ResNet-18) trained on the same augmented datasets is necessary to support the conclusion that synthetic data augmentation is what drives the improvement. Without such a control, the statement in Section VI that synthetic data is 'instrumental' is not established.
minor comments (6)
  1. [Section II.A and Section III] There are figure reference errors: Section II.A says 'Fig. II' but the model diagram is Figure 1, and Section III says 'as shown in Fig. IV' but the synthetic images appear in Figure 3.
  2. [Section II.A, Eq. (1)] Equation (1) introduces ws as the attention weights from the SE Block but does not specify its dimensions or the type of multiplication (element-wise or matrix); please clarify the notation.
  3. [Abstract] There is a grammatical error: 'These findings shows' should be 'These findings show.'
  4. [Section IV (Data Processing)] The paper states that synthetic images are resized to 64x64 'to align with the input requirements of the model,' but it is not stated that the real images are also resized to this resolution; please specify the input resolution for all training and test images.
  5. [Table II] The caption 'Aug. refers to augmentation' is uninformative; the table should state explicitly that the entries are target per-class sample counts after augmentation and define what 'Both' means in the Aug. 2-4 columns.
  6. [Section IV (Training Configuration)] No hyperparameters are given for the diffusion models used in data generation (e.g., number of denoising steps, guidance scale, or random seeds), so the synthetic data generation is not reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the augmentation claim is an empirical measurement, and the ResEmoteNet self-citation is code-backed and does not load-bear the result.

full rationale

The central claim is that diffusion-generated synthetic images, added to the FER2013 and RAF-DB training sets, improve ResEmoteNet's test accuracy to 96.47% and 99.23%. This is an empirical measurement, not a derivation. ResEmoteNet is the authors' prior architecture [11], but it is used as a fixed backbone with released source code, so the self-citation is independent support rather than a circular premise; no equation in the paper defines the reported accuracy in terms of the prompts, the diffusion model, or the ResEmoteNet weights. The baseline 'ResEmoteNet (Without Augmentation)' in Table IV is presented as an in-paper comparison, so the improvement is not a fitted parameter renamed as a prediction. The paper's statement that 'the validation and testing sets remained unchanged' and its use of early stopping are potential protocol concerns (the validation split is not described), but absent evidence that validation equals test, this is a correctness and benchmarking risk, not a circular reduction. No self-definitional, fitted-input, or imported-uniqueness step is exhibited, so the score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hand-selected synthetic data quantities (15,000 per class), prompt-derived labels without quality filtering, and an unstated evaluation split. None of these are derived from a principle or released as artifacts, so the reported numbers are not independently computable from the paper.

free parameters (4)
  • Synthetic images per class (Aug4) = 15,000
    Chosen by hand (Aug1 through Aug4 escalate to 15,000 per class); no experiment justifies the stopping point; the reported final accuracy depends on this count.
  • Text prompts for Stable Diffusion = Not fully disclosed (sample list only)
    Prompts are hand-crafted and determine synthetic labels; the paper lists only a few examples and does not release the full set or generation/filtering parameters.
  • Early stopping patience = 5 epochs
    Chosen manually; interacts with the unstated validation split, affecting the reported test accuracy.
  • Learning rate / batch size = 1e-3 / 16
    Standard choices inherited from ResEmoteNet; not tuned in this paper.
assumptions (4)
  • domain assumption Stable Diffusion generated images are labeled correctly by the emotion word in the prompt
    Section III: synthetic labels come from prompt text; no human or automated verification that the generated face actually displays the target emotion.
  • domain assumption The test sets of FER2013 and RAF-DB are used without modification and match the benchmarks compared in Table IV
    Section IV and Table IV: the paper never states whether the public or private FER2013 split is used, nor how validation is carved out; prior SOTA numbers may use a different split.
  • domain assumption Training on 100k+ synthetic images with cross-entropy loss transfers to real test images
    Section V: the reported gains assume synthetic data faithfully represents the real emotion distribution, with no domain-shift analysis.
  • standard math Cross-entropy loss and SGD with plateau scheduling are adequate optimizers for this task
    Section IV: these choices are standard and unproblematic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improvement in Facial Emotion Recognition using Synthetic Data Generated by Diffusion Model." pith.science (2026). https://pith.science/paper/NO742EB5

@misc{pith2026241110863,
  author       = {Pith},
  title        = {Pith review of: Improvement in Facial Emotion Recognition using Synthetic Data Generated by Diffusion Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NO742EB5}},
  note         = {Machine review of arXiv:2411.10863}
}
read the original abstract

Facial Emotion Recognition (FER) plays a crucial role in computer vision, with significant applications in human-computer interaction, affective computing, and areas such as mental health monitoring and personalized learning environments. However, a major challenge in FER task is the class imbalance commonly found in available datasets, which can hinder both model performance and generalization. In this paper, we tackle the issue of data imbalance by incorporating synthetic data augmentation and leveraging the ResEmoteNet model to enhance the overall performance on facial emotion recognition task. We employed Stable Diffusion 2 and Stable Diffusion 3 Medium models to generate synthetic facial emotion data, augmenting the training sets of the FER2013 and RAF-DB benchmark datasets. Training ResEmoteNet with these augmented datasets resulted in substantial performance improvements, achieving accuracies of 96.47% on FER2013 and 99.23% on RAF-DB. These findings shows an absolute improvement of 16.68% in FER2013, 4.47% in RAF-DB and highlight the efficacy of synthetic data augmentation in strengthening FER models and underscore the potential of advanced generative models in FER research and applications. The source code for ResEmoteNet is available at https://github.com/ArnabKumarRoy02/ResEmoteNet

Figures

Figures reproduced from arXiv: 2411.10863 by the authors.

Figure 2
Figure 2. Overall pipeline for data augmentation. where ws represents the attention weights derived from the SE Block. This results in a new feature map f1 ∈ R H×W×C . It is noted that the temporal order of f1 is in accordance with that of the input X. B. Residual Feature Extraction The Residual Network comprises three Residual Blocks, each with weight layers followed by ReLU activation and skip connections that iteratively l… view at source ↗
Figure 3
Figure 3. Examples of synthetic images representing each emotion class in the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Confusion matrices for the performance of the model on (a) Original FER2013, (b) FER2013 with Augmentation 4 (15000 samples in each class), [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 16 canonical work pages

  1. [8]

    Diffusion-based data augmen- tation for skin disease classification: Impact across original medical datasets to fully synthetic images,

    M. Akrout, B. Gyepesi, P. Holl ´o, A. Po´or, B. Kincs˝o, S. Solis, K. Cirone, J. Kawahara, D. Slade, L. Abid, et al., “Diffusion-based data augmen- tation for skin disease classification: Impact across original medical datasets to fully synthetic images,” in International Conference on Medical Image Computing and Computer-Assisted Intervention , pp. 99– 1...

  2. [9]

    Is synthetic data from generative models ready for image recognition?,

    R. He, S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. Qi, “Is synthetic data from generative models ready for image recognition?,” arXiv preprint arXiv:2210.07574 , 2022

  3. [1]

    Facial expression recognition using residual masking network,

    L. Pham, T. H. Vu, and T. A. Tran, “Facial expression recognition using residual masking network,” in 2020 25Th international conference on pattern recognition (ICPR) , pp. 4513–4519, IEEE, 2021

  4. [2]

    A dual-direction attention mixed feature network for facial expression recognition,

    S. Zhang, Y . Zhang, Y . Zhang, Y . Wang, and Z. Song, “A dual-direction attention mixed feature network for facial expression recognition,”MDPI Electronics, vol. 12, no. 17, p. 3595, 2023

  5. [3]

    Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices,

    S. Chen, Y . Liu, X. Gao, and Z. Han, “Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices,” in Biometric Recognition: 13th Chinese Conference, CCBR 2018, Urumqi, China, August 11-12, 2018, Proceedings 13 , pp. 428–438, Springer, 2018

  6. [4]

    Poster++: A simpler and stronger facial expression recognition network,

    J. Mao, R. Xu, X. Yin, Y . Chang, B. Nie, and A. Huang, “Poster++: A simpler and stronger facial expression recognition network,” arXiv preprint arXiv:2301.12149, 2023

  7. [5]

    From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos

    Y . Chen, J. Li, S. Shan, M. Wang, and R. Hong, “From static to dynamic: Adapting landmark-aware image models for facial expression recognition in videos,” arXiv preprint arXiv:2312.05447 , 2023

  8. [6]

    Improved classification for pneumonia detection using transfer learning with gan based synthetic image augmentation,

    D. Srivastav, A. Bajpai, and P. Srivastava, “Improved classification for pneumonia detection using transfer learning with gan based synthetic image augmentation,” in 2021 11th international conference on cloud computing, data science & engineering (confluence), pp. 433–437, IEEE, 2021

Show all 28 references
  1. [7]

    Enhancement of image classification using transfer learning and gan-based synthetic data augmentation,

    S. Chatterjee, D. Hazra, Y .-C. Byun, and Y .-W. Kim, “Enhancement of image classification using transfer learning and gan-based synthetic data augmentation,” Mathematics, vol. 10, no. 9, p. 1541, 2022

  2. [10]

    Spatial deep feature augmentation technique for fer using genetic algorithm,

    N. Nida, M. H. Yousaf, A. Irtaza, S. Javed, and S. A. Velastin, “Spatial deep feature augmentation technique for fer using genetic algorithm,” Neural Computing and Applications , vol. 36, no. 9, pp. 4563–4581, 2024

  3. [11]

    Resemotenet: Bridging accuracy and loss reduction in facial emotion recognition,

    A. K. Roy, H. K. Kathania, A. Sharma, A. Dey, and M. S. A. Ansari, “Resemotenet: Bridging accuracy and loss reduction in facial emotion recognition,” arXiv preprint arXiv:2409.10545 , 2024

  4. [12]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 10684–10695, June 2022

  5. [13]

    Scaling rectified flow transformers for high-resolution image synthesis,

    P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. M ¨uller, H. Saini, Y . Levi, D. Lorenz, A. Sauer, F. Boesel, et al. , “Scaling rectified flow transformers for high-resolution image synthesis,” in Forty-first International Conference on Machine Learning , 2024

  6. [14]

    Progressive distillation for fast sampling of diffusion models,

    T. Salimans and J. Ho, “Progressive distillation for fast sampling of diffusion models,” arXiv preprint arXiv:2202.00512 , 2022

  7. [15]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,

    C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, “Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,”arXiv preprint arXiv:2111.02114, 2021

  8. [16]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195–4205, 2023

  9. [17]

    Imagenet large scale visual recognition challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, pp. 211–252, 2015

  10. [18]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,

    S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, “Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 3558–3568, 2021

  11. [19]

    Challenges in representation learning,

    I. J. Goodfellow, D. Erhan, P. Luc Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y . Tang, D. Thaler, D.-H. Lee, et al. , “Challenges in representation learning,” Neural Networks, vol. 64, no. C, pp. 59–63, 2015

  12. [20]

    Reliable crowdsourcing and deep locality- preserving learning for expression recognition in the wild,

    S. Li, W. Deng, and J. Du, “Reliable crowdsourcing and deep locality- preserving learning for expression recognition in the wild,” in Proceed- ings of the IEEE conference on computer vision and pattern recognition, pp. 2852–2861, 2017

  13. [21]

    An overview of gradient descent optimization algorithms,

    S. Ruder, “An overview of gradient descent optimization algorithms,” arXiv preprint arXiv:1609.04747 , 2016

  14. [22]

    Generalized cross entropy loss for training deep neural networks with noisy labels,

    Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” Advances in neural information processing systems, vol. 31, 2018

  15. [23]

    A lightweight attention-based deep network via multi-scale feature fu- sion for multi-view facial expression recognition,

    A. Ezati, M. Dezyani, R. Rana, R. Rajabi, and A. Ayatollahi, “A lightweight attention-based deep network via multi-scale feature fu- sion for multi-view facial expression recognition,” arXiv preprint arXiv:2403.14318, 2024

  16. [24]

    Local learning with deep and handcrafted features for facial expression recognition,

    M.-I. Georgescu, R. T. Ionescu, and M. Popescu, “Local learning with deep and handcrafted features for facial expression recognition,” IEEE Access, vol. 7, pp. 64827–64836, 2019

  17. [25]

    Cross-task multi-branch vision transformer for facial expression and mask wearing classification,

    A. Zhu, K. Li, T. Wu, P. Zhao, W. Zhou, and B. Hong, “Cross-task multi-branch vision transformer for facial expression and mask wearing classification,” arXiv preprint arXiv:2404.14606 , 2024

  18. [26]

    Emonext: an adapted convnext for facial emotion recognition,

    Y . El Boudouri and A. Bohi, “Emonext: an adapted convnext for facial emotion recognition,” in 2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP) , pp. 1–6, IEEE, 2023

  19. [27]

    Mixcut: A data augmentation method for facial expression recognition,

    J. Yu, Y . Liu, R. Fan, and G. Sun, “Mixcut: A data augmentation method for facial expression recognition,” arXiv preprint arXiv:2405.10489 , 2024

  20. [28]

    Representation learning and identity adversarial training for facial behavior understanding,

    M. Ning, A. A. Salah, and I. O. Ertugrul, “Representation learning and identity adversarial training for facial behavior understanding,” arXiv preprint arXiv:2407.11243, 2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.