REVIEW 4 major objections 6 minor 28 references
Improvement in Facial Emotion Recognition using Synthetic Data Generated by Diffusion Model
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Adding synthetic faces generated by Stable Diffusion to the training set raises facial emotion recognition accuracy from 79.79% to 96.47% on FER2013 and from 94.76% to 99.23% on RAF-DB.
desk verdict The paper's diffusion-augmentation recipe is a reasonable thing to try and the gains are big, but the evaluation protocol is so under-specified that the headline numbers can't be taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is ResEmoteNet, a convolutional network that combines squeeze-and-excitation blocks with residual skip connections, trained on diffusion-augmented data. The squeeze-and-excitation blocks recalibrate channel-wise feature responses, the residual blocks preserve the input signal to ease training, and the augmentation step, using Stable Diffusion 2 and Stable Diffusion 3 Medium, generates new facial images from text prompts. The operation that carries the argument is the systematic balancing of class sizes: FER2013 and RAF-DB are augmented in stages up to 15,000 images per class, and accuracy rises monotonically with each stage.
What would settle it
Retrain ResEmoteNet under the same augmentation protocol but with the validation set taken only from the original training split and test accuracy computed on the official FER2013 private test set and the RAF-DB test set; if the gap over the unaugmented baseline shrinks materially below 16.68 and 4.47 percentage points, the central claim would need revision.
Extended reading notes
Core claim
The paper's claim is that diffusion-based synthetic augmentation, not a new architecture, is what drives the accuracy jump. The pipeline generates synthetic facial expressions from text prompts, resizes the generated images to the model input size, and merges them into the existing training distributions of FER2013 and RAF-DB so each of the seven emotion classes has up to 15,000 samples. On FER2013 the reported test accuracy rises from 79.79% to 96.47%, with the Fear class improving from 70% to 95% and Happy reaching 100%; on RAF-DB accuracy rises from 94.76% to 99.23%, with four classes at 100%. The authors interpret these results as evidence that balancing the class distribution with synthetic data makes the model generalize better across all emotions.
Load-bearing premise
The reported gains assume that the validation set used for early stopping is drawn only from the training portion and that the test accuracy is measured on the same FER2013 split used by the comparison methods; the paper does not specify either choice.
Editorial extensions
If this is right
- On FER2013, augmenting each emotion class to 15,000 samples yields 96.47% accuracy, with Happy at 100% and Fear up from 70% to 95%.
- On RAF-DB, the same augmentation reaches 99.23% accuracy, with Happy, Neutral, Sad, and Surprise all at 100%.
- The augmented ResEmoteNet outperforms the comparison methods listed in the paper's Table IV on both datasets.
- The reported gains are largest for the classes that were least represented originally, indicating that balancing alone accounts for much of the improvement.
- Training remains practical: roughly 6.5 hours for FER2013 and 5 hours for RAF-DB on a Tesla P100, with inference under 5 ms per image.
Reading between the lines
- A direct test of the claim would be to repeat the augmentation on a different backbone, such as a standard ResNet, to see whether the accuracy jump is specific to ResEmoteNet or transfers to other architectures.
- Because several classes reach 100% accuracy after augmentation, an independent check on a truly held-out test set would reveal whether this reflects genuine generalization or a distributional overlap between synthetic training images and test images.
- Comparing 48x48 resizing against the 64x64 resizing used here would clarify whether the improvement comes from class balancing or from a shift in input resolution.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a data-augmentation method for facial emotion recognition (FER) that uses Stable Diffusion 2 and Stable Diffusion 3 Medium to generate synthetic per-class training images, which are added to the FER2013 and RAF-DB training sets. The authors then train ResEmoteNet, their own previously proposed architecture, on the augmented data and report accuracies of 96.47% on FER2013 and 99.23% on RAF-DB, compared with 79.79% and 94.76% without augmentation. They compare these results with a set of prior methods in Table IV and conclude that diffusion-based synthetic augmentation substantially improves FER accuracy.
Significance. If the reported numbers are correct, the paper would demonstrate a striking benefit of text-to-image diffusion models for FER, with gains of roughly 17% and 5% on two standard benchmarks. The confusion matrices in Figure 4 are internally consistent, and the idea of balancing classes with synthetic data is well motivated. However, the paper provides no code for the augmentation pipeline, no seeds or error bars, no verification of synthetic label quality, and no control that separates the augmentation effect from the choice of ResEmoteNet. The significance of the empirical claim is therefore currently limited by reproducibility and evaluation concerns.
major comments (4)
- [Section IV (Training Configuration) and Section III (Diffusion Based Data Augmentation)] The evaluation protocol is not defined. Section IV says the model is trained for up to 80 epochs with early stopping on validation loss (patience 5), but never states how the validation set is formed from the 28,709 FER2013 training images or the 12,271 RAF-DB training images. Section III's sentence 'The validation and testing sets remained unchanged' is uninterpretable without an initial partition. The paper also does not state whether the FER2013 test accuracy is on the public or the private test split. As a result, the central claim of a 16.68% absolute improvement on FER2013 is not verifiable. The internal inconsistency in Table IV, where the unaugmented ResEmoteNet baseline (79.79%) already exceeds all listed SOTA methods (max 77.85%), strengthens the concern that the baseline and the comparison methods use different evaluation protocols.
- [Section V (Results and Discussion)] All results in Table III and Figure 4 are single-run point estimates. Because both the diffusion sampling and the network initialization are stochastic, the reported accuracies are meaningless without a measure of variance. The authors should provide the mean and standard deviation over at least three independent runs, and state the random seeds used for synthetic-data generation and training. This is necessary to determine whether the differences between Aug. 1 and Aug. 4 (e.g., 84.81% versus 96.47% on FER2013) are significant or within run-to-run noise.
- [Section III (Diffusion Based Data Augmentation) and Figure 3] The paper provides no verification that the synthetic images actually express the intended emotion. The prompts listed in Table I are generic and the text does not describe how many prompts were used, how many images were generated per prompt, or how the 15,000-images-per-class target was reached without excessive duplication. Without a manual or automated check of a sample of generated images, and without an analysis of the synthetic distribution's similarity to the real distribution, the accuracy gains in Table III could be attributed to mislabeled synthetic examples or to overfitting to synthetic artifacts rather than to corrected class balance.
- [Section V.A (Comparison with previous study) and Table IV] The experimental design cannot separate the effect of diffusion-based augmentation from the choice of ResEmoteNet as the backbone, because all experiments use ResEmoteNet only. Since ResEmoteNet is the authors' own prior architecture, a control experiment with a standard FER network (e.g., ResNet-18) trained on the same augmented datasets is necessary to support the conclusion that synthetic data augmentation is what drives the improvement. Without such a control, the statement in Section VI that synthetic data is 'instrumental' is not established.
minor comments (6)
- [Section II.A and Section III] There are figure reference errors: Section II.A says 'Fig. II' but the model diagram is Figure 1, and Section III says 'as shown in Fig. IV' but the synthetic images appear in Figure 3.
- [Section II.A, Eq. (1)] Equation (1) introduces ws as the attention weights from the SE Block but does not specify its dimensions or the type of multiplication (element-wise or matrix); please clarify the notation.
- [Abstract] There is a grammatical error: 'These findings shows' should be 'These findings show.'
- [Section IV (Data Processing)] The paper states that synthetic images are resized to 64x64 'to align with the input requirements of the model,' but it is not stated that the real images are also resized to this resolution; please specify the input resolution for all training and test images.
- [Table II] The caption 'Aug. refers to augmentation' is uninformative; the table should state explicitly that the entries are target per-class sample counts after augmentation and define what 'Both' means in the Aug. 2-4 columns.
- [Section IV (Training Configuration)] No hyperparameters are given for the diffusion models used in data generation (e.g., number of denoising steps, guidance scale, or random seeds), so the synthetic data generation is not reproducible.
Circularity Check
No significant circularity; the augmentation claim is an empirical measurement, and the ResEmoteNet self-citation is code-backed and does not load-bear the result.
full rationale
The central claim is that diffusion-generated synthetic images, added to the FER2013 and RAF-DB training sets, improve ResEmoteNet's test accuracy to 96.47% and 99.23%. This is an empirical measurement, not a derivation. ResEmoteNet is the authors' prior architecture [11], but it is used as a fixed backbone with released source code, so the self-citation is independent support rather than a circular premise; no equation in the paper defines the reported accuracy in terms of the prompts, the diffusion model, or the ResEmoteNet weights. The baseline 'ResEmoteNet (Without Augmentation)' in Table IV is presented as an in-paper comparison, so the improvement is not a fitted parameter renamed as a prediction. The paper's statement that 'the validation and testing sets remained unchanged' and its use of early stopping are potential protocol concerns (the validation split is not described), but absent evidence that validation equals test, this is a correctness and benchmarking risk, not a circular reduction. No self-definitional, fitted-input, or imported-uniqueness step is exhibited, so the score is 0.
Assumptions & free parameters
free parameters (4)
- Synthetic images per class (Aug4) =
15,000
- Text prompts for Stable Diffusion =
Not fully disclosed (sample list only)
- Early stopping patience =
5 epochs
- Learning rate / batch size =
1e-3 / 16
assumptions (4)
- domain assumption Stable Diffusion generated images are labeled correctly by the emotion word in the prompt
- domain assumption The test sets of FER2013 and RAF-DB are used without modification and match the benchmarks compared in Table IV
- domain assumption Training on 100k+ synthetic images with cross-entropy loss transfers to real test images
- standard math Cross-entropy loss and SGD with plateau scheduling are adequate optimizers for this task
Cite this review
Pith. "Pith review of Improvement in Facial Emotion Recognition using Synthetic Data Generated by Diffusion Model." pith.science (2026). https://pith.science/paper/NO742EB5
@misc{pith2026241110863,
author = {Pith},
title = {Pith review of: Improvement in Facial Emotion Recognition using Synthetic Data Generated by Diffusion Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/NO742EB5}},
note = {Machine review of arXiv:2411.10863}
}
read the original abstract
Facial Emotion Recognition (FER) plays a crucial role in computer vision, with significant applications in human-computer interaction, affective computing, and areas such as mental health monitoring and personalized learning environments. However, a major challenge in FER task is the class imbalance commonly found in available datasets, which can hinder both model performance and generalization. In this paper, we tackle the issue of data imbalance by incorporating synthetic data augmentation and leveraging the ResEmoteNet model to enhance the overall performance on facial emotion recognition task. We employed Stable Diffusion 2 and Stable Diffusion 3 Medium models to generate synthetic facial emotion data, augmenting the training sets of the FER2013 and RAF-DB benchmark datasets. Training ResEmoteNet with these augmented datasets resulted in substantial performance improvements, achieving accuracies of 96.47% on FER2013 and 99.23% on RAF-DB. These findings shows an absolute improvement of 16.68% in FER2013, 4.47% in RAF-DB and highlight the efficacy of synthetic data augmentation in strengthening FER models and underscore the potential of advanced generative models in FER research and applications. The source code for ResEmoteNet is available at https://github.com/ArnabKumarRoy02/ResEmoteNet
Figures
Reference graph
Works this paper leans on
-
[8]
M. Akrout, B. Gyepesi, P. Holl ´o, A. Po´or, B. Kincs˝o, S. Solis, K. Cirone, J. Kawahara, D. Slade, L. Abid, et al., “Diffusion-based data augmen- tation for skin disease classification: Impact across original medical datasets to fully synthetic images,” in International Conference on Medical Image Computing and Computer-Assisted Intervention , pp. 99– 1...
work page 2023
-
[9]
Is synthetic data from generative models ready for image recognition?,
R. He, S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. Qi, “Is synthetic data from generative models ready for image recognition?,” arXiv preprint arXiv:2210.07574 , 2022
arXiv 2022
-
[1]
Facial expression recognition using residual masking network,
L. Pham, T. H. Vu, and T. A. Tran, “Facial expression recognition using residual masking network,” in 2020 25Th international conference on pattern recognition (ICPR) , pp. 4513–4519, IEEE, 2021
work page 2020
-
[2]
A dual-direction attention mixed feature network for facial expression recognition,
S. Zhang, Y . Zhang, Y . Zhang, Y . Wang, and Z. Song, “A dual-direction attention mixed feature network for facial expression recognition,”MDPI Electronics, vol. 12, no. 17, p. 3595, 2023
work page 2023
-
[3]
Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices,
S. Chen, Y . Liu, X. Gao, and Z. Han, “Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices,” in Biometric Recognition: 13th Chinese Conference, CCBR 2018, Urumqi, China, August 11-12, 2018, Proceedings 13 , pp. 428–438, Springer, 2018
work page 2018
-
[4]
Poster++: A simpler and stronger facial expression recognition network,
J. Mao, R. Xu, X. Yin, Y . Chang, B. Nie, and A. Huang, “Poster++: A simpler and stronger facial expression recognition network,” arXiv preprint arXiv:2301.12149, 2023
arXiv 2023
-
[5]
Y . Chen, J. Li, S. Shan, M. Wang, and R. Hong, “From static to dynamic: Adapting landmark-aware image models for facial expression recognition in videos,” arXiv preprint arXiv:2312.05447 , 2023
work page Pith review arXiv 2023
-
[6]
D. Srivastav, A. Bajpai, and P. Srivastava, “Improved classification for pneumonia detection using transfer learning with gan based synthetic image augmentation,” in 2021 11th international conference on cloud computing, data science & engineering (confluence), pp. 433–437, IEEE, 2021
work page 2021
Show all 28 references
-
[7]
Enhancement of image classification using transfer learning and gan-based synthetic data augmentation,
S. Chatterjee, D. Hazra, Y .-C. Byun, and Y .-W. Kim, “Enhancement of image classification using transfer learning and gan-based synthetic data augmentation,” Mathematics, vol. 10, no. 9, p. 1541, 2022
2022
-
[10]
Spatial deep feature augmentation technique for fer using genetic algorithm,
N. Nida, M. H. Yousaf, A. Irtaza, S. Javed, and S. A. Velastin, “Spatial deep feature augmentation technique for fer using genetic algorithm,” Neural Computing and Applications , vol. 36, no. 9, pp. 4563–4581, 2024
2024
-
[11]
Resemotenet: Bridging accuracy and loss reduction in facial emotion recognition,
A. K. Roy, H. K. Kathania, A. Sharma, A. Dey, and M. S. A. Ansari, “Resemotenet: Bridging accuracy and loss reduction in facial emotion recognition,” arXiv preprint arXiv:2409.10545 , 2024
2024 arXiv
-
[12]
High- resolution image synthesis with latent diffusion models,
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 10684–10695, June 2022
2022
-
[13]
Scaling rectified flow transformers for high-resolution image synthesis,
P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. M ¨uller, H. Saini, Y . Levi, D. Lorenz, A. Sauer, F. Boesel, et al. , “Scaling rectified flow transformers for high-resolution image synthesis,” in Forty-first International Conference on Machine Learning , 2024
2024
-
[14]
Progressive distillation for fast sampling of diffusion models,
T. Salimans and J. Ho, “Progressive distillation for fast sampling of diffusion models,” arXiv preprint arXiv:2202.00512 , 2022
2022 arXiv
-
[15]
Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,
C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, “Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,”arXiv preprint arXiv:2111.02114, 2021
2021 arXiv
-
[16]
Scalable diffusion models with transformers,
W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195–4205, 2023
2023
-
[17]
Imagenet large scale visual recognition challenge,
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, pp. 211–252, 2015
2015
-
[18]
Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,
S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, “Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 3558–3568, 2021
2021
-
[19]
Challenges in representation learning,
I. J. Goodfellow, D. Erhan, P. Luc Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y . Tang, D. Thaler, D.-H. Lee, et al. , “Challenges in representation learning,” Neural Networks, vol. 64, no. C, pp. 59–63, 2015
2015
-
[20]
Reliable crowdsourcing and deep locality- preserving learning for expression recognition in the wild,
S. Li, W. Deng, and J. Du, “Reliable crowdsourcing and deep locality- preserving learning for expression recognition in the wild,” in Proceed- ings of the IEEE conference on computer vision and pattern recognition, pp. 2852–2861, 2017
2017
-
[21]
An overview of gradient descent optimization algorithms,
S. Ruder, “An overview of gradient descent optimization algorithms,” arXiv preprint arXiv:1609.04747 , 2016
2016 arXiv
-
[22]
Generalized cross entropy loss for training deep neural networks with noisy labels,
Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” Advances in neural information processing systems, vol. 31, 2018
2018
-
[23]
A lightweight attention-based deep network via multi-scale feature fu- sion for multi-view facial expression recognition,
A. Ezati, M. Dezyani, R. Rana, R. Rajabi, and A. Ayatollahi, “A lightweight attention-based deep network via multi-scale feature fu- sion for multi-view facial expression recognition,” arXiv preprint arXiv:2403.14318, 2024
2024 arXiv
-
[24]
Local learning with deep and handcrafted features for facial expression recognition,
M.-I. Georgescu, R. T. Ionescu, and M. Popescu, “Local learning with deep and handcrafted features for facial expression recognition,” IEEE Access, vol. 7, pp. 64827–64836, 2019
2019
-
[25]
Cross-task multi-branch vision transformer for facial expression and mask wearing classification,
A. Zhu, K. Li, T. Wu, P. Zhao, W. Zhou, and B. Hong, “Cross-task multi-branch vision transformer for facial expression and mask wearing classification,” arXiv preprint arXiv:2404.14606 , 2024
2024 arXiv
-
[26]
Emonext: an adapted convnext for facial emotion recognition,
Y . El Boudouri and A. Bohi, “Emonext: an adapted convnext for facial emotion recognition,” in 2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP) , pp. 1–6, IEEE, 2023
2023
-
[27]
Mixcut: A data augmentation method for facial expression recognition,
J. Yu, Y . Liu, R. Fan, and G. Sun, “Mixcut: A data augmentation method for facial expression recognition,” arXiv preprint arXiv:2405.10489 , 2024
2024 arXiv
-
[28]
Representation learning and identity adversarial training for facial behavior understanding,
M. Ning, A. A. Salah, and I. O. Ertugrul, “Representation learning and identity adversarial training for facial behavior understanding,” arXiv preprint arXiv:2407.11243, 2024
2024 arXiv
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.