REVIEW 3 major objections 4 minor 50 references
Benchmarking Face Recognition without Real Faces
T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read Two synthetic face datasets reproduce the ranking behavior of real face-recognition benchmarks across 24 models.
desk verdict First serious attempt at validating synthetic face datasets as real-benchmark substitutes; the core finding is credible, but the model-pool composition needs sensitivity analysis before the confidence intervals can be fully trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing instrument is the cross-dataset correlation analysis: for 24 models, the same verification protocol is run on each synthetic and each real dataset, and the resulting model-level scores (accuracy, EER, ZeroFMR, FMR100, FMR1000) are correlated between every dataset pair. Pearson's r measures linear agreement in absolute scores; Spearman's rho measures whether the ranking of models is preserved. FID scores and mated-versus-non-mated similarity distributions are used to explain why some synthetic datasets succeed (clear identity separability, moderate domain gap) and others fail.
What would settle it
Take a set of recent face recognition models not among the 24 (e.g., newly released transformer architectures from different training regimes) and compute their rankings on MorphFace, Vec2Face, and the seven real benchmarks. If Spearman's rho between the synthetic and real rankings falls below 0.85, or if the 95% confidence intervals no longer overlap the real-real agreement range, the central claim is falsified.
Extended reading notes
Core claim
The central claim is that well-constructed synthetic face datasets can serve as reliable substitutes for real benchmarks in comparative face-recognition evaluation. The evidence is a large cross-correlation study: for each of 24 models, scores on 12 synthetic and 7 real datasets are computed, and agreement between datasets is quantified by Pearson's r and Spearman's rho. MorphFace and Vec2Face consistently achieve mean r = 0.90 and rho = 0.87 across biometric metrics, with 95% confidence intervals overlapping the agreement observed among real datasets themselves. Datasets designed explicitly for benchmarking (ControlFace10k, SynMulti-PIE) are among the worst, while these training-oriented da
Load-bearing premise
The 24-model pool is treated as an unbiased sample of face recognition architectures and training regimes; if it is unrepresentative, the measured agreement between synthetic and real benchmarks could change.
Editorial extensions
If this is right
- Face recognition models can be benchmarked on synthetic data, removing the need to process or redistribute real biometric photos.
- The synthetic-benchmark approach enables fully synthetic pipelines, from training to evaluation, lowering legal and ethical exposure under GDPR and BIPA.
- The validated criteria (r and rho > 0.9 with lower confidence bounds > 0.75) give dataset creators concrete targets for future synthetic datasets.
- Model rankings produced on MorphFace and Vec2Face are expected to transfer to real photographs, so researchers can trust synthetic comparisons for model selection.
- The failure of some synthetic datasets warns that not all synthetic data works for benchmarking; identity separability and intra-class diversity are necessary ingredients.
Reading between the lines
- The specific winners, MorphFace and Vec2Face, may be superseded by future generators; the durable contribution is the validation methodology, which could become a standard acceptance test for synthetic benchmarks.
- Because the model pool includes seven CNNs trained on synthetic data, the correlations might be inflated; recomputing the analysis without those models is a direct robustness test the paper does not report.
- The same correlation-based validation could be applied to other biometric modalities (fingerprint, iris, voice) where synthetic data is emerging, potentially generalizing the privacy-preserving benchmarking argument.
- The 0.85–0.90 agreement level may not hold for specialized evaluation niches (extreme pose, cross-age, demographic subgroups) that real benchmarks capture differently; targeted synthetic datasets might be needed.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper asks whether synthetic face datasets can replace real benchmarks for face recognition evaluation. It compares 12 synthetic datasets against 7 established real benchmarks using 24 pre-trained models, computing accuracy, EER, FMR100, FMR1000, ZeroFMR, Pearson/Spearman correlations, FID, and similarity distributions. The main claim is that MorphFace and Vec2Face reproduce the relative ranking behaviour of real benchmarks, reaching agreement levels within the natural disagreement observed among real benchmarks themselves, thereby enabling fully synthetic benchmarking pipelines.
Significance. If the central claim holds, the paper would make a strong contribution to privacy-preserving face recognition: it would allow model evaluation without any real face data. The study is unusually comprehensive in scope: 12 synthetic datasets, 7 real benchmarks, 24 models spanning CNNs and ViTs, and multiple biometric operating points. The public code release and inclusion of the pair lists in the replication package are concrete reproducibility strengths. However, the headline conclusion rests on a small, partially non-independent model pool that includes models trained on the very synthetic datasets being evaluated, and at least one quantitative table is internally inconsistent. These issues are load-bearing and must be resolved before the claims can be accepted.
major comments (3)
- [§3.2, Table 2; §4.1–4.5] The 24-model pool includes 7 CNNs trained entirely on synthetic data, and Table 2 indicates that ResNet50 variants trained on Langevin-DisCo, IDiff-Face, HyperFace, and SFace are both pool members and candidate benchmark datasets. A model trained on a synthetic dataset is not an independent probe of that dataset as a benchmark; its score reflects training-distribution match. This can inflate the measured synthetic–real correlations and affect the 'within natural disagreement' claim. Please report a sensitivity analysis that excludes all models trained on the synthetic datasets under evaluation, or otherwise demonstrates that the headline correlations are unchanged.
- [§3.2, §3.4, §4.5] The 24 models are not 24 independent observations: the pool contains IResNet18/50/100, EdgeFace base/S/XS/XXS, TransFace S/L, and multiple ResNet50 variants. The bootstrap in §3.4 resamples models with replacement, treating near-duplicates as independent. This makes the 95% CIs narrower than they would be under an appropriate clustered bootstrap (e.g., by architecture family or by training dataset). Since §4.5 uses those CIs to claim overlap with real–real disagreement, the effective sample size should be stated and the CIs recomputed with a cluster-robust procedure.
- [Table 6, FMR100 row] For AgeDB-30, the table reports Meanρ=0.81 and MinLowCI=0.87; for CFP-FP it reports Meanρ=0.86 and MinLowCI=0.89. A lower confidence bound cannot exceed the point estimate of the corresponding pairwise correlation. Therefore one of the two quantities is mislabelled, computed on a different set of pairs, or simply erroneous. This table is part of the quantitative evidence for the 'no systematic gap' conclusion in §4.5, so the inconsistency must be fixed and the corrected values checked against the stated conclusion.
minor comments (4)
- [§4.3, Table 5] FID is computed with an IResNet18 face-recognition embedding rather than the standard Inception-v3. This is a legitimate choice, but it makes the FID values non-comparable to standard FID numbers in the literature. State this explicitly in the text and justify the choice.
- [§3.1, Table 1] The distinction between SFace and SFace2 should be made explicit: SFace is a model and SFace2 is a dataset. The current text can confuse the two, especially in Table 2.
- [§4.4, Figure 3] The text says 'SwinFace and IResNet18 produce similar performance levels while FaceNet CASIA clearly underperforms; Vec2Face and MorphFace reproduce this ordering.' Figure 3 presumably shows only a subset of the 24 models; please clarify which models are displayed and whether the statement is about the displayed subset or the full pool.
- [General] A minor typographical issue: the first affiliation line contains a stray accented character ('Wendkˆuuni'); the author name should appear as 'Wendkûuni' or similar.
Circularity Check
No significant circularity: headline correlations are independently measured against external real benchmarks.
full rationale
The paper's central claim is an empirical benchmark-fidelity comparison: 12 synthetic datasets are scored by 24 pretrained models and correlated (Eqs. 1-3) with 7 established real benchmarks. No parameter is fitted from the real-benchmark scores and then used to predict those same scores; MorphFace and Vec2Face are pre-existing datasets (refs [31,46]), not constructed by this paper. The self-citations to [4] frame dataset-quality notions (intra-class diversity, identity separability) and interpret ZeroFMR outliers, but they are not load-bearing: the rho/r values in Sections 4.1-4.5 are computed from the present measurements, not imported from [4]. The inclusion of 7 synthetic-trained ResNet50 models (Table 2) is a genuine model-pool confound — e.g., an IDiff-Face model scores IDiff-Face — but it is not a definitional circularity: no model in the pool is trained on MorphFace or Vec2Face, and no correlation is forced by construction. One data-integrity issue should be flagged separately: Table 6 reports for FMR100 AgeDB-30 'Mean rho = 0.81' and 'MinLowCI = 0.87', which is impossible if MinLowCI is a minimum lower 95% CI bound; this does not indicate circularity but should be corrected. Overall, the derivation is self-contained against external real benchmarks.
Assumptions & free parameters
assumptions (4)
- domain assumption The seven real benchmarks (LFW, CPLFW, CALFW, AgeDB-30, CFP-FP, IJB-B, IJB-C) provide the correct ground-truth ordering of face recognition models.
- domain assumption The 24 pre-trained models are a representative and sufficiently diverse sample of face recognition systems.
- domain assumption Pearson's r and Spearman's rho between dataset-level model scores are valid proxies for benchmark interchangeability.
- domain assumption The Unrestricted with Labeled Outside Data protocol on randomly sampled 3,000 mated and 3,000 non-mated pairs is a fair approximation of benchmarking behavior for synthetic datasets.
Cite this review
Pith. "Pith review of Benchmarking Face Recognition without Real Faces." pith.science (2026). https://pith.science/paper/OJ7VEPBL
@misc{pith2026260714932,
author = {Pith},
title = {Pith review of: Benchmarking Face Recognition without Real Faces},
year = {2026},
howpublished = {\url{https://pith.science/paper/OJ7VEPBL}},
note = {Machine review of arXiv:2607.14932}
}
read the original abstract
Synthetic face datasets have become effective enough to train face recognition models with accuracy rivaling that of models trained on real photographs. This progress sidesteps the ethical and legal burdens of collecting real biometric data, yet evaluation has not kept pace. Even studies that train entirely on synthetic images still rely on real-face benchmarks to measure performance, leaving the privacy problem only half solved. We ask whether synthetic datasets can replace real benchmarks for face recognition evaluation. We test 12 synthetic datasets against 7 established real benchmarks using 24 pre-trained models that span both convolutional and transformer architectures. Our evaluation covers biometric verification metrics, similarity score distributions, cross-model ranking consistency, and the underlying distributional properties of each dataset. Benchmarking fidelity varies widely across the synthetic candidates, but the two strongest, MorphFace and Vec2Face, reproduce the relative behavior of real benchmarks and reach agreement levels that fall within the natural disagreement already observed among the real benchmarks themselves. These results establish that well-constructed synthetic datasets can support reliable comparative evaluation for face recognition, moving the field closer to a fully synthetic and privacy-preserving pipeline for both training and benchmarking.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
X. An, X. Zhu, Y . Xiao, L. Wu, M. Zhang, Y . Gao, B. Qin, D. Zhang, and Y . Fu. Partial fc: Training 10 million identities on a single machine, 2021
2021
-
[2]
Baltsou, I
G. Baltsou, I. Sarridis, C. Koutlis, and S. Papadopoulos. Sdfd: Building a versatile synthetic face image dataset with diverse attributes. In2024 IEEE 18th International Con- ference on Automatic Face and Gesture Recognition (FG), pages 1–10. IEEE, 2024
2024
-
[3]
Behrmann, W
J. Behrmann, W. Grathwohl, R. T. Chen, D. Duvenaud, and J.-H. Jacobsen. Invertible residual networks. InInternational conference on machine learning, pages 573–582. PMLR, 2019
2019
-
[4]
Borsukiewicz, F
P. Borsukiewicz, F. Boutros, I. E. Olatunji, C. Beumier, W. C. Ou´edraogo, J. Klein, and T. F. Bissyand´e. Beyond real faces: synthetic datasets can achieve reliable recognition perfor- mance without privacy compromise.npj Artificial Intelli- gence, 2026
2026
-
[5]
Boutros, J
F. Boutros, J. H. Grebe, A. Kuijper, and N. Damer. Idiff-face: Synthetic-based face recognition through fizzy identity-conditioned diffusion model. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 19650–19661, 2023
2023
-
[6]
Boutros, M
F. Boutros, M. Huber, A. T. Luu, P. Siebke, and N. Damer. Sface2: Synthetic-based face recognition with w-space identity-driven sampling.IEEE Transactions on Biometrics, Behavior, and Identity Science, 2024
2024
-
[7]
Boutros, M
F. Boutros, M. Huber, P. Siebke, T. Rieber, and N. Damer. Sface: Privacy-friendly and accurate face recognition using synthetic data. In2022 IEEE International Joint Conference on Biometrics (IJCB), pages 1–11. IEEE, 2022
2022
-
[8]
Boutros, V
F. Boutros, V . Struc, J. Fierrez, and N. Damer. Synthetic data for face recognition: Current state and future prospects. Image and Vision Computing, 135:104688, 2023
2023
Show all 50 references
-
[9]
Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman. Vggface2: A dataset for recognising faces across pose and age, 2018
2018
-
[10]
Colbois, T
L. Colbois, T. de Freitas Pereira, and S. Marcel. On the use of automatically generated synthetic image datasets for bench- marking face recognition. In2021 IEEE International Joint Conference on Biometrics (IJCB), pages 1–8. IEEE, 2021
2021
-
[11]
J. Dan, Y . Liu, H. Xie, J. Deng, H. Xie, X. Xie, and B. Sun. Transface: Calibrating transformer training for face recog- nition from a data-centric perspective. InProceedings of the IEEE/CVF international conference on computer vision, pages 20642–20653, 2023
2023
-
[12]
J. Deng, J. Guo, N. Xue, and S. Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019
2019
-
[13]
J. Deng, J. Guo, D. Zhang, Y . Deng, X. Lu, and S. Shi. Lightweight face recognition challenge. In2019 IEEE/CVF International Conference on Computer Vision Workshop (IC- CVW), pages 2638–2646, 2019
2019
-
[14]
Regulation (EU) 2016/679 of the European Parliament and of the Council
European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council
2016
-
[15]
Geissb ¨uhler, H
D. Geissb ¨uhler, H. O. Shahreza, and S. Marcel. Syn- thetic face datasets generation via latent space explo- ration from brownian identity diffusion.arXiv preprint arXiv:2405.00228, 2024
2024 arXiv
-
[16]
George, C
A. George, C. Ecabert, H. O. Shahreza, K. Kotwal, and S. Marcel. Edgeface: Efficient face recognition model for edge devices.IEEE Transactions on Biometrics, Behavior, and Identity Science, 6(2):158–168, 2024
2024
-
[17]
George and S
A. George and S. Marcel. Digi2real: Bridging the realism gap in synthetic data face recognition via foundation models. InProceedings of the Winter Conference on Applications of Computer Vision, pages 1469–1478, 2025
2025
-
[18]
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio. Gen- erative adversarial nets.Advances in neural information pro- cessing systems, 27, 2014
2014
-
[19]
G. E. Granato. Statistics for stochastic modeling of volume reduction, hydrograph extension, and water-quality treat- ment by structural stormwater runoff best management prac- tices (bmps). Technical report, US Geological Survey, 2014
2014
-
[20]
Grother, P
P. Grother, P. Grother, A. Hom, M. Ngan, and K. Hanaoka. Face recognition vendor test (FRVT) Part 7: Identification for paperless travel and immigration. US Department of Commerce, National Institute of Standards and Technology, 2021
2021
-
[21]
Heusel, H
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017
2017
-
[22]
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion proba- bilistic models.Advances in neural information processing systems, 33:6840–6851, 2020
2020
-
[23]
G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller. La- beled faces in the wild: A database forstudying face recog- nition in unconstrained environments. InWorkshop on faces in’Real-Life’Images: detection, alignment, and recognition, 2008
2008
-
[24]
Public act 095-0994, 2008
Illinois General Assembly. Public act 095-0994, 2008
2008
-
[25]
Information technology – biometric performance testing and reporting – part 1: Prin- ciples and framework, 2021
ISO/IEC JTC 1/SC 37 Biometrics. Information technology – biometric performance testing and reporting – part 1: Prin- ciples and framework, 2021
2021
-
[26]
Karras, M
T. Karras, M. Aittala, S. Laine, E. H ¨ark¨onen, J. Hellsten, J. Lehtinen, and T. Aila. Alias-free generative adversarial networks.Advances in neural information processing sys- tems, 34:852–863, 2021
2021
-
[27]
M. Kim, F. Liu, A. Jain, and X. Liu. Dcface: Synthetic face generation with dual condition diffusion model. InProceed- ings of the ieee/cvf conference on computer vision and pat- tern recognition, pages 12715–12725, 2023
2023
-
[28]
Kortylewski, A
A. Kortylewski, A. Schneider, T. Gerig, B. Egger, A. Morel- Forster, and T. Vetter. Training deep face recognition systems with synthetic data.arXiv preprint arXiv:1802.05891, 2018
2018 arXiv
-
[29]
B. Maze, J. Adams, J. A. Duncan, N. Kalka, T. Miller, C. Otto, A. K. Jain, W. T. Niggel, J. Anderson, J. Cheney, et al. Iarpa janus benchmark-c: Face dataset and protocol. In2018 international conference on biometrics (ICB), pages 158–165. IEEE, 2018
2018
-
[30]
Melzi, C
P. Melzi, C. Rathgeb, R. Tolosana, R. Vera-Rodriguez, D. Lawatsch, F. Domin, and M. Schaubert. Gandiffface: Controllable generation of synthetic datasets for face recog- nition with realistic variations. InProceedings of the IEEE/CVF International Conference on Computer Vision...
2023
-
[31]
Y . Mi, Z. Zhong, Y . Huang, Q. Yuan, X. Zhao, J. Xu, S. Ding, S. Wang, R. Guo, and S. Zhou. Data synthesis with diverse styles for face recognition via 3dmm-guided diffusion. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 21203–21214, 2025
2025
-
[32]
Moschoglou, A
S. Moschoglou, A. Papaioannou, C. Sagonas, J. Deng, I. Kot- sia, and S. Zafeiriou. Agedb: the first manually collected, in-the-wild age database. Inproceedings of the IEEE confer- ence on computer vision and pattern recognition workshops, pages 51–59, 2017
2017
-
[33]
Nisevic, D
M. Nisevic, D. Milojevic, and D. Spajic. Synthetic data in medicine: Legal and ethical considerations for patient profil- ing.Computational and Structural Biotechnology Journal, 28:190–198, 2025
2025
-
[34]
Nzalasse, R
K. Nzalasse, R. Raj, E. Laird, and C. Clark. Sig: A syn- thetic identity generation pipeline for generating evaluation datasets for face recognition. InInternational Conference on Pattern Recognition, pages 299–313. Springer, 2024
2024
-
[35]
L. Qin, M. Wang, C. Deng, K. Wang, X. Chen, J. Hu, and W. Deng. Swinface: A multi-task transformer for face recog- nition, expression recognition, age estimation and attribute estimation.IEEE Transactions on Circuits and Systems for Video Technology, 34(4):2223–2234, 2023
2023
-
[36]
H. Qiu, B. Yu, D. Gong, Z. Li, W. Liu, and D. Tao. Syn- face: Face recognition with synthetic data. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 10880–10890, 2021
2021
-
[37]
Rombach, A
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Om- mer. High-resolution image synthesis with latent diffu- sion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684– 10695, 2022
2022
-
[38]
Sengupta, J.-C
S. Sengupta, J.-C. Chen, C. Castillo, V . M. Patel, R. Chel- lappa, and D. W. Jacobs. Frontal to profile face verification in the wild. In2016 IEEE winter conference on applications of computer vision (WACV), pages 1–9. IEEE, 2016
2016
-
[39]
H. O. Shahreza and S. Marcel. Hyperface: Generating syn- thetic face recognition datasets by exploring face embedding hypersphere.arXiv preprint arXiv:2411.08470, 2024
2024 arXiv
-
[40]
Smith and S
M. Smith and S. Miller. The ethical application of biometric facial recognition technology.Ai & Society, 37(1):167–175, 2022
2022
-
[41]
Sun and Z
Z. Sun and Z. Liu. Ensuring privacy in face recognition: a survey on data generation, inference and storage.Discover Applied Sciences, 7(5):441, 2025
2025
-
[42]
Z. Sun, S. Song, I. Patras, and G. Tzimiropoulos. Cemi- face: Center-based semi-hard synthetic face generation for face recognition.Advances in Neural Information Process- ing Systems, 37:35612–35638, 2024
2024
-
[43]
Szegedy, W
C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–9, 2015
2015
-
[44]
Szegedy, V
C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016
2016
-
[45]
Whitelam, E
C. Whitelam, E. Taborsky, A. Blanton, B. Maze, J. Adams, T. Miller, N. Kalka, A. K. Jain, J. A. Duncan, K. Allen, et al. Iarpa janus benchmark-b face dataset. Inproceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 90–98, 2017
2017
-
[46]
H. Wu, J. Singh, S. Tian, L. Zheng, and K. W. Bowyer. Vec2face: Scaling face dataset generation with loosely con- strained vectors.arXiv preprint arXiv:2409.02979, 2024
2024 arXiv
-
[47]
Yeung, T
M. Yeung, T. Teramoto, S. Wu, T. Fujiwara, K. Suzuki, and T. Kojima. Variface: Fair and diverse synthetic dataset gener- ation for face recognition.arXiv preprint arXiv:2412.06235, 2024
2024 arXiv
-
[48]
D. Yi, Z. Lei, S. Liao, and S. Z. Li. Learning face represen- tation from scratch.arXiv preprint arXiv:1411.7923, 2014
2014 arXiv
-
[49]
Zheng and W
T. Zheng and W. Deng. Cross-pose lfw: A database for studying cross-pose face recognition in unconstrained envi- ronments.Beijing University of Posts and Telecommunica- tions, Tech. Rep, 5(7):5, 2018
2018
-
[50]
Zheng, W
T. Zheng, W. Deng, and J. Hu. Cross-age lfw: A database for studying cross-age face recognition in unconstrained en- vironments.arXiv preprint arXiv:1708.08197, 2017
2017 arXiv
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.