REVIEW 4 major objections 5 minor 31 references
A two-CNN ensemble with a correlation-aware loss predicts face image quality in near-perfect agreement with human ratings, using only about two million parameters.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
An ensemble of MobileNetV3-Small and ShuffleNetV2 with a correlation-aware loss and test-time augmentation reaches SRCC 0.9829 and PLCC 0.9894 on the VQualA FIQA validation set.
T0 review reviewed 2026-08-04 challenge →
load-bearing objection A plausible lightweight FIQA recipe, but the headline 'SOTA' claim rests on an unfair comparison: the proposed model is fine-tuned on the target training set while baselines are used off-the-shelf, and all scores come from the validation set. the 4 major comments →
A Lightweight Ensemble-Based Face Image Quality Assessment Method with Correlation-Aware Loss
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that a two-branch ensemble of compact CNNs—MobileNetV3-Small and ShuffleNetV2, each with a small MLP regression head—trained with a loss that combines MSE with a Pearson-correlation regularizer, and evaluated with test-time augmentation over flips and color shifts, achieves state-of-the-art face image quality assessment on the VQualA challenge. The paper reports SRCC 0.9829 and PLCC 0.9894 on the validation set, outperforming the previous best face-specific method (TOPIQ Swin Face) by a wide margin, and does so with only ~2M trainable parameters and 0.4985 GFLOPs per sample. Ablations show that each component—ensemble, correlation-aware loss, and TTA—contributes incremen
What carries the argument
The load-bearing piece is MSECorrLoss, a joint objective L = L_MSE + α(1 − Pearson(q, q̂)), where the Pearson term is computed over the training batch. It forces the predicted scores to be both close to human labels in absolute value (MSE) and monotonically aligned with them (correlation), which is exactly what SRCC/PLCC measure. The second mechanism is the dual averaging at inference: predictions are averaged over T=3 augmented views (horizontal/vertical flips and slight color variations) and then over the two models, smoothing out individual model errors. Together they convert two generic lightweight backbones into a face-quality regressor that ranks images almost perfectly according to hu
Load-bearing premise
The paper's performance claims rest on scores computed on the 1,000-image validation set (and an ambiguous 80/20 re-split of the training data), because the competition test set was not accessible; if this validation set is not representative of the true test distribution, the reported state-of-the-art correlations will not hold on new data.
What would settle it
Run the trained ensemble on the official VQualA test set (if it becomes available) and compute SRCC/PLCC. If the scores fall noticeably below 0.9829/0.9894, or if the gap over TOPIQ Swin Face disappears, the central claim of state-of-the-art performance is not supported. Alternatively, re-train the model on the original 30k training split, freeze hyperparameters, and evaluate on the 1k validation images only once; a large drop from the reported numbers would indicate overfitting to the validation set.
If this is right
- On the VQualA FIQA validation set, the method reports SRCC 0.9829 and PLCC 0.9894, outperforming all compared baselines, including the face-specific TOPIQ Swin Face (0.9156/0.9416), by a large margin.
- The full system uses about 2 million trainable parameters and 0.4985 GFLOPs per sample, staying within the competition's efficiency budget.
- Ablations isolate three contributors: ensembling the two backbones improves the final score over the best single model; adding MSECorrLoss further improves it; and TTA gives the final boost.
- The method therefore offers a practical trade-off: near-SOTA perceptual quality prediction at a fraction of the computational cost of heavier FIQA models.
Where Pith is reading between the lines
- The reported numbers come from the validation set because the competition test set was inaccessible; if the validation set is not a faithful proxy for the hidden test distribution, the claimed state-of-the-art margin may not transfer. Re-evaluating on the official test set would be the direct check.
- MSECorrLoss is a generic objective: any regression task that cares about ranking (e.g., general no-reference IQA, aesthetic scoring, or image retrieval relevance) could adopt it without architectural changes, since it only modifies the training loss.
- The 80/20 re-split mentioned in Section 4.2 is ambiguous; if the validation set was itself used for hyperparameter selection (including the loss weight and TTA settings), the reported correlations are in-sample and could shrink on a clean holdout.
- The diversity of the two backbones (depthwise-separable MobileNet vs. channel-shuffle ShuffleNet) suggests that ensemble diversity, not raw capacity, is what drives the gain; testing the same loss and TTA with other small backbone pairs would show how far this recipe generalizes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a lightweight face image quality assessment (FIQA) method: an ensemble of MobileNetV3-Small and ShuffleNetV2, each fine-tuned on the VQualA FIQA training set, with a correlation-aware loss (MSECorrLoss) combining MSE and a Pearson correlation regularizer, and test-time augmentation (TTA) with averaging over augmented views and models. Experiments are reported on the VQualA FIQA challenge dataset, claiming SRCC 0.9829 and PLCC 0.9894 on the validation set, with about 2 million parameters and 0.4985 GFLOPs, and the paper states that this significantly outperforms existing baselines across all metrics.
Significance. If the reported numbers were obtained under a fair evaluation protocol, the method would be practically significant: a ~2M-parameter ensemble with sub-0.5 GFLOPs outperforming much heavier transformer-based FIQA models is a useful result for on-device deployment. The ablation structure is logical, and the method is straightforward to reproduce and independently testable. However, as presented, the central SOTA claim is not supported: (i) the comparison in Table 1 is not apples-to-apples because the proposed model is fine-tuned on the target domain while baselines are used off-the-shelf; (ii) all numbers are reported on the validation set that was also used for ablations and hyperparameter selection; and (iii) the loss directly optimizes the same Pearson correlation coefficient used as the PLCC evaluation metric. The contribution would be better framed as an efficient, target-domain-fine-tuned FIQA model with modest improvements, pending fair comparison.
major comments (4)
- [Sec. 4.3, Table 1] The claim 'our method significantly outperforms existing baselines across all metrics' is undermined by a train/protocol mismatch. The text states that for SOTA models 'pretrained weights were used without any additional retraining,' while the proposed ensemble is fine-tuned on the VQualA training split. Table 2 shows that even a single MobileNet with plain MSE (Baseline A) reaches SRCC 0.9662, already far above TOPIQ Swin Face's 0.9156. This indicates that most of the margin over baselines comes from target-domain fine-tuning, not from the ensemble, loss, or TTA. To support the SOTA claim, the authors should fine-tune the baselines on the same training split, or explicitly present Table 1 as an out-of-domain comparison and temper the conclusion.
- [Sec. 4.1 and Sec. 4.2] The evaluation protocol is ambiguous and likely optimistic. Sec. 4.1 says the test set was inaccessible and the validation set was used for experiments; Sec. 4.2 then states 'The datasets were divided with a 80%/20% splitting rate for the training and validation sets.' This is inconsistent with the reported 30k/1k/1k split. If an 80/20 re-split of the training data was used, the relation to the official validation set is unclear. Moreover, the same validation set was used for ablations, hyperparameter choices (learning rate, number of epochs, TTA), and final evaluation, which inflates apparent performance. No error bars or significance tests are provided. The authors should clarify the exact split, and if possible report results on the official hidden test set or on a held-out portion untouched by model selection.
- [Eqs. (4)-(7), Sec. 4.4.2] The correlation-aware loss is directly tied to the evaluation metric: L_Corr = 1 - Pearson(q_i, q_hat_i) is exactly the complement of the PLCC computed on a training batch. Since PLCC is the evaluation metric, part of the reported PLCC gain is optimized by construction. This is not inherently wrong, but the paper's framing that the loss 'tightens alignment with human perceptual judgments' should be tempered; the metric is not an independent measure of perceptual alignment. The ablation comparing MSE vs. MSECorrLoss is also performed on the same validation set, so the +0.003 SRCC and +0.003 PLCC gains should be accompanied by a statement about selection bias.
- [Sec. 3.1, Sec. 4.2, Sec. 4.4.3] The TTA configuration is not reproducible as described. Sec. 3.1 sets T=3, but Sec. 4.2 lists only two augmentations (Random Horizontal Flip and Random Vertical Flip, both with p=1.0), while Sec. 4.4.3 says TTA includes 'horizontal flips and slight color variations.' The exact list of T augmented views is missing, and the relationship between T=3 and the described flips is unexplained. Since TTA is one of the three claimed contributions, this needs to be specified precisely.
minor comments (5)
- [Eq. (3)] The final bias in Eq. (3) is written as b2, but it should be b3 to match the dimensions and the earlier definition of W3 and b3.
- [Abstract and Sec. 1] The contributions state a 'sub-million-parameter scale,' but Sec. 3 and Sec. 4.2 report approximately 2 million parameters. Please correct the inconsistency.
- [Table 1 and References] Table 1 lists 'NIMA [?]' with an unresolved citation marker; NIMA should be reference [20]. Also, reference [16] has a malformed title/author entry ('A Deep Bilinear Convolutional Neural Network. Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Network.').
- [Sec. 4.2] The phrase 'with a80%/20% splitting rate' has a typo ('a80%'), and the sentence 'without applying additional data augmentation' conflicts with the immediately following 'various augmentation strategies were tested.' Please clarify.
- [Sec. 3, Eq. (4)] The balancing hyperparameter alpha in Eq. (4) is never given a value or a sensitivity study. Also, the 'lower learning rate to the backbones' is mentioned but not quantified; please specify the exact learning rates and any decay schedule.
Circularity Check
No formal circular derivation; minor overlap between the correlation-aware training loss and the PLCC evaluation metric, plus one non-load-bearing self-citation.
specific steps
-
other
[Eq. (6) in Sec. 3.1; Table 1 in Sec. 4.3]
"LCorr = 1−Pearson(q_i, ˆq_i) (6) ... Ours 0.9829 0.9894 0.9862"
The loss term LCorr is the negative of the Pearson correlation between predictions and ground-truth scores, and PLCC is the same Pearson correlation computed on the evaluation set. Training with Eq. (6) directly optimizes a PLCC-like objective, so the PLCC improvement attributed to the correlation-aware loss is partly guaranteed by construction rather than being an independent discovery. This is a partial overlap between the training objective and the evaluation metric; because the reported PLCC is measured on a separate validation split, it remains standard supervised learning rather than a full circularity.
full rationale
The paper's central claim is empirical: a lightweight ensemble (MobileNetV3-Small + ShuffleNetV2) is fine-tuned on VQualA training data with an MSE+Pearson-correlation loss and evaluated with SRCC/PLCC. There is no formal derivation chain in which a prediction reduces by definition to a fitted input. The most notable overlap is that the proposed MSECorrLoss includes LCorr = 1−Pearson(q, q_hat), which is the same statistic as the PLCC evaluation metric; this makes part of the reported PLCC gain trained-in rather than independently predicted. However, evaluation is performed on a held-out validation split, so the method remains independently testable. The paper itself states a real limitation in Sec. 4.1: the competition test set was inaccessible, so all experiments use the validation set; Sec. 4.2 then adds an ambiguous 80/20 re-split, creating a generalization risk. Additionally, Sec. 4.3 compares the fine-tuned proposed model against baselines used 'without any additional retraining', which is an unfair protocol and a correctness risk, but not a circularity. Self-citation is limited to Ref. [30], a survey used only for background claims about generic NR-IQA; it is not load-bearing. Overall, there is no significant circularity.
Axiom & Free-Parameter Ledger
free parameters (3)
- alpha =
not reported
- TTA view count T =
3
- Input resize =
600x416
axioms (4)
- standard math Pearson correlation formula (Eq. 7) is a meaningful objective for perceptual quality.
- domain assumption Human MOS labels on VQualA are the correct ground truth for FIQA.
- domain assumption ImageNet pretrained features transfer to face quality regression.
- ad hoc to paper The validation set is a fair proxy for the hidden test set.
Cite this review
Pith. "Pith review of A Lightweight Ensemble-Based Face Image Quality Assessment Method with Correlation-Aware Loss." pith.science (2026). https://pith.science/paper/MSNAEUCA
@misc{pith2026250910114,
author = {Pith},
title = {Pith review of: A Lightweight Ensemble-Based Face Image Quality Assessment Method with Correlation-Aware Loss},
year = {2026},
howpublished = {\url{https://pith.science/paper/MSNAEUCA}},
note = {Machine review of arXiv:2509.10114}
}
read the original abstract
Face image quality assessment (FIQA) plays a critical role in face recognition and verification systems, especially in uncontrolled, real-world environments. Although several methods have been proposed, general-purpose no-reference image quality assessment techniques often fail to capture face-specific degradations. Meanwhile, state-of-the-art FIQA models tend to be computationally intensive, limiting their practical applicability. We propose a lightweight and efficient method for FIQA, designed for the perceptual evaluation of face images in the wild. Our approach integrates an ensemble of two compact convolutional neural networks, MobileNetV3-Small and ShuffleNetV2, with prediction-level fusion via simple averaging. To enhance alignment with human perceptual judgments, we employ a correlation-aware loss (MSECorrLoss), combining mean squared error (MSE) with a Pearson correlation regularizer. Our method achieves a strong balance between accuracy and computational cost, making it suitable for real-world deployment. Experiments on the VQualA FIQA benchmark demonstrate that our model achieves a Spearman rank correlation coefficient (SRCC) of 0.9829 and a Pearson linear correlation coefficient (PLCC) of 0.9894, remaining within competition efficiency constraints.
Figures
Reference graph
Works this paper leans on
-
[1]
Lorenzo Agnolucci Leonardo Galteri Marco Bertini. Quality-Aware Image-Text Alignment for Opinion- Unaware Image Quality Assessment.arXiv preprint arXiv:2403.11176, 2024. 5
Pith/arXiv arXiv 2024
-
[2]
Lacey Best-Rowden and Anil K. Jain. Learning Face Im- age Quality From Human Assessments.IEEE Transactions on Information Forensics and Security, 13(12):3064–3077,
-
[3]
CR-FIQA: Face Image Quality Assessment by Learning Sample Relative Classifiability
Fadi Boutros, Meiling Fang, Marcel Klemt, Biying Fu, and Naser Damer. CR-FIQA: Face Image Quality Assessment by Learning Sample Relative Classifiability. In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5836–5845, Vancouver, BC, Canada, 2023. IEEE. 2
2023
-
[4]
TOPIQ: A Top-Down Approach from Semantics to Distortions for Im- age Quality Assessment.IEEE Transactions on Image Pro- cessing, 2024
Chaofeng Chen, Jiadi Mo, Jingwen Hou, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin. TOPIQ: A Top-Down Approach from Semantics to Distortions for Im- age Quality Assessment.IEEE Transactions on Image Pro- cessing, 2024. 2, 5
2024
-
[5]
Maria Cuellar, Hon Kiu, To, and Arush Mehrotra. Ac- curacy and Fairness of Facial Recognition Technology in Low-Quality Police Images: An Experiment With Synthetic Faces, 2025. arXiv:2505.14320 [cs]. 1
Pith/arXiv arXiv 2025
-
[6]
A Deep Insight into Measuring Face Image Utility with General and Face-specific Image Quality Metrics
Biying Fu, Cong Chen, Olaf Henniger, and Naser Damer. A Deep Insight into Measuring Face Image Utility with General and Face-specific Image Quality Metrics. In2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1121–1130, Waikoloa, HI, USA,
-
[7]
Unsupervised Face Do- main Transfer for Low-Resolution Face Recognition.IEEE Signal Processing Letters, 27:156–160, 2020
Sungeun Hong and Jongbin Ryu. Unsupervised Face Do- main Transfer for Low-Resolution Face Recognition.IEEE Signal Processing Letters, 27:156–160, 2020. 1
2020
-
[8]
Woobin Im, Sungeun Hong, Sung-Eui Yoon, and Hyun S. Yang. Scale-Varying Triplet Ranking with Classification Loss for Facial Age Estimation. InComputer Vision – ACCV 2018, pages 247–259. Springer International Publish- ing, Cham, 2019. Series Title: Lecture Notes in Computer Science. 1
2018
-
[9]
IFQA: Interpretable FACE Quality Assessment
Byungho Jo, Donghyeon Cho, In Kyu Park, and Sungeun Hong. IFQA: Interpretable FACE Quality Assessment. In Proceedings of the IEEE/CVF winter conference on applica- tions of computer vision, pages 3444–3453, 2023. 2, 5
2023
-
[10]
Adam: A Method for Stochastic Optimization.arXiv preprint arXiv:1412.6980,
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization.arXiv preprint arXiv:1412.6980,
-
[11]
Improved Self-Training for Test-Time Adaptation
Jing Ma. Improved Self-Training for Test-Time Adaptation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23701–23710, 2024. 3
2024
-
[12]
Facial Retouching and Alteration Detection
Puspita Majumdar, Akshay Agarwal, Mayank Vatsa, and Richa Singh. Facial Retouching and Alteration Detection. InHandbook of Digital Face Manipulation and Detection, pages 367–387. Springer International Publishing, Cham,
-
[13]
Moorthy, and Alan C
Anish Mittal, Anush K. Moorthy, and Alan C. Bovik. Blind/Referenceless Image Spatial Quality Evaluator. In 2011 Conference Record of the Forty Fifth Asilomar Con- ference on Signals, Systems and Computers (ASILOMAR), pages 723–727, Pacific Grove, CA, USA, 2011. IEEE. 2, 5
2011
-
[14]
Mittal, A
A. Mittal, A. K. Moorthy, and A. C. Bovik. No- Reference Image Quality Assessment in the Spatial Domain. IEEE Transactions on Image Processing, 21(12):4695–4708,
-
[15]
Completely Blind
A. Mittal, R. Soundararajan, and A. C. Bovik. Making a “Completely Blind” Image Quality Analyzer.IEEE Signal Processing Letters, 20(3):209–212, 2013. 2, 5
2013
-
[16]
Blind Im- age Quality Assessment Using A Deep Bilinear Convolu- tional Neural Network.Deep Bilinear Convolutional Neural,
A Deep Bilinear Convolutional Neural Network. Blind Im- age Quality Assessment Using A Deep Bilinear Convolu- tional Neural Network.Deep Bilinear Convolutional Neural,
-
[17]
CLIB-FIQA: Face Image Quality Assessment with Confi- dence Calibration
Fu-Zhao Ou, Chongyi Li, Shiqi Wang, and Sam Kwong. CLIB-FIQA: Face Image Quality Assessment with Confi- dence Calibration. In2024 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 1694– 1704, 2024. ISSN: 2575-7075. 2
2024
-
[18]
Face Image Quality Assessment: A Literature Survey.ACM Computing Surveys, 54(10s):1–49, 2022
Torsten Schlett, Christian Rathgeb, Olaf Henniger, Javier Galbally, Julian Fierrez, and Christoph Busch. Face Image Quality Assessment: A Literature Survey.ACM Computing Surveys, 54(10s):1–49, 2022. 1, 2
2022
-
[19]
Sheikh and A.C
H.R. Sheikh and A.C. Bovik. Image Information and Vi- sual Quality.IEEE Transactions on Image Processing, 15 (2):430–444, 2006. 2
2006
-
[20]
NIMA: Neural Image Assessment.IEEE Transactions on Image Processing, 27(8): 3998–4011, 2018
Hossein Talebi and Peyman Milanfar. NIMA: Neural Image Assessment.IEEE Transactions on Image Processing, 27(8): 3998–4011, 2018. 2
2018
-
[21]
Face Quality Estimation and Its Correlation to Demographic and Non-Demographic Bias in Face Recognition
Philipp Terhorst, Jan Niklas Kolf, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper. Face Quality Estimation and Its Correlation to Demographic and Non-Demographic Bias in Face Recognition. In2020 IEEE International Joint Conference on Biometrics (IJCB), pages 1–11, Houston, TX, USA, 2020. IEEE. 1
2020
-
[22]
SER-FIQ: Unsupervised Estimation of Face Image Quality Based on Stochastic Em- bedding Robustness
Philipp Terhorst, Jan Niklas Kolf, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper. SER-FIQ: Unsupervised Estimation of Face Image Quality Based on Stochastic Em- bedding Robustness. In2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5650–5659, Seattle, W A, USA, 2020. IEEE. 1
2020
-
[23]
Channappayya, and Swarup S
Venkatanath N, Praneeth D, Maruthi Chandrasekhar Bh, Sumohana S. Channappayya, and Swarup S. Medasani. Blind Image Quality Evaluation using Perception based Fea- tures. In2015 Twenty First National Conference on Commu- nications (NCC), pages 1–6, Mumbai, India, 2015. IEEE. 2, 5
2015
-
[24]
Bovik, H.R
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image Quality Assessment: From Error Visibility to Struc- tural Similarity.IEEE Transactions on Image Processing, 13 (4):600–612, 2004. Conference Name: IEEE Transactions on Image Processing. 2
2004
-
[25]
Recent Advances in Rate Con- trol: From Optimization to Implementation and Beyond
Xuekai Wei, Mingliang Zhou, Heqiang Wang, Haoyan Yang, Lei Chen, and Sam Kwong. Recent Advances in Rate Con- trol: From Optimization to Implementation and Beyond. IEEE Transactions on Circuits and Systems for Video Tech- nology, 34(1):17–33, 2024. 2
2024
-
[26]
MANIQA: Multi-Dimension Attention Network for No- Reference Image Quality Assessment
Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. MANIQA: Multi-Dimension Attention Network for No- Reference Image Quality Assessment. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1191–1200, 2022. 5
2022
-
[27]
Face Anti- Spoofing: Model Matters, so Does Data
Xiao Yang, Wenhan Luo, Linchao Bao, Yuan Gao, Dihong Gong, Shibao Zheng, Zhifeng Li, and Wei Liu. Face Anti- Spoofing: Model Matters, so Does Data. In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3502–3511, Long Beach, CA, USA, 2019. IEEE. 1
2019
-
[28]
A Study on the Impact of Face Image Quality on Face Recognition in the Wild, 2023
Na Zhang. A Study on the Impact of Face Image Quality on Face Recognition in the Wild, 2023. arXiv:2307.02679 [cs]. 1
Pith/arXiv arXiv 2023
-
[29]
Efros, Eli Shecht- man, and Oliver Wang
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shecht- man, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018. ISSN: 2575-7075. 2
2018
-
[30]
Wei Zhou, Hadi Amirpour, Christian Timmerer, Guangtao Zhai, Patrick Le Callet, and Alan C. Bovik. Perceptual Vi- sual Quality Assessment: Principles, Methods, and Future Directions, 2025. Version Number: 1. 1
2025
-
[2022]
Series Title: Advances in Computer Vision and Pattern Recognition. 1
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.