{"id":"23bb7e44-9cca-4ea9-bbc3-b40091bc671c","arxiv_id":"2509.10114","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An ensemble of MobileNetV3-Small and ShuffleNetV2 with a correlation-aware loss and test-time augmentation reaches SRCC 0.9829 and PLCC 0.9894 on the VQualA FIQA validation set.","lead":"The paper proposes a lightweight ensemble of two compact neural networks that scores face image quality for recognition systems. It reports state-of-the-art agreement with human ratings on a public benchmark, at a fraction of the computational cost of existing methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unfair baseline comparison: proposed method is fine-tuned on VQualA training data while all baselines are used off-the-shelf, so the claimed SOTA margin may reflect fine-tuning benefit rather than architectural/loss superiority.","rationale":"The reader's weakest assumption correctly flags the validation-set-only evaluation and the ambiguous 80/20 re-split. However, I identify a distinct and more fundamental issue: the comparison against baselines is confounded with training protocol. The paper explicitly discloses that baselines use pretrained weights without retraining (Sec. 4.3), while the proposed model is fine-tuned on the VQualA training set. This alone can explain the large margin over TOPIQ Swin Face; indeed, the ablation's single MobileNet+MSE baseline (SRCC 0.9662) already exceeds that baseline without the proposed contributions. Therefore, the central 'significantly outperforms' claim is not established even on the validation set. This does not change the overall conditional verdict (the paper still has merit as a lightweight FIQA design with internal ablations), but it should be a prominent condition for acceptance. I did not select the parameter-count discrepancy (claimed ~2M vs. realistic ~4.8M for MobileNetV3-Small+ShuffleNetV2) as the primary concern because it is secondary to the performance claim, though it also warrants correction.","tokens_in":8624,"tokens_out":7186,"duration_ms":73902,"concrete_test":"Fine-tune the strongest face-specific baseline, TOPIQ Swin Face, on the VQualA training set using the same schedule (30 epochs, Adam, lr 5e-4) and evaluate on the same validation set. If its SRCC/PLCC rise from 0.9156/0.9416 to approach or exceed the proposed method's 0.9829/0.9894, the claimed advantage is attributable to fine-tuning rather than the proposed architecture, loss, or TTA. Reporting the proposed method without VQualA fine-tuning would further clarify the comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('significantly outperforms existing baselines across all metrics', Sec. 4.3) is undermined by a train/protocol mismatch. The proposed ensemble is fine-tuned on the VQualA training set (30k images), whereas Sec. 4.3 states that for SOTA models 'pretrained weights were used without any additional retraining.' Thus Table 1 compares a target-domain-tuned model to out-of-domain baselines. Even a single MobileNet baseline with plain MSE (Table 2, Baseline A) reaches SRCC 0.9662, already beating TOPIQ Swin Face's 0.9156, suggesting most of the margin comes from fine-tuning on VQualA, not from the proposed ensemble, loss, or TTA. The paper's headline numbers are also reported on the validation set (Sec. 4.1), which was used for ablations and hyperparameter selection, but the more acute problem is that the comparison to prior art is not apples-to-apples. Without fine-tuning the baselines on the same training data, the 'SOTA' claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a lightweight face image quality assessment (FIQA) method: an ensemble of MobileNetV3-Small and ShuffleNetV2, each fine-tuned on the VQualA FIQA training set, with a correlation-aware loss (MSECorrLoss) combining MSE and a Pearson correlation regularizer, and test-time augmentation (TTA) with averaging over augmented views and models. Experiments are reported on the VQualA FIQA challenge dataset, claiming SRCC 0.9829 and PLCC 0.9894 on the validation set, with about 2 million parameters and 0.4985 GFLOPs, and the paper states that this significantly outperforms existing baselines across all metrics.","tokens_in":8967,"tokens_out":4886,"duration_ms":53083,"significance":"If the reported numbers were obtained under a fair evaluation protocol, the method would be practically significant: a ~2M-parameter ensemble with sub-0.5 GFLOPs outperforming much heavier transformer-based FIQA models is a useful result for on-device deployment. The ablation structure is logical, and the method is straightforward to reproduce and independently testable. However, as presented, the central SOTA claim is not supported: (i) the comparison in Table 1 is not apples-to-apples because the proposed model is fine-tuned on the target domain while baselines are used off-the-shelf; (ii) all numbers are reported on the validation set that was also used for ablations and hyperparameter selection; and (iii) the loss directly optimizes the same Pearson correlation coefficient used as the PLCC evaluation metric. The contribution would be better framed as an efficient, target-domain-fine-tuned FIQA model with modest improvements, pending fair comparison.","major_comments":[{"comment":"The claim 'our method significantly outperforms existing baselines across all metrics' is undermined by a train/protocol mismatch. The text states that for SOTA models 'pretrained weights were used without any additional retraining,' while the proposed ensemble is fine-tuned on the VQualA training split. Table 2 shows that even a single MobileNet with plain MSE (Baseline A) reaches SRCC 0.9662, already far above TOPIQ Swin Face's 0.9156. This indicates that most of the margin over baselines comes from target-domain fine-tuning, not from the ensemble, loss, or TTA. To support the SOTA claim, the authors should fine-tune the baselines on the same training split, or explicitly present Table 1 as an out-of-domain comparison and temper the conclusion.","section":"Sec. 4.3, Table 1"},{"comment":"The evaluation protocol is ambiguous and likely optimistic. Sec. 4.1 says the test set was inaccessible and the validation set was used for experiments; Sec. 4.2 then states 'The datasets were divided with a 80%/20% splitting rate for the training and validation sets.' This is inconsistent with the reported 30k/1k/1k split. If an 80/20 re-split of the training data was used, the relation to the official validation set is unclear. Moreover, the same validation set was used for ablations, hyperparameter choices (learning rate, number of epochs, TTA), and final evaluation, which inflates apparent performance. No error bars or significance tests are provided. The authors should clarify the exact split, and if possible report results on the official hidden test set or on a held-out portion untouched by model selection.","section":"Sec. 4.1 and Sec. 4.2"},{"comment":"The correlation-aware loss is directly tied to the evaluation metric: L_Corr = 1 - Pearson(q_i, q_hat_i) is exactly the complement of the PLCC computed on a training batch. Since PLCC is the evaluation metric, part of the reported PLCC gain is optimized by construction. This is not inherently wrong, but the paper's framing that the loss 'tightens alignment with human perceptual judgments' should be tempered; the metric is not an independent measure of perceptual alignment. The ablation comparing MSE vs. MSECorrLoss is also performed on the same validation set, so the +0.003 SRCC and +0.003 PLCC gains should be accompanied by a statement about selection bias.","section":"Eqs. (4)-(7), Sec. 4.4.2"},{"comment":"The TTA configuration is not reproducible as described. Sec. 3.1 sets T=3, but Sec. 4.2 lists only two augmentations (Random Horizontal Flip and Random Vertical Flip, both with p=1.0), while Sec. 4.4.3 says TTA includes 'horizontal flips and slight color variations.' The exact list of T augmented views is missing, and the relationship between T=3 and the described flips is unexplained. Since TTA is one of the three claimed contributions, this needs to be specified precisely.","section":"Sec. 3.1, Sec. 4.2, Sec. 4.4.3"}],"minor_comments":[{"comment":"The final bias in Eq. (3) is written as b2, but it should be b3 to match the dimensions and the earlier definition of W3 and b3.","section":"Eq. (3)"},{"comment":"The contributions state a 'sub-million-parameter scale,' but Sec. 3 and Sec. 4.2 report approximately 2 million parameters. Please correct the inconsistency.","section":"Abstract and Sec. 1"},{"comment":"Table 1 lists 'NIMA [?]' with an unresolved citation marker; NIMA should be reference [20]. Also, reference [16] has a malformed title/author entry ('A Deep Bilinear Convolutional Neural Network. Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Network.').","section":"Table 1 and References"},{"comment":"The phrase 'with a80%/20% splitting rate' has a typo ('a80%'), and the sentence 'without applying additional data augmentation' conflicts with the immediately following 'various augmentation strategies were tested.' Please clarify.","section":"Sec. 4.2"},{"comment":"The balancing hyperparameter alpha in Eq. (4) is never given a value or a sensitivity study. Also, the 'lower learning rate to the backbones' is mentioned but not quantified; please specify the exact learning rates and any decay schedule.","section":"Sec. 3, Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The core idea is not particularly novel and the empirical evidence is currently insufficient to support the claimed state-of-the-art result. I recommend a major revision with the primary requirement being a fair comparison: either fine-tune the baselines on the same VQualA training split or clearly restrict the claims to 'fine-tuned on VQualA, evaluated on the validation set.' If the official test set remains inaccessible, cross-validation on the training set with all model selection conducted inside each fold would be a reasonable fallback. The paper might be suitable for publication after such a revision, though the contribution will likely be much more modest than currently stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a clean engineering study: two lightweight CNNs fine-tuned on the VQualA training set, ensemble averaging, a Pearson-correlation loss term, and test-time augmentation. The efficiency numbers (about 2M parameters, 0.5 GFLOPs) are genuinely attractive, and the ablations in Table 2 are done properly, showing each component adds a small but consistent increment. I also give the authors credit for being explicit that the competition test set was inaccessible and that the validation set was used for experiments.\n\nThe soft spots are real and load-bearing. The biggest issue is the comparison in Table 1. The proposed ensemble is fine-tuned on the VQualA training set, while the baselines are used with pretrained weights and no retraining. That is not apples-to-apples, and it likely explains most of the reported margin. Even a single MobileNet with plain MSE (Baseline A) reaches SRCC 0.9662, already beating TOPIQ Swin Face's 0.9156. So the claim of 'significantly outperforms existing baselines across all metrics' is unsupported as stated. A proper comparison would fine-tune the baselines on the same training data, or at least disclose the protocol mismatch upfront.\n\nSecond, all reported numbers come from the same validation set used for ablations and hyperparameter selection. The paper mentions an 80/20 split in Sec. 4.2, but the dataset description in Sec. 4.1 already defines train/val/test sizes. That ambiguity needs clearing up. Without error bars or significance tests, the small ablation gaps (0.002–0.008) could be noise.\n\nThird, the correlation-aware loss directly optimizes the Pearson correlation that is also the evaluation metric. That is not fatal, since evaluation happens on a separate split, but it is worth stating that part of the PLCC gain is trained into the model.\n\nWho is this for? Practitioners who want a very small FIQA model and want a solid recipe with component ablations. It is not a rigorous SOTA claim as it stands. The flaws are fixable—matched baseline training and a clean evaluation protocol would make the comparison meaningful. I would send it to peer review with a strong request for revision, because the engineering is competent and the efficiency angle is useful.","headline":"A plausible lightweight FIQA recipe, but the headline 'SOTA' claim rests on an unfair comparison: the proposed model is fine-tuned on the target training set while baselines are used off-the-shelf, and all scores come from the validation set.","tokens_in":9394,"tokens_out":1450,"would_cite":false,"duration_ms":16741,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-CNN ensemble with a correlation-aware loss predicts face image quality in near-perfect agreement with human ratings, using only about two million parameters.","keywords":["face image quality assessment","no-reference IQA","ensemble learning","correlation-aware loss","test-time augmentation","lightweight CNNs","MobileNetV3-Small","ShuffleNetV2"],"falsifier":"Run the trained ensemble on the official VQualA test set (if it becomes available) and compute SRCC/PLCC. If the scores fall noticeably below 0.9829/0.9894, or if the gap over TOPIQ Swin Face disappears, the central claim of state-of-the-art performance is not supported. Alternatively, re-train the model on the original 30k training split, freeze hyperparameters, and evaluate on the 1k validation images only once; a large drop from the reported numbers would indicate overfitting to the validation set.","tokens_in":8577,"feed_emoji":"📊","tokens_out":5108,"duration_ms":45218,"temperature":0.7,"pith_summary":"This paper sets out to show that face image quality can be assessed with a small, fast neural network ensemble without sacrificing correlation with human ratings. It pairs two lightweight image-classification backbones, MobileNetV3-Small and ShuffleNetV2, fine-tunes each to output a single quality score, and averages their predictions. A custom loss that mixes mean squared error with a Pearson-correlation penalty steers the model toward correct ranking as well as correct absolute scores, and test-time averaging over flipped and color-shifted copies adds stability. On the VQualA FIQA benchmark's validation set, the method reports a Spearman rank correlation of 0.9829 and a Pearson correlation of 0.9894, beating heavier face-specific models while using roughly two million parameters and under half a GFLOP per image. The implication is that lightweight, deployable FIQA can match or exceed much larger models.","feed_headline":"0.98 face-quality correlation from two small CNNs","feed_subtitle":"An ensemble of MobileNet and ShuffleNet with a correlation-aware loss beats heavier FIQA models at a fraction of the compute.","key_machinery":"The load-bearing piece is MSECorrLoss, a joint objective L = L_MSE + α(1 − Pearson(q, q̂)), where the Pearson term is computed over the training batch. It forces the predicted scores to be both close to human labels in absolute value (MSE) and monotonically aligned with them (correlation), which is exactly what SRCC/PLCC measure. The second mechanism is the dual averaging at inference: predictions are averaged over T=3 augmented views (horizontal/vertical flips and slight color variations) and then over the two models, smoothing out individual model errors. Together they convert two generic lightweight backbones into a face-quality regressor that ranks images almost perfectly according to hu","core_discovery":"The central claim is that a two-branch ensemble of compact CNNs—MobileNetV3-Small and ShuffleNetV2, each with a small MLP regression head—trained with a loss that combines MSE with a Pearson-correlation regularizer, and evaluated with test-time augmentation over flips and color shifts, achieves state-of-the-art face image quality assessment on the VQualA challenge. The paper reports SRCC 0.9829 and PLCC 0.9894 on the validation set, outperforming the previous best face-specific method (TOPIQ Swin Face) by a wide margin, and does so with only ~2M trainable parameters and 0.4985 GFLOPs per sample. Ablations show that each component—ensemble, correlation-aware loss, and TTA—contributes incremen","pith_inferences":["The reported numbers come from the validation set because the competition test set was inaccessible; if the validation set is not a faithful proxy for the hidden test distribution, the claimed state-of-the-art margin may not transfer. Re-evaluating on the official test set would be the direct check.","MSECorrLoss is a generic objective: any regression task that cares about ranking (e.g., general no-reference IQA, aesthetic scoring, or image retrieval relevance) could adopt it without architectural changes, since it only modifies the training loss.","The 80/20 re-split mentioned in Section 4.2 is ambiguous; if the validation set was itself used for hyperparameter selection (including the loss weight and TTA settings), the reported correlations are in-sample and could shrink on a clean holdout.","The diversity of the two backbones (depthwise-separable MobileNet vs. channel-shuffle ShuffleNet) suggests that ensemble diversity, not raw capacity, is what drives the gain; testing the same loss and TTA with other small backbone pairs would show how far this recipe generalizes."],"forward_implications":["On the VQualA FIQA validation set, the method reports SRCC 0.9829 and PLCC 0.9894, outperforming all compared baselines, including the face-specific TOPIQ Swin Face (0.9156/0.9416), by a large margin.","The full system uses about 2 million trainable parameters and 0.4985 GFLOPs per sample, staying within the competition's efficiency budget.","Ablations isolate three contributors: ensembling the two backbones improves the final score over the best single model; adding MSECorrLoss further improves it; and TTA gives the final boost.","The method therefore offers a practical trade-off: near-SOTA perceptual quality prediction at a fraction of the computational cost of heavier FIQA models."],"fun_headline_variants":["Lightweight ensemble hits 0.98 face-quality correlation","Two tiny CNNs, one 0.98 face-quality score","Correlation-aware loss lifts FIQA to 0.98 SRCC","MobileNet+ShuffleNet beat heavy FIQA with 0.98 correlation","Efficient FIQA: 0.9829 SRCC from two small CNNs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's performance claims rest on scores computed on the 1,000-image validation set (and an ambiguous 80/20 re-split of the training data), because the competition test set was not accessible; if this validation set is not representative of the true test distribution, the reported state-of-the-art correlations will not hold on new data.","fun_headline_variants_meta":{"raw":{"variants":["Lightweight ensemble hits 0.98 face-quality correlation","Two tiny CNNs, one 0.98 face-quality score","Correlation-aware loss lifts FIQA to 0.98 SRCC","MobileNet+ShuffleNet beat heavy FIQA with 0.98 correlation","Efficient FIQA: 0.9829 SRCC from two small CNNs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000705,"raw_usage":{"total_tokens":3028,"prompt_tokens":770,"completion_tokens":2258,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":2175}},"tokens_in":514,"tokens_out":2258,"duration_ms":17669,"temperature":1.0,"reasoning_tokens":2175,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T18:05:36.035351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained ensemble on the official VQualA test set (if it becomes available) and compute SRCC/PLCC. If the scores fall noticeably below 0.9829/0.9894, or if the gap over TOPIQ Swin Face disappears, the central claim of state-of-the-art performance is not supported. Alternatively, re-train the model on the original 30k training split, freeze hyperparameters, and evaluate on the 1k validation images only once; a large drop from the reported numbers would indicate overfitting to the validation set.","supporting_citations":[],"review_version":1}