{"id":"b8ec1ab1-a8e3-4558-900f-77fd6706c10b","arxiv_id":"2412.13155","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"FaceQ, a new 12K-image benchmark with multi-dimensional human preference scores, reveals that existing quality metrics poorly match human judgment on AI-generated faces, and F-Eval, an instruction-tuned LMM, outperforms them.","lead":"This paper introduces FaceQ, a database of 12,255 AI-generated face images with 491,000 human preference ratings, and uses it to benchmark 29 face generation, customization, and restoration models. It also shows that current image quality metrics align poorly with human preferences on these faces, and presents F-Eval, a multimodal evaluator that matches human scores more closely.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"F-Eval's 'optimal across all tasks' claim hinges on an unstated or random 80:20 split over a dataset with repeated prompts, identities, and LQ inputs; group-level leakage may inflate reported correlations.","rationale":"The paper's contribution is a benchmark plus a new evaluator. The benchmark's raw material—MOS distributions and model rankings—does not depend on the split, so the dataset and F-Bench analysis can stand even if F-Eval's numbers are leaky. What collapses is the stronger claim that F-Eval is the best one-for-all evaluator. The reader's weakest assumption already pointed at leakage; I agree. The paper's own text makes the risk concrete: same prompts reused across generation and customization, same LQ inputs across restoration models, and no deduplication or identity-aware splitting is described. Also, the split for F-Eval is not even stated, which should be fixed in any revision. This is not a disagreement with the AIGC-IQA consensus; it is an internal evaluation-protocol risk. A grouped split would settle it, and if the grouped split shows no material change, the conditional verdict can be lifted.","tokens_in":30013,"tokens_out":4578,"duration_ms":43572,"concrete_test":"Ask the authors to release the exact split and to rerun the full comparison under a group-disjoint protocol: before splitting 80/20, cluster FaceQ by source—(Gen) prompt text, (Cus) reference identity image, (Res) LQ image—and keep every image in a cluster on one side. Retrain F-Eval, MINTIQA*, MANIQA*, PromptIQA*, and TReS* under this protocol, and also recompute with F-Eval trained only on the training clusters. If the SRCC/KRCC/PLCC margins shrink materially (say, by more than 0.05 in any dimension-task cell of Tables 2-4) or the absolute numbers drop, the headline SOTA claim needs the 'optimal across all tasks and dimensions' wording qualified as within-distribution or source-leaky rather than general.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.3's claim that F-Eval achieves optimal performance across all tasks and dimensions rests on the evaluation protocol in Section 6.2. The paper says deep-learning baselines are 'retrained on FaceQ with an 80:20 training/testing split', but it never states which split is used for F-Eval itself; Section 6.1 just says F-Eval is trained for 10 epochs on FaceQ. If F-Eval was trained on the full dataset, its test images are training images, and its reported SRCC/KRCC/PLCC values cannot be compared with the baselines. Even under the same 80:20 split, the split appears to be random at image level while FaceQ is structured by shared sources. FaceQ-Gen (Sec. 3.1.1) generates multiple images from the same prompts across 14 models; FaceQ-Cus (Sec. 3.1.2) reuses a shared reference identity and prompt list across six customization models; FaceQ-Res (Sec. 3.1.3) restores the same synthetic and real-world low-quality inputs with nine restoration methods. Thus the same prompt, reference identity, or LQ image can appear in both train and test. A model with a text encoder and face encoder can memorize source-level quality patterns, inflating especially the correspondence and ID-fidelity dimensions. Since F-Eval includes both prompt text and reference identity as inputs, it is the method best positioned to exploit this leakage, so the gap over baselines may partly be an artifact of the split rather than genuine one-for-all generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces FaceQ, a large-scale database of 12,255 AI-generated, customized, and restored face images with 491,130 human ratings aggregated into 32,742 MOS values across four dimensions (quality, authenticity, ID fidelity, correspondence), covering 29 generative models. Using FaceQ, the authors construct F-Bench, a benchmark that (i) compares the 29 models across tasks and prompt categories, and (ii) evaluates 26 existing IQA/FQA/AIGCIQA/preference metrics against human MOS. The paper further proposes F-Eval, a Qwen-VL-based LMM with four dimension-specific LoRA experts and a router, which takes the image(s), prompt text, and optional reference image as input and outputs per-dimension quality scores. Reported results show that existing metrics correlate weakly with human MOS on FaceQ, and that F-Eval attains the highest SRCC/KRCC/PLCC in nearly all task-dimension pairs.","tokens_in":30368,"tokens_out":7124,"duration_ms":60563,"significance":"If the dataset and benchmark are released as described, FaceQ would be the first large-scale face-focused AIGC quality database with multi-dimensional human preference labels, and F-Bench would be a practically useful resource for the face generation, customization, and restoration communities. The observation that existing metrics—including recent AIGC-specific ones such as CLIPScore, ImageReward, and MINTIQA—correlate poorly with human MOS on AI-generated faces is an important cautionary result that appears well supported by the reported correlations, which are derived from independent human ratings and are not circular. The paper also provides substantial methodological detail (annotation interface, ITU-R BT.500-14 procedure, degradation pipelines, model implementation details) that increases the usability of the resource. The proposed F-Eval architecture, combining a frozen ArcFace face encoder with dimension-specific LoRA experts, is a reasonable design for a one-for-all evaluator.","major_comments":[{"comment":"The evaluation protocol for F-Eval is not specified. Section 6.2 states that the deep-learning baselines are \"retrained on FaceQ with an 80:20 training/testing split,\" but Section 6.1, which describes F-Eval's training, does not mention any split and says only that the network is trained for 10 epochs on FaceQ. The reader therefore cannot determine whether F-Eval's test images in Tables 2–4 were held out during training. If F-Eval was trained on the full dataset, its reported SRCC/KRCC/PLCC values reflect memorization, which would invalidate the Section 6.3 claim that \"Our method achieves optimal performance across all tasks and dimensions.\" The authors must state the exact split used for F-Eval and, if it was not the same 80:20 split used for the baselines, re-run the comparison under a shared protocol.","section":"Section 6.2 / 6.3"},{"comment":"The dataset is structured by shared source content, and a random image-level split can leak source-level information. FaceQ-Gen (Section 3.1.1) generates multiple images from a common prompt list across 14 models; FaceQ-Cus (Section 3.1.2) reuses reference identities and a subset of the same prompts; FaceQ-Res (Section 3.1.3) restores the same synthetic and real-world low-quality inputs with nine methods. Under an image-level 80:20 split, images derived from the same prompt, identity, or LQ source can appear in both training and test partitions. Because F-Eval explicitly receives the prompt text and reference image as inputs, it is the method best positioned to exploit such leakage, particularly for the correspondence and ID-fidelity dimensions. The manuscript does not report any group-wise splitting, source-based deduplication, or identity-aware partitioning. The authors should re-evaluate F-Eval and the fine-tuned baselines on a group-wise split (e.g., by prompt, identity, and LQ source) or justify why image-level splitting is adequate given the shared-source structure.","section":"Section 6.2 / 3.1"},{"comment":"The state-of-the-art claim is based on a single 80:20 split with no statistical significance assessment. The reported margins between F-Eval and the best fine-tuned baseline are small in several cells (e.g., FaceQ-Gen Quality SRCC 0.8486 vs. 0.8312; FaceQ-Res synthetic Quality SRCC 0.8692 vs. 0.8656), and no confidence intervals, bootstrap tests, or multiple-seed results are provided. Without variance estimates or significance tests, the claim that F-Eval \"achieves optimal performance across all tasks and dimensions\" is not supported beyond the specific split shown. The authors should report mean±std over multiple splits or provide significance tests for the key comparisons.","section":"Table 2 / Section 6.3"}],"minor_comments":[{"comment":"The main text says \"outlier detection based on Kurtosis is conducted with a rejection rate of 3% [78],\" while the supplementary (Section 9.1) says \"3% invalid data are removed and no subject is removed\"; these statements should be reconciled to specify whether ratings or subjects were rejected.","section":"Section 3.3 / Supplementary 9.1"},{"comment":"Tables 3 and 4 cite SER-FIQ as [25], but in the reference list [25] is TReS (Golestaneh et al.); the correct reference for SER-FIQ is [85]. The citation inconsistency should be fixed.","section":"Tables 3 and 4"},{"comment":"The abstract contains a typo: \"restorted faces\" should be \"restored faces.\" The introduction also uses \"state-of-art\" where \"state-of-the-art\" is intended.","section":"Abstract / Introduction"},{"comment":"\"The comparison QA models falls into four categories\" should be \"The comparison QA models fall into four categories.\"","section":"Section 6.2"},{"comment":"The paper states that the FaceQ database \"will be publicly available upon publication,\" but no data or evaluation code are provided in the current version, which limits independent verification of the reported correlations and benchmark findings; the authors should clarify the release timeline and, if possible, provide evaluation code to enable reproducibility.","section":"Availability"}],"recommendation":"major_revision","confidential_remarks":"The dataset and benchmark contributions are potentially valuable and appear to support the main observation that existing quality metrics align poorly with human preferences on AI-generated faces. However, the F-Eval state-of-the-art claim currently hinges on an underspecified evaluation protocol; the missing split specification and the group-level leakage risk are load-bearing and need to be addressed with a clear description and, ideally, a group-wise split experiment. The paper is within the scope of this venue and deserves a revision opportunity rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Know this: the FaceQ dataset is the real contribution, and it should be taken seriously. The F-Eval SOTA claim is not yet trustworthy as reported, because of an unclear and possibly leaking train/test split. I would send the paper to review, but only with the expectation that the authors release data/code, use a leakage-aware split, and report significance or cross-split results before the numbers are cited.\n\nWhat is new: FaceQ is the first large-scale AIGC face quality database covering generation, customization, and restoration, with 12,255 images from 29 models and 491k human ratings across quality, authenticity, ID fidelity, and correspondence. That fills a real gap; AGIQA-3K, AIGCIQA, and RichHF-18K are general AIGC databases, and PIQ23 is real faces. The F-Bench analysis of model strengths and prompt categories is solid and genuinely informative. The finding that NIQE, CLIPScore, BLIPScore, and ImageReward correlate near zero with human MOS on these faces is plausible and, if the data check out, an important caution for practitioners.\n\nWhere it is soft. The main soft spot is exactly what the stress-test note says. Section 6.2 says deep-learning baselines are retrained on FaceQ with an 80:20 training/testing split; Section 6.1 just says F-Eval is trained for 10 epochs on FaceQ, without stating which split is used for the reported results. If F-Eval's numbers come from the full dataset or the training split, they are not comparable with the baselines. Even assuming it is evaluated on the same 20% test split, the split appears random at image level while FaceQ is structured by shared sources: the same prompts appear across 14 generation models, the customization subset reuses a shared reference identity and prompt list across 6 models, and the restoration subset applies 9 methods to the same low-quality inputs. F-Eval takes the prompt and reference image as inputs, so it is the method best positioned to memorize source-level patterns rather than learn quality. That can inflate correspondence and ID fidelity numbers in particular. The authors do not describe deduplication, identity-aware splitting, or group-level cross-validation, and their stated limitations only mention privacy and resolution. This is load-bearing for the 'optimal across all tasks and dimensions' claim. It does not undercut the database itself or the existing-metrics-fail result, both of which are derived from independent human MOS and are not circular.\n\nAlso, all F-Eval comparisons are on a single split with no significance intervals, and data and code are not released yet. Those are fixable.\n\nBottom line: FaceQ/F-Bench deserves a serious referee. The F-Eval comparison needs revision before acceptance. The benchmark analysis and the negative result on existing metrics are worth citing once verified.","headline":"FaceQ is a genuinely useful new benchmark resource, but F-Eval's 'optimal across all tasks' claim rests on an unexplained 80:20 split with likely group-level leakage, so the SOTA numbers need revision before they can be trusted.","tokens_in":30904,"tokens_out":2072,"would_cite":true,"duration_ms":21168,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces FaceQ, a large-scale database of human preference ratings for AI-generated, customized, and restored faces, and F-Eval, a single evaluator that outperforms existing quality metrics on all four dimensions.","keywords":["AI-generated face images","face generation","face customization","face restoration","image quality assessment","human preference","benchmark","large multimodal model"],"falsifier":"Re-split FaceQ by clustering images on source identity, prompt, and low-quality input so no near-duplicate crosses the split, then retrain and re-evaluate F-Eval; if its SRCC or PLCC drops substantially relative to the reported tables, the claim that F-Eval is the best one-for-all evaluator on new faces is not supported.","tokens_in":29808,"feed_emoji":"🧑🎨","tokens_out":7714,"duration_ms":68206,"temperature":0.7,"pith_summary":"The paper sets out to establish that human preferences for AI-generated, customized, and restored faces are multidimensional and are not captured by existing image-quality metrics. It introduces FaceQ, a database of 12,255 face images produced by 29 models, with 491,130 human ratings converted into 32,742 mean opinion scores across four dimensions: quality, authenticity, identity fidelity, and text-image correspondence. On this data it benchmarks 26 existing IQA, face IQA, and preference metrics and finds that standard metrics correlate weakly with human judgment on authenticity, identity fidelity, and correspondence. It then builds F-Eval, a large-multimodal-model evaluator with a frozen face encoder and dimension-specific adapters, and reports that F-Eval outperforms all baselines on all three tasks and all four dimensions. If this holds, the paper supplies the first large-scale benchmark and one-for-all evaluator for how well generative face models match human preferences.","feed_headline":"F-Bench rates AI faces on four human-preference axes","feed_subtitle":"Existing image-quality metrics correlate poorly with human judgment; F-Eval tops them across generation, customization, restoration.","key_machinery":"The load-bearing object is FaceQ, whose subjective scores come from 180 annotators rating images under ITU-R BT.500-style protocols, with kurtosis-based outlier rejection and Z-score rescaling to a 0-100 MOS range. The four rating dimensions define the evaluation space: quality is overall perceptual fidelity, authenticity is how much the image resembles a real photograph, identity fidelity is how well the reference identity survives customization or restoration, and correspondence is text-image alignment. The evaluator F-Eval is an instruction-tuned large multimodal model, built on Qwen2.5-VL, that accepts an image, optional reference image, and prompt, and returns a score per requested dimension. Its distinctive mechanism is a mixture of LoRA experts: four low-rank adapters, one per dimension, are activated by a router that classifies the user's question, so the same weights serve all tasks without cross-dimension interference. A frozen ArcFace face encoder supplies identity-structure features that the vision encoder alone would miss.","core_discovery":"The central claim is that the perceived quality of a generated face is not a single number: authenticity, identity fidelity, and text-image correspondence move independently, and a usable evaluator must score them separately. FaceQ-Gen covers 14 text-to-face generators, FaceQ-Cus covers six identity-driven customization methods with a reference image, and FaceQ-Res covers nine restoration models on both synthetic and real-world low-quality inputs. Benchmarking on these subsets shows that generation models now follow prompts well but lag in authenticity, customization models that preserve identity often sacrifice quality, and restoration models maintain quality better than identity. F-Eval is designed to predict all four dimensions in one pass by fusing a vision encoder, a frozen face encoder (ArcFace), and a pretrained large language model, with four LoRA experts chosen by a trainable router according to the dimension being asked. The paper reports that F-Eval achieves the best correlation with human scores across every task and dimension, with the largest gains on customization, correspondence, and identity fidelity.","pith_inferences":["Editorial extension: if FaceQ's 80/20 split is not deduplicated by identity, prompt, or low-quality source image, the reported superiority of F-Eval could shrink under an identity-aware re-split.","Editorial extension: the same four-dimension, multi-expert instruction-tuning design could transfer to evaluating AI-generated full-body images, videos, or avatar faces, where authenticity and identity drift also matter.","The paper notes in its supplementary material that F-Eval degrades on very small or extreme side-view faces and does not localize distorted regions; a natural follow-up is a face encoder trained on such poses and a region-level explanation head.","Editorial extension: FaceQ's multi-dimensional MOSs could be reused as a preference-aligned reward signal for tuning generation models, though the paper itself does not test that use."],"forward_implications":["FaceQ gives the first standardized way to rank face generation, customization, and restoration models by human preference across separate dimensions rather than by a single quality score.","Researchers evaluating generative face models should treat authenticity, identity fidelity, and correspondence as distinct report cards, since models that score high on one can score low on another.","Existing IQA and face-IQA methods are not reliable proxies for human preference on AI-generated faces, so benchmark claims based on them need re-examination.","F-Eval provides a single instruction-tuned model that can be applied directly to new generated, customized, or restored face images without retraining per dimension."],"supporting_citations":[{"why":"AGIQA-3K is the prior large-scale AIGC quality database whose scope and dimensions FaceQ extends to faces.","marker":"[51]"},{"why":"AIGCIQA2023 supplies the quality-authenticity-correspondence dimension design that FaceQ adapts and augments with identity fidelity.","marker":"[89]"},{"why":"RichHF-18K provides the large-scale human-feedback AIGC dataset that defines the scale and protocol context for FaceQ.","marker":"[59]"},{"why":"PhotoMaker contributes reference identity images and serves as one of the six customized-face models benchmarked in FaceQ-Cus.","marker":"[57]"},{"why":"MINTIQA is the strongest AIGC-specific baseline that F-Eval must outperform in the comparison tables.","marker":"[91]"},{"why":"Qwen2.5-VL is the pretrained large multimodal model that serves as F-Eval's language and vision-fusion backbone.","marker":"[11]"},{"why":"ArcFace is the frozen face encoder that supplies identity and facial-structure features to F-Eval.","marker":"[22]"}],"fun_headline_variants":["F-Eval beats existing metrics on AI face quality","AI face quality: F-Eval tops human-preference correlation","F-Bench rethinks face evaluation with four human axes","Old metrics fail on AI identity, F-Eval succeeds","F-Eval: best predictor of human preference for AI faces"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"F-Eval's reported accuracy assumes the 80/20 training-test split of FaceQ keeps near-duplicate views, repeated identities, and the same prompts or restoration sources out of both sides; if that separation leaks, the headline correlations overstate performance on genuinely new faces.","fun_headline_variants_meta":{"raw":{"variants":["F-Eval beats existing metrics on AI face quality","AI face quality: F-Eval tops human-preference correlation","F-Bench rethinks face evaluation with four human axes","Old metrics fail on AI identity, F-Eval succeeds","F-Eval: best predictor of human preference for AI faces"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00087,"raw_usage":{"total_tokens":3802,"prompt_tokens":1010,"completion_tokens":2792,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":2709}},"tokens_in":626,"tokens_out":2792,"duration_ms":20236,"temperature":1.0,"reasoning_tokens":2709,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:22:57.980360+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-split FaceQ by clustering images on source identity, prompt, and low-quality input so no near-duplicate crosses the split, then retrain and re-evaluate F-Eval; if its SRCC or PLCC drops substantially relative to the reported tables, the claim that F-Eval is the best one-for-all evaluator on new faces is not supported.","supporting_citations":[{"cited_title":"Aigciqa2023: A large-scale im- age quality assessment database for ai generated images: from the perspectives of quality, authenticity and corre- spondence","cited_arxiv_id":null,"evidence_quote":"AIGCIQA2023 supplies the quality-authenticity-correspondence dimension design that FaceQ adapts and augments with identity fidelity."},{"cited_title":"Rich human feedback for text-to-image generation","cited_arxiv_id":null,"evidence_quote":"RichHF-18K provides the large-scale human-feedback AIGC dataset that defines the scale and protocol context for FaceQ."}],"review_version":1}