{"id":"eb413fcc-76c4-4ea4-86a4-199376d9a021","arxiv_id":"2505.02182","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A CLIP-based deepfake detector that adds conditional value-at-risk and AUC ranking losses reaches AUC 0.969 on the DFWild-Cup test set, modestly above two CLIP baselines.","lead":"This paper combines a frozen CLIP image encoder with a CVaR-based loss and an AUC ranking loss to detect AI-generated faces under class imbalance. It reports higher accuracy on an IEEE Signal Processing Cup benchmark, but the gains over one baseline are small and the statistical support is thin.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.004 AUC gap over Baseline 2 (Table II) is within sampling noise for a 3,072-image test set, so 'significantly outperforming' is not established without error bars or a paired significance test.","rationale":"The reader identified the same weakest assumption: the 0.004 AUC advantage over Baseline 2 is assumed reproducible and significant despite a single seed and a 3,072-image test set. I agree and quantified the concern: with roughly 1,548 real and 1,524 fake test images, the asymptotic standard error of AUC is about 0.003–0.004, so the observed gap is not statistically significant. This is load-bearing because the paper's headline conclusion rests on this margin. Additional red flags—the inconsistent γ values (0.6 vs 0.9) and the near-self Baseline 2—reinforce the concern. However, the method itself is a plausible combination of established components, and the missing statistical evidence could be supplied by a paired bootstrap or multi-seed runs, so the reader's CONDITIONAL verdict remains appropriate. My read does not change that verdict; it strengthens the justification for requesting the concrete test.","tokens_in":12291,"tokens_out":5625,"duration_ms":66084,"concrete_test":"Run a paired bootstrap on the test-set predictions of the proposed model and Baseline 2: resample the 1,548 real and 1,524 fake test images with replacement 10,000 times, compute the ROC-AUC for each model and their difference each time, and report the 95% percentile interval. If the interval contains 0, the claim of superior AUC over Baseline 2 is not established at the 5% level. For a stronger check, retrain both models from scratch with seeds 8079, 1, 2, 3, 4 under the same validation-based tuning and report mean ± std for AUC, accuracy, and EER; if the mean AUC difference is smaller than the pooled standard deviation, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that the proposed CVaR+AUC framework outperforms both CLIP baselines, with the key margin being AUC 0.969067 vs 0.964797 over Baseline 2 (Table II), a difference of 0.00427. The test set contains only 1,548 real and 1,524 fake images. A standard asymptotic standard error for AUC at these sample sizes is roughly 0.003–0.004, so the observed gap is not statistically significant at the 5% level (rough z ≈ 1.0–1.3, p > 0.1). The paper reports no confidence intervals, no multiple seeds (only seed 8079, Section IV-A4), and no paired comparison. Hyperparameters α and γ are selected on a validation set, yet the paper is internally inconsistent about the chosen γ: Section IV-A4 states γ = 0.6, while Table III and Section IV-D use or recommend γ = 0.9. Baseline 2 is not an independent published method but a variant of the authors' own framework [15] with the losses replaced, so the two-baseline comparison set cannot support the abstract and conclusion claim of 'superior generalization' and 'state-of-the-art performance.' If the 0.004 AUC gap is noise, the empirical superiority claim collapses, leaving only an unvalidated method description on a single split.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a deepfake face detection framework that combines frozen CLIP ViT-L/14 features with a trainable MLP, latent feature augmentation, a Conditional Value-at-Risk (CVaR) loss built on a vector-scaling (VS) loss, an AUC ranking loss, and sharpness-aware minimization (SAM). The method is evaluated on the DFWild-Cup benchmark against two CLIP-based baselines, and the authors report a test AUC of 0.969067, accuracy of 0.913737, and EER of 0.093176, concluding that their approach achieves state-of-the-art performance and superior generalization under imbalanced data. The code is released.","tokens_in":12567,"tokens_out":4423,"duration_ms":59183,"significance":"If the reported results were substantiated, the framework would be a useful integration of distributionally robust optimization and ranking-based losses for deepfake detection, and the public code release is a positive contribution. The method is clearly motivated and the paper includes ablations on loss composition and MLP depth. However, the empirical evidence as presented does not currently support the strong claims of statistical superiority and state-of-the-art performance: the key margin over Baseline 2 is small, the experiments use a single split and a single seed, and there are internal inconsistencies in the reported hyperparameters.","major_comments":[{"comment":"The central empirical claim is that the proposed method outperforms Baseline 2, with the key margin being AUC 0.969067 versus 0.964797, a difference of 0.00427, on a test set of only 1,548 real and 1,524 fake images. The paper reports no confidence intervals, no multiple seeds, and no paired significance test. At these sample sizes the standard error of AUC is roughly 0.003-0.004, so the observed gap is within sampling noise and the claim of 'significantly outperforming' is not established. The authors should report confidence intervals or error bars from multiple seeds and perform a paired test (e.g., DeLong's test for AUC) before drawing this conclusion.","section":"Section IV-B, Table II"},{"comment":"There is a direct contradiction about the optimal value of gamma. Section IV-A4 states that the grid search found alpha = 0.9 and gamma = 0.6 as optimal. However, Table III and Section IV-D use gamma = 0.9 for the proposed method, and Table V reports validation AUCs of 0.962341 for gamma = 0.6 and 0.955130 for gamma = 0.9, neither of which equals the reported test AUC of 0.969067. This makes it impossible to determine which configuration produced Table II and undermines the reproducibility of the central result. The authors must reconcile these numbers and specify which gamma corresponds to the test-set results.","section":"Section IV-A4 and Section IV-C/IV-D"},{"comment":"The stated motivation is that deepfake datasets contain 'far fewer fake samples compared to authentic ones,' but the DFWild-Cup training split used in this paper has 219,470 fake samples and 42,690 real samples. The minority class in the actual experimental data is the real class, not the fake class. The paper should either explicitly acknowledge the direction of imbalance in the benchmark or reframe the motivation, because the current text misdescribes the data that the proposed loss functions are applied to.","section":"Table I and Section I"},{"comment":"Baseline 2 is not an independent published method; it is constructed from the authors' own framework in reference [15] by replacing the cross-entropy loss with the VS loss and substituting the AUC loss. The comparison therefore isolates the loss changes but does not constitute a comparison against an external state-of-the-art detector. Consequently, the conclusion's claim of 'state-of-the-art performance' is not supported by the two-baseline comparison set. The authors should either add independent published baselines or temper the claim to a loss-component analysis.","section":"Section IV-A3"},{"comment":"The conclusion claims 'resilience to distribution shifts' and 'superior generalization,' but the experiments are conducted on a single test split of a single benchmark. No cross-dataset evaluation is presented, and the Future Work section explicitly defers evaluation on other datasets. Claims about distribution-shift robustness and generalization should be removed or supported by cross-dataset experiments, or at minimum the paper should state that such claims are not evaluated.","section":"Section IV and Section V"}],"minor_comments":[{"comment":"The description of the AUC loss should clarify how the surrogate logistic loss replaces the indicator function and whether m+ and m- are computed per mini-batch or over the full augmented dataset, since the normalization affects the gradient scale.","section":"Section III-D, Eq. (2)"},{"comment":"The text describes the ROC curve as 'near-perfect with an AUC close to 1,' but the reported test AUC is 0.969. If the figure shows validation performance, this should be stated explicitly to avoid confusion.","section":"Section IV-B, Fig. 2a"},{"comment":"The VS loss formula is difficult to parse because of missing parentheses and inconsistent subscript notation for the class-dependent scaling factors. Please write the loss in a cleaner form with explicit definitions of zeta_0, zeta_1, delta_0, and delta_1.","section":"Section III-D"},{"comment":"The accuracy and F1 values for the 6-layer and deeper MLPs (e.g., accuracy 0.596 for 6 layers) are dramatically lower than their AUC values, which suggests training collapse or a thresholding artifact. The paper should report the evaluation protocol and explain these discrepancies.","section":"Table IV"},{"comment":"The URL in the abstract is written with an underscore ('SP_CUP') while the text on page 1 writes it without ('SP CUP'). Please make the URL consistent and correct throughout.","section":"Abstract and Footnote"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about statistical significance lands: the 0.004 AUC gap over Baseline 2 is within sampling noise for the given test size, and the internal gamma inconsistency means the central number may not be reproducible. The imbalance-direction mismatch in Table I is also a substantive issue that should be fixed before further review. The manuscript is suitable for resubmission if the authors add proper uncertainty quantification, reconcile the hyperparameter reporting, and temper the generalization claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this quickly. It's a competition-style deepfake detection paper from a Purdue group using frozen CLIP ViT-L/14 features, an MLP head, and a composite loss: CVaR over a vector-scaling loss plus an AUC surrogate, trained with SAM. The combination on the DFWild-Cup benchmark is new, and the paper is mostly honest: they run ablations on loss composition and MLP depth, report sensitivity to gamma, and close with a real limitation (needs class priors for the VS loss). Code link is provided.\n\nWhat is actually new is small. Every component is known, and the closest baseline (Baseline 2) is their own earlier Robust CLIP detector with the loss replaced. Against that baseline the gain on the test set is 0.004 AUC and 0.007 accuracy. The test set has only 3,072 images, so that gap is within sampling noise; they report no error bars, no multiple seeds, no paired test. The abstract and conclusion calling this 'superior' and 'state-of-the-art' is stronger than the evidence. There is also an internal inconsistency: Section IV-A4 says grid search picked gamma=0.6, but the sensitivity analysis and ablation recommend gamma=0.9, and Table III uses gamma=0.9 for both. That matters because the headline numbers come from the gamma=0.6 setting, if I read it correctly. Minor issues: 'beta' appears in Algorithm 1's required list but is never defined or used, and there are typos ('moise', 'feature back'). The comparison set is only two CLIP baselines, one being their own model repurposed, so the 'imbalanced data' claim is not tested across datasets or against standard deepfake detectors.\n\nOn balance the idea is plausible: CVaR reweighting plus AUC ranking on frozen CLIP features is a sensible way to handle imbalance, and the paper is clearly written at the level of an undergraduate competition entry. It does not hold up as a standalone research claim without statistical evidence and resolved hyperparameter reporting. If the authors add multiple seeds, confidence intervals, a proper paired comparison, and fix the gamma inconsistency, it could be a credible workshop result. For now, I would treat the numbers as indicative, not established. I'd send it to review only if you want to force the authors to add rigor; otherwise desk-reject with the statistical concerns stated.","headline":"A plausible recombined loss for CLIP-based deepfake detection whose 0.004 AUC edge over their own baseline is within noise; needs error bars and a fix to the gamma inconsistency before the superiority claim can stand.","tokens_in":13164,"tokens_out":2372,"would_cite":false,"duration_ms":29761,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A CLIP-based detector that combines worst-case loss reweighting with AUC ranking achieves the best accuracy and lowest error rate on the DFWild-Cup test set among the compared CLIP baselines.","keywords":["deepfake detection","AI-generated face detection","CLIP","class imbalance","CVaR loss","AUC loss","sharpness-aware minimization","feature augmentation"],"falsifier":"Run the same training procedure on the DFWild-Cup training set with ten different random seeds and compute test AUC confidence intervals; if the intervals for the proposed model and Baseline 2 overlap, the claim of outperforming the baselines fails.","tokens_in":12055,"feed_emoji":"🎭","tokens_out":7278,"duration_ms":81975,"temperature":0.7,"pith_summary":"The paper claims that the usual weakness of deepfake detectors under class imbalance can be addressed by changing the training objective rather than resampling. On CLIP ViT-L/14 features, it trains a small three-layer classifier with a loss that reweights the hardest examples (CVaR over a vector-scaling loss) and adds an AUC ranking term, then flattens the loss landscape with sharpness-aware minimization. On the DFWild-Cup benchmark, this combination reports the highest AUC (0.969), accuracy (0.914), and F1, and the lowest equal error rate (0.093) among the two CLIP baselines. The claim matters because real-world deepfake data is heavily skewed toward real images, and detectors trained on balanced benchmarks often fail when deployment data is imbalanced.","feed_headline":"0.969 AUC for deepfake faces via CVaR loss on CLIP features","feed_subtitle":"Worst-case loss weighting plus AUC ranking beats both CLIP baselines on the imbalanced DFWild-Cup test set.","key_machinery":"The load-bearing mechanism is the composite loss $L = L_{\\mathrm{CVaR}} + \\gamma L_{\\mathrm{AUC}}$. The CVaR loss minimizes the average loss over the hardest $\\alpha$-fraction of examples in each batch, using the hinge form $[\\ell(\\theta; F_i^*, Y_i^*) - \\lambda]_+$, and inside it a vector-scaling loss applies different multiplicative and additive logit factors per class to correct imbalance. The AUC loss is a smoothed ranking loss that pushes real samples above fake samples, directly optimizing the AUC metric. Training uses sharpness-aware minimization to minimize the loss at a perturbed parameter point, seeking a flattened loss landscape for generalization. All of this sits on frozen CLIP ViT-L/14 features augmented with additive Gaussian noise before the classifier head.","core_discovery":"The central discovery is that the CVaR-plus-AUC objective improves CLIP-based deepfake detection under imbalance, with the final model beating both baselines on every metric in Table II: AUC 0.969067 versus 0.964797 for the stronger baseline, accuracy 0.913737 versus 0.906901, and EER 0.093176 versus 0.096457. Ablations attribute the gain to the CVaR loss: replacing it with cross-entropy while keeping the AUC term drops AUC to 0.964737 and accuracy to 0.901367. The paper also reports that a three-layer MLP is optimal; deeper heads overfit.","pith_inferences":["Beyond the paper's experiments, the same CVaR-plus-ranking objective should transfer to other frozen feature extractors, since nothing in the loss depends on CLIP specifically; testing it on a non-CLIP backbone would isolate the loss's contribution.","The paper evaluates only one benchmark and one split, so the robustness claim for unseen generators is extrapolated from CLIP's pretraining; evaluating on newly released diffusion models would be a direct test.","The requirement to know the empirical class distribution for the VS loss weights, which the paper itself notes as a limitation, means the method is not directly usable when class priors are unknown or change over time.","The reported gains come from a single-seed run, so multi-seed evaluation with confidence intervals is the natural next check before deploying the detector in practice."],"forward_implications":["On the DFWild-Cup test set, the proposed model beats both CLIP baselines on all six reported metrics (AUC 0.969067, accuracy 0.913737, F1 0.913693, precision 0.914162, recall 0.913617, EER 0.093176).","The ablation shows the CVaR term is the main source of improvement: replacing it with cross-entropy while keeping the AUC term drops AUC to 0.964737 and accuracy to 0.901367.","The architecture study shows that a three-layer MLP is the best head; deeper heads degrade, with AUC falling to 0.9446 at fifteen layers, indicating CLIP features already encode most of the discriminative signal.","The sensitivity analysis shows the model is stable across gamma values in the tested range, with F1 between 0.893 and 0.908, so the approach does not depend on a fragile loss-weight setting."],"supporting_citations":[{"why":"supplies the CLIP pretraining method whose frozen ViT-L/14 image encoder produces the 768-dimensional features used throughout.","marker":"[20]"},{"why":"defines Baseline 1, the CLIP-based AI-generated image detector with a linear classifier that the paper compares against.","marker":"[21]"},{"why":"defines Baseline 2, the robust CLIP-based detector with latent augmentation and SAM whose losses the paper replaces.","marker":"[15]"},{"why":"provides the vector-scaling loss with per-class logit factors that handles label imbalance inside the CVaR term.","marker":"[57]"},{"why":"supplies the AUC ranking loss used as the second term of the total objective.","marker":"[58]"},{"why":"provides sharpness-aware minimization, the optimizer that flattens the loss landscape for generalization.","marker":"[61]"},{"why":"supplies the DeepfakeBench data compilation, including the train and test splits used for evaluation.","marker":"[63]"},{"why":"provides the pretrained CLIP ViT-L/14 model weights used as the frozen feature encoder.","marker":"[44]"}],"fun_headline_variants":["CVaR loss lifts deepfake AUC to 0.969","Imbalance-proof deepfake detection with CVaR-AUC","Combining CVaR and AUC beats CLIP baselines","Robust face forgery detection despite class imbalance","Deepfake detector uses worst-case loss for robust AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported edge over the stronger baseline is a difference of 0.004 in AUC, measured on one test split with one training seed and hyperparameters selected on the validation set; if that edge is noise, the central comparative claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["CVaR loss lifts deepfake AUC to 0.969","Imbalance-proof deepfake detection with CVaR-AUC","Combining CVaR and AUC beats CLIP baselines","Robust face forgery detection despite class imbalance","Deepfake detector uses worst-case loss for robust AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000471,"raw_usage":{"total_tokens":2290,"prompt_tokens":838,"completion_tokens":1452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":1370}},"tokens_in":454,"tokens_out":1452,"duration_ms":11987,"temperature":1.0,"reasoning_tokens":1370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:58:36.637699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same training procedure on the DFWild-Cup training set with ten different random seeds and compute test AUC confidence intervals; if the intervals for the proposed model and Baseline 2 overlap, the claim of outperforming the baselines fails.","supporting_citations":[{"cited_title":"Rais- ing the bar of ai-generated image detection with clip,","cited_arxiv_id":null,"evidence_quote":"defines Baseline 1, the CLIP-based AI-generated image detector with a linear classifier that the paper compares against."},{"cited_title":"Robust clip-based detector for exposing diffusion model-generated images,","cited_arxiv_id":null,"evidence_quote":"defines Baseline 2, the robust CLIP-based detector with latent augmentation and SAM whose losses the paper replaces."},{"cited_title":"Label- imbalanced and group-sensitive classification under overparameteriza- tion,","cited_arxiv_id":null,"evidence_quote":"provides the vector-scaling loss with per-class logit factors that handles label imbalance inside the CVaR term."},{"cited_title":"Minimax auc fairness: Efficient algorithm with provable convergence,","cited_arxiv_id":null,"evidence_quote":"supplies the AUC ranking loss used as the second term of the total objective."},{"cited_title":"Deepfakebench: A comprehensive benchmark of deepfake detection,","cited_arxiv_id":null,"evidence_quote":"supplies the DeepfakeBench data compilation, including the train and test splits used for evaluation."},{"cited_title":"OpenCLIP,","cited_arxiv_id":null,"evidence_quote":"provides the pretrained CLIP ViT-L/14 model weights used as the frozen feature encoder."}],"review_version":1}