{"id":"0a9f89ef-6e8f-4483-8c1c-e43370b5cfad","arxiv_id":"2509.04150","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Finetuned CLIP-pretrained ConvNeXt-base and ViT-b32 classifiers reach 81% accuracy on Deepfake-Eval-2024, within noise of the leading commercial detector's 82%.","lead":"By tuning hyperparameters and finetuning CLIP-pretrained open vision models, the authors reach 81% accuracy on the in-the-wild Deepfake-Eval-2024 benchmark, close to a leading commercial detector's 82%. The result suggests simple open-source detectors were undertuned, but part of the gain over the original 63% baseline comes from a method change, not only tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark label validity is the load-bearing risk: Deepfake-Eval-2024 mixes AI-generated, edited, and source-judged images, so the 81% accuracy may not measure deepfake detection.","rationale":"The reader's weakest assumption identified both benchmark label validity and small-test-set noise. I agree that label validity is a serious concern, and it is explicitly supported by the paper's own Section 4.4. However, I do not view the small-test-set noise as equally load-bearing: the 18-point improvement over the 63% baseline is many standard errors (paired difference SE ~2.2%), so the claim that simple baselines are much better is statistically robust. The 1-point gap to the commercial detector is within noise, but the paper only claims 'competing', which remains defensible even with a few points of uncertainty. The deeper issue is construct validity: if labels are based on source trustworthiness or any editing, the model may not be learning deepfake detection at all. This threatens the interpretation of the result, not just its precision. The reader's verdict of CONDITIONAL is appropriate; my concern reinforces the need for label-quality analysis but does not push toward rejection, since the paper transparently discloses the limitation and the empirical result on the benchmark stands. I chose 'partial' agreement because the reader bundles label validity with test noise, whereas I would prioritize label validity as the decisive condition.","tokens_in":8460,"tokens_out":10776,"duration_ms":101140,"concrete_test":"Have two independent forensic analysts re-annotate a random sample of ~200 Deepfake-Eval-2024 test images into three classes: (a) AI-generated, (b) conventionally edited/manipulated, (c) authentic. Compute inter-annotator agreement (e.g., Cohen's kappa) between the original benchmark labels and the consensus re-labels, and compute the ConvNeXt-base/ViT-b32 model's accuracy separately on the subset confirmed as AI-generated. If accuracy on the confirmed AI-generated subset is substantially below the reported 81% (e.g., <70%), or if agreement is low (kappa < 0.6), the central claim is not supported; if accuracy remains high, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that simple open-source detectors are competitive with commercial deepfake detectors—rests on Deepfake-Eval-2024's ground truth measuring AI-generated content. Section 4.4 explicitly states: 'Not all fake images were AI-generated' and 'TrueMedia.org reviewers often labeled the ground truth of these image based on the trustworthiness of the original source or if the photo appears to be edited in any manner.' This means the 'fake' class is a mixture of AI-generated images, conventional edits, and images distrusted due to source reputation. If a meaningful fraction of test labels reflect source trustworthiness rather than manipulation, the reported 81% accuracy and the 1-point gap to the commercial detector describe detection of 'inauthentic/suspicious' images, not deepfakes. The paper's own error analysis demonstrates the issue (e.g., a real photo modified with image-editing software labeled fake), making it central rather than peripheral. Without showing performance on a cleanly-defined AI-generated subset, the headline is ambiguous and the comparison to commercial detectors does not establish the intended conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper revisits the Ojha et al. baseline for deepfake detection on the Deepfake-Eval-2024 benchmark. Using standard pretrained backbones (ResNet-50, ViT-b32, ConvNeXt-base) with a final dropout + linear layer, the authors tune hyperparameters such as learning rate, scheduler, pretraining source, and finetuning vs. freezing. They report that a CLIP-pretrained ConvNeXt-base or ViT-b32, finetuned with a low learning rate and cosine annealing, reaches 81% test accuracy on the benchmark's single 60/40 split, compared to the 63% originally reported for the same baseline and the 82% of the leading commercial detector. They also report computational costs, ROC AUC, and GradCAM visualizations, and release code.","tokens_in":8734,"tokens_out":3010,"duration_ms":31580,"significance":"If correct, this is a practically useful demonstration that simple open-source detectors can be competitive with proprietary systems on an in-the-wild benchmark, and it highlights the sensitivity of such baselines to hyperparameter choices. The paper's strengths include a clear hyperparameter ablation, evaluation on an external benchmark (not circular), a focus on compute/interpretability tradeoffs, and released code. However, the central comparative claim rests on small accuracy differences on a single test split and on benchmark labels that the paper itself acknowledges are not purely AI-generated; as presented, the evidence does not firmly establish that open-source detectors 'compete' with the commercial detector on the construct of deepfake detection.","major_comments":[{"comment":"The headline comparison (81% vs. 82%, 79% vs. 81%) is reported without confidence intervals, bootstrap estimates, or multiple random seeds. With n=789 test images, the standard error of accuracy is roughly sqrt(0.81*0.19/789) ≈ 1.4 percentage points, so a 1–2 point gap is within sampling noise. The claim that these open-source models 'closely follow' or 'compete with' the commercial detector is not statistically supported. Please report CIs, bootstrap intervals, or at least results over several seeds and random splits.","section":"Section 4.3 / Table 2"},{"comment":"The paper states: 'Not all fake images were AI-generated' and that TrueMedia.org reviewers 'labeled the ground truth of these image based on the trustworthiness of the original source or if the photo appears to be edited in any manner.' This means the 81% accuracy is for a 'fake/inauthentic' class that mixes AI generation, conventional editing, and source-reputation judgments, not purely deepfakes. The central claim about deepfake-detection competitiveness is therefore ambiguous. Please re-analyze results on a subset of images that are clearly AI-generated (or otherwise filter by label rationale), and temper the conclusions accordingly.","section":"Section 4.4"},{"comment":"Hyperparameters were selected using only 116 validation images drawn from the same single split, and the final test result is from that same split. This creates a risk of selection overfitting and does not measure generalization to other splits. Additionally, the paper compares its 81% to the 63% reported in [5] without rerunning the original baseline under the same protocol (augmentation, resolution, epochs, optimizer). To support an '18% improvement' claim, please include a controlled comparison where the Ojha-style baseline is retrained with the same preprocessing and training setup (except the hyperparameters under study), and ideally evaluate across multiple splits.","section":"Section 3.1 / Table 1"}],"minor_comments":[{"comment":"Typo: 'commerial' should be 'commercial.' Also, 'signficant' in Section 4.2 and 'to deployed' in Section 4.3 need correction.","section":"Section 1 / Abstract"},{"comment":"The training resolution is 256 pixels after random crop, while pretrained models often expect 224; please clarify how the positional/input embeddings of ViT-b32 handle this discrepancy.","section":"Section 3.3"},{"comment":"'mean average precision' should be defined as AP averaged over classes, and it is used later as 'mean precision' in Figure 4; please standardize terminology.","section":"Section 4.1"},{"comment":"The last row of Figure 5 is described as 'appears be a real photo' — grammatical fix and a clearer description of what is shown would help.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The label-validity concern is real and lands on the paper's own Section 4.4. The authors should either demonstrate performance on a cleanly AI-generated subset or substantially reframe the claims as 'inauthentic image detection.' The lack of uncertainty quantification also needs to be addressed in any revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clean, honest hyperparameter sweep showing that finetuned CLIP backbones can hit 81% on Deepfake-Eval-2024, within noise of the 82% commercial detector. The result is real as a number, but the headline overstates it: the benchmark's ground truth is a mix of AI-generated images, conventional edits, and images labeled by source trustworthiness, as the authors themselves note in Section 4.4. So 81% measures 'inauthentic/suspicious' detection on this dataset, not necessarily deepfake detection.\n\nWhat's new: not a new method or benchmark, but a careful tuning of Ojha et al.'s pretrained-feature classifier, with full finetuning, low learning rate, cosine annealing, and CLIP pretraining. They also report compute and speed tradeoffs across three architectures, plus GradCAM analysis. That's useful practical information for people deploying open-source detectors.\n\nWhat it does well: it's transparent. The paper includes learning curves, validation numbers, error analysis, and explicitly flags the label problem. It doesn't hide that frozen-feature probes underperform finetuning, which is part of the 63-to-81 jump. Code is public. That's solid.\n\nSoft spots, in order of severity:\n\n1. Label validity. Section 4.4 says 'Not all fake images were AI-generated' and that TrueMedia reviewers labeled by source trustworthiness or apparent editing. That undermines the 'deepfake' construct. The stress-test note is right that this is load-bearing, not peripheral. The authors acknowledge it but still frame the conclusion as competing with commercial deepfake detectors. The abstract should have been hedged to 'inauthentic images on this benchmark.'\n\n2. Statistical fragility. One split, 789 test images, no seeds or confidence intervals. 81% vs 82% is within binomial noise. The 18-point gain over the 63% baseline is likely real, but it's not a controlled rerun of the exact same training recipe, so part of the gain comes from different training choices (full finetuning vs frozen, etc.).\n\n3. Contribution size. It's a hyperparameter scan. That's fine as a practical note, but the conclusion should not oversell it as 'setting a new standard.'\n\nWho this is for: people working on deepfake detection benchmarks or open-source detection deployment. It doesn't change the theoretical landscape, but it's a solid empirical data point with honest reporting.\n\nRecommendation: send it to peer review. It deserves a serious referee, not a desk reject. The label-validity discussion is genuinely useful. But the authors should be asked to add confidence intervals or seed variance, clarify the construct measured, and temper the abstract claims. Given the paper's own Section 4.4, the central argument is conditional, not broken.","headline":"Useful tuning study on a new in-the-wild benchmark, but the 81% headline is fragile because the benchmark labels mix AI-generated, edited, and source-trust-judged images—and the paper itself admits that.","tokens_in":9253,"tokens_out":2548,"would_cite":true,"duration_ms":24139,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Finetuning a standard pretrained vision model—a single linear head on a CLIP backbone, trained at a low learning rate with cosine annealing—lifts Deepfake-Eval-2024 accuracy from 63% to 81%, within one point of the leading commercial detect","keywords":["deepfake detection","in-the-wild benchmark","Deepfake-Eval-2024","CLIP pretraining","cosine annealing","fine-tuning","ViT-b32","ConvNeXt-base"],"falsifier":"Run the same finetuning recipe on a freshly drawn 60/40 split of Deepfake-Eval-2024, or on an independent batch of 2024 social-media images with labels assigned by forensic review rather than source trustworthiness. If accuracy falls back toward the original 63%, or the 81%-versus-82% ordering flips across splits, the central claim fails.","tokens_in":8347,"feed_emoji":"🕵️","tokens_out":6169,"duration_ms":51296,"temperature":0.7,"pith_summary":"The paper tries to show that the weak open-source baseline on the Deepfake-Eval-2024 benchmark was not inherently weak—it was undertuned. Taking the same simple recipe of a standard pretrained vision model with a dropout layer and one linear classifier, and finetuning the whole network at a low learning rate with cosine annealing, pushes test accuracy from the previously reported 63% to 81%. That result lands within one point of the leading commercial detector (82%) and beats the previous best open-source model (69%) by 12 points. The authors also show CLIP pretraining matters more than ImageNet pretraining, and that a ViT-b32 model delivers near-top accuracy at a fraction of the compute of ConvNeXt-base. A reader should care because it suggests open-source, inspectable detectors can compete with proprietary ones on realistic social-media data.","feed_headline":"Simple finetune lifts deepfake detection from 63% to 81%","feed_subtitle":"A low learning rate and cosine annealing put an open-source detector within one point of the top commercial model.","key_machinery":"The mechanism is a two-part adaptation of standard pretrained vision models: a lightweight classification head (dropout plus a single linear layer) on a CLIP-pretrained backbone, with the entire network unfrozen and finetuned at a low initial learning rate (1e-5) under cosine-annealing warm restarts. The low learning rate and annealing schedule let the pretrained features shift toward deepfake-specific cues rather than being overwritten or stuck in sharp minima; unfreezing the backbone is what enables both the accuracy gain and the GradCAM interpretability. CLIP pretraining on a large 2021 web-scale image-text corpus supplies features better matched to social-media imagery than ImageNet feat","core_discovery":"The central claim is that the 18-point gap between the simple pretrained-features baseline and commercial detectors on Deepfake-Eval-2024 was an artifact of poor hyperparameter choice, not a ceiling of the method. Finetuning all weights of CLIP-pretrained ViT-b32 and ConvNeXt-base models with an initial learning rate of 1e-5 and cosine annealing with warm restarts yields 81% accuracy on the benchmark's fixed 40% test split, close to the 82% reported for the best commercial model. The same tuning raises the method's own prior result from 63% and outperforms all previously reported open-source models. The paper further claims that this gain comes with practical tradeoffs: ResNet-50 is smaller","pith_inferences":["Because Deepfake-Eval-2024's labels partly reflect source trustworthiness and conventional editing, the 81% figure may be measuring 'inauthentic or untrustworthy image' rather than strictly AI-generated deepfake; retesting on a purely generative corpus would separate those constructs.","The single 60/40 split with 789 test images means the 81%-versus-82% gap is within plausible sampling error; the robust conclusion is parity, not superiority, and a multi-split or bootstrapped evaluation would sharpen it.","If the parity holds, deployment incentives shift: organizations could run transparent local detectors instead of paid proprietary APIs, and detection benchmarks could emphasize calibration and uncertainty rather than a single accuracy number.","The optimization finding suggests other 'simple baselines' on in-the-wild benchmarks may be similarly undertuned; low-rate cosine-annealing finetuning is a cheap check before declaring a method saturated."],"forward_implications":["On Deepfake-Eval-2024, open-source detectors are competitive with the leading commercial detector: 81% vs 82%, with the previous open-source best at 69%.","Hyperparameter choices—initial learning rate, scheduler, and whether the backbone is frozen—are decisive for this baseline; cosine annealing at 1e-5 is worth roughly 18 points over the original report.","CLIP-pretrained backbones generalize better than ImageNet-pretrained or randomly initialized ones on this social-media benchmark, so detector performance depends heavily on the pretraining distribution's recency and scale.","Deployment tradeoffs separate the models: ViT-b32 gives near-best accuracy with about 8.8 GFLOPs and 16 ms inference, while ConvNeXt-base gives the best operating-characteristic curve but costs about 30.7 GFLOPs; ResNet-50 remains an option for memory-limited settings at 79%.","A ConvNeXt-base detector can flag more than 30% of fakes with zero false positives, making it usable as a precision-first prefilter for human review."],"supporting_citations":[{"why":"Supplies the in-the-wild benchmark, the 60/40 split, and the 63%/69%/82% reference accuracies the paper beats or matches.","marker":"[5]"},{"why":"Introduces the simple classifier-on-pretrained-features baseline that the paper revisits with better hyperparameters.","marker":"[17]"},{"why":"Provides the ViT-b32 architecture that reaches 81% accuracy.","marker":"[8]"},{"why":"Provides the ConvNeXt-base architecture that ties for the best accuracy.","marker":"[14]"},{"why":"Provides the ResNet-50 architecture used as the smallest comparison model.","marker":"[11]"},{"why":"Gives the parameter counts, GFLOPs, and inference-speed numbers used to compare deployment tradeoffs.","marker":"[10]"},{"why":"Documents the CLIP contrastive pretraining on LAION-400M that the top models are initialized from.","marker":"[6]"}],"fun_headline_variants":["Hyperparameter fix boosts open-source deepfake detector to 81%","Better tuning closes gap with commercial deepfake detectors","Finetuning trick lifts deepfake detection baseline to 81%","Simple hyperparameters put open-source detector on par with commercial","In-the-wild deepfake detection: simple tuning yields 81% accuracy"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The benchmark's ground-truth labels reliably measure what we mean by deepfake, and the fixed 60/40 split's 789 test images give a stable estimate of accuracy; if labels capture source trustworthiness or any editing rather than AI generation, or if the split is noisy, the 81% claim and the one-point gap to the commercial detector weaken.","fun_headline_variants_meta":{"raw":{"variants":["Hyperparameter fix boosts open-source deepfake detector to 81%","Better tuning closes gap with commercial deepfake detectors","Finetuning trick lifts deepfake detection baseline to 81%","Simple hyperparameters put open-source detector on par with commercial","In-the-wild deepfake detection: simple tuning yields 81% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":2803,"prompt_tokens":756,"completion_tokens":2047,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":1974}},"tokens_in":500,"tokens_out":2047,"duration_ms":13866,"temperature":1.0,"reasoning_tokens":1974,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:20:23.014520+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same finetuning recipe on a freshly drawn 60/40 split of Deepfake-Eval-2024, or on an independent batch of 2024 social-media images with labels assigned by forensic review rather than source trustworthiness. If accuracy falls back toward the original 63%, or the 81%-versus-82% ordering flips across splits, the central claim fails.","supporting_citations":[{"cited_title":"Towards Uni- versal Fake Image Detectors that Generalize Across Gen- erative Models","cited_arxiv_id":null,"evidence_quote":"Introduces the simple classifier-on-pretrained-features baseline that the paper revisits with better hyperparameters."},{"cited_title":"OpenCLIP Model Profiles","cited_arxiv_id":null,"evidence_quote":"Gives the parameter counts, GFLOPs, and inference-speed numbers used to compare deployment tradeoffs."},{"cited_title":"Reproducible scal- ing laws for contrastive language-image learning","cited_arxiv_id":null,"evidence_quote":"Documents the CLIP contrastive pretraining on LAION-400M that the top models are initialized from."}],"review_version":1}