REVIEW 3 major objections 4 minor 27 references
Finetuning a standard pretrained vision model—a single linear head on a CLIP backbone, trained at a low learning rate with cosine annealing—lifts Deepfake-Eval-2024 accuracy from 63% to 81%, within one point of the leading commercial detect
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Finetuned CLIP-pretrained ConvNeXt-base and ViT-b32 classifiers reach 81% accuracy on Deepfake-Eval-2024, within noise of the leading commercial detector's 82%.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection Useful tuning study on a new in-the-wild benchmark, but the 81% headline is fragile because the benchmark labels mix AI-generated, edited, and source-trust-judged images—and the paper itself admits that. the 3 major comments →
Revisiting Simple Baselines for In-The-Wild Deepfake Detection
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that the 18-point gap between the simple pretrained-features baseline and commercial detectors on Deepfake-Eval-2024 was an artifact of poor hyperparameter choice, not a ceiling of the method. Finetuning all weights of CLIP-pretrained ViT-b32 and ConvNeXt-base models with an initial learning rate of 1e-5 and cosine annealing with warm restarts yields 81% accuracy on the benchmark's fixed 40% test split, close to the 82% reported for the best commercial model. The same tuning raises the method's own prior result from 63% and outperforms all previously reported open-source models. The paper further claims that this gain comes with practical tradeoffs: ResNet-50 is smaller
What carries the argument
The mechanism is a two-part adaptation of standard pretrained vision models: a lightweight classification head (dropout plus a single linear layer) on a CLIP-pretrained backbone, with the entire network unfrozen and finetuned at a low initial learning rate (1e-5) under cosine-annealing warm restarts. The low learning rate and annealing schedule let the pretrained features shift toward deepfake-specific cues rather than being overwritten or stuck in sharp minima; unfreezing the backbone is what enables both the accuracy gain and the GradCAM interpretability. CLIP pretraining on a large 2021 web-scale image-text corpus supplies features better matched to social-media imagery than ImageNet feat
Load-bearing premise
The benchmark's ground-truth labels reliably measure what we mean by deepfake, and the fixed 60/40 split's 789 test images give a stable estimate of accuracy; if labels capture source trustworthiness or any editing rather than AI generation, or if the split is noisy, the 81% claim and the one-point gap to the commercial detector weaken.
What would settle it
Run the same finetuning recipe on a freshly drawn 60/40 split of Deepfake-Eval-2024, or on an independent batch of 2024 social-media images with labels assigned by forensic review rather than source trustworthiness. If accuracy falls back toward the original 63%, or the 81%-versus-82% ordering flips across splits, the central claim fails.
If this is right
- On Deepfake-Eval-2024, open-source detectors are competitive with the leading commercial detector: 81% vs 82%, with the previous open-source best at 69%.
- Hyperparameter choices—initial learning rate, scheduler, and whether the backbone is frozen—are decisive for this baseline; cosine annealing at 1e-5 is worth roughly 18 points over the original report.
- CLIP-pretrained backbones generalize better than ImageNet-pretrained or randomly initialized ones on this social-media benchmark, so detector performance depends heavily on the pretraining distribution's recency and scale.
- Deployment tradeoffs separate the models: ViT-b32 gives near-best accuracy with about 8.8 GFLOPs and 16 ms inference, while ConvNeXt-base gives the best operating-characteristic curve but costs about 30.7 GFLOPs; ResNet-50 remains an option for memory-limited settings at 79%.
- A ConvNeXt-base detector can flag more than 30% of fakes with zero false positives, making it usable as a precision-first prefilter for human review.
Where Pith is reading between the lines
- Because Deepfake-Eval-2024's labels partly reflect source trustworthiness and conventional editing, the 81% figure may be measuring 'inauthentic or untrustworthy image' rather than strictly AI-generated deepfake; retesting on a purely generative corpus would separate those constructs.
- The single 60/40 split with 789 test images means the 81%-versus-82% gap is within plausible sampling error; the robust conclusion is parity, not superiority, and a multi-split or bootstrapped evaluation would sharpen it.
- If the parity holds, deployment incentives shift: organizations could run transparent local detectors instead of paid proprietary APIs, and detection benchmarks could emphasize calibration and uncertainty rather than a single accuracy number.
- The optimization finding suggests other 'simple baselines' on in-the-wild benchmarks may be similarly undertuned; low-rate cosine-annealing finetuning is a cheap check before declaring a method saturated.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper revisits the Ojha et al. baseline for deepfake detection on the Deepfake-Eval-2024 benchmark. Using standard pretrained backbones (ResNet-50, ViT-b32, ConvNeXt-base) with a final dropout + linear layer, the authors tune hyperparameters such as learning rate, scheduler, pretraining source, and finetuning vs. freezing. They report that a CLIP-pretrained ConvNeXt-base or ViT-b32, finetuned with a low learning rate and cosine annealing, reaches 81% test accuracy on the benchmark's single 60/40 split, compared to the 63% originally reported for the same baseline and the 82% of the leading commercial detector. They also report computational costs, ROC AUC, and GradCAM visualizations, and release code.
Significance. If correct, this is a practically useful demonstration that simple open-source detectors can be competitive with proprietary systems on an in-the-wild benchmark, and it highlights the sensitivity of such baselines to hyperparameter choices. The paper's strengths include a clear hyperparameter ablation, evaluation on an external benchmark (not circular), a focus on compute/interpretability tradeoffs, and released code. However, the central comparative claim rests on small accuracy differences on a single test split and on benchmark labels that the paper itself acknowledges are not purely AI-generated; as presented, the evidence does not firmly establish that open-source detectors 'compete' with the commercial detector on the construct of deepfake detection.
major comments (3)
- [Section 4.3 / Table 2] The headline comparison (81% vs. 82%, 79% vs. 81%) is reported without confidence intervals, bootstrap estimates, or multiple random seeds. With n=789 test images, the standard error of accuracy is roughly sqrt(0.81*0.19/789) ≈ 1.4 percentage points, so a 1–2 point gap is within sampling noise. The claim that these open-source models 'closely follow' or 'compete with' the commercial detector is not statistically supported. Please report CIs, bootstrap intervals, or at least results over several seeds and random splits.
- [Section 4.4] The paper states: 'Not all fake images were AI-generated' and that TrueMedia.org reviewers 'labeled the ground truth of these image based on the trustworthiness of the original source or if the photo appears to be edited in any manner.' This means the 81% accuracy is for a 'fake/inauthentic' class that mixes AI generation, conventional editing, and source-reputation judgments, not purely deepfakes. The central claim about deepfake-detection competitiveness is therefore ambiguous. Please re-analyze results on a subset of images that are clearly AI-generated (or otherwise filter by label rationale), and temper the conclusions accordingly.
- [Section 3.1 / Table 1] Hyperparameters were selected using only 116 validation images drawn from the same single split, and the final test result is from that same split. This creates a risk of selection overfitting and does not measure generalization to other splits. Additionally, the paper compares its 81% to the 63% reported in [5] without rerunning the original baseline under the same protocol (augmentation, resolution, epochs, optimizer). To support an '18% improvement' claim, please include a controlled comparison where the Ojha-style baseline is retrained with the same preprocessing and training setup (except the hyperparameters under study), and ideally evaluate across multiple splits.
minor comments (4)
- [Section 1 / Abstract] Typo: 'commerial' should be 'commercial.' Also, 'signficant' in Section 4.2 and 'to deployed' in Section 4.3 need correction.
- [Section 3.3] The training resolution is 256 pixels after random crop, while pretrained models often expect 224; please clarify how the positional/input embeddings of ViT-b32 handle this discrepancy.
- [Section 4.1] 'mean average precision' should be defined as AP averaged over classes, and it is used later as 'mean precision' in Figure 4; please standardize terminology.
- [Section 4.4] The last row of Figure 5 is described as 'appears be a real photo' — grammatical fix and a clearer description of what is shown would help.
Circularity Check
No significant circularity: the 81% test accuracy is an empirical result on a fixed external benchmark split, not a fitted value, a renamed known result, or a self-citation chain.
full rationale
The paper's central claim is empirical, not definitional. Section 3.1 states: 'we adhere to the same finetuning split prescribed by Deepfake-Eval-2024 which designates 60% (1161 images) for training and 40% (789 images) for testing,' and additionally: 'From the training data, we further reserve 10% (116 images) of training samples for validation during our hyperparameter search.' Hyperparameters are selected on the validation subset, and the reported 81% is measured on the held-out test split. No test accuracy is used as a training signal; no fitted parameter is renamed as a prediction. The comparison to the previously reported 63% baseline is a comparison to an externally published number obtained under a different training protocol, which is a measurement-comparability/fairness concern, not circularity. The model family (simple linear classifier over pretrained features) is attributed to Ojha et al. [17], and the benchmark to Chandra et al. [5]; no load-bearing self-citation occurs in the method or evaluation chain. Section 4.4 explicitly flags the benchmark's label-validity limits ('Not all fake images were AI-generated', 'TrueMedia.org reviewers often labeled the ground truth of these image based on the trustworthiness of the original source or if the photo appears to be edited in any manner'), which is an external-validity caveat, not a circular step. The 81% claim therefore does not reduce by construction to the paper's inputs; it is a genuine held-out evaluation result.
Axiom & Free-Parameter Ledger
free parameters (7)
- initial learning rate =
1e-5 (best for CLIP ViT-b32 and ConvNeXt-base), from grid {1e-3, 1e-4, 1e-5}
- learning rate scheduler =
cosine annealing (vs step decay)
- backbone pretraining source =
CLIP on LAION-400M (vs ImageNet, random init)
- backbone finetuning vs frozen =
finetune all weights
- dropout rate and L2 penalty =
moderate levels, exact values not reported
- early stopping patience =
5 epochs
- random crop scale and resolution =
50-100% crop, resize to 256
axioms (3)
- domain assumption Deepfake-Eval-2024 ground-truth labels are valid and correct
- domain assumption The single 60/40 split gives a stable accuracy estimate
- domain assumption Third-party pretrained checkpoints (ImageNet, CLIP/OpenCLIP) are valid and their training exposure is as described
Cite this review
Pith. "Pith review of Revisiting Simple Baselines for In-The-Wild Deepfake Detection." pith.science (2026). https://pith.science/paper/CSC267TB
@misc{pith2026250904150,
author = {Pith},
title = {Pith review of: Revisiting Simple Baselines for In-The-Wild Deepfake Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/CSC267TB}},
note = {Machine review of arXiv:2509.04150}
}
read the original abstract
The widespread adoption of synthetic media demands accessible deepfake detectors and realistic benchmarks. While most existing research evaluates deepfake detectors on highly controlled datasets, we focus on the recently released "in-the-wild" benchmark, Deepfake-Eval-2024. Initial reporting on Deepfake-Eval-2024 showed that three finetuned open-source models achieve accuracies between 61% and 69%, significantly lagging behind the leading commercial deepfake detector with 82% accuracy. Our work revisits one of these baseline approaches, originally introduced by Ojha et al., which adapts standard pretrained vision backbones to produce generalizable deepfake detectors. We demonstrate that with better-tuned hyperparameters, this simple approach actually yields much higher performance -- 81% accuracy on Deepfake-Eval-2024 -- surpassing the previously reported accuracy of this baseline approach by 18% and competing with commercial deepfake detectors. We discuss tradeoffs in accuracy, computational costs, and interpretability, focusing on how practical these deepfake detectors might be when deployed in real-world settings. Our code can be found at https://github.com/Deepfake-Detection-KKO/deepfake-detection.
Figures
Reference graph
Works this paper leans on
-
[1]
Fakhar Abbas and Araz Taeihagh. Unmasking deepfakes: A systematic review of deepfake detection and generation techniques using artificial intelligence. Expert Systems with Applications, 252:124260, 2024. 2
work page 2024
-
[2]
MesoNet: a Compact Facial Video Forgery Detec- tion Network
Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. MesoNet: a Compact Facial Video Forgery Detec- tion Network. pages 1–7, Dec. 2018. arXiv:1809.00888 [cs]. 2, 7
Pith/arXiv arXiv 2018
-
[3]
Arnesh Batra, Anushk Kumar, Jashn Khemani, Arush Gum- ber, Arhan Jain, and Somil Gupta. Socialdf: Benchmark dataset and detection model for mitigating harmful deepfake content on social media platforms, 2025. 2
work page 2025
-
[4]
Real-time deepfake detection in the real-world, 2024
Bar Cavia, Eliahu Horwitz, Tal Reiss, and Yedid Hoshen. Real-time deepfake detection in the real-world, 2024. 2
work page 2024
-
[5]
Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circu- lated in 2024, May 2025
Nuria Alina Chandra, Ryan Murtfeldt, Lin Qiu, Arnab Kar- makar, Hannah Lee, Emmanuel Tanumihardja, Kevin Farhat, Ben Caffee, Sejin Paik, Changyeon Lee, Jongwook Choi, Aerin Kim, and Oren Etzioni. Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circu- lated in 2024, May 2025. arXiv:2503.02857 [cs]. 1, 2, 3, 5, 6
Pith/arXiv arXiv 2024
-
[6]
Reproducible scal- ing laws for contrastive language-image learning
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page 2818–2829. IEEE, June 2023. 4
work page 2023
-
[7]
Le, Jiwon Kim, Simon Woo, Shahroz Tariq, Alsharif Abuadbba, and Kristen Moore
Beomsang Cho, Binh M. Le, Jiwon Kim, Simon Woo, Shahroz Tariq, Alsharif Abuadbba, and Kristen Moore. To- wards understanding of deepfake videos in the wild. In Pro- ceedings of the 32nd ACM International Conference on In- formation and Knowledge Management , CIKM ’23, page 4530–4537, New York, NY , USA, 2023. Association for Computing Machinery. 2
work page 2023
-
[8]
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021. arXiv:2010.11929 [cs]. 2
Pith/arXiv arXiv 2021
-
[9]
Unmasking deepfakes with simple features,
Ricard Durall, Margret Keuper, Franz-Josef Pfreundt, and Janis Keuper. Unmasking deepfakes with simple features,
-
[10]
ML Foundations. OpenCLIP Model Profiles. https://github.com/mlfoundations/open clip/blob/main/ docs/model profile.csv, 2023. [Accessed 25-07-2025]. 4, 5, 6
work page 2023
-
[11]
Deep Residual Learning for Image Recognition, Dec
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, Dec. 2015. arXiv:1512.03385 [cs]. 2
Pith/arXiv arXiv 2015
-
[12]
Deepfake detection using deep learning meth- ods: A systematic and comprehensive review
Arash Heidari, Nima Jafari Navimipour, Hasan Dag, and Mehmet Unal. Deepfake detection using deep learning meth- ods: A systematic and comprehensive review. WIREs Data Mining and Knowledge Discovery, 14(2):e1520, 2024. 2
work page 2024
-
[13]
Asvspoof 2021: Towards spoofed and deepfake speech detection in the wild
Xuechen Liu, Xin Wang, Md Sahidullah, Jose Patino, H´ector Delgado, Tomi Kinnunen, Massimiliano Todisco, Junichi Yamagishi, Nicholas Evans, Andreas Nautsch, and Kong Aik Lee. Asvspoof 2021: Towards spoofed and deepfake speech detection in the wild. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2507–2522, 2023. 2
work page 2021
-
[14]
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s, Mar. 2022. arXiv:2201.03545 [cs]. 2
Pith/arXiv arXiv 2022
-
[15]
The creation and detection of deepfakes: A survey
Yisroel Mirsky and Wenke Lee. The creation and detection of deepfakes: A survey. ACM Comput. Surv. , 54(1), Jan
-
[16]
A survey on the detection and impacts of deepfakes in visual, audio, and tex- tual formats
Rami Mubarak, Tariq Alsboui, Omar Alshaikh, Isa Inuwa- Dutse, Saad Khan, and Simon Parkinson. A survey on the detection and impacts of deepfakes in visual, audio, and tex- tual formats. IEEE Access, 11:144497–144529, 2023. 2
work page 2023
-
[17]
Towards Uni- versal Fake Image Detectors that Generalize Across Gen- erative Models
Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards Uni- versal Fake Image Detectors that Generalize Across Gen- erative Models. In 2023 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 24480– 24489, Vancouver, BC, Canada, June 2023. IEEE. 1, 2
work page 2023
-
[18]
Evaluating deep- fake detectors in the wild, 2025
Viacheslav Pirogov and Maksim Artemev. Evaluating deep- fake detectors in the wild, 2025. 2
work page 2025
-
[19]
Deepfake videos in the wild: Analysis and detection
Jiameng Pu, Neal Mangaokar, Lauren Kelly, Parantapa Bhat- tacharya, Kavya Sundaram, Mobin Javed, Bolun Wang, and Bimal Viswanath. Deepfake videos in the wild: Analysis and detection. In Proceedings of the Web Conference 2021, WWW ’21, page 981–992, New York, NY , USA, 2021. As- sociation for Computing Machinery. 2
work page 2021
-
[20]
Do Vision Trans- formers See Like Convolutional Neural Networks?, Mar
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do Vision Trans- formers See Like Convolutional Neural Networks?, Mar
-
[21]
Md Shohel Rana, Mohammad Nur Nobi, Beddhu Murali, and Andrew H. Sung. Deepfake Detection: A Systematic Literature Review. IEEE Access, 10:25494–25513, 2022. 2
work page 2022
-
[22]
Con- tinuous fake media detection: Adapting deepfake detectors to new generative techniques
Francesco Tassone, Luca Maiano, and Irene Amerini. Con- tinuous fake media detection: Adapting deepfake detectors to new generative techniques. Computer Vision and Image Understanding, 249:104143, 2024. 2
work page 2024
-
[23]
Deepfake Video De- tection Using Convolutional Vision Transformer, Mar
Deressa Wodajo and Solomon Atnafu. Deepfake Video De- tection Using Convolutional Vision Transformer, Mar. 2021. arXiv:2102.11126 [cs]. 7
Pith/arXiv arXiv 2021
-
[24]
T. Zhang. Deepfake generation and detection, a survey. Multimedia Tools and Applications , 81:6259–6276, Febru- ary 2022. 2
work page 2022
-
[25]
Tianfei Zhou, Wenguan Wang, Zhiyuan Liang, and Jianbing Shen. Face forensics in the wild. CoRR, abs/2103.16076,
work page internal anchor Pith review Pith/arXiv arXiv
-
[26]
Wilddeepfake: A challenging real-world dataset for deepfake detection
Bojia Zi, Minghao Chang, Jingjing Chen, Xingjun Ma, and Yu-Gang Jiang. Wilddeepfake: A challenging real-world dataset for deepfake detection. In Proceedings of the 28th ACM International Conference on Multimedia , MM ’20, page 2382–2390, New York, NY , USA, 2020. Association for Computing Machinery. 2
work page 2020
-
[2022]
arXiv:2108.08810 [cs]. 5, 7
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.