REVIEW 2 major objections 5 minor 27 references
Exploring Active Data Selection Strategies for Continuous Training in Deepfake Detection
T0 review · 2 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper shows that repeatedly fine-tuning a deepfake detector on the small subset of new images it is most unsure about cuts test error from 22.5% to about 2.5% equal error rate, using only 15% of the available pool.
desk verdict A plausible proof-of-concept with a large headline improvement, but the key active-vs-random comparison rests on a single run without error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the negative energy-based confidence score $c_m = -T \log \sum_{j=1}^{J} \exp(l_{m,j}/T)$, a negative log-sum-exp over the detector's final-layer logits with temperature $T=1$; lower values mean the model is less certain. Algorithm 1 sorts the pool by this score, takes the $L$ lowest-scoring samples, removes them from the pool, adds them to the training set, and fine-tunes the model on the combined data, repeating for $K$ iterations. The score doubles as an out-of-distribution detector, which is why it tends to pick samples from generators the model has not seen.
What would settle it
A concrete falsifying experiment would compare the ranking of pool samples by negative energy against the ranking by how much adding each sample individually improves validation EER; if the two rankings do not correlate positively, the confidence score is not actually selecting the most useful data. A simpler check would run the same continuous-training protocol with many random-seed pool orderings and see whether random selection ever matches the active method's final EER.
Extended reading notes
Core claim
The central claim is that a model's own confidence score, specifically the negative energy score computed from its output logits, can serve as the selector for what to add during continuous training. The authors show that low-confidence samples from a redundant pool are the ones whose addition improves detection most: after repeated rounds of adding 10,000 selected images per round, the EER fell from 22.5% to just under 2.5%, and the selected 15% of the pool outperformed a randomly chosen 15% once the easy samples were exhausted. The authors emphasize that the update is not domain adaptation, because the model is fine-tuned on the union of the original and selected data, preserving previously learned detection while absorbing new methods.
Load-bearing premise
The loop only works as written if every pool image, especially each one selected, has a known real/fake label, and if low confidence really flags the samples most worth training on; remove either assumption and the algorithm's selection signal or its training step breaks.
Editorial extensions
If this is right
- A detector that starts at 22.5% EER on new spoofing methods can reach about 2.5% EER after continuous training on only 15% of a redundant pool.
- Negative-energy selection beats random selection in later iterations, once the first easy-to-learn batches have been consumed.
- Because only a fraction of the pool is added, per-epoch training stays far cheaper than retraining on all pool data: 1303 seconds per epoch versus 2798 seconds per epoch in the paper's setup.
- The same loop can be triggered again whenever a new generator becomes prevalent, keeping the detector current without rebuilding the master set.
Reading between the lines
- The selection order the authors observe, with newer generators like Stable Diffusion chosen first, older methods like FF++ later, and KoDF mostly skipped, suggests the negative energy score is effectively measuring novelty; a deployed system could use that ordering to decide which new generators need data collection first.
- In a deployed setting where labels are not guaranteed, the same confidence signal could route low-scoring images to human labelers or drive a semi-supervised pseudo-labeling variant, neither of which the paper tests.
- Because random selection wins the first two rounds, a hybrid strategy that starts with random warm-up and then switches to active selection may converge faster than either alone; this is a testable variant, not a paper claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an active data selection method for continuously updating a deepfake detector. Starting from a detector trained on ForgeryNet, a pool of real and fake face images from FF++, Google DFD, YouTube DF, KoDF, and Stable Diffusion is scored with a negative-energy confidence measure; the L lowest-confidence samples are added to the training set, the model is fine-tuned, and the process is repeated. Experiments report that the base model has 22.5% EER on the combined test set, while the actively updated model reaches just under 2.5% EER after adding 15% of the pool, and the paper states that active selection outperforms random selection in later iterations.
Significance. If the active-versus-random advantage is reproducible, the paper offers a practical, low-cost way to prioritize data for regular model updates in deepfake detection. The magnitude of the improvement over the base model is substantial, the algorithm is clearly specified, and all datasets are public, which aids reproducibility. The self-referential nature of using the model's own confidence for selection is not a circularity problem because the method is compared against random selection and against the base model. However, the key comparative claim currently rests on a single run with no uncertainty quantification, so the significance is conditional on additional multi-seed experiments.
major comments (2)
- [Section 3.4, Fig. 2] The central comparative claim that AL negE outperforms AL random is supported by a single run. No random seeds, repeated runs, error bars, or significance tests are reported, and the paper itself states that random selection was better in the first two iterations. Since each iteration involves stochastic fine-tuning on different selected batches, the later crossover could be within run-to-run variance. Please report mean and standard deviation over multiple seeds, together with a paired significance test or bootstrap confidence intervals at the reported 15% selection point and, ideally, for each iteration.
- [Section 2.1, Algorithm 1] The method requires labels y_n for every pool sample, including the selected ones, in order to perform supervised fine-tuning at line 9. This is explicitly acknowledged as reasonable when the pool data come from public datasets or are generated via APIs, but the abstract and introduction frame the selection as 'automatic' and motivated by new deepfake methods in the wild. Please state the label-availability assumption and its scope in the abstract or introduction, or provide an experiment or extension for unlabeled pool data; otherwise the applicability of the method is narrower than the framing suggests.
minor comments (5)
- [Fig. 2] The caption and the surrounding text state that the horizontal axis encodes the iteration index, the number of selected samples, and its percentage, but the figure itself appears to show only one axis; make the mapping explicit and readable.
- [Section 3.1, Table 1] For video-based pools such as YouTube DF and KoDF, the face extraction procedure must include a frame-sampling step; please specify the frame selection strategy and the number of frames per video.
- [Section 3.4, footnote 4] The reported training-time comparison (2798 vs. 1303 sec/epoch) lacks hardware and batch-size details; state these conditions or remove the cost claim.
- [Section 2.2] The sentence 'This method is compared with a random selection from a pool set.3' has a misplaced footnote marker; move the marker to the end of the sentence or the paragraph.
- [Algorithm 1] The notation 'argmin-sort_m({..., c_m, ...})[0:L]' is nonstandard; define it as returning the indices of the L smallest confidence scores.
Circularity Check
No significant circularity: the reported EER improvements are measured outcomes of an implemented active-learning loop, not predictions derived from fitted constants or self-citations.
full rationale
The paper's central claim, that continuous training with actively selected pool data reduces EER from 22.5% (Base) to about 2.5% (AL negE), is an empirical result reported in Section 3.4 and Figure 2. No equation in the paper derives these EER values from the selection criterion; the negative-energy score in Eq. (1) is only used to rank pool samples in Algorithm 1, and the EER is then measured on an independent test set. The comparison against AL random and against the untrained Base model provides an external baseline, so the selection rule is not equivalent to the evaluation metric by construction. The only self-referential aspect is that the model uses its own confidence scores to choose training data, but this is the method being tested rather than a fitted parameter renamed as a prediction. The self-citation [WY23] in Section 2.2 is used only as background support for choosing negative-energy scoring; the paper's active-vs-random comparison and the Base-model improvement do not depend on that citation, so it is not load-bearing. The paper itself acknowledges limitations: labels in the pool are assumed known (Section 2.1), random selection was better in the first two iterations (Section 3.4), and no error bars or significance tests are reported. These are robustness and statistical-evidence concerns, not circularity, because they do not amount to a reduction of the claimed result to its own inputs.
Assumptions & free parameters
free parameters (4)
- L (number of selected samples per iteration) =
10000
- K (number of continuous-training iterations) =
6 (implied)
- Fine-tuning epochs per iteration =
3
- Softmax temperature T =
1
assumptions (4)
- domain assumption Labels of all pool data, including selected samples, are known and available for supervised fine-tuning.
- domain assumption The pool set is pre-balanced to 40,000 images per source dataset.
- domain assumption Test sets are drawn from the same source datasets as the pool set.
- domain assumption The negative energy score with T=1 is a valid confidence metric for selecting useful training data.
Cite this review
Pith. "Pith review of Exploring Active Data Selection Strategies for Continuous Training in Deepfake Detection." pith.science (2026). https://pith.science/paper/7EP2J64K
@misc{pith2026250207269,
author = {Pith},
title = {Pith review of: Exploring Active Data Selection Strategies for Continuous Training in Deepfake Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/7EP2J64K}},
note = {Machine review of arXiv:2502.07269}
}
read the original abstract
In deepfake detection, it is essential to maintain high performance by adjusting the parameters of the detector as new deepfake methods emerge. In this paper, we propose a method to automatically and actively select the small amount of additional data required for the continuous training of deepfake detection models in situations where deepfake detection models are regularly updated. The proposed method automatically selects new training data from a \textit{redundant} pool set containing a large number of images generated by new deepfake methods and real images, using the confidence score of the deepfake detection model as a metric. Experimental results show that the deepfake detection model, continuously trained with a small amount of additional data automatically selected and added to the original training set, significantly and efficiently improved the detection performance, achieving an EER of 2.5% with only 15% of the amount of data in the pool set.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" initialize.prev.this.status FUNCTION begin.bib " write newline preamble empty 'skip preamble write newline if " thebibliography " longest.label * " " * write newline " [1] #1 " write newline " url@samestyle " write newline " " write newline " [2] #2 " write newline " =0pt " write newline " " ALTinterwordstretchfactor * " " * write newli...
-
[2]
4" FUNCTION default.is.dash.repeated.names #1 FUNCTION default.name.format.string
11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...
work page 2007
-
[3]
write newline
" write newline "" initialize.prev.this.status FUNCTION begin.bib " write newline preamble empty 'skip preamble write newline if " thebibliography " longest.label * " " * write newline " [1] #1 " write newline " url@samestyle " write newline " " write newline " [2] #2 " write newline " =0pt " write newline " " ALTinterwordstretchfactor * " " * write newli...
-
[4]
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[5]
Biometrics, ISO/IEC JTC1 SC37: , ISO/IEC 2382-37: 2017 Information Technology-Vocabulary-Part 37: Biometrics, 2017
work page 2017
-
[6]
Dissertation, Universidad aut \'o noma de Madrid, 2007
Castro, Daniel Ramos: Forensic Evaluation of the Evidence Using Automatic Speaker Recognition Systems. Dissertation, Universidad aut \'o noma de Madrid, 2007
work page 2007
- [7]
- [8]
Show all 27 references
-
[9]
In: Proc
Deng, Jia; Dong, Wei; Socher, Richard; Li, Li-Jia; Li, Kai; Fei-Fei, Li: ImageNet : A large-scale hierarchical image database. In: Proc. CVPR. IEEE, S. 248--255, 2009
2009
-
[10]
In: Proc
Deng, Jiankang; Guo, Jia; Ververas, Evangelos; Kotsia, Irene; Zafeiriou, Stefanos: RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild. In: Proc. CVPR. S. 5202--5211, 2020
2020
-
[11]
https://ai.googleblog.com/2019/09/contributing-data-to-deepfake-detection.html, 9 2019
Dufour, Nick; Gully, Andrew: , Contributing Data to Deepfake Detection Research. https://ai.googleblog.com/2019/09/contributing-data-to-deepfake-detection.html, 9 2019
2019
-
[12]
Journal of King Saud University-Computer and Information Sciences, 35(4):90--99, 2023
Dong, Fengkai; Zou, Xiaoqiang; Wang, Jiahui; Liu, Xiyao: Contrastive learning-based general Deepfake detection with multi-scale RGB frequency clues. Journal of King Saud University-Computer and Information Sciences, 35(4):90--99, 2023
2023
-
[13]
In: Proc
He, Yinan; Gan, Bei; Chen, Siyu; Zhou, Yichun; Yin, Guojun; Song, Luchuan; Sheng, Lu; Shao, Jing; Liu, Ziwei: ForgeryNet : A versatile benchmark for comprehensive forgery analysis. In: Proc. CVPR. S. 4360--4369, 2021
2021
-
[14]
In: Proc
Karras, Tero; Laine, Samuli; Aila, Timo: A style-based generator architecture for generative adversarial networks. In: Proc. CVPR. S. 4401--4410, 2019
2019
-
[15]
a \"a , Hannu; Hautam \
Kukanov, Ivan; Karttunen, Janne; Sillanp \"a \"a , Hannu; Hautam \"a ki, Ville: Cost sensitive optimization of deepfake detector. In: Proc. APSIPA ASC. IEEE, S. 1300--1303, 2020
2020
-
[16]
In: Proc
Kwon, Patrick; You, Jaeseong; Nam, Gyuhyeon; Park, Sungwoo; Chae, Gyeongsu: Kodf: A large-scale korean deepfake detection dataset. In: Proc. ICCV. S. 10744--10753, 2021
2021
-
[17]
In: Proc
Larue, Nicolas; Vu, Ngoc-Son; Struc, Vitomir; Peer, Peter; Christophides, Vassilis: SeeABLE: Soft Discrepancies and Bounded Contrastive Learning for Exposing Deepfakes. In: Proc. ICCV. S. 21011--21021, 2023
2023
-
[18]
In: Proc
Liu, Weitang; Wang, Xiaoyun; Owens, John; Li, Yixuan: Energy-based Out-of-distribution Detection . In: Proc. NIPS. Jgg. 33, S. 21464--21475, 2020
2020
-
[19]
In: Proc
Liu, Huan; Tan, Zichang; Tan, Chuangchuang; Wei, Yunchao; Zhao, Yao; Wang, Jingdong: Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection. In: Proc. CVPR. 2023
2023
-
[20]
In: Proc
Ma, Haoxin; Yi, Jiangyan; Tao, Jianhua; Bai, Ye; Tian, Zhengkun; Wang, Chenglong: Continual learning for fake audio detection. In: Proc. INTERSPEECH. 2021
2021
-
[21]
In: Proc
Rossler, Andreas; Cozzolino, Davide; Verdoliva, Luisa; Riess, Christian; Thies, Justus; Nie ner, Matthias: FaceForensics++ : Learning to detect manipulated facial images. In: Proc. ICCV. S. 1--11, 2019
2019
-
[22]
In: Proc
Rombach, Robin; Blattmann, Andreas; Lorenz, Dominik; Esser, Patrick; Ommer, Bj \"o rn: High-resolution image synthesis with latent diffusion models. In: Proc. CVPR. S. 10684--10695, 2022
2022
-
[23]
In: Proc
Song, Wentang; Lin, Yuzhen; Li, Bin: Towards Generic Deepfake Detection with Dynamic Curriculum. In: Proc. ICASSP. IEEE, S. 4500--4504, 2024
2024
-
[24]
In (Meila, Marina; Zhang, Tong, Hrsg.): Proceedings of the 38th International Conference on Machine Learning
Tan, Mingxing; Le, Quoc: EfficientNetV2: Smaller Models and Faster Training. In (Meila, Marina; Zhang, Tong, Hrsg.): Proceedings of the 38th International Conference on Machine Learning. Jgg. 139 in Proc. Machine Learning Research. PMLR, S. 10096--10106, 18--24 Jul 2021
2021
-
[25]
In: Proc
Wang, Xin; Yamagishi, Junichi: Investigating Active-learning-based Training Data Selection for Speech Spoofing Countermeasure . In: Proc. SLT . S. 585--592, 2023
2023
-
[26]
In: Proc
Yan, Zhiyuan; Zhang, Yong; Yuan, Xinhang; Lyu, Siwei; Wu, Baoyuan: DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection. In: Proc. NeurIPS Datasets and Benchmarks Track. 2023
2023
-
[27]
Advances in Neural Information Processing Systems, 36, 2024
Yan, Zhiyuan; Zhang, Yong; Yuan, Xinhang; Lyu, Siwei; Wu, Baoyuan: DeepfakeBench : A Comprehensive Benchmark of Deepfake Detection. Advances in Neural Information Processing Systems, 36, 2024
2024
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.