Pith. sign in

REVIEW 2 major objections 5 minor 27 references

Exploring Active Data Selection Strategies for Continuous Training in Deepfake Detection

T0 review · 2 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read The paper shows that repeatedly fine-tuning a deepfake detector on the small subset of new images it is most unsure about cuts test error from 22.5% to about 2.5% equal error rate, using only 15% of the available pool.

desk verdict A plausible proof-of-concept with a large headline improvement, but the key active-vs-random comparison rests on a single run without error bars. read the letter →

arxiv 2502.07269 v1 pith:7EP2J64K submitted 2025-02-11 cs.CV

classification cs.CV
keywords deepfakedetectionactivelearningcontinuoustrainingdataselectionnegativeenergyscorecertaintyscoringout-of-distribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a way to keep a deepfake detector current as new forgery methods appear: instead of retraining on all new data, score each candidate image by the detector's own uncertainty, select a small batch of the least certain images, and fine-tune on those plus the original training set. On a proof-of-concept setup, a detector trained only on ForgeryNet scored 22.5% EER on a test set containing newer, unseen spoofing methods. After repeated rounds of active selection and continuous training, the same detector reached about 2.5% EER while using only 15% of the pool set, and it outperformed random selection in later rounds. The practical point is that a deployed detector could be updated cheaply and automatically whenever a new deepfake generator appears, without bloating the master training set.

What carries the argument

The mechanism is the negative energy-based confidence score $c_m = -T \log \sum_{j=1}^{J} \exp(l_{m,j}/T)$, a negative log-sum-exp over the detector's final-layer logits with temperature $T=1$; lower values mean the model is less certain. Algorithm 1 sorts the pool by this score, takes the $L$ lowest-scoring samples, removes them from the pool, adds them to the training set, and fine-tunes the model on the combined data, repeating for $K$ iterations. The score doubles as an out-of-distribution detector, which is why it tends to pick samples from generators the model has not seen.

What would settle it

A concrete falsifying experiment would compare the ranking of pool samples by negative energy against the ranking by how much adding each sample individually improves validation EER; if the two rankings do not correlate positively, the confidence score is not actually selecting the most useful data. A simpler check would run the same continuous-training protocol with many random-seed pool orderings and see whether random selection ever matches the active method's final EER.

Watch

Extended reading notes

Core claim

The central claim is that a model's own confidence score, specifically the negative energy score computed from its output logits, can serve as the selector for what to add during continuous training. The authors show that low-confidence samples from a redundant pool are the ones whose addition improves detection most: after repeated rounds of adding 10,000 selected images per round, the EER fell from 22.5% to just under 2.5%, and the selected 15% of the pool outperformed a randomly chosen 15% once the easy samples were exhausted. The authors emphasize that the update is not domain adaptation, because the model is fine-tuned on the union of the original and selected data, preserving previously learned detection while absorbing new methods.

Load-bearing premise

The loop only works as written if every pool image, especially each one selected, has a known real/fake label, and if low confidence really flags the samples most worth training on; remove either assumption and the algorithm's selection signal or its training step breaks.

Editorial extensions

If this is right

  • A detector that starts at 22.5% EER on new spoofing methods can reach about 2.5% EER after continuous training on only 15% of a redundant pool.
  • Negative-energy selection beats random selection in later iterations, once the first easy-to-learn batches have been consumed.
  • Because only a fraction of the pool is added, per-epoch training stays far cheaper than retraining on all pool data: 1303 seconds per epoch versus 2798 seconds per epoch in the paper's setup.
  • The same loop can be triggered again whenever a new generator becomes prevalent, keeping the detector current without rebuilding the master set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The selection order the authors observe, with newer generators like Stable Diffusion chosen first, older methods like FF++ later, and KoDF mostly skipped, suggests the negative energy score is effectively measuring novelty; a deployed system could use that ordering to decide which new generators need data collection first.
  • In a deployed setting where labels are not guaranteed, the same confidence signal could route low-scoring images to human labelers or drive a semi-supervised pseudo-labeling variant, neither of which the paper tests.
  • Because random selection wins the first two rounds, a hybrid strategy that starts with random warm-up and then switches to active selection may converge faster than either alone; this is a testable variant, not a paper claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes an active data selection method for continuously updating a deepfake detector. Starting from a detector trained on ForgeryNet, a pool of real and fake face images from FF++, Google DFD, YouTube DF, KoDF, and Stable Diffusion is scored with a negative-energy confidence measure; the L lowest-confidence samples are added to the training set, the model is fine-tuned, and the process is repeated. Experiments report that the base model has 22.5% EER on the combined test set, while the actively updated model reaches just under 2.5% EER after adding 15% of the pool, and the paper states that active selection outperforms random selection in later iterations.

Significance. If the active-versus-random advantage is reproducible, the paper offers a practical, low-cost way to prioritize data for regular model updates in deepfake detection. The magnitude of the improvement over the base model is substantial, the algorithm is clearly specified, and all datasets are public, which aids reproducibility. The self-referential nature of using the model's own confidence for selection is not a circularity problem because the method is compared against random selection and against the base model. However, the key comparative claim currently rests on a single run with no uncertainty quantification, so the significance is conditional on additional multi-seed experiments.

major comments (2)
  1. [Section 3.4, Fig. 2] The central comparative claim that AL negE outperforms AL random is supported by a single run. No random seeds, repeated runs, error bars, or significance tests are reported, and the paper itself states that random selection was better in the first two iterations. Since each iteration involves stochastic fine-tuning on different selected batches, the later crossover could be within run-to-run variance. Please report mean and standard deviation over multiple seeds, together with a paired significance test or bootstrap confidence intervals at the reported 15% selection point and, ideally, for each iteration.
  2. [Section 2.1, Algorithm 1] The method requires labels y_n for every pool sample, including the selected ones, in order to perform supervised fine-tuning at line 9. This is explicitly acknowledged as reasonable when the pool data come from public datasets or are generated via APIs, but the abstract and introduction frame the selection as 'automatic' and motivated by new deepfake methods in the wild. Please state the label-availability assumption and its scope in the abstract or introduction, or provide an experiment or extension for unlabeled pool data; otherwise the applicability of the method is narrower than the framing suggests.
minor comments (5)
  1. [Fig. 2] The caption and the surrounding text state that the horizontal axis encodes the iteration index, the number of selected samples, and its percentage, but the figure itself appears to show only one axis; make the mapping explicit and readable.
  2. [Section 3.1, Table 1] For video-based pools such as YouTube DF and KoDF, the face extraction procedure must include a frame-sampling step; please specify the frame selection strategy and the number of frames per video.
  3. [Section 3.4, footnote 4] The reported training-time comparison (2798 vs. 1303 sec/epoch) lacks hardware and batch-size details; state these conditions or remove the cost claim.
  4. [Section 2.2] The sentence 'This method is compared with a random selection from a pool set.3' has a misplaced footnote marker; move the marker to the end of the sentence or the paragraph.
  5. [Algorithm 1] The notation 'argmin-sort_m({..., c_m, ...})[0:L]' is nonstandard; define it as returning the indices of the L smallest confidence scores.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported EER improvements are measured outcomes of an implemented active-learning loop, not predictions derived from fitted constants or self-citations.

full rationale

The paper's central claim, that continuous training with actively selected pool data reduces EER from 22.5% (Base) to about 2.5% (AL negE), is an empirical result reported in Section 3.4 and Figure 2. No equation in the paper derives these EER values from the selection criterion; the negative-energy score in Eq. (1) is only used to rank pool samples in Algorithm 1, and the EER is then measured on an independent test set. The comparison against AL random and against the untrained Base model provides an external baseline, so the selection rule is not equivalent to the evaluation metric by construction. The only self-referential aspect is that the model uses its own confidence scores to choose training data, but this is the method being tested rather than a fitted parameter renamed as a prediction. The self-citation [WY23] in Section 2.2 is used only as background support for choosing negative-energy scoring; the paper's active-vs-random comparison and the Base-model improvement do not depend on that citation, so it is not load-bearing. The paper itself acknowledges limitations: labels in the pool are assumed known (Section 2.1), random selection was better in the first two iterations (Section 3.4), and no error bars or significance tests are reported. These are robustness and statistical-evidence concerns, not circularity, because they do not amount to a reduction of the claimed result to its own inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several explicit modeling choices: labeled pool data, an equalized pool, same-domain test sets, and a borrowed confidence metric. No new physical or mathematical entities are postulated. The free parameters are standard training and selection hyperparameters, none of which are fitted to the test set, though several are chosen by hand.

free parameters (4)
  • L (number of selected samples per iteration) = 10000
    Section 3.3 states that 10,000 samples are selected from the pool in each continuous-training iteration. This value is chosen by hand and directly controls the reported 15% pool usage.
  • K (number of continuous-training iterations) = 6 (implied)
    The paper never states K explicitly. The reported result uses 60,000 selected images, which is 15% of the 400,000-image pool at 10,000 per iteration, implying about six iterations. The stopping criterion affects both efficiency and final EER.
  • Fine-tuning epochs per iteration = 3
    Section 3.3 says fine-tuning is conducted for three epochs per continuous-training iteration. This is a hand-chosen hyperparameter that affects how much the model changes at each step.
  • Softmax temperature T = 1
    Equation (1) includes the softmax temperature T, set to 1 following [Li20]. It is not fitted in this paper but is a fixed hyperparameter in the confidence score.
assumptions (4)
  • domain assumption Labels of all pool data, including selected samples, are known and available for supervised fine-tuning.
    Stated in Section 2.1: labels {y_n} of the selected data are assumed to be known, and Algorithm 1 uses them in continuous training. If false, the proposed method cannot be applied as described.
  • domain assumption The pool set is pre-balanced to 40,000 images per source dataset.
    Footnote 3 states that the amount of data per dataset in the pool set was adjusted and equalized beforehand. This removes pool-distribution bias and directly affects both random-selection and active-selection behavior.
  • domain assumption Test sets are drawn from the same source datasets as the pool set.
    Table 1 and Section 3.4 evaluate on 1,000 held-out images per dataset. The result measures adaptation to pool-domain distributions, not generalization to unseen spoofing domains, which the authors acknowledge as future work in Section 4.
  • domain assumption The negative energy score with T=1 is a valid confidence metric for selecting useful training data.
    Section 2.2 adopts the negative energy-based score from [Li20] and, following [WY23], uses it as the selection metric. Its effectiveness is assumed from prior literature rather than validated against other selection metrics in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Active Data Selection Strategies for Continuous Training in Deepfake Detection." pith.science (2026). https://pith.science/paper/7EP2J64K

@misc{pith2026250207269,
  author       = {Pith},
  title        = {Pith review of: Exploring Active Data Selection Strategies for Continuous Training in Deepfake Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7EP2J64K}},
  note         = {Machine review of arXiv:2502.07269}
}
read the original abstract

In deepfake detection, it is essential to maintain high performance by adjusting the parameters of the detector as new deepfake methods emerge. In this paper, we propose a method to automatically and actively select the small amount of additional data required for the continuous training of deepfake detection models in situations where deepfake detection models are regularly updated. The proposed method automatically selects new training data from a \textit{redundant} pool set containing a large number of images generated by new deepfake methods and real images, using the confidence score of the deepfake detection model as a metric. Experimental results show that the deepfake detection model, continuously trained with a small amount of additional data automatically selected and added to the original training set, significantly and efficiently improved the detection performance, achieving an EER of 2.5% with only 15% of the amount of data in the pool set.

Figures

Figures reproduced from arXiv: 2502.07269 by the authors.

Figure 1
Figure 1. Active data selection for continuous training of deepfake detection models. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. EERs (%) on the evaluation set across different continuous training iterations. Numbers along [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Percentage indicating from which dataset the image was selected in each iteration. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 25 canonical work pages

  1. [1]

    write newline

    " write newline "" initialize.prev.this.status FUNCTION begin.bib " write newline preamble empty 'skip preamble write newline if " thebibliography " longest.label * " " * write newline " [1] #1 " write newline " url@samestyle " write newline " " write newline " [2] #2 " write newline " =0pt " write newline " " ALTinterwordstretchfactor * " " * write newli...

  2. [2]

    4" FUNCTION default.is.dash.repeated.names #1 FUNCTION default.name.format.string

    11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

  3. [3]

    write newline

    " write newline "" initialize.prev.this.status FUNCTION begin.bib " write newline preamble empty 'skip preamble write newline if " thebibliography " longest.label * " " * write newline " [1] #1 " write newline " url@samestyle " write newline " " write newline " [2] #2 " write newline " =0pt " write newline " " ALTinterwordstretchfactor * " " * write newli...

  4. [4]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  5. [5]

    Biometrics, ISO/IEC JTC1 SC37: , ISO/IEC 2382-37: 2017 Information Technology-Vocabulary-Part 37: Biometrics, 2017

  6. [6]

    Dissertation, Universidad aut \'o noma de Madrid, 2007

    Castro, Daniel Ramos: Forensic Evaluation of the Evidence Using Automatic Speaker Recognition Systems. Dissertation, Universidad aut \'o noma de Madrid, 2007

  7. [7]

    In: Proc

    Chung, Joon Son; Nagrani, Arsha; Zisserman, Andrew: VoxCeleb2 : Deep Speaker Recognition. In: Proc. INTERSPEECH . S. 1086--1090, 2018

  8. [8]

    In: Proc

    Das, Srijan; Jain, Tanmay; Reilly, Dominick; Balaji, Pranav; Karmakar, Soumyajit; Marjit, Shyam; Li, Xiang; Das, Abhijit; Ryoo, Michael S: Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders. In: Proc. WACV. S. 6878--6888, 2024

Show all 27 references
  1. [9]

    In: Proc

    Deng, Jia; Dong, Wei; Socher, Richard; Li, Li-Jia; Li, Kai; Fei-Fei, Li: ImageNet : A large-scale hierarchical image database. In: Proc. CVPR. IEEE, S. 248--255, 2009

  2. [10]

    In: Proc

    Deng, Jiankang; Guo, Jia; Ververas, Evangelos; Kotsia, Irene; Zafeiriou, Stefanos: RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild. In: Proc. CVPR. S. 5202--5211, 2020

  3. [11]

    https://ai.googleblog.com/2019/09/contributing-data-to-deepfake-detection.html, 9 2019

    Dufour, Nick; Gully, Andrew: , Contributing Data to Deepfake Detection Research. https://ai.googleblog.com/2019/09/contributing-data-to-deepfake-detection.html, 9 2019

  4. [12]

    Journal of King Saud University-Computer and Information Sciences, 35(4):90--99, 2023

    Dong, Fengkai; Zou, Xiaoqiang; Wang, Jiahui; Liu, Xiyao: Contrastive learning-based general Deepfake detection with multi-scale RGB frequency clues. Journal of King Saud University-Computer and Information Sciences, 35(4):90--99, 2023

  5. [13]

    In: Proc

    He, Yinan; Gan, Bei; Chen, Siyu; Zhou, Yichun; Yin, Guojun; Song, Luchuan; Sheng, Lu; Shao, Jing; Liu, Ziwei: ForgeryNet : A versatile benchmark for comprehensive forgery analysis. In: Proc. CVPR. S. 4360--4369, 2021

  6. [14]

    In: Proc

    Karras, Tero; Laine, Samuli; Aila, Timo: A style-based generator architecture for generative adversarial networks. In: Proc. CVPR. S. 4401--4410, 2019

  7. [15]

    a \"a , Hannu; Hautam \

    Kukanov, Ivan; Karttunen, Janne; Sillanp \"a \"a , Hannu; Hautam \"a ki, Ville: Cost sensitive optimization of deepfake detector. In: Proc. APSIPA ASC. IEEE, S. 1300--1303, 2020

  8. [16]

    In: Proc

    Kwon, Patrick; You, Jaeseong; Nam, Gyuhyeon; Park, Sungwoo; Chae, Gyeongsu: Kodf: A large-scale korean deepfake detection dataset. In: Proc. ICCV. S. 10744--10753, 2021

  9. [17]

    In: Proc

    Larue, Nicolas; Vu, Ngoc-Son; Struc, Vitomir; Peer, Peter; Christophides, Vassilis: SeeABLE: Soft Discrepancies and Bounded Contrastive Learning for Exposing Deepfakes. In: Proc. ICCV. S. 21011--21021, 2023

  10. [18]

    In: Proc

    Liu, Weitang; Wang, Xiaoyun; Owens, John; Li, Yixuan: Energy-based Out-of-distribution Detection . In: Proc. NIPS. Jgg. 33, S. 21464--21475, 2020

  11. [19]

    In: Proc

    Liu, Huan; Tan, Zichang; Tan, Chuangchuang; Wei, Yunchao; Zhao, Yao; Wang, Jingdong: Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection. In: Proc. CVPR. 2023

  12. [20]

    In: Proc

    Ma, Haoxin; Yi, Jiangyan; Tao, Jianhua; Bai, Ye; Tian, Zhengkun; Wang, Chenglong: Continual learning for fake audio detection. In: Proc. INTERSPEECH. 2021

  13. [21]

    In: Proc

    Rossler, Andreas; Cozzolino, Davide; Verdoliva, Luisa; Riess, Christian; Thies, Justus; Nie ner, Matthias: FaceForensics++ : Learning to detect manipulated facial images. In: Proc. ICCV. S. 1--11, 2019

  14. [22]

    In: Proc

    Rombach, Robin; Blattmann, Andreas; Lorenz, Dominik; Esser, Patrick; Ommer, Bj \"o rn: High-resolution image synthesis with latent diffusion models. In: Proc. CVPR. S. 10684--10695, 2022

  15. [23]

    In: Proc

    Song, Wentang; Lin, Yuzhen; Li, Bin: Towards Generic Deepfake Detection with Dynamic Curriculum. In: Proc. ICASSP. IEEE, S. 4500--4504, 2024

  16. [24]

    In (Meila, Marina; Zhang, Tong, Hrsg.): Proceedings of the 38th International Conference on Machine Learning

    Tan, Mingxing; Le, Quoc: EfficientNetV2: Smaller Models and Faster Training. In (Meila, Marina; Zhang, Tong, Hrsg.): Proceedings of the 38th International Conference on Machine Learning. Jgg. 139 in Proc. Machine Learning Research. PMLR, S. 10096--10106, 18--24 Jul 2021

  17. [25]

    In: Proc

    Wang, Xin; Yamagishi, Junichi: Investigating Active-learning-based Training Data Selection for Speech Spoofing Countermeasure . In: Proc. SLT . S. 585--592, 2023

  18. [26]

    In: Proc

    Yan, Zhiyuan; Zhang, Yong; Yuan, Xinhang; Lyu, Siwei; Wu, Baoyuan: DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection. In: Proc. NeurIPS Datasets and Benchmarks Track. 2023

  19. [27]

    Advances in Neural Information Processing Systems, 36, 2024

    Yan, Zhiyuan; Zhang, Yong; Yuan, Xinhang; Lyu, Siwei; Wu, Baoyuan: DeepfakeBench : A Comprehensive Benchmark of Deepfake Detection. Advances in Neural Information Processing Systems, 36, 2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.