Pith. sign in

REVIEW 3 major objections 5 minor 28 references

Spotting tell-tale visual artifacts in face swapping videos: strengths and pitfalls of CNN detectors

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CNN detectors for face-swap videos achieve near-perfect accuracy within a dataset but fail to generalize across datasets, learning dataset-specific cues rather than occlusion artifacts.

desk verdict Useful new dataset and a credible cross-dataset negative result, but the claim that models ignore occlusion cues rests on a confounded comparison. read the letter →

arxiv 2506.16497 v1 pith:LMRGK2LL submitted 2025-06-19 cs.CV cs.AIcs.CR

Riccardo Ziglio , Cecilia Pasquini , Silvio Ranise This is my paper · ORCID
classification cs.CVcs.AIcs.CR
keywords faceswappingdetectionocclusionartifactscross-datasetgeneralizationchallenge-responseCNNdetectorsvideoforensicsFOWSdatasetdeepfake
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether CNN-based detectors can exploit occlusion-induced visual artifacts to spot face-swapped videos in remote video calls. Using a newly collected corpus of challenge-based videos and an existing one, it shows that within a single dataset the detectors are nearly perfect, with balanced accuracy often above 99 percent. Across datasets, however, the same models collapse to roughly 50 to 60 percent balanced accuracy, near chance. The authors conclude that the networks learn dataset-specific statistical cues rather than the occlusion artifacts that the challenge-response setup is meant to expose, so specialized detection strategies are needed.

What carries the argument

The load-bearing experiment is the cross-category transfer test, which compares equal error rates when training and testing on the same frame category versus swapping categories (occ and no-occ). Since occlusion artifacts appear only in occ frames, a detector that truly relied on them should fail when trained on no-occ and tested on occ; instead the equal error rates stay near zero in most configurations, showing the models are not using the occlusion cues. This is paired with a cross-dataset protocol built on two corpora, a new one (FOWS) with hand and object occlusions and three swapping algorithms, and an existing one (GOTCHA), with five CNN architectures including two previously specialized for deepfake detection.

What would settle it

Re-run the cross-category test on a dataset where the same subject, pose, illumination, and motion are held fixed while only the occlusion is toggled on and off; if models trained on no-occ still detect occ frames as well as occ-trained models, the paper's interpretation would be supported, and if performance collapses, the confound would explain the result.

Watch

Extended reading notes

Core claim

The central claim is that general-purpose CNN detectors for face-swap videos, when trained and tested on the same data source, achieve excellent separation (balanced accuracy near 100 percent), yet this skill does not transfer. In cross-dataset tests the same architectures drop to near-chance balanced accuracy around 50 to 60 percent, with error patterns that depend on the direction of transfer. The paper's sharper finding comes from cross-category tests: models trained on frames without occlusions detect occlusion-containing manipulated frames almost as well as models trained on occluded frames, and models trained on occluded frames also do well on frames without occlusions. The authors take this as evidence that the discriminative cues are largely shared, dataset-specific statistics rather than the occlusion-related artifacts that motivated the challenge-response approach, a reading supported by activation maps that avoid the occluded regions.

Load-bearing premise

The conclusion that models ignore occlusion-specific cues assumes that occluded and non-occluded frames differ only in the presence of occlusion artifacts, but in the collected data they also differ in motion, pose, and detector behavior, so the cross-category result could be driven by other shared cues.

Editorial extensions

If this is right

  • Near-perfect in-domain results should not be read as evidence that a face-swap detector will work in a live video call, since unseen sources shift the input distribution.
  • Default 0.5 thresholds learned on one dataset are miscalibrated on another, so even when scores separate, classification with the original threshold fails.
  • The challenge-response approach remains valuable for human observers, but automated detectors need architectures or training objectives that explicitly target occlusion artifacts.
  • Prior training on generic large-scale deepfake datasets does not confer an advantage when the model is retrained on occlusion-based data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The large threshold shifts shown in the score distributions suggest that per-dataset calibration or unsupervised adaptation on unlabeled test frames could recover a meaningful portion of the lost accuracy.
  • Because occlusion is confounded with motion and camera dynamics, a controlled synthetic corpus that toggles only the occlusion would isolate whether occlusion artifacts are learnable at all.
  • Forcing the network to attend to the occluded region during training is a direct test of whether occlusion cues are sufficient for cross-dataset detection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper benchmarks five CNN-based face-swap detectors on two video corpora: the newly collected FOWS dataset (7 subjects, hand/object occlusion challenges, three swapping algorithms) and the existing GOTCHA dataset. In a fully aligned setting, where training and test frames come from the same dataset and the same frame category (occ or no-occ), most architectures achieve near-perfect balanced accuracy on both datasets. In cross-dataset tests, balanced accuracy drops to roughly 50-60% for most models, and the paper documents score-distribution shifts that make a fixed 0.5 threshold unreliable. Additional cross-category experiments and GradCAM++ visualizations are used to argue that the models learn dataset-specific statistical cues rather than genuine occlusion-related artifacts. The authors conclude that specialized detection strategies are needed to exploit challenge-based occlusion artifacts.

Significance. If its empirical results hold, this paper makes a useful contribution: it introduces a new publicly available dataset with systematic face occlusions, evaluates five architectures under aligned and cross-dataset protocols, and clearly demonstrates that high in-dataset performance does not transfer across acquisition sources. The cross-dataset generalization failure is an important negative result for challenge-based deepfake detection and supports the authors' call for specialized approaches. The paper also ships code and data, which aids reproducibility. However, the interpretive claim that the models do not learn occlusion-related cues rests on cross-category and attribution evidence that is confounded, so the paper's strongest conclusion is currently underdetermined.

major comments (3)
  1. [Section IV-B, Table IV, and Section V] The cross-category evidence does not isolate occlusion as the only variable that changes between the occ and no-occ frame sets. In FOWS, no-occ frames are sampled from a static temporal segment, while occ frames are taken from the dynamic segment and only from frames where the BlazeFace detector fails under strict tolerance. The two categories therefore differ simultaneously in motion, pose, background, scale, and detector-success statistics, in addition to the presence of occlusion-induced artifacts. As a result, the near-zero EER values in Table IV for no-occ-trained models tested on occ frames could be explained by these confounds or by generic face-swap artifacts (e.g., blending seams, color/texture inconsistencies) that appear in both categories. The statement in Section V that models learn mostly dataset-specific statistical cues rather than actual occlusion-related cues is stronger than the evidence supports. Please add a matched control, for example by sampling no-occ and occ frames with comparable motion statistics or by using static frames with synthetic occlusions, and supplement the GradCAM++ examples with a quantitative measure of overlap between the highlighted regions and the occluder area.
  2. [Section IV-A, Tables II and III] All results appear to come from a single training run per configuration; no repeated seeds, confidence intervals, or significance tests are reported. Given the FOWS test set size (4,800 frames per category) and the stochasticity of CNN training, point-estimate differences such as EffNetB4 B-ACC 59.92 versus MobNet B-ACC 62.34 in the cross-dataset setting may not be meaningful. The central claim of a large aligned-to-cross-dataset performance drop is robust to this issue, but the paper also compares model families and comments on relative strengths; those comparisons need variance information. Please report mean and standard deviation over multiple runs, or at least a statistical test for the main cross-dataset comparisons.
  3. [Section III-B] The choice of the two FOWS testing subjects is deliberate but narrow: they are the only woman and the only man with a thick beard in the dataset. This makes the FOWS test partition unrepresentative of the general population and fully confounds subject attributes with the test split. The paper should state this as an explicit limitation and discuss whether the aligned and cross-dataset conclusions could be affected by the demographic makeup of the test set; reporting per-subject results would strengthen the analysis.
minor comments (5)
  1. [Abstract] The sentence 'Face swapping manipulations in video streams represents an increasing threat' should read 'represent an increasing threat' to agree with the plural subject.
  2. [Section III-C] 'The dataset and part of the code used used in this study' contains a duplicated word; it should be 'code used in this study'.
  3. [Section IV-A] In the sentence 'when training on GOTCHA and training on FOWS, models like MobNet and Xception retain rather good values of AUC and EER', the second occurrence of 'training' should be 'testing'.
  4. [Table III] In the upper-left and lower-left blocks of Table III, the header 'ACC(OR F)' appears although the test set is GOTCHA; the column should presumably read 'ACC(OR G)' for consistency with the right-hand blocks.
  5. [Section IV-B] The GradCAM++ discussion reports only two models and sample frames; the observation that activations are 'often not concentrated on the occluded part' is anecdotal and should be labeled as illustrative rather than as a quantitative finding.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the core results are an empirical cross-dataset benchmark with independent training and test partitions.

full rationale

The paper reports an empirical benchmark, not a derivation, and no load-bearing step reduces to its own inputs. The central claims are that CNN detectors achieve near-perfect accuracy in aligned settings and that this accuracy collapses in cross-dataset settings (Table III). These results come from training on one dataset and evaluating on a different dataset or on held-out subjects, with no fitted parameter later relabeled as a prediction. The newly collected FOWS dataset is independent of the GOTCHA dataset from [16], which was created by different authors. The interpretive claim in Section V that models 'learn mostly dataset-specific statistical cues rather than actual occlusion-related cues' is supported by cross-category transfer results (Table IV), but even if those results are confounded by static/dynamic segment differences, a validity concern is not a circularity concern. There is no equation that defines X in terms of Y, no fitted input called a prediction, and no self-citation chain that carries the argument. The only self-citation is a routine reference to prior work by two of the authors ([18]) in the related-work context, and it is not load-bearing. The paper is self-contained against external benchmark data and does not smuggle in an ansatz via citation. Therefore no circular step is present, and the score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

This is an empirical study; the central claim rests on dataset construction and evaluation choices rather than on fitted mathematical parameters. The main assumptions are that occlusion frames are correctly identified by detector failure, that artifacts are truly present and similar across datasets, and that the selected CNN architectures represent current practice. These assumptions are standard but not fully verified, and the absence of repeated runs and the unreported augmentation tuning add uncertainty.

free parameters (4)
  • Face-box expansion factor = 1.3
    Used to expand the BlazeFace bounding box when extracting faces. Chosen per [23,24], not tuned here, but it affects the input to all detectors and could influence results.
  • Number of sampled frames per user/category = 100
    Sampling 100 frames from static/dynamic segments yields 600 frames per category per user. This is a hand-chosen number that sets dataset size and could affect training dynamics.
  • Training epochs = 10 (FOWS), 15 (GOTCHA)
    Hand-selected epochs with early stopping; different across datasets, potentially affecting convergence.
  • Data augmentation configuration = random resize crop (224,224), flip, rotation [-5,5], color jitter
    Selected after unreported preliminary tests (Section III-D), it influences model training and could bias results if tuned on test data.
assumptions (4)
  • domain assumption The BlazeFace failure-based selection reliably separates occ from no-occ frames.
    In Section III-B, frames where BlazeFace fails with strict tolerance are labeled as occluded. This assumes detector failure is a faithful proxy for occlusion, ignoring other causes of failure such as motion blur.
  • domain assumption Occlusion-based artifacts are present in all manipulated videos.
    Section II states 'In agreement with [16], occlusion-based artifacts are present in all manipulated videos.' This is taken from prior work without independent quantitative verification in FOWS.
  • domain assumption The five CNN architectures are representative of current detection approaches.
    Section III-C selects baselines and two specialized detectors based on prior benchmarks; the paper does not cover all published methods, so results may not generalize to other architectures.
  • domain assumption Default 0.5 decision threshold is the relevant operating point.
    Section III-D fixes the threshold at 0.5 for accuracy metrics. The paper separately reports AUC/EER, but the practical conclusion about failure is partly driven by this threshold choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Spotting tell-tale visual artifacts in face swapping videos: strengths and pitfalls of CNN detectors." pith.science (2026). https://pith.science/paper/LMRGK2LL

@misc{pith2026250616497,
  author       = {Pith},
  title        = {Pith review of: Spotting tell-tale visual artifacts in face swapping videos: strengths and pitfalls of CNN detectors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMRGK2LL}},
  note         = {Machine review of arXiv:2506.16497}
}
read the original abstract

Face swapping manipulations in video streams represents an increasing threat in remote video communications, due to advances in automated and real-time tools. Recent literature proposes to characterize and exploit visual artifacts introduced in video frames by swapping algorithms when dealing with challenging physical scenes, such as face occlusions. This paper investigates the effectiveness of this approach by benchmarking CNN-based data-driven models on two data corpora (including a newly collected one) and analyzing generalization capabilities with respect to different acquisition sources and swapping algorithms. The results confirm excellent performance of general-purpose CNN architectures when operating within the same data source, but a significant difficulty in robustly characterizing occlusion-based visual cues across datasets. This highlights the need for specialized detection strategies to deal with such artifacts.

Figures

Figures reproduced from arXiv: 2506.16497 by the authors.

Figure 1
Figure 1. (a) Pipeline of the video recording with the guiding videos for the hand occlusion challenge. (b) Examples of original [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Histogram of the decision score for MobNet (left) and Xception (right) trained GOTCHA and tested on FOWS [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the accuracy on all partitions of the models trained on [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: GradCAM++ activation maps for MobNet trained on GOTCHA and tested on FOWS (top row), and for ICPR2020 trained on FOWS and tested on GOTCHA (bottom row). V. DISCUSSIONS AND CONCLUSIONS We have performed an experimental analysis exploring the effectiveness of challenge-b…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 25 canonical work pages

  1. [1]

    Amerini, M

    I. Amerini, M. Barni, S. Battiato, P. Bestagini, G. Boato, T. Sari Bonaventura, V . Bruni, R. Caldelli, F. De Natale, R. De Nicola, L. Guarnera, S. Mandelli, G. L. Marcialis, M. Micheletto, A. Montibeller, G. Orru’, A. Ortis, P. Perazzo, G. Puglisi, D. Salvi, S. Tubaro, C. Melis Tonti, M. Villari, and D. Vitulano. Deepfake media forensics: State of the ar...

    arXiv 2024

  2. [2]

    Bonettini, E

    N. Bonettini, E. D. Cannas, S. Mandelli, L. Bondi, P. Bestagini, and S. Tubaro. Video face manipulation detection through ensemble of CNNs. In International Conference on Pattern Recognition (ICPR) , 2021. Available at https://github.com/polimi-ispl/icpr2020dfdc/tree/master

    work page 2021

  3. [3]

    SimSwap: An efficient framework for high fidelity face swapping

    Renwang C., Xuanhong C., Bingbing N., and Yanhao G. SimSwap: An efficient framework for high fidelity face swapping. In ACM International Conference on Multimedia , 2020

    work page 2020

  4. [4]

    J. P. Cardenuto, J. Yang, R. Padilha, R. Wan, D. Moreira, H. Li, S. Wang, F. Andal ´o, S. Marcel, and A. Rocha. The age of synthetic realities: Challenges and opportunities. APSIPA Transactions on Signal and Information Processing , 12(1), 2023

    work page 2023

  5. [5]

    F. Chollet. Xception: Deep Learning with Depthwise Separable Convolutions . In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

    work page 2017

  6. [6]

    Remote ID Proofing

    ENISA. Remote ID Proofing. ENISA Report, March 2021

    work page 2021

  7. [7]

    Remote Identity Proofing – Attacks & Countermeasures

    ENISA. Remote Identity Proofing – Attacks & Countermeasures. ENISA Report, January 2022

    work page 2022

  8. [8]

    C. R. Gerstner and H. Farid. Detecting real-time deep-fake videos using active illumination. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , 2022

    work page 2022

Show all 28 references
  1. [9]

    Pytorch library for cam methods

    Jacob Gildenblat and contributors. Pytorch library for cam methods. 2021. Available at https://github.com/jacobgil/pytorch-grad-cam

    2021

  2. [10]

    Mediapipe face detector python

    Google. Mediapipe face detector python. Available at https://developers.google.com/mediapipe/solutions/vision/face detector

  3. [11]

    Groshev, A

    A. Groshev, A. Maltseva, D. Chesakov, A. Kuznetsov, and D. Dimitrov. GHOST—a new face swap approach for image and video domains. IEEE Access, 2022

    2022

  4. [12]

    Haasnoot, Luuk J

    E. Haasnoot, Luuk J. Spreeuwers, and Raymond N. J. Veldhuis. Presentation attack detection and biometric recognition in a challenge-response formalism. EURASIP Journal on Information Security , (1):5, 2022

    2022

  5. [13]

    C. Li, L. Wang, S. Ji, X. Zhang, Z. Xi, S. Guo, and T. Wang. Seeing is Living? Rethinking the Security of Facial Liveness Verification in the Deepfake Era. In USENIX Security 22 , pages 2673–2690, 2022

    2022

  6. [14]

    T. Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence , 42(2):318–327, 2020

    2020

  7. [15]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019

    2019

  8. [16]

    Mittal, C

    G. Mittal, C. Hegde, and N. Memon. Gotcha: Real-time video deepfake detection via challenge-response. In IEEE European Symposium on Security and Privacy (EuroS&P) , 2024. Available at https://github.com/mittalgovind/GOTCHA-Deepfakes

    2024

  9. [17]

    FSGAN: Subject agnostic face swapping and reenactment

    Yuval N., Yosi K., and Tal H. FSGAN: Subject agnostic face swapping and reenactment. IEEE/CVF International Conference on Computer Vision (ICCV), 2019

    2019

  10. [18]

    Pasquini, M

    C. Pasquini, M. Pernpruner, G. Sciarretta, and S. Ranise. Towards a fine-grained threat model for video-based remote identity proofing. In Machine Learning and Principles and Practice of Knowledge Discovery in Databases - International Workshops of ECML PKDD , 2023

    2023

  11. [19]

    Perov, D

    I. Perov, D. Gao, N. Chervoniy, K. Liu, S. Marangonda, C. Um ´e, Dpfks, C. Shift Facenheim, L. RP, J. Jiang, S. Zhang, P. Wu, B. Zhou, and W. Zhang. DeepFaceLab: Integrated, flexible and extensible face-swapping framework. arXiv: 2005.05535 , 2021

    2005 arXiv

  12. [20]

    M. M. Pic, G. Mahfoudi, A. Trabelsi, and J. Dugelay. Face Manipulation Detection in Remote Operational Systems . Springer International Publishing, 2022

    2022

  13. [21]

    Ramachandra and C

    R. Ramachandra and C. Busch. Presentation attack detection methods for face recognition systems: A comprehensive survey. ACM Computing Surveys , 50(1), 2017

    2017

  14. [22]

    Rosberg, E

    F. Rosberg, E. E. Aksoy, F. Alonso-Fernandez, and C. Englund. FaceDancer: Pose- and occlusion-aware high fidelity face swapping. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , 2023

    2023

  15. [23]

    R ¨ossler, D

    A. R ¨ossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner. Faceforensics++: Learning to detect manipulated facial images. In ICCV, 2019

    2019

  16. [24]

    Visual realism assessment for face-swap videos

    Xianyun S., Beibei D., Caiyong W., Bo P., and Jing D. Visual realism assessment for face-swap videos. arXiv: 2302.00918 , 2023

    2023 arXiv

  17. [25]

    Tolosana, R

    R. Tolosana, R. Vera-Rodr ´ıguez, J. Fierrez, A. Morales, and J. Ortega-Garcia. Deepfakes and beyond: A survey of face manipulation and fake detection. arXiv: 2001.00179 , 2020

    2001 arXiv

  18. [26]

    Danial Samadi Vahdati, Tai Duc Nguyen, and Matthew C. Stamm. Defending low-bandwidth talking head videoconferencing systems from real-time puppeteering attacks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , 2023

    2023

  19. [27]

    Verdoliva

    L. Verdoliva. Media forensics and deepfakes: An overview. IEEE Journal of Selected Topics in Signal Processing , 14(5):910–932, 2020

    2020

  20. [28]

    Z. Yan, Y . Zhang, Xi. Yuan, S. Lyu, and B. Wu. Deepfakebench: A comprehensive benchmark of deepfake detection. In Advances in Neural Information Processing Systems, volume 36, pages 4534–4565, 2023. Available at https://github.com/SCLBD/DeepfakeBench

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.