Pith. sign in

REVIEW 4 major objections 5 minor 107 references

A 100-image benchmark finds that high ROC-AUC can overstate how deployable a deepfake detector really is.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:16 UTC pith:2GYWQ4FW

load-bearing objection A useful first cross-paradigm deepfake benchmark, but the headline miscalibration claim is partly an artifact of the authors' own score normalization, especially for TruthScan. the 4 major comments →

arxiv 2607.06254 v2 pith:2GYWQ4FW submitted 2026-07-07 cs.CV cs.AI

VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

Sharayu N. Deshmukh , Md Rashidunnabi , Nelton Tiago Gemo , Kurundkar G. D. , Mahamune M. R. , Nilesh K. Deshmukh This is my paper
classification cs.CV cs.AI
keywords deepfake detectionAI-generated image detectioncross-paradigm benchmarkingcommercial detection APIsvision-language modelsopen-source detectorsMatthews correlation coefficientmodel calibration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds a single adversarial 100-image corpus and evaluates 36 deepfake detectors from three paradigms — commercial APIs, zero-shot vision-language models, and open-source detectors — under one shared output schema and metric protocol. It argues that on this corpus, commercial APIs are the strongest on average, vision LLMs sit in the middle, and most open-source detectors trail, while the best open-source model outperforms every vision LLM on ranking ability. The more consequential claim is narrower: several otherwise strong rankers separate fakes from reals well (ROC-AUC above 0.79) yet make miscalibrated decisions at their shipped 0.5 threshold (MCC between 0.23 and 0.32), so a high headline AUC can overstate real-world deployability. The paper also shows that on an imbalanced 79:21 corpus, accuracy and F1 are gameable by an always-fake predictor, arguing that MCC and specificity should be reported alongside AUC and accuracy.

Core claim

On the paper's own terms, the central discovery is the one-directional gap between ranking power and operating-point quality. Across 36 models, MCC and ROC-AUC correlate strongly (Pearson r≈0.86), so for most detectors the two metrics move together; but TruthScan, Hive, DRCT, and Claude Opus 4.8 each reach ROC-AUC above 0.79 while their MCC at the shared 0.5 threshold sits at 0.230–0.322. The paper interprets this as threshold miscalibration rather than weak discrimination: confidence scores separate the classes correctly in principle, but the shipped default threshold does not translate that separation into hard-label agreement. The mirror-image pattern — strong MCC with weak AUC — never ap

What carries the argument

The carrying mechanism is the benchmark protocol itself: a fixed 100-image corpus organized into eight edge-case families (face swaps, text-to-video stills, avatar compositing, on-device photo edits, opaque-provenance images, and compressed research frames), a unified output schema that normalizes every model's native output to a shared P(fake) in [0,1], and a dual-metric evaluation that ranks by MCC at the fixed 0.5 threshold while reporting ROC-AUC as a threshold-free tiebreak. The divergence between those two metrics — a band of models below the MCC–AUC diagonal — is the instrument that exposes the miscalibration finding; the 79:21 imbalance is what makes accuracy and F1 vulnerable to the

Load-bearing premise

The rankings and the miscalibration finding stand or fall on the paper's reconstruction of each vendor's native output as a shared P(fake) score; if a mapping misrepresents a vendor's actual decision rule, every downstream comparison is an artifact of that mapping.

What would settle it

Re-run the five commercial APIs using each vendor's own documented threshold and native score instead of the paper's normalized P(fake) at 0.5; if TruthScan and Hive's MCCs rise above ~0.6 while Neural Defend stays near 0.88, the miscalibration result is an artifact of the normalization mapping, not a property of the models. Independently re-verifying the ground-truth labels for the six partially verified or opaque fake source groups would serve as a second check.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Commercial APIs as a group post the strongest median operating-point quality (median MCC 0.322), and the leading API is also the fastest, so buying a service and buying speed need not trade off.
  • Paradigm membership is not destiny: the open-source detector DRCT reaches ROC-AUC 0.866, ahead of every vision LLM tested, even though the open-source median MCC is the weakest of the three tracks.
  • A high ROC-AUC on an adversarial corpus is necessary but not sufficient for trustworthy default-threshold decisions; procurement should weigh MCC and specificity alongside AUC.
  • Reporting accuracy or F1 alone on an imbalanced corpus can rank an always-fake predictor as a reasonable detector; reporting MCC and specificity prevents that.
  • Recalibrating per-model decision thresholds rather than using shipped defaults could raise operating-point quality without changing underlying discriminative ability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: recompute the commercial track using each vendor's native decision rule instead of the paper's reconstructed P(fake); if TruthScan and Hive's MCCs rise sharply, the miscalibration finding is partly a normalization artifact rather than pure model behavior.
  • Because the corpus is hand-selected for difficulty and skewed 79:21, the benchmark is a diagnostic stress test, not an estimate of field accuracy; practitioners should expect different absolute performance on easier, more representative data.
  • The one-directional gap suggests calibration metrics be reported by vendors themselves; a standard 'MCC at shipped threshold' could become a useful procurement datum.
  • If the corpus grows to several hundred images as the paper suggests, per-generator-family error analysis could reveal which paradigm fails on which edge case, turning the aggregate ranking into actionable design guidance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces VendorBench-100, a 100-image benchmark (79 fake, 21 real) for deepfake image detection, evaluated across three paradigms: 5 commercial APIs, 7 zero-shot vision LLMs, and 24 open-source detectors. It proposes a unified output schema and normalization, uses MCC as the primary ranking metric and ROC-AUC as a threshold-free tiebreak, and reports a full leaderboard. The main empirical claims are: commercial APIs have the strongest median operating-point quality, vision LLMs are mid-tier, open-source detectors are most variable; MCC and ROC-AUC are strongly correlated (Pearson r≈0.86); a subset of strong rankers (TruthScan, Hive, DRCT, Claude Opus 4.8) have high ROC-AUC but low MCC at the 0.5 threshold, indicating miscalibration at their shipped default threshold; and accuracy/F1 are misleading under the 79:21 imbalance because an always-fake predictor scores well on both. The paper is unusually explicit about its limitations: single run per model, no confidence intervals or significance testing, small per-family samples, partial provenance verification, and normalization validated only on a sample of raw responses.

Significance. If the findings hold, the benchmark is a useful contribution: it addresses a real gap in cross-paradigm evaluation, provides an adversarial corpus with an eight-family taxonomy and provenance registry, and releases code and data. The explicit limitation statements and the release of the evaluation harness are notable strengths. However, the headline miscalibration finding rests on author-chosen normalizations and a common 0.5 threshold rather than on vendor-shipped decision rules, and the small single-run design makes the specific subset identification fragile. With verification of vendor-native outputs and uncertainty quantification, the paper could be a valuable reference; in its current form the central claim is not yet fully established.

major comments (4)
  1. [Section 3.2, Table 1] The TruthScan adapter is load-bearing for the miscalibration finding. The paper states: 'TruthScan reports a numeric score on a documented 0–100 scale together with a categorical final_result field; we take P(fake)=score/100 and use this same value, thresholded at 0.5, for both the hard label and ROC-AUC.' The paper never reports the vendor's final_result or demonstrates that 50/100 is TruthScan's shipped decision threshold. The Table 1 row (MCC 0.322, recall 0.354) and TruthScan's inclusion in the 'miscalibrated at shipped default threshold' subset in Section 5 therefore depend on an author-chosen threshold. Please report the confusion matrix using final_result, and if it differs, re-evaluate the miscalibration claim. At minimum, justify why score/100 ≥ 0.5 is the vendor's operational rule.
  2. [Section 3.2, Table 1] The Neural Defend adapter determines the top-ranked model and the 'commercial APIs lead' conclusion. P(fake) is reconstructed as the maximum over sub-checks of each check's confidence (or one minus confidence for 'genuine' predictions). This max-aggregation is never shown to match Neural Defend's native verdict. Since Neural Defend leads the leaderboard (Table 1, MCC 0.876, AUC 0.979), the ranking may be an artifact of the chosen fusion rule. Please validate the reconstruction against the vendor's returned verdicts on all 100 images, or report sensitivity of the ranking to alternative aggregations (e.g., mean, min, product, or the vendor's own label). The current Section 6 admission that normalization was validated only on a sample of raw responses is insufficient for the benchmark's central comparison.
  3. [Section 5, Eq. (6)] The paper labels the MCC-versus-ROC-AUC gap as 'miscalibrated at their shipped default threshold.' This conflates threshold operating-point selection with probability calibration. A model whose scores are systematically shifted away from 0.5 can have low MCC at 0.5 without being miscalibrated in the standard calibration sense, and a perfectly calibrated model can have low MCC if 0.5 is not the optimal operating point for a 79:21 prior. The finding should be reframed as 'operating-point quality at the common 0.5 threshold lags ranking ability' unless the authors provide a calibration analysis (e.g., reliability diagrams or expected calibration error) and evidence that 0.5 is each model's shipped threshold. As written, the one-directional miscalibration claim overstates what the reported metrics establish.
  4. [Table 1, Section 6] The identification of the miscalibrated subset (TruthScan, Hive, DRCT, Claude Opus 4.8) is based on point estimates from a single run on 100 images. The paper explicitly forgoes confidence intervals and significance testing. Given that the gap between MCC 0.230–0.322 and AUC 0.79–0.92 is the paper's most consequential finding, please provide bootstrap confidence intervals for MCC, ROC-AUC, and their difference for the named models, and state whether subset membership survives. Without this, the one-directional claim is not distinguishable from sampling noise, especially for models whose MCC values are within 0.1 of one another.
minor comments (5)
  1. [Abstract] Typographical spacing: 'V endorBench-100' appears in the abstract and should read 'VendorBench-100'.
  2. [Section 3.2] The phrase 'shipped 0.5 threshold' is used for all models. For commercial APIs and open-source checkpoints, the paper should clarify whether 0.5 is the model's native default or the study's common threshold; otherwise the phrase 'shipped default' is misleading.
  3. [Section 2.4 / Table 1] Several vision-LLM entries are underspecified: qwen, nemotron_nano_vl, zai_glm52, claude_opus48, and llama4_maverick lack exact version identifiers. Please add the precise model versions and collection dates so the results are reproducible.
  4. [Section 3.1.2] Six of 21 fake source groups are partially verified or deliberately opaque. The released provenance registry should include a per-image verification confidence or label-confidence score so readers can assess the impact of possible label noise on the reported metrics.
  5. [Section 6] The statement that score normalization was 'validated by cross-checking each vendor's documented output schema against a sample of raw responses' should report the sample size and the criteria used for validation; as written, the scope of the validation is not assessable.

Circularity Check

0 steps flagged

No circular derivation: all findings are point measurements of real model outputs under a disclosed protocol; the miscalibration finding is protocol-contingent but not constructed from its own inputs.

full rationale

This is an empirical benchmark report, not a derivation chain. No parameter is fitted to a subset and then re-predicted; no self-citations appear anywhere in the reference list (authors do not overlap with any cited work); no uniqueness theorem or ansatz is imported via citation; and the MCC-vs-AUC correlation (r≈0.86) and the always-fake 'accuracy trap' are computed directly from the confusion matrices and the 79:21 split, respectively. The closest thing to a constructed finding is the headline 'miscalibrated at their shipped default threshold' claim. For TruthScan the paper states it 'take[s] P(fake) = score/100 and use[s] this same value, thresholded at 0.5' while noting the API also returns 'a categorical final_result field' that is never used (Section 3.2); the 0.5 operating point is therefore the authors' protocol choice, and Section 5's attribution of this point to a 'shipped 0.5 threshold' is an interpretation rather than a measured vendor decision rule. Section 6 partially discloses this by admitting the normalization was 'validated by cross-checking each vendor's documented output schema against a sample of raw responses.' Because the confusion matrices themselves are genuine measurements (TruthScan's scores really do separate classes at AUC 0.923 while sitting mostly below 0.5), the finding is protocol-dependent validity risk, not circularity: the conclusion is not equivalent by construction to the input. Likewise, the always-fake demonstration (79% accuracy, 0.88 F1, 0.00 MCC) is a mathematical consequence of the corpus composition used as motivation and confirmed empirically on zai_glm52, dima806_ai_vs_real, and ateeqq_siglip2, so it is a restatement of a known statistical fact, not a renamed prediction. Score 2 reflects the minor protocol-contingent framing of the central 'shipped threshold' claim; no step reduces to its own input.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The benchmark depends on several load-bearing choices made by the authors: the shared 0.5 threshold, the 79:21 split, and the vendor output mappings. These are not fitted parameters in the usual sense, but they act as free choices that shape the results. The ground-truth label and anti-leakage premises are unproven but stated. No new entities are postulated.

free parameters (3)
  • Decision threshold = 0.5
    All 36 models are scored at the same 0.5 threshold on P(fake); this author-chosen threshold underpins every MCC and the miscalibration finding (Section 3.3).
  • Corpus class ratio = 79:21 fake:real
    The deliberately imbalanced split makes the accuracy trap possible and is a design choice, not a measured quantity (Section 3.1). Different splits would change the severity of the trap.
  • Vendor normalization mappings = Neural Defend: max over sub-check confidences; TruthScan: score/100
    Author-chosen conversions from native vendor schemas to the shared scale; results would change under different conversions (Section 3.2). Validated on a sample of responses only (Section 6).
axioms (3)
  • domain assumption MCC is the appropriate primary ranking metric for imbalanced detection benchmarks.
    The paper adopts MCC per the recommendation of Chicco & Jurman [78] and the always-fake argument in Section 3.3; this normative choice drives the leaderboard order.
  • domain assumption Ground-truth labels of the corpus are correct, including the 2 opaque and 4 partially verified fake source groups.
    All metrics are computed against these labels; the paper reports partial verification for 6 of 21 fake source groups (Section 3.1.2, Section 6).
  • domain assumption The anti-leakage protocol prevents models from reading label-bearing metadata.
    Neutral filenames are served (Section 3.1.2), but the paper cannot prove models ignore EXIF or other metadata; vision LLMs could in principle access such markers.

pith-pipeline@v1.3.0-alltime-deepseek · 21669 in / 12955 out tokens · 115679 ms · 2026-08-02T08:16:18.261540+00:00 · methodology

0 comments
read the original abstract

Deepfake image detection is served by three fundamentally different paradigms - commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors - that are rarely evaluated under a common protocol, making direct comparison difficult. We introduce VendorBench-100, a cross-paradigm benchmark that evaluates 36 representative models using a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework. Models are ranked primarily by the Matthews correlation coefficient (MCC), with ROC-AUC reported as a threshold-independent measure of ranking ability. Rather than maximizing size, it emphasizes real-world difficulty through a taxonomy of eight edge-case families such as face swaps, text-to-video stills, AI photo edits, avatar compositing, opaque-provenance images, and compressed research frames. Commercial APIs achieve the strongest median performance, followed by vision LLMs and open-source detectors, though individual open-source models remain competitive with the best LLMs. Across all 36 models, MCC and ROC-AUC are strongly correlated (Pearson r ~ 0.86); the more consequential finding is narrower and one-directional: a subset of otherwise strong rankers are miscalibrated at their shipped default threshold, so a high ROC-AUC can overstate real-world deployability. Separately, raw accuracy and F1 are unreliable on this corpus's imbalanced class split, since a model that predicts "fake" indiscriminately scores deceptively well on both while offering no real discriminative skill. No single metric is safe in isolation: MCC and specificity should always accompany ROC-AUC and accuracy. We release the complete evaluation framework and results. Code and data: https://github.com/sharayu-20/vendorbench-100

Figures

Figures reproduced from arXiv: 2607.06254 by Kurundkar G. D., Mahamune M. R., Md Rashidunnabi, Nelton Tiago Gemo, Nilesh K. Deshmukh, Sharayu N. Deshmukh.

Figure 1
Figure 1. Figure 1: Overview of VendorBench-100. Left: the accuracy trap, a naive always-fake predictor scores 79% accuracy [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: End-to-end evaluation architecture of VendorBench-100. Left: outputs from all three paradigms are [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of the 79 fake images across the eight edge-case families in VendorBench-100. The corpus [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Provenance verification statistics for the fake-image corpus. Left: verification status of the 21 source groups. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Unified leaderboard of all 36 evaluated models across the three benchmark tracks. Models are ranked by [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of MCC values across the three benchmark tracks. Commercial APIs show the highest median [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of MCC values across the three tracks. Commercial APIs show the highest median operating-point [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Latency comparison of the five commercial APIs. Left: mean inference latency on a logarithmic scale. Right: [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Sensitivity–specificity plane for all evaluated models. Models near the bottom-right corner correctly identify [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Relationship between operating-point quality (MCC) and threshold-independent ranking ability (ROC-AUC). [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 9
Figure 9. Figure 9: Relationship between operating-point quality (MCC) and ranking ability (ROC-AUC). Most of the 36 models [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Confusion composition for representative detectors. Each stacked bar shows how the 100 predictions are [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 10
Figure 10. Figure 10: Confusion composition for representative detectors. Each stacked bar splits the decided images into [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

107 extracted references · 13 linked inside Pith

  1. [1]

    Finance worker pays out $25 million after video call with deepfake ‘chief financial officer’

    Heather Chen and Kathleen Magramo. Finance worker pays out $25 million after video call with deepfake ‘chief financial officer’. CNN, February 2024

    2024

  2. [2]

    Berryessa

    Rebecca Umbach, Nicola Henry, Gemma Beard, and Colleen M. Berryessa. Non-consensual synthetic intimate imagery: Prevalence, attitudes, and knowledge in 10 countries. InProceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24), pages 1–20, 2024

    2024

  3. [3]

    How AI is being abused to create child sexual abuse imagery

    Internet Watch Foundation. How AI is being abused to create child sexual abuse imagery. Technical report, October 2023

    2023

  4. [4]

    Misinformation and elections: Provisional findings from four countries in 2024

    Vian Bakir et al. Misinformation and elections: Provisional findings from four countries in 2024. Technical report, London School of Economics and Political Science, 2025

    2024

  5. [5]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. Technical report, OpenAI, 2024

    2024

  6. [6]

    Gemini: A family of highly capable multimodal models

    Gemini Team, Google. Gemini: A family of highly capable multimodal models. Technical report, Google DeepMind, 2024

    2024

  7. [7]

    Drct: Diffusion reconstruction contrastive training towards universal detection of diffusion generated images

    Baoying Chen, Jishen Zeng, Jianquan Yang, and Rui Yang. Drct: Diffusion reconstruction contrastive training towards universal detection of diffusion generated images. InProceedings of the 41st International Conference on Machine Learning (ICML), 2024. ICML 2024 Spotlight. Code: https://github.com/beibuwandeluori/ DRCT

    2024

  8. [8]

    Leveraging representations from intermediate encoder-blocks for synthetic image detection

    Christos Koutlis and Symeon Papadopoulos. Leveraging representations from intermediate encoder-blocks for synthetic image detection. InProceedings of the European Conference on Computer Vision (ECCV), 2024. arXiv:2402.19091. Code:https://github.com/mever-team/rine. 18 APREPRINT- JULY30, 2026

    Pith/arXiv arXiv 2024

  9. [9]

    Deepfake-eval- 2024: A multi-modal in-the-wild benchmark of deepfakes circulated in 2024, 2025

    Nuria Alina Chandra, Ryan Murtfeldt, Lin Qiu, Arnab Karmakar, Hannah Lee, Emmanuel Tanumihardja, Kevin Farhat, Ben Caffee, Sejin Paik, Changyeon Lee, Jongwook Choi, Aerin Kim, and Oren Etzioni. Deepfake-eval- 2024: A multi-modal in-the-wild benchmark of deepfakes circulated in 2024, 2025

    2024

  10. [10]

    Tianyi Wang, Xin Liao, K. P. Chow, Xiaodong Lin, and Yinglong Wang. Deepfake detection: A comprehensive survey from the reliability perspective, 2022

    2022

  11. [11]

    FaceForensics++: Learning to detect manipulated facial images

    Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. FaceForensics++: Learning to detect manipulated facial images. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1–11, 2019

    2019

  12. [12]

    GenImage: A million-scale benchmark for detecting AI-generated image

    Mingjian Zhu, Hanting Chen, Qiangyu Yan, Xudong Huang, Guanyu Lin, Wei Li, Zhijun Tu, Hailin Hu, Jie Hu, and Yunhe Wang. GenImage: A million-scale benchmark for detecting AI-generated image. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  13. [13]

    The state of deepfakes: Landscape, threats, and impact

    Henry Ajder, Giorgio Patrini, Francesco Cavalli, Laetitia Cattivelli, Rebecca Jessop, Adam Falck, Emma Chagas, Sophie Miller, Alex Theophilus, and Mika Westerlund. The state of deepfakes: Landscape, threats, and impact. Technical report, Deeptrace (now Sensity AI), October 2019

    2019

  14. [14]

    The 2023 state of deepfakes: Realities, threats, and impact, 2023

    Home Security Heroes. The 2023 state of deepfakes: Realities, threats, and impact, 2023

    2023

  15. [15]

    Fincen alert on fraud schemes involving deepfake media targeting financial institutions

    Financial Crimes Enforcement Network (FinCEN). Fincen alert on fraud schemes involving deepfake media targeting financial institutions. Technical Report FIN-2024-Alert004, U.S. Department of the Treasury, November 2024

    2024

  16. [16]

    Global cybersecurity outlook 2025

    World Economic Forum. Global cybersecurity outlook 2025. Technical report, 2025

    2025

  17. [17]

    Children and deepfakes

    Mar Negreiro. Children and deepfakes. Technical Report PE 775.855, European Parliamentary Research Service, July 2025

    2025

  18. [18]

    Unmasking cybercrime: Strengthening digital identity verification against deepfakes, January 2026

    World Economic Forum Cybercrime Atlas. Unmasking cybercrime: Strengthening digital identity verification against deepfakes, January 2026

    2026

  19. [19]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. InAdvances in Neural Information Processing Systems, volume 27, pages 2672–2680, 2014

    2014

  20. [20]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4401–4410, June 2019

    2019

  21. [21]

    Analyzing and improving the image quality of StyleGAN

    Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8110–8119, June 2020

    2020

  22. [22]

    Jain, and Pieter Abbeel

    Jonathan Ho, Ajay N. Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. InAdvances in Neural Information Processing Systems, volume 33, pages 6840–6851, 2020

    2020

  23. [23]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022

    2022

  24. [24]

    Fast face-swap using convolutional neural networks

    Iryna Korshunova, Wenzhe Shi, Joni Dambre, and Lucas Theis. Fast face-swap using convolutional neural networks. InProceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3677–3685, Oct 2017

    2017

  25. [25]

    Face2face: Real-time face capture and reenactment of RGB videos

    Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of RGB videos. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2387–2395, June 2016

    2016

  26. [26]

    Deferred neural rendering: Image synthesis using neural textures.ACM Transactions on Graphics (TOG), 38(4):66:1–66:12, 2019

    Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures.ACM Transactions on Graphics (TOG), 38(4):66:1–66:12, 2019

    2019

  27. [27]

    Deepfakes: a new threat to face recognition? assessment and detection

    Pavel Korshunov and Sébastien Marcel. Deepfakes: a new threat to face recognition? assessment and detection. arXiv preprint arXiv:1812.08685, 2018

    Pith/arXiv arXiv 2018

  28. [28]

    Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A. Efros. CNN-Generated Images Are Surprisingly Easy to Spot... for Now. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8695–8704, 2020

    2020

  29. [29]

    Leveraging frequency analysis for deep fake image recognition

    Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. Leveraging frequency analysis for deep fake image recognition. InProceedings of the 37th International Conference on Machine Learning (ICML), volume 119, pages 3247–3258, 2020. 19 APREPRINT- JULY30, 2026

    2020

  30. [30]

    Watch your up-convolution: CNN based generative deep neural networks are failing to reproduce spectral distributions

    Ricard Durall, Margret Keuper, and Janis Keuper. Watch your up-convolution: CNN based generative deep neural networks are failing to reproduce spectral distributions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7893–7902, 2020

    2020

  31. [31]

    Towards universal fake image detectors that generalize across generative models

    Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards universal fake image detectors that generalize across generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24480–24489, 2023

    2023

  32. [32]

    Raising the bar of AI-generated image detection with CLIP.arXiv preprint arXiv:2312.00195, 2023

    Davide Cozzolino, Giovanni Poggi, Riccardo Corvi, Matthias Nießner, and Luisa Verdoliva. Raising the bar of AI-generated image detection with CLIP.arXiv preprint arXiv:2312.00195, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    DIRE for diffusion-generated image detection.arXiv preprint arXiv:2303.09295, 2023

    Zhendong Wang, Jianmin Bao, Wengang Zhou, Weilun Wang, Hezhen Hu, Hong Chen, and Houqiang Li. DIRE for diffusion-generated image detection.arXiv preprint arXiv:2303.09295, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    AEROBLADE: Training-free detection of latent diffusion images using autoencoder reconstruction error.arXiv preprint arXiv:2401.17879, 2024

    Jonas Ricker, Denis Lukovnikov, and Asja Fischer. AEROBLADE: Training-free detection of latent diffusion images using autoencoder reconstruction error.arXiv preprint arXiv:2401.17879, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    C2p-clip: Injecting cate- gory common prompt in clip to enhance generalization in deepfake detection.arXiv preprint arXiv:2408.09647,

    Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, Ping Liu, and Yunchao Wei. C2p-clip: Injecting cate- gory common prompt in clip to enhance generalization in deepfake detection.arXiv preprint arXiv:2408.09647,

    Pith/arXiv arXiv

  36. [36]

    FaceForensics: A large-scale video dataset for forgery detection in human faces

    Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. FaceForensics: A large-scale video dataset for forgery detection in human faces. 2018

    2018

  37. [37]

    The Deepfake detection challenge (DFDC) preview dataset.arXiv preprint arXiv:1910.08854, 2019

    Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, and Cristian Canton Ferrer. The Deepfake detection challenge (DFDC) preview dataset.arXiv preprint arXiv:1910.08854, 2019

    Pith/arXiv arXiv 1910

  38. [38]

    The DeepFake detection challenge (DFDC) dataset.arXiv preprint arXiv:2006.07397, 2020

    Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, Cristian Canton Ferrer, et al. The DeepFake detection challenge (DFDC) dataset.arXiv preprint arXiv:2006.07397, 2020

    Pith/arXiv arXiv 2006

  39. [39]

    WildDeepfake: A challenging real- world dataset for deepfake detection

    Bojia Zi, Minghao Chang, Jingjing Chen, Xingjun Ma, and Yu-Gang Jiang. WildDeepfake: A challenging real- world dataset for deepfake detection. InProceedings of the 28th ACM International Conference on Multimedia, pages 2382–2390, 2020

    2020

  40. [40]

    Celeb-DF: A large-scale challenging dataset for DeepFake forensics

    Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. Celeb-DF: A large-scale challenging dataset for DeepFake forensics. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3204–3213, 2020

    2020

  41. [41]

    ForgeryNet: A versatile benchmark for comprehensive forgery analysis

    Yinan He, Bei Gan, Siyu Chen, Yichun Zhou, Guojun Yin, Luchuan Song, Lu Sheng, Jing Shao, and Ziwei Liu. ForgeryNet: A versatile benchmark for comprehensive forgery analysis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4360–4369, 2021

    2021

  42. [42]

    DF40: Toward next-generation deepfake detection

    Zhiyuan Yan, Taiping Yao, Shen Chen, Yandan Zhao, Xinghe Fu, Junwei Zhu, Donghao Luo, Li Yuan, Chengjie Wang, Shouhong Ding, Yunsheng Wu, et al. DF40: Toward next-generation deepfake detection. InAdvances in Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2024

    2024

  43. [43]

    Deepfakebench: A comprehensive benchmark of deepfake detection

    Zhiyuan Yan, Yong Zhang, Xinhang Yuan, Siwei Lyu, and Baoyuan Wu. Deepfakebench: A comprehensive benchmark of deepfake detection. InAdvances in Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2023

    2023

  44. [44]

    Claude opus 4.8 model card, 2026

    Anthropic. Claude opus 4.8 model card, 2026. Model-card reference for claude_opus48; exact model-card URL not independently verified

    2026

  45. [45]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023. Cited as the Qwen vision-language family anchor; the exact Qwen version used forqwenis not specified i...

    Pith/arXiv arXiv 2023

  46. [46]

    Llama 4: Multimodal intelligence, openly available, 2025

    Meta AI. Llama 4: Multimodal intelligence, openly available, 2025. Reference for llama4_maverick; blog/model-card URL not independently verified

    2025

  47. [47]

    Nemotron: Nvidia’s family of open foundation models, 2024

    NVIDIA. Nemotron: Nvidia’s family of open foundation models, 2024. Reference for nemotron_nano_vl; exact model variant not independently verified

    2024

  48. [48]

    Glm: General language model family (zhipu ai / z.ai), 2024

    Zhipu AI. Glm: General language model family (zhipu ai / z.ai), 2024. Reference for zai_glm52; exact GLM-5.2 release details not independently verified

    2024

  49. [49]

    C2PA technical specification

    Coalition for Content Provenance and Authenticity. C2PA technical specification. Technical Specification 1.0, C2PA, December 2021

    2021

  50. [50]

    Content credentials: C2PA technical specification

    Coalition for Content Provenance and Authenticity (C2PA). Content credentials: C2PA technical specification. Technical specification v2.4, 2024. 20 APREPRINT- JULY30, 2026

    2024

  51. [51]

    Scalable watermarking for identifying large language model outputs.Nature, 634(8035):818–823, 2024

    Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, Jamie Hayes, Nidhi Vyas, Majd Al Merey, Jonah Brown-Cohen, Rudy Bunel, Borja Balle, Taylan Cemgil, Zahra Ahmed, Kitty Stacpoole, Ilia Shumailov, Ciprian Baetu, Sven Gowal, Demis Hassabis, and Pu...

    2024

  52. [52]

    Sven Gowal, Rudy Bunel, Florian Stimberg, David Stutz, Guillermo Ortiz-Jimenez, Christina Kouridi, Mel Vecerík, Jamie Hayes, Sylvestre-Alvise Rebuffi, Paul Bernard, Chris Gamble, Miklós Z. Horváth, Fabian Kaczmarczyck, Alex Kaskasoli, Aleksandar Petrov, Ilia Shumailov, Meghana Thotakuri, Olivia Wiles, Jessica Yung, Zahra Ahmed, Victor Martin, Simon Rosen,...

    arXiv 2025

  53. [53]

    RealAPI product documentation

    Reality Defender. RealAPI product documentation. https://www.realitydefender.com/product/ realapi, 2024

    2024

  54. [54]

    Ai-generated image and video detection documentation

    Hive AI. Ai-generated image and video detection documentation. https://docs.thehive.ai/docs/ ai-image-and-video-detection, 2024

    2024

  55. [55]

    Ai-generated image detection API documentation

    Sightengine. Ai-generated image detection API documentation. https://sightengine.com/docs/ ai-generated-image-detection, 2024

    2024

  56. [56]

    Ai image detection API documentation

    TruthScan. Ai image detection API documentation. https://truthscan.com/ truthscan-ai-image-detection-api-documentation, 2024

    2024

  57. [57]

    Ai generated image detection (DeepScan API)

    Neural Defend. Ai generated image detection (DeepScan API). https://neuraldefend.gitbook.io/ neural-defend/ai-generated-image-detection, 2024

    2024

  58. [58]

    Deepfake detection platform.https://sensity.ai/, 2024

    Sensity AI. Deepfake detection platform.https://sensity.ai/, 2024

    2024

  59. [59]

    Ai generated image detection

    Illuminarty. Ai generated image detection. https://www.illuminarty.ai/en/image/ ai-generated-image-detection.html, 2024

    2024

  60. [60]

    Image detection API reference

    AI or Not. Image detection API reference. https://docs.aiornot.com/api-reference/ reports-by-modality/image.md, 2024

    2024

  61. [61]

    Intel introduces real-time deepfake detector (FakeCatcher)

    Intel Corporation. Intel introduces real-time deepfake detector (FakeCatcher). https://newsroom.intel. com/artificial-intelligence/intel-introduces-real-time-deepfake-detector, 2022

    2022

  62. [62]

    Poskitt, and Xingmei Wang

    Boquan Li, Jun Sun, Christopher M. Poskitt, and Xingmei Wang. How generalizable are deepfake image detectors? an empirical study, 2023

    2023

  63. [63]

    Assessment framework for deepfake detection in real-world situations, 2023

    Yuhang Lu and Touradj Ebrahimi. Assessment framework for deepfake detection in real-world situations, 2023

    2023

  64. [64]

    Fake or JPEG? revealing common biases in generated image detection datasets, 2024

    Patrick Grommelt, Louis Weiss, Franz-Josef Pfreundt, and Janis Keuper. Fake or JPEG? revealing common biases in generated image detection datasets, 2024

    2024

  65. [65]

    A sanity check for AI-generated image detection

    Shilin Yan, Ouxiang Li, Jiayin Cai, Yanbin Hao, Xiaolong Jiang, Yao Hu, and Weidi Xie. A sanity check for AI-generated image detection. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  66. [66]

    The adversarial AI-Art: Understanding, generation, detection, and benchmarking

    Fang Li et al. The adversarial AI-Art: Understanding, generation, detection, and benchmarking. https: //arxiv.org/abs/2404.14581, 2024

    Pith/arXiv arXiv 2024

  67. [67]

    Organic or diffused: Can we distinguish human art from AI-generated images?https://doi.org/10.1145/3658644.3670306, 2024

    Anna Yoo Jeong Ha, Josephine Passananti, et al. Organic or diffused: Can we distinguish human art from AI-generated images?https://doi.org/10.1145/3658644.3670306, 2024

    arXiv 2024

  68. [68]

    The visual counter turing test (VCT2): A benchmark for evaluating AI-Generated image detection and the visual AI index.https://arxiv.org/abs/2411.16754, 2024

    Nasrin Imanpour Chakraborty et al. The visual counter turing test (VCT2): A benchmark for evaluating AI-Generated image detection and the visual AI index.https://arxiv.org/abs/2411.16754, 2024

    arXiv 2024

  69. [69]

    Ai-genbench: A new ongoing benchmark for ai-generated image detection

    Lorenzo Pellegrini, Davide Cozzolino, Serafino Pandolfini, Davide Maltoni, Matteo Ferrara, Luisa Verdoliva, Marco Prati, and Marco Ramilli. Ai-genbench: A new ongoing benchmark for ai-generated image detection. In 2025 International Joint Conference on Neural Networks (IJCNN), pages 1–9, 2025

    2025

  70. [70]

    A. S. Iufereva and Peter Mozelius. Deepfake detection: Human performance versus AI tools – a comparison of accuracy and effectiveness. InProceedings of the International Conference on AI Research (ICAIR), 2025

    2025

  71. [71]

    Adversarial deepfakes: Evaluating vulnerability of deepfake detectors to adversarial examples

    Shehzeen Hussain, Paarth Neekhara, Malhar Jere, Farinaz Koushanfar, and Julian McAuley. Adversarial deepfakes: Evaluating vulnerability of deepfake detectors to adversarial examples. InIEEE Winter Conference on Applications of Computer Vision (WACV), pages 3348–3357, 2021

    2021

  72. [72]

    Chee K. Chow. On optimum recognition error and reject tradeoff.IEEE Transactions on Information Theory, 16(1):41–46, 1970

    1970

  73. [73]

    On the foundations of noise-free selective classification.Journal of Machine Learning Research, 11(53):1605–1641, 2010

    Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification.Journal of Machine Learning Research, 11(53):1605–1641, 2010. 21 APREPRINT- JULY30, 2026

    2010

  74. [74]

    Selective classification for deep neural networks

    Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. InAdvances in Neural Information Processing Systems (NeurIPS), volume 30, 2017

    2017

  75. [75]

    SelectiveNet: A deep neural network with an integrated reject option

    Yonatan Geifman and Ran El-Yaniv. SelectiveNet: A deep neural network with an integrated reject option. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 2151–2159, 2019

    2019

  76. [76]

    Learning and evaluating classifiers under sample selection bias

    Bianca Zadrozny. Learning and evaluating classifiers under sample selection bias. InProceedings of the 21st International Conference on Machine Learning (ICML), pages 903–910, 2004

    2004

  77. [77]

    Counterfactually comparing abstaining classifiers

    Yo Joong Choe, Aditya Gangrade, and Aaditya Ramdas. Counterfactually comparing abstaining classifiers. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023

    2023

  78. [78]

    The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation.BMC Genomics, 21(1):6, 2020

    Davide Chicco and Giuseppe Jurman. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation.BMC Genomics, 21(1):6, 2020

    2020

  79. [79]

    Brodersen, Cheng Soon Ong, Klaas E

    Kay H. Brodersen, Cheng Soon Ong, Klaas E. Stephan, and Joachim M. Buhmann. The balanced accuracy and its posterior distribution. InProceedings of the 20th International Conference on Pattern Recognition (ICPR), pages 3121–3124, 2010

    2010

  80. [80]

    A systematic analysis of performance measures for classification tasks

    Marina Sokolova and Guy Lapalme. A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4):427–437, 2009

    2009

Showing first 80 references.