The reviewed record of science sign in
Pith

arxiv: 2607.06254 · v1 · pith:2GYWQ4FW · submitted 2026-07-07 · cs.CV · cs.AI

VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

Sharayu N. Deshmukh , Md Rashidunnabi , Nelton Tiago Gemo , Kurundkar G. D. , Mahamune M. R. , Nilesh K. Deshmukh This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 11:47 UTCglm-5.2pith:2GYWQ4FWrecord.jsonopen to challenge →

classification cs.CV cs.AI
keywords benchmarkmodelsvendorbench-100evaluationimagellmsopen-sourceranking
0
0 comments X

The pith

Best deepfake detector by AUC ranks 32nd by accuracy

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

VendorBench-100 evaluates 36 deepfake image detectors across three paradigms—commercial APIs, zero-shot vision-language models, and open-source checkpoints—on a single 100-image adversarial corpus. The benchmark ranks models by the Matthews correlation coefficient (MCC), which measures decision quality at a model's default threshold, with ROC-AUC as a threshold-free tiebreak. The central finding is that ranking power (ROC-AUC) and operating-point quality (MCC) can diverge sharply and in opposite directions across all three paradigms. TruthScan achieves the highest ROC-AUC (0.915) yet posts a negative MCC (−0.130), ranking 32nd of 36, because its default threshold flags essentially every image as fake. Neural Defend shows the mirror pattern: best MCC (0.876) with near-chance ROC-AUC (0.516). The paper argues that any single-metric leaderboard systematically mischaracterizes at least one real failure or success mode for deployed detectors.

Core claim

The paper identifies a consistent divergence between a detector's ability to rank fake images above real ones (ROC-AUC) and its ability to make correct decisions at its shipped default threshold (MCC). This divergence is demonstrated through two anchor cases operating at opposite extremes: TruthScan separates the classes well in principle but defaults to predicting fake on nearly every input, while Neural Defend makes excellent hard-label decisions from confidence scores that barely separate the two classes. The paper shows this is not an isolated artifact but recurs across commercial APIs, vision LLMs, and open-source detectors, and that the class imbalance (79 fake, 21 real) creates a base

What carries the argument

Dual-metric evaluation protocol pairing MCC (operating-point quality at default threshold) with ROC-AUC (threshold-free ranking ability), applied uniformly across 36 models normalized into a shared prediction schema.

If this is right

  • Any deepfake detection benchmark reporting only accuracy, F1, or ROC-AUC risks hiding a default-threshold failure mode where a model flags nearly every image as fake yet appears competent.
  • Procurement decisions for detection systems should evaluate both ranking power and operating-point quality side by side, since optimizing either alone can select a model that fails in the other dimension.
  • Calibrated per-model decision thresholds, rather than shipped defaults, could substantially improve deployed detector reliability without changing the underlying discriminative signal.
  • The finding that a single open-source detector (DRCT) outperforms all vision LLMs on ranking power suggests that careful academic detector design remains competitive with expensive general-purpose foundation models for this task.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the MCC-AUC divergence is structural rather than noise, then threshold calibration—recalibrating each model's decision boundary on a held-out validation set—should close much of the gap between ranking power and operating-point quality, potentially flattening the leaderboard significantly.
  • The base-rate exploitation pathology (predicting fake on everything) may be an adaptive response by model developers to deployment environments that are also heavily fake-skewed; a detector that defaults to fake might be intentionally tuned for high-prevalence settings rather than being miscalibrated in absolute terms.
  • The finding could extend to other binary detection tasks with class imbalance—malware detection, spam filtering, medical screening—where single-metric leaderboards may similarly mask threshold-calibration failures.

Load-bearing premise

The 100-image corpus (79 fake, 21 real) is assumed sufficient to support cross-paradigm conclusions about detector behavior. With only 21 real images, a single misclassification shifts specificity by roughly 4.8 percentage points, making per-model MCC estimates volatile and the specific magnitudes of the AUC-MCC divergence difficult to distinguish from sampling noise without confidence intervals or repeated runs.

What would settle it

Run the same 36 models on a larger, balanced corpus with bootstrap confidence intervals. If the MCC-AUC divergence for TruthScan and Neural Defend shrinks to within noise of agreement, the structural claim weakens to a small-sample artifact.

Figures

Figures reproduced from arXiv: 2607.06254 by Kurundkar G. D., Mahamune M. R., Md Rashidunnabi, Nelton Tiago Gemo, Nilesh K. Deshmukh, Sharayu N. Deshmukh.

Figure 1
Figure 1. Figure 1: Overview of VendorBench-100. Left: the accuracy trap, a naive always-fake predictor scores 79% accuracy [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: End-to-end evaluation architecture of VendorBench-100. Left: outputs from all three paradigms are [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of the 79 fake images across the eight edge-case families in VendorBench-100. The corpus [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Provenance verification statistics for the fake-image corpus. Left: verification status of the 21 source groups. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Unified leaderboard of all 36 evaluated models across the three benchmark tracks. Models are ranked by [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of MCC values across the three benchmark tracks. Commercial APIs show the highest median [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Latency comparison of the five commercial APIs. Left: mean inference latency on a logarithmic scale. Right: [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Sensitivity–specificity plane for all evaluated models. Models near the bottom-right corner correctly identify [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Relationship between operating-point quality (MCC) and threshold-independent ranking ability (ROC-AUC). [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Confusion composition for representative detectors. Each stacked bar shows how the 100 predictions are [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
read the original abstract

Deepfake image detection is currently served by three fundamentally different paradigms: commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors. Despite their widespread use, these paradigms are rarely evaluated under a common protocol, making direct comparison difficult. We introduce VendorBench-100, a cross-paradigm benchmark that evaluates 36 representative models using a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework. To ensure reliable assessment under the corpus's intentional class imbalance, models are ranked primarily by the Matthews correlation coefficient (MCC), with ROC-AUC reported as a threshold-independent measure of ranking ability. Rather than maximizing dataset size, VendorBench-100 emphasizes challenging real-world scenarios through a curated taxonomy of eight edge-case families, including face swaps, text-to-video stills, AI photo edits, avatar compositing, opaque-provenance images, and compressed research frames. Our evaluation shows that commercial APIs achieve the strongest median performance, followed by vision LLMs and open-source detectors. However, individual open-source models remain competitive with the best vision LLMs. More importantly, we identify a consistent divergence between ranking ability (ROC-AUC) and operating-point quality (MCC), demonstrating that strong score discrimination does not necessarily produce reliable default-threshold decisions. This metric disagreement, rather than any single leaderboard ranking, is the central finding of the benchmark. We release the complete evaluation framework and benchmark results to support reproducible future research. The source code and data are available at: https://github.com/sharayu-20/vendorbench-100

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 6 minor

Summary. The paper introduces VendorBench-100, a cross-paradigm benchmark evaluating 36 deepfake image detectors—5 commercial APIs, 7 zero-shot vision LLMs, and 24 open-source models—on a single fixed 100-image adversarial corpus (79 fake, 21 real). The corpus is organized around eight edge-case families (face swaps, text-to-video stills, AI photo edits, avatar compositing, etc.) and all models are evaluated under a unified output schema with MCC as the primary ranking metric and ROC-AUC as a threshold-free tiebreak. The central empirical finding is a divergence between ranking ability (ROC-AUC) and operating-point quality (MCC): TruthScan achieves the highest ROC-AUC (0.915) yet has negative MCC (−0.130) because its default threshold flags all 21 real images as fake, while Neural Defend shows the mirror pattern (best MCC 0.876, near-chance AUC 0.516). The paper releases code, data, and per-image evidence.

Significance. The cross-paradigm design—placing commercial APIs, vision LLMs, and open-source detectors on common ground—is a genuine contribution to the deepfake detection evaluation literature, which has largely evaluated these paradigms in isolation. The MCC-versus-AUC divergence insight is well-motivated and practically actionable for practitioners making procurement decisions. The release of the evaluation harness, per-image provenance registry, and complete results on GitHub is a strength for reproducibility. The anti-leakage protocol (neutral filenames, separate ground-truth manifest) is a thoughtful design choice. However, the significance of the specific quantitative findings is substantially limited by the corpus size, as detailed below.

major comments (3)
  1. §5, 'MCC and ROC-AUC Diverge' subsection, and Table 1: The central finding—that ranking power and operating-point quality 'can point in opposite directions'—is demonstrated most concretely by TruthScan (AUC 0.915, MCC −0.130, specificity 0/21) and Neural Defend (MCC 0.876, AUC 0.516). With only 21 real images, a single reclassification shifts specificity by ~4.8 percentage points. For TruthScan specifically, if just 2 of 21 real images were correctly classified (TN=2, FP=19), recomputing Eq. (6) yields MCC ≈ +0.03, which would eliminate the 'negative MCC despite high AUC' headline. The paper acknowledges this limitation in §6 ('the corpus is small... without confidence intervals'), but §5's language ('consistent,' 'recurring,' 'cuts across all three paradigms') overstates what 21 real images can support. The structural insight that AUC and threshold-dependent MCC can diverge is sound and
  2. §5 and Table 1: The claim that the divergence 'cuts across all three paradigms' is supported primarily by two commercial APIs (TruthScan, Neural Defend) and one vision LLM (zai_glm52, specificity 0/21, MCC −0.152). Among open-source detectors, the AUC-MCC divergence is less dramatic: the strongest open-source ranker (DRCT, AUC 0.866) has MCC 0.230—positive but modest, consistent with a conservative threshold rather than a qualitative divergence. The paper should either (a) provide per-paradigm examples of the divergence that are comparably dramatic to TruthScan/Neural Defend, or (b) qualify the claim to note that the most striking divergences are concentrated in specific commercial and LLM models rather than being uniformly distributed across all three paradigms.
  3. §3.3 and §5: No uncertainty quantification is provided for any reported metric. The paper states in §3.3 that pairwise significance testing across 630 comparisons was not performed, and §6 acknowledges the absence of bootstrap confidence intervals. Given that the paper's contribution is empirical (benchmark results and model rankings), the absence of any uncertainty estimate on the headline MCC and AUC values is a load-bearing gap. At minimum, bootstrap confidence intervals on MCC for the key models (TruthScan, Neural Defend, and the top-5 by MCC) would allow readers to assess whether the reported rankings are distinguishable from sampling noise. This is feasible without acquiring a larger corpus and would substantially strengthen the paper's claims.
minor comments (6)
  1. Table 1: The model identifiers are inconsistent in formatting (e.g., 'neural_defend' vs 'reality_defender' vs 'ntire2026_deepfake' vs 'dima806_ai_vs_real'). A brief note on naming conventions or consistent formatting would improve readability.
  2. §3.2, Vision LLMs paragraph: Results were 'collected out-of-band, some via vendor API and some via browser automation.' The paper acknowledges this as a limitation (§6) but does not specify which models used which method. This information should be included, as browser automation may introduce rendering or timing artifacts that differ from API access.
  3. Figure 2: The MCC value for TruthScan is rendered as '˘22120.13' which appears to be a formatting artifact. It should read '−0.13.' Please fix.
  4. §3.1.1: The edge-case taxonomy lists eight families but the abstract mentions only six by name. A complete enumeration in the abstract or a cross-reference would help readers navigate the taxonomy.
  5. References [44], [46], [47], [48]: Several model references note that 'exact model variant not independently verified' or 'model-card URL not independently verified.' For a benchmark paper, exact model versions are essential for reproducibility. Please verify and specify the exact model versions/commit hashes used.
  6. §4.3, Table 3: Latency is reported as a single mean per provider. The number of requests averaged and any variance measure (std or IQR) should be included, as latency for commercial APIs can be highly variable.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for a careful and constructive report. The referee correctly identifies the cross-paradigm design, the MCC-versus-AUC divergence insight, and the reproducibility artifacts as the paper's main contributions. The three major comments all concern the statistical fragility of results drawn from a 100-image corpus with only 21 real images. We agree with the substance of all three comments and will revise accordingly: (1) we will soften the language in Section 5 to avoid overstating what 21 real images can support and will add explicit sensitivity analysis; (2) we will qualify the 'cuts across all three paradigms' claim to acknowledge that the most dramatic divergences are concentrated in specific commercial and LLM models; (3) we will add bootstrap confidence intervals for key models. We cannot, within this revision cycle, expand the corpus itself, which is the one standing limitation.

read point-by-point responses

  1. Referee: §5, 'MCC and ROC-AUC Diverge' subsection, and Table 1: With only 21 real images, a single reclassification shifts specificity by ~4.8 percentage points. For TruthScan, if just 2 of 21 real images were correctly classified (TN=2, FP=19), recomputing Eq. (6) yields MCC ≈ +0.03, which would eliminate the 'negative MCC despite high AUC' headline. The paper acknowledges this limitation in §6, but §5's language ('consistent,' 'recurring,' 'cuts across all three paradigms') overstates what 21 real images can support.

    Authors: The referee's arithmetic is correct, and we accept this point. With 21 real images, each reclassification shifts specificity by approximately 4.8 percentage points, and the TruthScan headline (AUC 0.915, MCC −0.130) is indeed sensitive to the classification of just two real images. We will make three concrete changes in the revision. First, we will add an explicit sensitivity analysis showing how TruthScan's MCC changes under small perturbations of its real-image classifications (TN=0 through TN=4), making the fragility of the specific point estimate transparent to the reader. Second, we will soften the language in Section 5: 'consistent' will become 'observed,' 'recurring' will become 'present in multiple models,' and we will remove language that implies the divergence is robustly established by this corpus alone. Third, we will reframe the central claim as a structural observation—that AUC and threshold-dependent MCC *can* diverge, with TruthScan and Neural Defend as concrete illustrations—rather than as a statistically robust empirical finding. The structural argument (that ranking ability and operating-point quality measure different things and therefore *can* disagree) holds regardless of corpus size, but the referee is right that our current language implies a stronger empirical claim than 21 real images support. revision: yes

  2. Referee: §5 and Table 1: The claim that the divergence 'cuts across all three paradigms' is supported primarily by two commercial APIs (TruthScan, Neural Defend) and one vision LLM (zai_glm52, specificity 0/21, MCC −0.152). Among open-source detectors, the AUC-MCC divergence is less dramatic: the strongest open-source ranker (DRCT, AUC 0.866) has MCC 0.230—positive but modest, consistent with a conservative threshold rather than a qualitative divergence. The paper should either (a) provide per-paradigm examples of the divergence that are comparably dramatic to TruthScan/Neural Defend, or (b) qualify the claim to note that the most striking divergences are concentrated in specific commercial and LLM models rather than being uniformly distributed across all three paradigms.

    Authors: We accept this point and will adopt option (b). The referee is correct that the most dramatic divergences—TruthScan (AUC 0.915, MCC −0.130) and Neural Defend (AUC 0.516, MCC 0.876)—are concentrated in the commercial track, with zai_glm52 (AUC 0.470, MCC −0.152) as the LLM-track instance. Among open-source detectors, the divergence is qualitatively different: DRCT (AUC 0.866, MCC 0.230) shows a positive-but-modest MCC consistent with a conservative threshold rather than a sign reversal. We do not have a comparably dramatic open-source example of AUC-MCC divergence in the current results, and we will not claim one. In the revision, we will qualify the 'cuts across all three paradigms' language to state that the divergence *is present* in all three paradigms but that the most striking instances are concentrated in specific commercial and LLM models. We will also note explicitly that the open-source track exhibits a milder form of the same phenomenon—conservative thresholds suppressing MCC despite reasonable ranking power—rather than the qualitative sign-reversal seen in TruthScan. revision: yes

  3. Referee: §3.3 and §5: No uncertainty quantification is provided for any reported metric. The paper states in §3.3 that pairwise significance testing across 630 comparisons was not performed, and §6 acknowledges the absence of bootstrap confidence intervals. Given that the paper's contribution is empirical (benchmark results and model rankings), the absence of any uncertainty estimate on the headline MCC and AUC values is a load-bearing gap. At minimum, bootstrap confidence intervals on MCC for the key models (TruthScan, Neural Defend, and the top-5 by MCC) would allow readers to assess whether the reported rankings are distinguishable from sampling noise.

    Authors: We agree that the absence of uncertainty quantification is a load-bearing gap, and we will add bootstrap confidence intervals in the revision. Specifically, we will compute nonparametric bootstrap CIs (10,000 resamples) for MCC and ROC-AUC on the key models the referee identifies: TruthScan, Neural Defend, and the top-5 models by MCC. We will report these CIs in an expanded Table 1 (or a supplementary table) and will discuss in Section 5 whether the headline rankings are distinguishable from sampling noise. We expect that some pairwise rankings—particularly among models with close MCC values in the middle of the leaderboard—will not be statistically distinguishable, and we will state this plainly. We will not attempt pairwise significance testing across all 630 comparisons, as the multiple-comparisons problem on a 100-image corpus would render most such tests uninformative; however, the bootstrap CIs on key models are feasible without new data collection and will directly address the referee's concern. revision: yes

standing simulated objections not resolved
  • We cannot expand the corpus beyond 100 images within this revision cycle. The corpus was hand-curated with per-image provenance verification across 21 source groups, and expanding it while maintaining the same quality of provenance documentation and adversarial difficulty requires substantial new data collection and verification effort. The referee's concern about corpus size is valid, and we address it through sensitivity analysis, language qualification, and bootstrap CIs, but a larger corpus is explicitly listed as future work in Section 6 and cannot be completed for this revision.

Circularity Check

0 steps flagged

No circularity: standard benchmark paper with externally constructed corpus and textbook metrics

full rationale

VendorBench-100 is an evaluation/benchmark paper with no derivation chain to audit for circularity. The metrics (Eqs. 1–6: accuracy, precision, recall, specificity, F1, MCC) are standard textbook formulas applied directly to confusion-matrix counts; none is defined in terms of the paper's own outputs. The 100-image corpus is externally hand-curated and not fitted to any model's parameters. No self-citations appear in the reference list—the authors (Deshmukh, Rashidunnabi, Gemo, Kurundkar, Mahamune, Deshmukh) do not cite their own prior work as load-bearing evidence. The central finding (MCC–AUC divergence) is an empirical observation about specific models' behavior on the corpus, not a derived result that could reduce to its inputs by construction. The paper is transparent about its statistical limitations (§6: small corpus, single run, no significance testing), but those are correctness/statistical-power concerns, not circularity. The derivation chain is self-contained against external benchmarks, so the circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new entities are postulated. The paper is an empirical evaluation.

free parameters (3)
  • Corpus composition (100 images, 79:21 split) = 79 fake, 21 real across 8 edge-case families
    The specific 100 images were hand-selected by the authors to be adversarial. The class ratio, category distribution, and image selection are all author choices that directly determine every reported metric.
  • Decision threshold (0.5 on P(fake)) = 0.5
    All threshold-dependent metrics (MCC, accuracy, F1, precision, recall, specificity) are computed at the standard 0.5 threshold. This is a conventional choice but is not derived from the data; it directly affects the MCC-AUC divergence finding since a different threshold could reduce or eliminate the divergence for specific models.
  • Shared forensic verdict prompt for vision LLMs = Shared schema requesting status, AI-manipulated judgment, and probability estimate
    The specific prompt used for all 7 vision LLMs is an author-designed artifact. Different prompts could produce different zero-shot performance, and the prompt was not tuned per model.
axioms (4)
  • domain assumption The 100-image corpus is representative of real-world adversarial deepfake scenarios despite its small size.
    Invoked throughout §3.1 and §5. The paper claims the corpus probes 'distinct failure modes' and reflects 'the difficult, modern middle ground.' Every cross-paradigm conclusion depends on this representativeness, which is asserted rather than tested.
  • domain assumption A single run per model produces stable enough estimates to support qualitative cross-paradigm conclusions.
    Invoked implicitly in §4 where all results are single-run point estimates. The paper acknowledges in §6 that no multi-seed runs or bootstrap CIs were computed, but still draws qualitative conclusions about paradigm-level ordering and metric divergence patterns.
  • domain assumption The 0.5 threshold on normalized P(fake) is the appropriate operating point for evaluating all models.
    Invoked in §3.3 where all threshold-dependent metrics use the 0.5 cutoff. The MCC-AUC divergence finding is partly an artifact of this specific threshold choice; models with poor calibration at 0.5 might perform well at a tuned threshold.
  • ad hoc to paper Vision LLM outputs collected via different methods (API vs browser automation) are comparable.
    §3.2 states LLM results were 'collected out-of-band, some via vendor API and some via browser automation.' The paper treats these as comparable despite different collection methods potentially introducing systematic differences in model behavior.

pith-pipeline@v1.1.0-glm · 24191 in / 4132 out tokens · 282256 ms · 2026-07-08T11:47:25.712394+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

107 extracted references · 107 canonical work pages · 13 internal anchors

  1. [1]

    Finance worker pays out $25 million after video call with deepfake ‘chief financial officer’

    Heather Chen and Kathleen Magramo. Finance worker pays out $25 million after video call with deepfake ‘chief financial officer’. CNN, February 2024

    work page 2024

  2. [2]

    Berryessa

    Rebecca Umbach, Nicola Henry, Gemma Beard, and Colleen M. Berryessa. Non-consensual synthetic intimate imagery: Prevalence, attitudes, and knowledge in 10 countries. InProceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24), pages 1–20, 2024

    work page 2024

  3. [3]

    How AI is being abused to create child sexual abuse imagery

    Internet Watch Foundation. How AI is being abused to create child sexual abuse imagery. Technical report, October 2023. 17 APREPRINT- JULY8, 2026

    work page 2023

  4. [4]

    Misinformation and elections: Provisional findings from four countries in 2024

    Vian Bakir et al. Misinformation and elections: Provisional findings from four countries in 2024. Technical report, London School of Economics and Political Science, 2025

    work page 2024

  5. [5]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. Technical report, OpenAI, 2024

    work page 2024

  6. [6]

    Gemini: A family of highly capable multimodal models

    Gemini Team, Google. Gemini: A family of highly capable multimodal models. Technical report, Google DeepMind, 2024

    work page 2024

  7. [7]

    Drct: Diffusion reconstruction contrastive training towards universal detection of diffusion generated images

    Baoying Chen, Jishen Zeng, Jianquan Yang, and Rui Yang. Drct: Diffusion reconstruction contrastive training towards universal detection of diffusion generated images. InProceedings of the 41st International Conference on Machine Learning (ICML), 2024. ICML 2024 Spotlight. Code: https://github.com/beibuwandeluori/ DRCT

    work page 2024

  8. [8]

    Leveraging Representations from Intermediate Encoder-blocks for Synthetic Image Detection

    Christos Koutlis and Symeon Papadopoulos. Leveraging representations from intermediate encoder-blocks for synthetic image detection. InProceedings of the European Conference on Computer Vision (ECCV), 2024. arXiv:2402.19091. Code:https://github.com/mever-team/rine

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    Deepfake-eval- 2024: A multi-modal in-the-wild benchmark of deepfakes circulated in 2024, 2025

    Nuria Alina Chandra, Ryan Murtfeldt, Lin Qiu, Arnab Karmakar, Hannah Lee, Emmanuel Tanumihardja, Kevin Farhat, Ben Caffee, Sejin Paik, Changyeon Lee, Jongwook Choi, Aerin Kim, and Oren Etzioni. Deepfake-eval- 2024: A multi-modal in-the-wild benchmark of deepfakes circulated in 2024, 2025

    work page 2024

  10. [10]

    Tianyi Wang, Xin Liao, K. P. Chow, Xiaodong Lin, and Yinglong Wang. Deepfake detection: A comprehensive survey from the reliability perspective, 2022

    work page 2022

  11. [11]

    FaceForensics++: Learning to detect manipulated facial images

    Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. FaceForensics++: Learning to detect manipulated facial images. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1–11, 2019

    work page 2019

  12. [12]

    GenImage: A million-scale benchmark for detecting AI-generated image

    Mingjian Zhu, Hanting Chen, Qiangyu Yan, Xudong Huang, Guanyu Lin, Wei Li, Zhijun Tu, Hailin Hu, Jie Hu, and Yunhe Wang. GenImage: A million-scale benchmark for detecting AI-generated image. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    work page 2023

  13. [13]

    The state of deepfakes: Landscape, threats, and impact

    Henry Ajder, Giorgio Patrini, Francesco Cavalli, Laetitia Cattivelli, Rebecca Jessop, Adam Falck, Emma Chagas, Sophie Miller, Alex Theophilus, and Mika Westerlund. The state of deepfakes: Landscape, threats, and impact. Technical report, Deeptrace (now Sensity AI), October 2019

    work page 2019

  14. [14]

    The 2023 state of deepfakes: Realities, threats, and impact, 2023

    Home Security Heroes. The 2023 state of deepfakes: Realities, threats, and impact, 2023

    work page 2023

  15. [15]

    Fincen alert on fraud schemes involving deepfake media targeting financial institutions

    Financial Crimes Enforcement Network (FinCEN). Fincen alert on fraud schemes involving deepfake media targeting financial institutions. Technical Report FIN-2024-Alert004, U.S. Department of the Treasury, November 2024

    work page 2024

  16. [16]

    Global cybersecurity outlook 2025

    World Economic Forum. Global cybersecurity outlook 2025. Technical report, 2025

    work page 2025

  17. [17]

    Children and deepfakes

    Mar Negreiro. Children and deepfakes. Technical Report PE 775.855, European Parliamentary Research Service, July 2025

    work page 2025

  18. [18]

    Unmasking cybercrime: Strengthening digital identity verification against deepfakes, January 2026

    World Economic Forum Cybercrime Atlas. Unmasking cybercrime: Strengthening digital identity verification against deepfakes, January 2026

    work page 2026

  19. [19]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. InAdvances in Neural Information Processing Systems, volume 27, pages 2672–2680, 2014

    work page 2014

  20. [20]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4401–4410, June 2019

    work page 2019

  21. [21]

    Analyzing and improving the image quality of StyleGAN

    Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8110–8119, June 2020

    work page 2020

  22. [22]

    Jain, and Pieter Abbeel

    Jonathan Ho, Ajay N. Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. InAdvances in Neural Information Processing Systems, volume 33, pages 6840–6851, 2020

    work page 2020

  23. [23]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022

    work page 2022

  24. [24]

    Fast face-swap using convolutional neural networks

    Iryna Korshunova, Wenzhe Shi, Joni Dambre, and Lucas Theis. Fast face-swap using convolutional neural networks. InProceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3677–3685, Oct 2017. 18 APREPRINT- JULY8, 2026

    work page 2017

  25. [25]

    Face2face: Real-time face capture and reenactment of RGB videos

    Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of RGB videos. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2387–2395, June 2016

    work page 2016

  26. [26]

    Deferred neural rendering: Image synthesis using neural textures.ACM Transactions on Graphics (TOG), 38(4):66:1–66:12, 2019

    Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures.ACM Transactions on Graphics (TOG), 38(4):66:1–66:12, 2019

    work page 2019

  27. [27]

    DeepFakes: a New Threat to Face Recognition? Assessment and Detection

    Pavel Korshunov and Sébastien Marcel. Deepfakes: a new threat to face recognition? assessment and detection. arXiv preprint arXiv:1812.08685, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  28. [28]

    Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A. Efros. CNN-Generated Images Are Surprisingly Easy to Spot... for Now. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8695–8704, 2020

    work page 2020

  29. [29]

    Leveraging frequency analysis for deep fake image recognition

    Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. Leveraging frequency analysis for deep fake image recognition. InProceedings of the 37th International Conference on Machine Learning (ICML), volume 119, pages 3247–3258, 2020

    work page 2020

  30. [30]

    Watch your up-convolution: CNN based generative deep neural networks are failing to reproduce spectral distributions

    Ricard Durall, Margret Keuper, and Janis Keuper. Watch your up-convolution: CNN based generative deep neural networks are failing to reproduce spectral distributions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7893–7902, 2020

    work page 2020

  31. [31]

    Towards universal fake image detectors that generalize across generative models

    Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards universal fake image detectors that generalize across generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24480–24489, 2023

    work page 2023

  32. [32]

    Raising the Bar of AI-generated Image Detection with CLIP

    Davide Cozzolino, Giovanni Poggi, Riccardo Corvi, Matthias Nießner, and Luisa Verdoliva. Raising the bar of AI-generated image detection with CLIP.arXiv preprint arXiv:2312.00195, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  33. [33]

    DIRE for Diffusion-Generated Image Detection

    Zhendong Wang, Jianmin Bao, Wengang Zhou, Weilun Wang, Hezhen Hu, Hong Chen, and Houqiang Li. DIRE for diffusion-generated image detection.arXiv preprint arXiv:2303.09295, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [34]

    AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error

    Jonas Ricker, Denis Lukovnikov, and Asja Fischer. AEROBLADE: Training-free detection of latent diffusion images using autoencoder reconstruction error.arXiv preprint arXiv:2401.17879, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection

    Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, Ping Liu, and Yunchao Wei. C2p-clip: Injecting cate- gory common prompt in clip to enhance generalization in deepfake detection.arXiv preprint arXiv:2408.09647,

    work page internal anchor Pith review Pith/arXiv arXiv

  36. [36]

    Code:https://github.com/chuangchuangtan/C2P-CLIP-DeepfakeDetection

    AAAI 2025. Code:https://github.com/chuangchuangtan/C2P-CLIP-DeepfakeDetection

    work page 2025

  37. [37]

    FaceForensics: A large-scale video dataset for forgery detection in human faces

    Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. FaceForensics: A large-scale video dataset for forgery detection in human faces. 2018

    work page 2018

  38. [38]

    The Deepfake Detection Challenge (DFDC) Preview Dataset

    Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, and Cristian Canton Ferrer. The Deepfake detection challenge (DFDC) preview dataset.arXiv preprint arXiv:1910.08854, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  39. [39]

    The DeepFake Detection Challenge (DFDC) Dataset

    Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, Cristian Canton Ferrer, et al. The DeepFake detection challenge (DFDC) dataset.arXiv preprint arXiv:2006.07397, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  40. [40]

    WildDeepfake: A challenging real- world dataset for deepfake detection

    Bojia Zi, Minghao Chang, Jingjing Chen, Xingjun Ma, and Yu-Gang Jiang. WildDeepfake: A challenging real- world dataset for deepfake detection. InProceedings of the 28th ACM International Conference on Multimedia, pages 2382–2390, 2020

    work page 2020

  41. [41]

    Celeb-DF: A large-scale challenging dataset for DeepFake forensics

    Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. Celeb-DF: A large-scale challenging dataset for DeepFake forensics. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3204–3213, 2020

    work page 2020

  42. [42]

    ForgeryNet: A versatile benchmark for comprehensive forgery analysis

    Yinan He, Bei Gan, Siyu Chen, Yichun Zhou, Guojun Yin, Luchuan Song, Lu Sheng, Jing Shao, and Ziwei Liu. ForgeryNet: A versatile benchmark for comprehensive forgery analysis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4360–4369, 2021

    work page 2021

  43. [43]

    DF40: Toward next-generation deepfake detection

    Zhiyuan Yan, Taiping Yao, Shen Chen, Yandan Zhao, Xinghe Fu, Junwei Zhu, Donghao Luo, Li Yuan, Chengjie Wang, Shouhong Ding, Yunsheng Wu, et al. DF40: Toward next-generation deepfake detection. InAdvances in Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2024

    work page 2024

  44. [44]

    Deepfakebench: A comprehensive benchmark of deepfake detection

    Zhiyuan Yan, Yong Zhang, Xinhang Yuan, Siwei Lyu, and Baoyuan Wu. Deepfakebench: A comprehensive benchmark of deepfake detection. InAdvances in Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2023

    work page 2023

  45. [45]

    Claude opus 4.8 model card, 2026

    Anthropic. Claude opus 4.8 model card, 2026. Model-card reference for claude_opus48; exact model-card URL not independently verified. 19 APREPRINT- JULY8, 2026

    work page 2026

  46. [46]

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023. Cited as the Qwen vision-language family anchor; the exact Qwen version used forqwenis not specified i...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  47. [47]

    Llama 4: Multimodal intelligence, openly available, 2025

    Meta AI. Llama 4: Multimodal intelligence, openly available, 2025. Reference for llama4_maverick; blog/model-card URL not independently verified

    work page 2025

  48. [48]

    Nemotron: Nvidia’s family of open foundation models, 2024

    NVIDIA. Nemotron: Nvidia’s family of open foundation models, 2024. Reference for nemotron_nano_vl; exact model variant not independently verified

    work page 2024

  49. [49]

    Glm: General language model family (zhipu ai / z.ai), 2024

    Zhipu AI. Glm: General language model family (zhipu ai / z.ai), 2024. Reference for zai_glm52; exact GLM-5.2 release details not independently verified

    work page 2024

  50. [50]

    C2PA technical specification

    Coalition for Content Provenance and Authenticity. C2PA technical specification. Technical Specification 1.0, C2PA, December 2021

    work page 2021

  51. [51]

    Content credentials: C2PA technical specification

    Coalition for Content Provenance and Authenticity (C2PA). Content credentials: C2PA technical specification. Technical specification v2.4, 2024

    work page 2024

  52. [52]

    Scalable watermarking for identifying large language model outputs.Nature, 634(8035):818–823, 2024

    Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, Jamie Hayes, Nidhi Vyas, Majd Al Merey, Jonah Brown-Cohen, Rudy Bunel, Borja Balle, Taylan Cemgil, Zahra Ahmed, Kitty Stacpoole, Ilia Shumailov, Ciprian Baetu, Sven Gowal, Demis Hassabis, and Pu...

    work page 2024

  53. [53]

    Sven Gowal, Rudy Bunel, Florian Stimberg, David Stutz, Guillermo Ortiz-Jimenez, Christina Kouridi, Mel Vecerík, Jamie Hayes, Sylvestre-Alvise Rebuffi, Paul Bernard, Chris Gamble, Miklós Z. Horváth, Fabian Kaczmarczyck, Alex Kaskasoli, Aleksandar Petrov, Ilia Shumailov, Meghana Thotakuri, Olivia Wiles, Jessica Yung, Zahra Ahmed, Victor Martin, Simon Rosen,...

    work page arXiv 2025

  54. [54]

    RealAPI product documentation

    Reality Defender. RealAPI product documentation. https://www.realitydefender.com/product/ realapi, 2024

    work page 2024

  55. [55]

    Ai-generated image and video detection documentation

    Hive AI. Ai-generated image and video detection documentation. https://docs.thehive.ai/docs/ ai-image-and-video-detection, 2024

    work page 2024

  56. [56]

    Ai-generated image detection API documentation

    Sightengine. Ai-generated image detection API documentation. https://sightengine.com/docs/ ai-generated-image-detection, 2024

    work page 2024

  57. [57]

    Ai image detection API documentation

    TruthScan. Ai image detection API documentation. https://truthscan.com/ truthscan-ai-image-detection-api-documentation, 2024

    work page 2024

  58. [58]

    Ai generated image detection (DeepScan API)

    Neural Defend. Ai generated image detection (DeepScan API). https://neuraldefend.gitbook.io/ neural-defend/ai-generated-image-detection, 2024

    work page 2024

  59. [59]

    Deepfake detection platform.https://sensity.ai/, 2024

    Sensity AI. Deepfake detection platform.https://sensity.ai/, 2024

    work page 2024

  60. [60]

    Ai generated image detection

    Illuminarty. Ai generated image detection. https://www.illuminarty.ai/en/image/ ai-generated-image-detection.html, 2024

    work page 2024

  61. [61]

    Image detection API reference

    AI or Not. Image detection API reference. https://docs.aiornot.com/api-reference/ reports-by-modality/image.md, 2024

    work page 2024

  62. [62]

    Intel introduces real-time deepfake detector (FakeCatcher)

    Intel Corporation. Intel introduces real-time deepfake detector (FakeCatcher). https://newsroom.intel. com/artificial-intelligence/intel-introduces-real-time-deepfake-detector, 2022

    work page 2022

  63. [63]

    Poskitt, and Xingmei Wang

    Boquan Li, Jun Sun, Christopher M. Poskitt, and Xingmei Wang. How generalizable are deepfake image detectors? an empirical study, 2023

    work page 2023

  64. [64]

    Assessment framework for deepfake detection in real-world situations, 2023

    Yuhang Lu and Touradj Ebrahimi. Assessment framework for deepfake detection in real-world situations, 2023

    work page 2023

  65. [65]

    Fake or JPEG? revealing common biases in generated image detection datasets, 2024

    Patrick Grommelt, Louis Weiss, Franz-Josef Pfreundt, and Janis Keuper. Fake or JPEG? revealing common biases in generated image detection datasets, 2024

    work page 2024

  66. [66]

    A sanity check for AI-generated image detection

    Shilin Yan, Ouxiang Li, Jiayin Cai, Yanbin Hao, Xiaolong Jiang, Yao Hu, and Weidi Xie. A sanity check for AI-generated image detection. InInternational Conference on Learning Representations (ICLR), 2025

    work page 2025

  67. [67]

    The Adversarial AI-Art: Understanding, Generation, Detection, and Benchmarking

    Fang Li et al. The adversarial AI-Art: Understanding, generation, detection, and benchmarking. https: //arxiv.org/abs/2404.14581, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  68. [68]

    Organic or diffused: Can we distinguish human art from AI-generated images?https://doi.org/10.1145/3658644.3670306, 2024

    Anna Yoo Jeong Ha, Josephine Passananti, et al. Organic or diffused: Can we distinguish human art from AI-generated images?https://doi.org/10.1145/3658644.3670306, 2024. 20 APREPRINT- JULY8, 2026

    work page doi:10.1145/3658644.3670306 2024

  69. [69]

    The visual counter turing test (VCT2): A benchmark for evaluating AI-Generated image detection and the visual AI index.https://arxiv.org/abs/2411.16754, 2024

    Nasrin Imanpour Chakraborty et al. The visual counter turing test (VCT2): A benchmark for evaluating AI-Generated image detection and the visual AI index.https://arxiv.org/abs/2411.16754, 2024

    work page arXiv 2024

  70. [70]

    Ai-genbench: A new ongoing benchmark for ai-generated image detection

    Lorenzo Pellegrini, Davide Cozzolino, Serafino Pandolfini, Davide Maltoni, Matteo Ferrara, Luisa Verdoliva, Marco Prati, and Marco Ramilli. Ai-genbench: A new ongoing benchmark for ai-generated image detection. In 2025 International Joint Conference on Neural Networks (IJCNN), pages 1–9, 2025

    work page 2025

  71. [71]

    A. S. Iufereva and Peter Mozelius. Deepfake detection: Human performance versus AI tools – a comparison of accuracy and effectiveness. InProceedings of the International Conference on AI Research (ICAIR), 2025

    work page 2025

  72. [72]

    Adversarial deepfakes: Evaluating vulnerability of deepfake detectors to adversarial examples

    Shehzeen Hussain, Paarth Neekhara, Malhar Jere, Farinaz Koushanfar, and Julian McAuley. Adversarial deepfakes: Evaluating vulnerability of deepfake detectors to adversarial examples. InIEEE Winter Conference on Applications of Computer Vision (WACV), pages 3348–3357, 2021

    work page 2021

  73. [73]

    Chee K. Chow. On optimum recognition error and reject tradeoff.IEEE Transactions on Information Theory, 16(1):41–46, 1970

    work page 1970

  74. [74]

    On the foundations of noise-free selective classification.Journal of Machine Learning Research, 11(53):1605–1641, 2010

    Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification.Journal of Machine Learning Research, 11(53):1605–1641, 2010

    work page 2010

  75. [75]

    Selective classification for deep neural networks

    Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. InAdvances in Neural Information Processing Systems (NeurIPS), volume 30, 2017

    work page 2017

  76. [76]

    SelectiveNet: A deep neural network with an integrated reject option

    Yonatan Geifman and Ran El-Yaniv. SelectiveNet: A deep neural network with an integrated reject option. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 2151–2159, 2019

    work page 2019

  77. [77]

    Learning and evaluating classifiers under sample selection bias

    Bianca Zadrozny. Learning and evaluating classifiers under sample selection bias. InProceedings of the 21st International Conference on Machine Learning (ICML), pages 903–910, 2004

    work page 2004

  78. [78]

    Counterfactually comparing abstaining classifiers

    Yo Joong Choe, Aditya Gangrade, and Aaditya Ramdas. Counterfactually comparing abstaining classifiers. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023

    work page 2023

  79. [79]

    The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation.BMC Genomics, 21(1):6, 2020

    Davide Chicco and Giuseppe Jurman. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation.BMC Genomics, 21(1):6, 2020

    work page 2020

  80. [80]

    Brodersen, Cheng Soon Ong, Klaas E

    Kay H. Brodersen, Cheng Soon Ong, Klaas E. Stephan, and Joachim M. Buhmann. The balanced accuracy and its posterior distribution. InProceedings of the 20th International Conference on Pattern Recognition (ICPR), pages 3121–3124, 2010

    work page 2010

Showing first 80 references.