Pith. sign in

REVIEW 3 major objections 4 minor 5 cited by

This paper introduces P2V, a stress-test dataset of perturbed public voices, and reports that 22 current audio deepfake detectors lose 43% average performance on it, while detectors trained on P2V stay accurate and generalize to existing be

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A new audio deepfake benchmark, P2V, is reported to cut the mean performance of 22 current detectors by 43%, while detectors trained on it stay robust and generalize to older datasets.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The abstract promises a timely robustness benchmark but provides no way to check the headline numbers; as presented, the claims are unfalsifiable. the 3 major comments →

arxiv 2508.10949 v1 pith:AXS4FXFN submitted 2025-08-13 cs.SD eess.AS

Perturbed Public Voices (P$^{2}$V): A Dataset for Robust Audio Deepfake Detection

Chongyang Gao , Marco Postiglione , Isabel Gortner , Sarit Kraus , V.S. Subrahmanian This is my paper
classification cs.SD eess.AS
keywords audio deepfake detectionvoice cloningadversarial noiseLLM-generated transcriptsbenchmark datasetdistribution shiftdetector robustnesspublic voices
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that audio deepfake detectors trained on existing benchmarks are far less reliable once the test set resembles real malicious fakes. It builds P2V, an IRB-approved dataset whose samples combine three properties: LLM-generated transcripts consistent with the speaker's identity, environmental and adversarial noise, and voice-cloning technology from 2020 to 2025. On that dataset, 22 recent detectors lose 43% on average, measured as the mean of F1 on deepfake audio, AUC, and 1-EER. Detectors trained on P2V, by contrast, keep their performance under these perturbations and also generalize to the older datasets. If this holds, current benchmarks overstate detector reliability and P2V provides a more realistic training and evaluation distribution.

Core claim

The central discovery is that moving from existing benchmarks to P2V causes a 43% average performance drop across 22 audio deepfake detectors, where performance is the mean of F1 on deepfake audio, AUC, and 1-EER. P2V builds its samples from public voices along three axes: identity-consistent transcripts generated by LLMs, environmental and adversarial noise, and state-of-the-art voice cloning released between 2020 and 2025. Simple adversarial perturbations alone degrade performance by up to 16%, and advanced cloning lowers detectability by 20-30%. Detectors trained on P2V do not show this collapse: they remain accurate on P2V and still generalize to existing datasets. The paper offers P2V a

What carries the argument

P2V (Perturbed Public Voices) is the central object carrying the argument. It is a dataset built from public voices and perturbed along three construction axes: LLM-generated transcripts that match the target identity, environmental and adversarial noise added to the audio, and 2020-2025 voice cloning systems. The dataset functions as both the stress test and the fix: it defines the distribution-shifted evaluation that produces the 43% drop, and it serves as the training set that keeps detectors accurate and transferable to older benchmarks. The reported scores aggregate three metrics (F1 on deepfake audio, AUC, and 1-EER) across 22 detectors.

Load-bearing premise

The load-bearing premise is that the three ways P2V changes voices—LLM-written identity-consistent transcripts, added noise, and recent voice cloning—match what real malicious deepfakes look like; if an attacker produces fakes outside those three dimensions, the reported 43% drop and the training fix do not transfer.

What would settle it

A concrete check would be to assemble an independent corpus of real-world malicious deepfakes (for example from reported scam calls or political disinformation) and run both the original 22 detectors and P2V-trained models on it. If P2V-trained models show a performance drop comparable to the original 43% on that independent corpus, the claim that P2V represents real malicious deepfakes fails. A simpler ablation: remove each of P2V's three axes in turn; if removing noise alone eliminates most of the 43% drop, then identity-consistent transcripts and recent cloning are not carrying the difficul

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Existing benchmark results should be read as optimistic: the same detectors that look strong on current datasets lose 43% average performance on P2V.
  • Training on P2V is a concrete hardening step: it preserves accuracy on existing datasets while improving resistance to noise, adversarial perturbations, and recent cloning.
  • Evaluation of audio deepfake detectors should include distribution-shifted sets like P2V, otherwise gains measured on a single benchmark may not transfer.
  • Because recent cloning alone cuts detectability by 20-30%, detectors need to be re-checked as new cloning systems appear.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper, the identity-consistent transcript axis suggests that detectors may currently rely on textual anomalies (grammar, repetition, context drift) that disappear when an LLM writes speaker-consistent text; a detector combining voice-print matching with speaker-specific language patterns would be a natural next step.
  • An untested extension is to build P2V-style datasets for non-English voices and dialects, where public-voice availability and cloning-system training data differ; the paper's generalization results only cover the legacy datasets it tested.
  • A useful stress test would be to track P2V-trained detectors against post-2025 cloning systems; if the 20-30% detectability reduction grows, the benchmark must be continuously refreshed to stay representative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The abstract announces a new dataset, Perturbed Public Voices (P2V), for audio deepfake detection, built along three axes: LLM-generated identity-consistent transcripts, environmental/adversarial noise, and 2020-2025 voice cloning. It claims that 22 current detectors lose 43% mean performance (mean of F1 on deepfake audio, AUC, and 1-EER) when evaluated on P2V, that simple adversarial perturbations induce up to 16% degradation, that advanced cloning reduces detectability by 20-30%, and that P2V-trained models retain robustness while generalizing to existing datasets. The manuscript is reviewed here in abstract-only form; no full text, dataset, or evaluation protocol is available.

Significance. If substantiated, P2V would be a valuable contribution: a public, IRB-approved stress-test dataset targeting realistic conditions that current benchmarks omit, with results suggesting a substantial generalization gap in audio deepfake detection and a potential training remedy. The planned public release and the use of recent cloning systems are concrete strengths. However, the central claims are currently unverifiable from the abstract alone. The significance is therefore conditional on the full manuscript providing a reproducible protocol and external validation.

major comments (3)
  1. [Abstract, headline results] The 43% mean performance drop is asserted without any visible protocol: no list of the 22 detectors, no dataset size, no train/test split, no class balance, no speaker-overlap policy, and no error bars. F1 depends on a decision threshold; averaging F1 with AUC and 1-EER can inflate apparent degradation if detectors use default or poorly calibrated thresholds on out-of-distribution audio. The abstract must state the exact evaluation protocol, including whether thresholds are tuned per detector on P2V, and give per-metric and per-detector breakdowns.
  2. [Abstract, data and code availability] The abstract states that P2V 'will be publicly released upon acceptance by a conference/journal.' At the time of review, neither the dataset nor the evaluation code is available. This makes the headline numbers impossible to replicate or audit. For a benchmark-result claim, the evaluation protocol, the exact perturbation parameters, and the detector configurations must be released or otherwise specified in the full manuscript before the results can be assessed.
  3. [Abstract, external validity and circularity] The three construction axes are described as 'critical aspects of malicious deepfakes,' but the abstract provides no evidence linking P2V to real-world deepfake cases (e.g., phone scams, election disinformation). Adversarial perturbations are explicitly worst-case, so the 16% degradation figure does not by itself support a real-world reliability claim. Additionally, the robustness result is partly self-referential: models trained on P2V are tested against perturbations from the same construction pipeline. The generalization result on existing datasets provides an external anchor, but the robustness claim needs independent held-out perturbation sets or a comparison with perturbations generated from a different pipeline to rule out benchmark overfitting.
minor comments (4)
  1. [Abstract, wording] Phrases such as 'cannot be trusted' and 'alarming' are rhetorical. Please quantify: what failure rate or performance level constitutes 'cannot be trusted' in a deployed setting?
  2. [Abstract, metric definition] Specify the exact aggregation of F1, AUC, and 1-EER. Is the reported 43% a simple average of three numbers across detectors, or a macro-average of per-detector composites? Provide the formula.
  3. [Abstract, cloning degradation] The '20-30% reduction in detectability' needs a baseline. Relative to what condition: clean audio, audio with environmental noise, or a specific cloning system? Please state the comparison and error bars.
  4. [Abstract, dataset composition] Give basic dataset statistics: number of voices, number of clips per condition, duration, language distribution, and how identity-consistent transcripts are generated and validated.

Circularity Check

0 steps flagged

No circularity identifiable from the abstract; reported results are empirical benchmark evaluations, not derivations from their own inputs.

full rationale

The abstract reports empirical measurements: 22 detectors lose 43% performance on P2V, adversarial perturbations degrade performance by up to 16%, and cloning reduces detectability by 20-30%. These are evaluation outcomes, not quantities derived from the dataset's definition or from fitted parameters. The statement that 'P2V-trained models maintain robustness against these attacks' is an in-distribution evaluation result: training and testing on the same constructed perturbation pipeline would be expected to show robustness, but the abstract does not assert this as a theoretical consequence, and it also reports generalization to existing datasets, which is an external and falsifiable anchor. No equation, self-citation, uniqueness theorem, or ansatz-smuggling step is present in the available text. Concerns about dataset availability, protocol transparency, and external validity are legitimate evidentiary worries but are not circularity under the required standard of exhibiting a specific reduction of a claimed prediction to its inputs. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central claims rest entirely on the abstract's construction choices and evaluation protocol, none of which are specified in detail: perturbation budgets, noise configurations, cloning mix, and the composite metric are all inputs the reader must take on faith. The dataset itself is not yet released.

free parameters (4)
  • perturbation strength ranges (adversarial noise magnitudes)
    Abstract reports up to 16% degradation from 'simple adversarial perturbations' but does not state perturbation budgets; these are hand-chosen construction parameters.
  • environmental noise types and SNRs
    The noise axis is asserted without disclosed configurations such as SNR levels or noise sources, which determine difficulty.
  • cloning model mix (2020-2025)
    Which cloning systems and how many voices or clips per system are not disclosed; the 20-30% reduction depends on this mix.
  • number of voices and clips per condition
    Dataset scale is not stated; sample counts shape both the 43% drop and the robustness claims.
axioms (4)
  • domain assumption The three axes (identity-consistent transcripts, noise, modern cloning) capture the critical aspects of real malicious deepfakes.
    The abstract's opening and dataset motivation rely on this representativeness premise; if P2V perturbations are artificial, the real-world failure claim collapses.
  • domain assumption The 22 detectors are a representative sample of current audio deepfake detectors.
    The 43% average drop is computed over an unlisted set; a biased selection would move the headline number.
  • domain assumption Mean of F1, AUC, and 1-EER is an appropriate composite robustness measure.
    The single performance number is a mean over three metrics; the abstract gives no aggregation justification.
  • domain assumption IRB approval implies voice data was collected with proper consent and can be legally redistributed.
    The dataset release claim rests on the adequacy of the IRB approval process for voice-cloning training data.
invented entities (1)
  • P2V dataset no independent evidence
    purpose: Benchmark for evaluating and training robust audio deepfake detectors with realistic malicious-deepfake conditions.
    The dataset is described and promised for future release, but no samples, metadata, or download are available in the abstract, so it cannot be independently inspected.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Perturbed Public Voices (P$^{2}$V): A Dataset for Robust Audio Deepfake Detection." pith.science (2026). https://pith.science/paper/AXS4FXFN

@misc{pith2026250810949,
  author       = {Pith},
  title        = {Pith review of: Perturbed Public Voices (P$^2$V): A Dataset for Robust Audio Deepfake Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AXS4FXFN}},
  note         = {Machine review of arXiv:2508.10949}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Current audio deepfake detectors cannot be trusted. While they excel on controlled benchmarks, they fail when tested in the real world. We introduce Perturbed Public Voices (P$^{2}$V), an IRB-approved dataset capturing three critical aspects of malicious deepfakes: (1) identity-consistent transcripts via LLMs, (2) environmental and adversarial noise, and (3) state-of-the-art voice cloning (2020-2025). Experiments reveal alarming vulnerabilities of 22 recent audio deepfake detectors: models trained on current datasets lose 43% performance when tested on P$^{2}$V, with performance measured as the mean of F1 score on deepfake audio, AUC, and 1-EER. Simple adversarial perturbations induce up to 16% performance degradation, while advanced cloning techniques reduce detectability by 20-30%. In contrast, P$^{2}$V-trained models maintain robustness against these attacks while generalizing to existing datasets, establishing a new benchmark for robust audio deepfake detection. P$^{2}$V will be publicly released upon acceptance by a conference/journal.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Multi-Backbone Self-Supervised Ensembles for Audio Deepfake Detection and a Cross-Track Analysis of Generation-Detection Asymmetry

    cs.SD 2026-08 conditional novelty 6.0

    A four-backbone SSL ensemble achieves near-perfect deepfake detection in the ImageCLEF 2026 track, while the same team's generated audio ranks first in the generation track, revealing an OR-versus-AND asymmetry betwee...

  2. Quantum Kernels for Audio Deepfake Detection Using Spectrogram Patch Features

    cs.SD 2026-05 unverdicted novelty 6.0

    Q-Patch quantum kernels on spectrogram patches achieve AUROC 0.87 for audio spoofing detection, outperforming RBF-SVM at 0.82 on the same features.

  3. RADAR Challenge 2026: Robust Audio Deepfake Recognition under Media Transformations

    eess.AS 2026-05 unverdicted novelty 5.0

    The RADAR Challenge 2026 provides a multilingual benchmark for audio deepfake detection under media transformations and finds that robust performance remains an open problem.

  4. RADAR Challenge 2026: Robust Audio Deepfake Recognition under Media Transformations

    eess.AS 2026-05 unverdicted novelty 4.0

    RADAR Challenge 2026 describes a benchmark with over 100,000 multilingual utterances and media transformations for audio deepfake detection, reporting results from 22 teams that highlight ongoing robustness issues.

  5. RADAR Challenge 2026: Robust Audio Deepfake Recognition under Media Transformations

    eess.AS 2026-05 unverdicted novelty 4.0

    RADAR Challenge 2026 organizes a multilingual audio deepfake detection benchmark with media transformations, reporting participation from 33 development and 22 evaluation teams using EER metric.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.