REVIEW 2 major objections 6 minor 34 references
A new score reveals when audio deepfake detectors fail because of speaker identity, not synthesis artifacts.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 06:34 UTC pith:M2XEKTLS
load-bearing objection A useful new per-utterance diagnostic for speaker sensitivity in deepfake detectors, with a solid error-prediction core, but the identity-specific interpretation is not yet cleanly separated from boundary proximity. the 2 major comments →
Probing Speaker Identity Sensitivity in Audio Deepfake Detectors
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that a deepfake detector's score on a single utterance is partly a function of the speaker's identity, and that this identity-dependence is measurable without ground-truth labels. The paper's Identity Sensitivity Score (ISS) does this by computing p_fake(A|S_i)=σ(ℓ(A)+α·cos(A,S_i)) for K enrolled speakers and taking the interquartile range of those scores. On ASVspoof 2019/2021 LA with AASIST and RawNet2, incorrect utterances show 29–52× higher ISS (AUC up to 0.954), and voice conversion moves high-ISS utterances' scores 19.2–30.7× more than low-ISS ones. The paper interprets this as evidence that identity sensitivity is real detector behavior, not just a proxy for confi
What carries the argument
The Identity Sensitivity Score (ISS): for a query utterance A, take the detector's log-odds ℓ(A), add α times the cosine similarity between A's speaker embedding (from a pretrained ECAPA-TDNN) and each of K enrolled speakers' mean embeddings, pass each sum through the sigmoid, and record the interquartile range of the resulting K probabilities. The logit-space addition is a Bayesian score-fusion-style probe that leaves the audio untouched; IQR makes the spread robust to outlier speakers. ISS is doing the work of converting a single scalar score into a family of identity-conditioned scores, so that identity-driven instability becomes visible as spread.
Load-bearing premise
The load-bearing premise is that comparing high-ISS to low-ISS utterances isolates identity sensitivity, but those groups are not matched for distance to the decision boundary, and the sigmoid slope is steepest near p=0.5, so the large voice-conversion response could partly reflect boundary proximity rather than identity.
What would settle it
Rematch the voice-conversion groups so high- and low-ISS utterances have identical distributions of |p_fake−0.5| (or identical entropy, margin, or confidence); if the 19–30× differential collapses or drops sharply, ISS is largely a confidence proxy rather than an identity measure.
If this is right
- At inference, utterances with ISS above a threshold can be flagged for human review without any labels, with AUC up to 0.954 for predicting misclassification.
- ISS monitors distribution shift: AASIST's 21× EER jump from 2019 to 2021 LA coincides with a 24× amplification of the ISS separation ratio, so rising ISS can signal that a deployment environment is drifting.
- Attack-type analysis shows TTS and VC errors carry strong ISS elevation, while RawNet2's hybrid-attack errors show none (ratio 0.2×), letting practitioners distinguish identity-driven failures from acoustic-artifact failures.
- Because ISS is complementary to entropy, margin, and confidence, it can be combined with those baselines rather than replace them for a fuller picture of why a prediction is uncertain.
Where Pith is reading between the lines
- If identity sensitivity is a learned shortcut, training on data where speaker identity is partially unlinked from the genuine/synthetic label (for example, pairing each synthesis system with many speakers and balancing speaker demographics) should reduce both ISS and cross-dataset EER; this is a testable intervention the paper notes but does not run.
- ISS depends on the external embedding space of ECAPA-TDNN; applying the same probe with a different speaker encoder, or with the detector's own internal representations, would reveal whether the measured identity sensitivity is a property of the detector or of the probe's notion of identity.
- The logit-space identity perturbation is an assessment, not a cause: nothing in the paper shows that removing identity sensitivity would improve generalization, only that the two are correlated. A training-level ablation that measures ISS before and after identity-invariant regularization would close that gap.
- Because ISS requires only one extra forward pass through a speaker encoder, it could be repurposed as a regularizer during training, penalizing high-ISS predictions to discourage identity shortcuts rather than merely diagnosing them.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Identity Sensitivity Score (ISS), a per-utterance diagnostic intended to quantify how much an audio deepfake detector's output depends on speaker identity. ISS is computed by taking the logit of the detector's p_fake, adding α·cos(A,S_i) for K alternative speaker prototypes from an ECAPA-TDNN embedding space, and measuring the interquartile range of the resulting sigmoid probabilities. The authors report that incorrectly classified utterances have 29–52× higher ISS than correctly classified ones, that ISS predicts misclassification with AUC up to 0.954, and that voice-converted versions of high-ISS utterances shift the detector score 19–30× more than low-ISS utterances. They interpret the voice-conversion result as evidence that ISS captures identity sensitivity rather than being a proxy for prediction confidence.
Significance. If the interpretation holds, ISS would be a useful label-free, inference-time diagnostic for speaker-dependent failure analysis in deepfake detection. The paper has genuine strengths: a clear problem formulation, a conceptually interesting logit-space perturbation, two architecturally distinct detectors, two datasets, an honest comparison to entropy/margin/confidence baselines, ablation over K, and a breakdown by attack type. The RawNet2 hybrid-attack result (ISS ratio 0.2×, p=1.00) is a nice example of selectivity. However, the central validation — the voice-conversion experiment — is confounded by decision-boundary proximity, so the paper's main interpretive claim is not currently established. The issue is fixable with matched-group or covariate-controlled analysis, which is why I recommend major revision rather than rejection.
major comments (2)
- [§4.4, Table 4; also §3.5] The voice-conversion validation does not control for the base score's distance to the decision boundary. For small α, ISS(A) ≈ α·IQR_i[cos(A,S_i)]·σ(ℓ)(1−σ(ℓ)), so high-ISS utterances are preferentially those with p_fake near 0.5. The measured Δscore = |p_fake(A_conv)−p_fake(A_src)| is similarly bounded by the sigmoid slope. Thus the 19.2×/30.7× differential between high- and low-ISS groups may be a slope artifact rather than evidence of identity-specific sensitivity. The acknowledgment in §3.5 that ISS is coupled to boundary proximity does not remove this confound. Please re-run the VC experiment with high- and low-ISS groups matched on p_fake (e.g., stratified within narrow p_fake bins), or include base p_fake as a covariate, and report the p_fake distributions of both groups. The attack-type results in §5.6 are suggestive but do not resolve this because those groups are defined by err
- [§3.4] ISS as defined requires sampling K alternative speakers while 'excluding A's true speaker.' At inference time, the true speaker identity is generally not known; if the query speaker is one of the enrolled speakers, this exclusion requires exactly the kind of label the method claims to avoid. The paper should clarify whether the evaluation speakers are disjoint from the enrollment pool. If they are disjoint, the exclusion step is vacuous and the method is only validated for out-of-enrollment speakers; if they are not disjoint, the label-free claim is problematic. Either way, the deployment scenario needs to be stated precisely.
minor comments (6)
- [§4.3, Table 7] The reported ISS ratios and AUCs use α=5.0, which is selected on the development set by maximizing the correct/incorrect ISS ratio. This is disclosed, but the paper should also present the evaluation-set results across the α grid (or a nested selection) to show that the headline numbers are not sensitive to the exact selection criterion. The 'bonafide/spoof ordering inverts' criterion is somewhat fragile and worth explaining.
- [§2.3] The baseline called 'confidence' is defined as 1−max(p_fake,1−p_fake), which is actually minimized at the boundary, not maximized. Although it is correctly grouped with entropy and margin as a monotone transform of |p_fake−0.5|, the name is confusing; consider calling it 'uncertainty' or defining it as max(p,1−p).
- [§4.4] The protocol says '500 utterances per detector' but the description is 250 high-ISS + 250 low-ISS. Please make the counting explicit.
- [Table 3] The 'Advantage' row reports differences in AUC percentage points (e.g., +5.3). Please state the units to avoid confusion with ratios.
- [§6.1] The p_fake=0.51 example is repeated nearly verbatim from §5.4. One occurrence would be sufficient.
- [Table 7 footnote] The dev-set ratios (6,200×–17,700×) versus evaluation ratios (29×) are attributed to class composition. The explanation is plausible but would be more convincing with a brief table of dev/eval ISS_cor and ISS_inc medians.
Circularity Check
No circular derivation: ISS is defined from external speaker embeddings and the base detector score, the main empirical claims are held-out, and the VC validation confound is a validity issue rather than a circular reduction.
full rationale
The derivation of ISS (Eqs. 3-4) is self-contained: it combines the detector's base logit with cosine similarities from an external ECAPA-TDNN speaker encoder and defines ISS as an IQR over K alternative-speaker contexts. It does not use error labels in its definition, so the observed 29-52x separation between correct and incorrect utterances and the AUC of 0.954 are empirical findings on held-out evaluation partitions, not algebraic consequences of the definition. The alpha=5 perturbation strength is tuned on the 2019 LA development set by maximizing the correct/incorrect ISS ratio; this is standard hyperparameter selection, and the headline numbers are reported on evaluation partitions, so this is not a fitted parameter being renamed as a prediction. The voice-conversion validation (Sec. 4.4, 5.5) has a genuine confound: high-ISS and low-ISS groups are selected by ISS rank without matching on p_fake, and since ISS = IQR{sigma(ell+alpha cos)} is amplified by the sigmoid slope sigma'(ell)=sigma(ell)(1-sigma(ell)) near p=0.5, the 19-30x Delta-score differential could partly reflect boundary proximity rather than identity. However, this is a validity/confound problem with an experimental test, not a circular derivation: Delta-score = |p_fake(A_conv)-p_fake(A_src)| is an external measurement and is not equal to Eq. (3) by construction. The only self-citation ([5]) is contextual background and is not load-bearing; no uniqueness theorem or prior ansatz is invoked to force the core claim. The authors themselves note in Sec. 6.2 that the VC validation does not establish a causal training-level link. Accordingly, no step in the paper's derivation chain reduces to its own inputs, and the circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (4)
- α (logit perturbation strength) =
5.0
- K (number of alternative speaker contexts) =
10
- N (enrollment utterances per speaker prototype) =
5
- VC high/low ISS group thresholds =
AASIST ISS>0.05 / <0.01; RawNet2 medians 3.1e-3 vs 5.8e-6
axioms (6)
- domain assumption ECAPA-TDNN cosine similarity is a valid proxy for speaker identity as relevant to the deepfake detector's decision.
- ad hoc to paper Adding α·cos(A,S_i) in logit space produces meaningful identity-conditioned probabilities.
- ad hoc to paper IQR of the identity-conditioned scores is the right measure of identity sensitivity.
- domain assumption ASVspoof 2019 LA training data correlates speaker identity with genuine/synthetic labels.
- domain assumption α tuned on a labeled development set transfers to the unlabeled deployment setting.
- ad hoc to paper The true speaker of a query can be excluded from the K sampled contexts at inference.
read the original abstract
Audio deepfake detectors are trained to distinguish genuine speech from synthetic speech and often perform well on standard benchmarks. Yet the same detector that achieves less than 1% error on one dataset can see its error rate increase twentyfold when evaluated on a different dataset. We argue that one contributing factor is speaker-identity reliance: standard training corpora correlate speaker identity with the genuine/synthetic label, allowing detectors to partially rely on speaker-related cues rather than synthesis artifacts alone. We propose the Identity Sensitivity Score (ISS), a per-utterance diagnostic that quantifies how much a detector's output changes across different speaker identity contexts. ISS requires no ground-truth labels at inference time and can be computed from the detector score and a pool of reference speaker examples. Across two detectors and two datasets, incorrectly classified utterances have ISS scores 29 to 52 times higher than correctly classified utterances, and ISS alone predicts misclassification with area-under-curve (AUC) up to 0.954. To test whether ISS actually captures identity-sensitive behavior rather than serving only as a proxy for prediction confidence, we apply voice conversion to 500 utterances and measure the resulting detector-score shift. Utterances flagged as identity-sensitive by ISS respond 19 to 30 times more strongly to this manipulation than utterances flagged as stable. These results position ISS as a practical inference-time diagnostic for speaker-dependent failure analysis in audio deepfake detection.
Figures
Reference graph
Works this paper leans on
-
[1]
ASVspoof 2021 baseline systems
ASVspoof Challenge. ASVspoof 2021 baseline systems. https : / / github . com / asvspoof - challenge / 2021, 2021
2021
-
[2]
Br ¨ummer and J
N. Br ¨ummer and J. du Preez. Application-independent eval- uation of speaker detection.Computer Speech & Language, 20(2–3):230–275, 2006
2006
-
[3]
Y . Chen, J. Yi, J. Xue, C. Wang, X. Zhang, S. Dong, S. Zeng, J. Tao, L. Zhao, and C. Fan. RawBMamba: End-to-end bidirectional state space model for audio deepfake detection. arXiv preprint arXiv:2406.06086, 2024
Pith/arXiv arXiv 2024
-
[4]
Cheng, H
Z. Cheng, H. Yang, Y . Xiong, and X. Hu. Explainable AI for forensic speech authentication within cognitive and compu- tational neuroscience.Frontiers in Neuroscience, 19, 2025
2025
-
[5]
D. K. Dar, Q. Yan, L. Xiao, and A. Ross. Impact of phonetics on speaker identity in adversarial voice attack. InProceed- ings IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 13462–13466, 2026
2026
-
[6]
Delgado, N
H. Delgado, N. Evans, T. Kinnunen, K. A. Lee, V . Vestman, A. Nautsch, M. Todisco, X. Wang, M. Sahidullah, J. Yam- agishi, and D. A. Reynolds. ASVspoof 2021: Automatic speaker verification spoofing and countermeasures challenge evaluation plan. InProceedings ASVspoof Workshop, 2021
2021
-
[7]
Desplanques, J
B. Desplanques, J. Thienpondt, and K. Demuynck. ECAPA- TDNN: Emphasized channel attention, propagation and ag- gregation in TDNN based speaker verification. InProceed- ings Interspeech, pages 3830–3834, 2020
2020
-
[8]
Frank and L
J. Frank and L. Sch ¨onherr. WaveFake: A data set to facilitate audio deepfake detection. InProceedings NeurIPS Datasets and Benchmarks Track, 2021
2021
-
[9]
Gal and Z
Y . Gal and Z. Ghahramani. Dropout as a Bayesian approxi- mation: Representing model uncertainty in deep learning. In Proceedings International Conference on Machine Learning (ICML), pages 1050–1059, 2016
2016
-
[10]
H. Gu, J. Yi, C. Wang, Y . Ren, J. Tao, X. Yan, Y . Chen, and X. Zhang. Utilizing speaker profiles for impersonation audio detection. InProceedings 32nd ACM International Conference on Multimedia, pages 1961–1970, 2024
1961
-
[11]
P. Kawa, M. Plata, and P. Syga. Improved deepfake detection using whisper features. InProceedings Interspeech, pages 4009–4013, 2023
2023
-
[12]
Khanjani, G
Z. Khanjani, G. Watson, and V . P. Janeja. Audio deepfakes: A survey.Frontiers in Big Data, 5:1001063, 2023
2023
-
[13]
Lakshminarayanan, A
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep en- sembles.Advances in Neural Information Processing Sys- tems, 2017
2017
-
[14]
J. Li, W. Lin, L. Chen, and Z. Wu. FreeVC: Towards high- quality text-free one-shot voice conversion. InProceedings IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2023
2023
-
[15]
X. Liu, M. Liu, L. Wang, K. A. Lee, H. Zhang, and J. Dang. Leveraging positional-related local-global dependency for synthetic speech detection. InProceedings IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 1–5, 2023
2023
-
[16]
X. Liu, X. Wang, M. Sahidullah, J. Patino, H. Del- gado, T. Kinnunen, M. Todisco, J. Yamagishi, N. Evans, A. Nautsch, and K. A. Lee. ASVspoof 2021: To- wards spoofed and deepfake speech detection in the wild. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2507–2522, 2023
2021
-
[17]
M ¨uller, P
N. M ¨uller, P. Czempin, F. Dieckmann, A. Froghyar, and K. B¨ottinger. Does audio deepfake detection generalize? In Proceedings Interspeech, pages 2783–2787, 2022
2022
-
[18]
Nagrani, J
A. Nagrani, J. S. Chung, and A. Zisserman. V oxCeleb: A large-scale speaker identification dataset. InProceedings In- terspeech, pages 2616–2620, 2017
2017
-
[19]
J. Pan, S. Nie, H. Zhang, S. He, K. Zhang, S. Liang, X. Zhang, and J. Tao. Speaker recognition-assisted robust audio deepfake detection. InProceedings Interspeech, pages 4202–4206, 2022
2022
-
[20]
Pianese, D
A. Pianese, D. Cozzolino, G. Poggi, and L. Verdoliva. Deep- fake audio detection by speaker verification. InProceedings IEEE International Workshop on Information Forensics and Security (WIFS), pages 1–6, 2022
2022
-
[21]
Ravanelli, T
M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cor- nell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, et al. Speechbrain: A general-purpose speech toolkit, 2021
2021
-
[22]
Sisman, J
B. Sisman, J. Yamagishi, S. King, and H. Li. An overview of voice conversion and its challenges: From statistical mod- eling to deep learning.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:132–157, 2021
2021
-
[23]
H. Tak, J.-w. Jung, J. Patino, M. Kamble, M. Todisco, and N. Evans. End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detection.arXiv preprint arXiv:2107.12710, 2021
Pith/arXiv arXiv 2021
-
[24]
H. Tak, J. Patino, M. Todisco, A. Nautsch, N. Evans, and A. Larcher. End-to-end anti-spoofing with RawNet2. In Proceedings IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6369–6373, 2021
2021
-
[25]
H. Tak, M. Todisco, X. Wang, J. Yamagishi, and N. Evans. Automatic speaker verification spoofing and deepfake de- tection using wav2vec 2.0 and data augmentation. InPro- ceedings The Speaker and Language Recognition Workshop (Odyssey), pages 112–119, 2022
2022
-
[26]
X. Tan, T. Qin, F. Soong, and T.-Y . Liu. A survey on neural speech synthesis.arXiv preprint arXiv:2106.15561, 2021
Pith/arXiv arXiv 2021
-
[27]
Todisco, X
M. Todisco, X. Wang, V . Vestman, M. Sahidullah, H. Del- gado, A. Nautsch, J. Yamagishi, N. Evans, T. Kinnunen, and K. A. Lee. ASVspoof 2019: Future horizons in spoofed and fake audio detection. InProceedings Interspeech, pages 1008–1012, 2019
2019
-
[28]
X. Wang, J. Yamagishi, M. Todisco, H. Delgado, A. Nautsch, N. Evans, M. Sahidullah, V . Vestman, T. Kinnunen, K. A. Lee, L. Juvela, P. Alku, Y .-H. Peng, H.-T. Hwang, Y . Tsao, H.-M. Wang, S. L. Maguer, M. Becker, F. Henderson, R. Clark, Y . Zhang, and Q. Wang. ASVspoof 2019: A large- scale public database of synthesized, converted and replayed speech.Com...
2019
-
[29]
weon Jung, S
J. weon Jung, S. bin Kim, H. jin Shim, J. ho Kim, and H.-J. Yu. RawNet: Advanced end-to-end deep neural network us- ing raw waveforms for text-independent speaker verification. InProceedings Interspeech, pages 1268–1272, 2019
2019
-
[30]
weon Jung, H.-S
J. weon Jung, H.-S. Heo, H. Tak, H. jin Shim, J. S. Chung, B.-J. Lee, H.-J. Yu, and N. Evans. AASIST: Audio anti- spoofing using integrated spectro-temporal graph attention networks. InProceedings IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6367–6371, 2022
2022
-
[31]
J. Yi, Y . Bai, J. Tao, H. Ma, Z. Tian, C. Wang, T. Wang, and R. Fu. ADD 2022: The first audio deep fake detection challenge. InProceedings IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 9216–9220, 2022
2022
-
[32]
J. Yi, J. Tao, R. Fu, X. Yan, C. Wang, T. Wang, C. Y . Zhang, X. Zhang, Y . Zhao, Y . Ren, L. Xu, J. Zhou, H. Gu, Z. Wen, S. Liang, Z. Lian, S. Nie, and H. Li. ADD 2023: The second audio deepfake detection challenge.arXiv preprint arXiv:2305.13774, 2023
Pith/arXiv arXiv 2023
-
[33]
J. Yi, C. Wang, J. Tao, X. Zhang, C. Y . Zhang, and Y . Zhao. Audio deepfake detection: A survey.arXiv preprint arXiv:2308.14970, 2023
Pith/arXiv arXiv 2023
-
[34]
Zhang, S
Q. Zhang, S. Wen, and T. Hu. Audio deepfake detection with self-supervised XLS-R and SLS classifier. InProceedings 32nd ACM International Conference on Multimedia, pages 6765–6773, 2024
2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.