pith. sign in

arxiv: 2606.20680 · v1 · pith:YUDLQWT2new · submitted 2026-06-13 · 💻 cs.CV · eess.AS

Beyond ROC-AUC: Operating-Point Performance Reporting for Biometric Verification

Ajan Ahmed , Masudul H. Imtiaz This is my paper

Pith reviewed 2026-06-27 04:04 UTC · model grok-4.3

classification 💻 cs.CV eess.AS
keywords biometric verificationROC-AUCDET curvefalse match rateoperating pointperformance reportingbootstrap intervals
0
0 comments X

The pith

Full ROC-AUC can reverse which biometric matcher performs better at the low false-match rates where systems actually operate.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Biometric verifiers are typically restricted by a strict false match budget to a narrow low-FMR slice of the score range. The full ROC-AUC averages the true match rate equally across the entire FMR range from zero to one, placing almost all its weight on regions the system never uses. This averaging can hide weak low-FMR behavior and even reverse the apparent order of two systems. The paper demonstrates the effect on seven pretrained matchers across face, voice, iris, and fingerprint, with bootstrap confidence intervals and paired tests. For face, FaceNet shows a higher full AUC while ArcFace shows a significantly higher TMR at FMR of 10 to the minus three, with non-overlapping intervals.

Core claim

The full ROC-AUC averages TMR with equal weight over the FMR range from 0 to 1, so almost all of its weight is placed where the system is never operated; low-FMR behavior can then be hidden, and the order of two systems can even be reversed. Tested on seven matchers, a system stronger on full AUC proved significantly worse at FMR = 10^-3, with the face example of FaceNet versus ArcFace confirmed by non-overlapping bootstrap intervals.

What carries the argument

The DET curve together with FNMR reported at a fixed low FMR, each with bootstrap confidence intervals and paired bootstrap tests.

If this is right

  • System rankings obtained from full ROC-AUC can differ from rankings obtained at the low-FMR operating points actually used.
  • The DET curve and FNMR at a stated FMR become the primary performance report, with ROC-AUC and EER retained only as supplementary context.
  • Bootstrap intervals and paired tests are required to establish that observed differences at a chosen operating point are significant.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Model training objectives that optimize full AUC may need re-examination when the intended deployment point lies far from the middle of the score range.
  • The same reporting mismatch could affect other threshold-based systems that are constrained to one tail of their error curve.
  • Public leaderboards that publish only AUC may therefore select the wrong system for security-sensitive applications.

Load-bearing premise

Biometric verifiers are deployed under a strict false match budget that restricts operation to a narrow low-FMR slice of the score range.

What would settle it

A set of biometric matchers in which full ROC-AUC rankings match the rankings at FMR = 10^-3 across all four modalities, with overlapping intervals.

Figures

Figures reproduced from arXiv: 2606.20680 by Ajan Ahmed, Masudul H. Imtiaz.

Figure 1
Figure 1. Figure 1: The same face system (ArcFace on LFW) under two FMR axes. Top, a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Mated and non-mated score distributions per modality, with the fixed FMR thresholds at [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: ROC curves on a logarithmic FMR axis, top row, and DET curves, bottom row, with the two matchers of each modality overlaid. The DET curve is [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: TMR and FNMR at fixed FMR targets. Deployment-relevant performance is represented by the operating points, which become difficult to resolve [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Four of the metrics across all seven systems; the remaining metrics [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
read the original abstract

A biometric verifier is often deployed with a strict false match budget, so only a narrow, low false match rate (FMR) slice of the score range is used. A reporting standard for this setting already exists. ISO/IEC 19795-1 asks for error rates at stated operating points, for the detection error tradeoff (DET) curve as the view of the trade-off between FMR and the false non-match rate (FNMR), and for an interval of uncertainty on every value. In practice, a single area under the receiver operating characteristic curve (ROC-AUC), the equal error rate (EER), or a verification accuracy is still reported as the resolution, which is a threshold-independent summary that the standard does not endorse. The full ROC-AUC averages the true match rate (TMR) with equal weight over the whole FMR range from 0 to 1, so almost all of its weight is placed where the system is never operated; low-FMR behavior can then be hidden, and the order of two systems can even be reversed. The guideline is revisited in this paper and tested against seven pretrained matchers across four modalities, face, voice, iris, and fingerprint, each reported with bootstrap confidence intervals and paired bootstrap tests. A system that looks stronger on full ROC-AUC is shown to be significantly worse at FMR = 10^-3. For face, a higher full AUC was obtained by FaceNet, whereas a higher TMR at FMR = 10^-3 was obtained by ArcFace, and both gaps were significant with non-overlapping intervals. Hence, the DET curve and the FNMR at a fixed FMR are re-iterated in this paper as the primary report, with ROC-AUC and EER retained as supplementary context.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 1 minor

Summary. The paper claims that full ROC-AUC is an inappropriate summary metric for biometric verification systems, which are typically deployed under strict false-match-rate (FMR) budgets that restrict operation to a narrow low-FMR slice of the score range. It shows that ROC-AUC can reverse system rankings relative to performance at a fixed low-FMR operating point (e.g., FMR = 10^{-3}), demonstrates this reversal empirically with seven pretrained matchers across face, voice, iris, and fingerprint modalities using bootstrap confidence intervals and paired bootstrap tests, and reiterates the ISO/IEC 19795-1 recommendation to report the DET curve and FNMR at stated operating points as primary, with ROC-AUC and EER as supplementary.

Significance. If the reported ranking reversal holds, the result is significant for the biometrics community because it supplies concrete, statistically grounded evidence (non-overlapping bootstrap intervals and paired tests) that a widely used threshold-independent metric can produce misleading comparisons under realistic deployment constraints. The explicit alignment with the existing ISO standard and the provision of reproducible statistical procedures strengthen the case for changing reporting practices.

minor comments (1)
  1. The abstract would be strengthened by a brief statement of the dataset sizes, matcher versions, and any exclusion rules used for the seven matchers, even if these details appear in the methods section.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive review, accurate summary of the contribution, and recommendation to accept. The manuscript's emphasis on operating-point reporting, statistical procedures, and alignment with ISO/IEC 19795-1 is correctly identified as the core message.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper contains no derivation, fitted model, or self-referential equations. Its central claim rests on direct empirical comparison of seven external pretrained matchers (FaceNet, ArcFace, etc.) across four modalities against the external ISO/IEC 19795-1 standard, using bootstrap CIs and paired tests. No self-citations are load-bearing, no ansatz is smuggled, and no prediction reduces to a fitted input by construction. The reported ranking reversal follows immediately from the data once the low-FMR deployment premise is granted; the work is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper invokes the ISO/IEC 19795-1 standard as background and relies on the existence of publicly available pretrained matchers; no free parameters, new axioms, or invented entities are introduced.

pith-pipeline@v0.9.1-grok · 5863 in / 1226 out tokens · 56948 ms · 2026-06-27T04:04:07.456933+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    An introduction to biometric recognition,

    A. K. Jain, A. Ross, and S. Prabhakar, “An introduction to biometric recognition,”IEEE Trans. Circuits and Systems for Video Technology, vol. 14, no. 1, pp. 4–20, 2004

    2004

  2. [2]

    FPGA-based implementations of biometric recognition: A review,

    A. Kia, A. Ahmed, and M. H. Imtiaz, “FPGA-based implementations of biometric recognition: A review,”Electronics, vol. 15, no. 11, p. 2145, 2026

    2026

  3. [3]

    Contactless fingerprint recognition using deep learning—a systematic review,

    A. M. M. Chowdhury and M. H. Imtiaz, “Contactless fingerprint recognition using deep learning—a systematic review,”Journal of Cy- bersecurity and Privacy, vol. 2, no. 3, pp. 714–730, 2022

    2022

  4. [4]

    Channel reduction for an EEG-based authentication system while performing motor movements,

    E. C. Ketola, M. Barankovich, S. Schuckers, A. Ray-Dowling, D. Hou, and M. H. Imtiaz, “Channel reduction for an EEG-based authentication system while performing motor movements,”Sensors, vol. 22, no. 23, p. 9156, 2022

    2022

  5. [5]

    Information technology — biometric performance testing and reporting — part 1: Principles and framework,

    ISO/IEC, “Information technology — biometric performance testing and reporting — part 1: Principles and framework,” International Organiza- tion for Standardization, 2021

    2021

  6. [6]

    An introduction to ROC analysis,

    T. Fawcett, “An introduction to ROC analysis,”Pattern Recognition Letters, vol. 27, no. 8, pp. 861–874, 2006

    2006

  7. [7]

    The meaning and use of the area under a receiver operating characteristic (ROC) curve,

    J. A. Hanley and B. J. McNeil, “The meaning and use of the area under a receiver operating characteristic (ROC) curve,”Radiology, vol. 143, no. 1, pp. 29–36, 1982

    1982

  8. [8]

    A Review of the FBI’s Handling of the Brandon Mayfield Case,

    U.S. Department of Justice, Office of the Inspector General, “A Review of the FBI’s Handling of the Brandon Mayfield Case,” https://oig. justice.gov/sites/default/files/legacy/special/s0601/exec.pdf, 2006, ac- cessed May 2026

    2006

  9. [9]

    Maltoni, D

    D. Maltoni, D. Maio, A. K. Jain, and S. Prabhakar,Handbook of Fingerprint Recognition, 2nd ed. Springer, 2009

    2009

  10. [10]

    Williams v. City of Detroit: Face Recognition False Arrest,

    American Civil Liberties Union, “Williams v. City of Detroit: Face Recognition False Arrest,” https://www.aclu.org/cases/ williams-v-city-of-detroit-face-recognition-false-arrest, 2024, accessed May 2026

    2024

  11. [11]

    Face Recognition Technology Evaluation (FRTE) 1:1 Verification,

    National Institute of Standards and Technology, “Face Recognition Technology Evaluation (FRTE) 1:1 Verification,” https://pages.nist.gov/ frvt/html/frvt11.html, 2026, accessed May 2026

    2026

  12. [12]

    HSBC V oice Recognition System Breached by Customer’s Twin,

    P. Collinson, “HSBC V oice Recognition System Breached by Customer’s Twin,” https://www.theguardian.com/business/2017/may/19/ hsbc-voice-recognition-system-breached-by-customers-twin, 2017, ac- cessed May 2026

    2017

  13. [13]

    ASVspoof 2019: A Large-Scale Public Database of Synthesized, Converted and Replayed Speech,

    X. Wang, J. Yamagishi, M. Todisco, H. Delgado, A. Nautsch, N. Evans, M. Sahidullah, V . Vestman, T. Kinnunen, K. A. Leeet al., “ASVspoof 2019: A Large-Scale Public Database of Synthesized, Converted and Replayed Speech,”Computer Speech & Language, vol. 64, p. 101114, 2020

    2019

  14. [14]

    Classical Machine Learning Baselines for Deepfake Audio Detection on the Fake-or-Real Dataset

    F. Ahmad, A. Ahmed, and M. H. Imtiaz, “Classical machine learning baselines for deepfake audio detection on the Fake-or-Real dataset,” https://arxiv.org/abs/2604.13400, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  15. [15]

    Descriptor: Extended-length audio dataset for synthetic voice detection and speaker recognition (ELAD-SVDSR),

    R. Vijaykumar, A. Ahmed, J. Parker, D. Pendyala, A. Collins, S. Schuck- ers, and M. H. Imtiaz, “Descriptor: Extended-length audio dataset for synthetic voice detection and speaker recognition (ELAD-SVDSR),” https://arxiv.org/abs/2510.00218, 2025

    work page arXiv 2025

  16. [16]

    How iris recognition works,

    J. Daugman, “How iris recognition works,”IEEE Trans. Circuits and Systems for Video Technology, vol. 14, no. 1, pp. 21–30, 2004

    2004

  17. [17]

    Image understanding for iris biometrics: A survey,

    K. W. Bowyer, K. Hollingsworth, and P. J. Flynn, “Image understanding for iris biometrics: A survey,”Computer Vision and Image Understand- ing, vol. 110, no. 2, pp. 281–307, 2008

    2008

  18. [18]

    Facenet: A unified embed- ding for face recognition and clustering,

    F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embed- ding for face recognition and clustering,” inProc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2015, pp. 815–823

    2015

  19. [19]

    Arcface: Additive angular margin loss for deep face recognition,

    J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” inProc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4690– 4699

    2019

  20. [20]

    Face recognition vendor test (FRVT) part 3: Demographic effects,

    P. Grother, M. Ngan, and K. Hanaoka, “Face recognition vendor test (FRVT) part 3: Demographic effects,” National Institute of Standards and Technology, Tech. Rep. NISTIR 8280, 2019

    2019

  21. [21]

    Front- end factor analysis for speaker verification,

    N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front- end factor analysis for speaker verification,”IEEE Trans. Audio, Speech, and Language Processing, vol. 19, no. 4, pp. 788–798, 2011

    2011

  22. [22]

    X-vectors: Robust DNN embeddings for speaker recognition,

    D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust DNN embeddings for speaker recognition,” inProc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 5329–5333

    2018

  23. [23]

    ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” inProc. Interspeech, 2020, pp. 3830–3834

    2020

  24. [24]

    V oxCeleb: A large-scale speaker identification dataset,

    A. Nagrani, J. S. Chung, and A. Zisserman, “V oxCeleb: A large-scale speaker identification dataset,” inProc. Interspeech, 2017, pp. 2616– 2620

    2017

  25. [25]

    NIST 2018 speaker recognition evaluation plan,

    National Institute of Standards and Technology, “NIST 2018 speaker recognition evaluation plan,” https://www.nist.gov/itl/iad/mig/ speaker-recognition, 2018

    2018

  26. [26]

    Application-independent evaluation of speaker detection,

    N. Br ¨ummer and J. du Preez, “Application-independent evaluation of speaker detection,”Computer Speech & Language, vol. 20, no. 2-3, pp. 230–275, 2006

    2006

  27. [27]

    High confidence visual recognition of persons by a test of statistical independence,

    J. G. Daugman, “High confidence visual recognition of persons by a test of statistical independence,”IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1148–1161, 1993

    1993

  28. [28]

    FVC2002: Second fingerprint verification competition,

    D. Maio, D. Maltoni, R. Cappelli, J. L. Wayman, and A. K. Jain, “FVC2002: Second fingerprint verification competition,” inProc. 16th Int. Conf. Pattern Recognition (ICPR), vol. 3, 2002, pp. 811–814

    2002

  29. [29]

    Analyzing a portion of the ROC curve,

    D. K. McClish, “Analyzing a portion of the ROC curve,”Medical Decision Making, vol. 9, no. 3, pp. 190–195, 1989

    1989

  30. [30]

    Efron and R

    B. Efron and R. J. Tibshirani,An Introduction to the Bootstrap. Chap- man & Hall/CRC, 1993

    1993

  31. [31]

    On calibration of modern neural networks,

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” inProc. 34th Int. Conf. Machine Learning (ICML), 2017, pp. 1321–1330

    2017

  32. [32]

    Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,

    Y . Gal and Z. Ghahramani, “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,” inProc. 33rd Int. Conf. Machine Learning (ICML), 2016, pp. 1050–1059

    2016

  33. [33]

    The DET curve in assessment of detection task performance,

    A. Martin, G. Doddington, T. Kamm, M. Ordowski, and M. Przy- bocki, “The DET curve in assessment of detection task performance,” inProc. 5th European Conf. Speech Communication and Technology (Eurospeech), 1997, pp. 1895–1898

    1997

  34. [34]

    Quantifying the relationship between speech quality metrics and biometric speaker recognition performance under acoustic degradation,

    A. Ahmed, M. J. A. Khondkar, A. Herrick, S. Schuckers, and M. H. Imtiaz, “Quantifying the relationship between speech quality metrics and biometric speaker recognition performance under acoustic degradation,” Signals, vol. 7, no. 1, p. 7, 2026

    2026

  35. [35]

    Descriptor: V oice pre-processing and quality assessment dataset (VPQAD),

    ——, “Descriptor: V oice pre-processing and quality assessment dataset (VPQAD),”IEEE Data Descriptions, 2024, iEEE Xplore document 10747764

    2024

  36. [36]

    A comparative analysis of deep-learning-based speech enhancement models: Assessing biometric speaker verification in real-world noisy environments,

    M. J. A. Khondkar, A. Ahmed, S. Schuckers, and M. H. Imtiaz, “A comparative analysis of deep-learning-based speech enhancement models: Assessing biometric speaker verification in real-world noisy environments,”Big Data and Cognitive Computing, vol. 10, no. 3, p. 98, 2026

    2026

  37. [37]

    A CROC stronger than ROC: Measuring, visualizing and optimizing early re- trieval,

    S. J. Swamidass, C.-A. Azencott, K. Daily, and P. Baldi, “A CROC stronger than ROC: Measuring, visualizing and optimizing early re- trieval,”Bioinformatics, vol. 26, no. 10, pp. 1348–1356, 2010

    2010

  38. [38]

    Evaluating virtual screening methods: Good and bad metrics for the “early recognition

    J.-F. Truchon and C. I. Bayly, “Evaluating virtual screening methods: Good and bad metrics for the “early recognition” problem,”Journal of Chemical Information and Modeling, vol. 47, no. 2, pp. 488–508, 2007

    2007

  39. [39]

    The relationship between precision-recall and ROC curves,

    J. Davis and M. Goadrich, “The relationship between precision-recall and ROC curves,” inProc. 23rd Int. Conf. Machine Learning (ICML), 2006, pp. 233–240

    2006

  40. [40]

    The precision-recall plot is more informa- tive than the ROC plot when evaluating binary classifiers on imbalanced datasets,

    T. Saito and M. Rehmsmeier, “The precision-recall plot is more informa- tive than the ROC plot when evaluating binary classifiers on imbalanced datasets,”PLOS ONE, vol. 10, no. 3, p. e0118432, 2015

    2015

  41. [41]

    Labeled faces in the wild: A database for studying face recognition in uncon- strained environments,

    G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in uncon- strained environments,” University of Massachusetts, Amherst, Tech. Rep. 07-49, 2007

    2007

  42. [42]

    CA- SIA iris image database version 4.0 (CASIA-Iris-Thousand),

    Chinese Academy of Sciences, Institute of Automation (CASIA), “CA- SIA iris image database version 4.0 (CASIA-Iris-Thousand),” http: //biometrics.idealtest.org/, 2010

    2010

  43. [43]

    Joint face detection and alignment using multitask cascaded convolutional networks,

    K. Zhang, Z. Zhang, Z. Li, and Y . Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,”IEEE Signal Processing Letters, vol. 23, no. 10, pp. 1499–1503, 2016

    2016

  44. [44]

    SpeechBrain: A general-purpose speech toolkit,

    M. Ravanelli, T. Parcollet, P. Plantingaet al., “SpeechBrain: A general- purpose speech toolkit,”arXiv preprint arXiv:2106.04624, 2021

    work page arXiv 2021

  45. [45]

    IRIS: Open iris recognition inference system,

    Worldcoin Foundation, “IRIS: Open iris recognition inference system,” https://github.com/worldcoin/open-iris, 2024

    2024

  46. [46]

    SourceAFIS: Fingerprint recognition library,

    R. Va ˇzan, “SourceAFIS: Fingerprint recognition library,” https:// sourceafis.machinezoo.com/, 2023

    2023

  47. [47]

    Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach,

    E. R. DeLong, D. M. DeLong, and D. L. Clarke-Pearson, “Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach,”Biometrics, vol. 44, no. 3, pp. 837– 845, 1988

    1988

  48. [48]

    Scikit-learn: Machine learning in Python,

    F. Pedregosa, G. Varoquaux, A. Gramfortet al., “Scikit-learn: Machine learning in Python,”Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011

    2011