Pith. sign in

REVIEW 2 major objections 5 minor 18 references

Second Competition on Presentation Attack Detection on ID Card

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper reports the outcome of the second open competition on ID-card presentation attack detection, claiming that the field improved sharply: the best open-track system reached AVRank 14.76% and EER 6.36%, versus 74.30% and 21.87% in…

desk verdict Useful second-edition competition report with real assets (synthetic training set, two-track protocol, longitudinal benchmark), but the Track 2 winner claim rests on a 0.04-point margin that needs uncertainty analysis before it can be taken at face value. read the letter →

arxiv 2507.20404 v1 pith:MFCMJUZA submitted 2025-07-27 cs.CV

Juan E. Tapia , Mario Nieto , Juan M. Espin , Alvaro S. Rocamora , Javier Barrachina , Naser Damer , Christoph Busch , Marija Ivanovska
show 25 more authors
Leon Todorov Renat Khizbullin Lazar Lazarevich Aleksei Grishin Daniel Schulz Sebastian Gonzalez Amir Mohammadi Ketan Kotwal Sebastien Marcel Raghavendra Mudgalgundurao Kiran Raja Patrick Schuch Sushrut Patwardhan Raghavendra Ramachandra Pedro Couto Pereira Joao Ribeiro Pinto Mariana Xavier Andrés Valenzuela Rodrigo Lara Borut Batagelj Marko Peterlin Peter Peer Ajnas Muhammed Diogo Nunes Nuno Gonçalves
This is my paper · ORCID
classification cs.CV
keywords presentationattackdetectionIDcardbiometricsbenchmarkAVRanksequesteredtestsetsyntheticdatasetspoof
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports the outcome of the second open competition on detecting presentation attacks on ID cards, meaning attempts to pass off printed, displayed, or altered copies of identity documents as genuine. Its central claim is that the field is improving: on a sequestered test set of 23,851 images, the best submitted system in the open-data track reached an average ranking of 14.76% and an equal-error rate of 6.36%, compared with 74.30% and 21.87% for the best system in the first edition. The paper also introduces a shared synthetic training set and an automatic evaluation platform, and documents that the hardest remaining cases are screen attacks in the open track and unseen-country generalization, especially Panama. A sympathetic reader takes away a concrete benchmark number to beat, not a proof that any particular architecture is best.

What carries the argument

The load-bearing mechanism is the evaluation protocol: a sequestered test set of 23,851 images from Chile, Guatemala, Panama, and Mexico, with bona fide samples plus screen, print, and composite attacks; an automated evaluation platform that scores each submission through a Docker API; and a headline metric AVRank = 0.2·BPCER10 + 0.3·BPCER20 + 0.5·BPCER100, which weights the most security-critical operating points most heavily. The paper also provides baselines, an EfficientNetV2-S trained on the shared synthetic data for Track 1 and three MobileViTv2 baselines trained on private and open data for Track 2, so that submitted models are compared against reproducible reference points.

What would settle it

If the organizers make per-image scores public, a bootstrap test on AVRank would settle the headline claim: if the 95% confidence intervals for Incode and Baseline-1 overlap, the Track 2 win is not statistically distinguishable under the chosen metric; an even simpler check is to recompute the leaderboard with equal weights on the three BPCER points and see whether the ranking flips.

Watch

Extended reading notes

Core claim

The discovery is the measured state of ID-card presentation attack detection in 2025, established by evaluating 74 submitted models on a fixed sequestered test set from four countries. In Track 1, where all teams trained on the shared synthetic dataset, the Dragons team won with AVRank 40.48% and EER 11.34%, using a CLIP visual encoder fine-tuned with LoRA and a YOLOv8-based document crop. In Track 2, where teams could use any data, Incode won with AVRank 14.76% and EER 6.36%, using an ensemble of two CNNs and one transformer trained on roughly 218,000 proprietary images. These numbers improve on the first edition's best AVRank of 74.30% and EER of 21.87%, and the paper interprets the gap as evidence that larger stores of bona fide images, not only attack diversity, drive generalization.

Load-bearing premise

The ranking of winners rests on the competition's hand-chosen weighted average of error rates at three operating points, and a different weighting, or any uncertainty estimate, could reorder the top teams, especially in Track 2 where the winner's margin over a baseline is 0.04 points.

Editorial extensions

If this is right

  • Future submissions can use AVRank 40.48% and 14.76% as the published state of the art for the two tracks on this test set.
  • The Track 1 winner's approach, segmentation-based cropping plus a fine-tuned CLIP encoder, shows that pretrained visual-language features transfer to unseen ID-card countries when training data is limited.
  • The Track 2 result ties performance mainly to training data scale and diversity: the winning ensemble used about 218,000 images and beat all three baselines, while smaller-data submissions remained far behind.
  • The organizers' call for future editions to track inference time recognizes that several foundation-model submissions take 1–2 minutes per image, too slow for online onboarding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the organizers release per-image scores, the same leaderboard could be recomputed under alternative weighting schemes, turning the competition's single ranking into a sensitivity analysis; under some weightings the Track 2 order may differ.
  • A direct next experiment is to train the Track 1 shared synthetic dataset with the Track 2 winner's ensemble recipe; the comparison would isolate how much of the two-track gap comes from data scale versus architecture.
  • The paper's emphasis on bona fide image count suggests a cheap test: add synthetic bona fide variations of real card templates and measure whether AVRank on Panama and screen attacks drops.
  • The Track 1 result also suggests that contrastive language-image pretraining can serve as a data-efficient starting point for ID-card PAD; replacing CLIP with a CNN or supervised ViT in the same pipeline would test that hypothesis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper reports the organization and results of the second Presentation Attack Detection competition on ID cards (PAD-ID Card 2025). Track 1 provides a shared synthetic training set of 12,000 images; Track 2 allows participants to use any open or private data. All submissions are evaluated on a sequestered test set of 23,851 images from four countries using ISO/IEC 30107-3 metrics, with the winner determined by AVRank, a weighted combination of BPCER10, BPCER20, and BPCER100. The reported winners are Dragons (Track 1, AVRank 40.48%) and Incode (Track 2, AVRank 14.76%), with Incode improving on the 2024 competition's best AVRank of 74.30%. The paper also describes the dataset generation, baselines, participant methods, and DET curves.

Significance. The main contribution is an independent, standardized benchmark of current ID-card PAD algorithms on a sequestered test set, together with a shared synthetic training dataset and an automated evaluation platform. These assets are valuable to the community, and the reported improvement over the first edition suggests genuine progress in the field. The paper is also honest about limitations, explicitly noting in Section 7 that model efficiency and the cost of inference need consideration in future editions. However, the significance of the headline claims is conditional on the statistical robustness of the winner ranking, which the paper currently does not establish.

major comments (2)
  1. [§4, Eq. (3); Table 7] The Track 2 winner claim is not statistically supported. Incode's AVRank (14.76%) exceeds Baseline-1 (14.80%) by only 0.04 percentage points, while Baseline-1 has a lower EER (6.07% vs. 6.36%) and a lower BPCER20 (7.90% vs. 9.08%). The decisive BPCER100 advantage (23.04% vs. 23.64%) corresponds to roughly 30 of the 5,000 bona fide test images. Without confidence intervals, bootstrap estimates, or a sensitivity analysis over the AVRank weights in Eq. (3), the reported winner ranking cannot be distinguished from noise; an equal-weight version of Eq. (3) would rank Baseline-1 ahead (11.533% vs. 11.560%). The paper should report uncertainty estimates and show that the winner determination is robust to reasonable changes in the metric weights.
  2. [§3; §2.3] The protocol's best-of-N selection and the unchanged test set weaken the comparative claims. Section 3 states that 'only the best submission per participant is shown and reported,' and Section 2.3 states that the sequestered test set 'remains unchanged from the previous edition.' Since participants could submit multiple models, the reported margins—particularly the 0.04-point Track 2 margin—reflect selection over submissions as well as algorithm quality. The reuse of the 2024 test set also allows participants to have tuned to the exact evaluation distribution. The paper should report all submissions or at least the number of submissions per team and discuss the effect of this selection on the reported results.
minor comments (5)
  1. [Abstract; Table 6] The abstract states an EER of 11.44% for the Track 1 winner, while Table 6 reports 11.34%; please correct the inconsistency and ensure that all reported numbers match between the abstract, tables, and text.
  2. [Table 3] In Table 3, the Validation count for Composite is written as '15.900'; this should be '15,900' to match the comma format used in the other entries.
  3. [§5.3.5] The IDCH description contains a typo: 'boan fide' should be 'bona fide'.
  4. [Figure 2; Figure 3] The captions for Figures 2 and 3 refer to per-country panels '(e),(f), and (g)', but each figure has eight panels (a)–(h); please update the caption text to reference all per-country panels.
  5. [§7] Section 7 raises model efficiency as a concern for deployment but provides no runtime data; either add timing results or explicitly mark this as an open direction for future work.

Circularity Check

0 steps flagged · score 2.0 of 10

Competition results are empirical measurements on a sequestered test set, not derivations; no load-bearing circular step found, though the reused test set and organizer-selected AVRank weights are self-referential design choices.

full rationale

The paper's central claims are empirical: 74 models were evaluated on a fixed sequestered test set of 23,851 images, and the winner is determined by a stated weighted metric AVRank = 0.2*BPCER10 + 0.3*BPCER20 + 0.5*BPCER100 (Eq. 3). The winner is not derived from any fitted parameter or from the definition of the metric; the numbers in Tables 6 and 7 are measurements of independently trained submitted systems against an external test set. The test set and the AVRank weighting are chosen by the organizers (Section 2.3 says the test set 'remains unchanged from the previous edition' [14], and Section 4 defines the hand-selected weights), which is self-referential design rather than circular derivation. The Track 2 winner margin (Incode 14.76% vs Baseline-1 14.80%) is within plausible sampling noise, and the abstract's Dragons EER (11.44%) disagrees with Table 6 (11.34%), but these are correctness/robustness concerns, not circularity: the values do not reduce to their inputs by construction, and no fit is renamed as a prediction. The self-citation of the prior competition [14] for the reused test set and baselines is transparent and does not carry the load of the winner claim, which rests on the measured scores of independently submitted models. Therefore no specific circular step can be quoted, and the appropriate score is low.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced; the load-bearing choices are the metric weights and dataset assumptions.

free parameters (1)
  • AVRank weighting factors = 0.2 (BPCER10), 0.3 (BPCER20), 0.5 (BPCER100)
    These hand-chosen weights define the competition winner. Track 2's top two results differ by only 0.04 AVRank points, so the winner depends on these exact weights.
assumptions (3)
  • domain assumption The sequestered test set remains unseen by participants despite being unchanged from the 2024 edition.
    Stated in Section 2.3; if participants iterated against leaderboard feedback through the EvalAI platform, the assumption fails and results reflect test-set overfitting.
  • domain assumption Any image that fails model processing is scored as an attack.
    Stated in Section 3: 'Any processing error is treated with a score of 0, i.e., an attack is detected'; this convention partly determines error rates for models that reject low-quality inputs.
  • ad hoc to paper The chosen AVRank weighting is an accepted summary of real-world security requirements.
    Equation 3 assigns weights to BPCER10/20/100 without external validation, and the Track 2 ranking is sensitive to these weights.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Second Competition on Presentation Attack Detection on ID Card." pith.science (2026). https://pith.science/paper/MFCMJUZA

@misc{pith2026250720404,
  author       = {Pith},
  title        = {Pith review of: Second Competition on Presentation Attack Detection on ID Card},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MFCMJUZA}},
  note         = {Machine review of arXiv:2507.20404}
}
read the original abstract

This work summarises and reports the results of the second Presentation Attack Detection competition on ID cards. This new version includes new elements compared to the previous one. (1) An automatic evaluation platform was enabled for automatic benchmarking; (2) Two tracks were proposed in order to evaluate algorithms and datasets, respectively; and (3) A new ID card dataset was shared with Track 1 teams to serve as the baseline dataset for the training and optimisation. The Hochschule Darmstadt, Fraunhofer-IGD, and Facephi company jointly organised this challenge. 20 teams were registered, and 74 submitted models were evaluated. For Track 1, the "Dragons" team reached first place with an Average Ranking and Equal Error rate (EER) of AV-Rank of 40.48% and 11.44% EER, respectively. For the more challenging approach in Track 2, the "Incode" team reached the best results with an AV-Rank of 14.76% and 6.36% EER, improving on the results of the first edition of 74.30% and 21.87% EER, respectively. These results suggest that PAD on ID cards is improving, but it is still a challenging problem related to the number of images, especially of bona fide images.

Figures

Figures reproduced from arXiv: 2507.20404 by the authors.

Figure 1
Figure 1. Examples of the different presentation and manipulation [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Track 1 global, per PAIS (b),(c) and, (d) and per country DET curve results (e),(f), and (g). [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Track 2 global, per PAIS (b),(c) and, (d) and per country DET curve results (e),(f) and, (g). [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 17 canonical work pages

  1. [1]

    V . L. Blankers, C. E. v. d. Heuvel, K. Y . Franke, and L. G. Vuurpijl. ICDAR signature verification competition. In 10th International Conference on Document Analysis and Recog- nition, pages 1403–1407, 2009

    work page 2009

  2. [2]

    Cheema and S

    U. Cheema and S. Moon. Sejong face database: A multi- modal disguise face database. Computer Vision and Image Understanding, 208-209:103218, 2021

    work page 2021

  3. [3]

    Cherti, R

    M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Il- harco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jit- sev. Reproducible scaling laws for contrastive language- image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2818– 2829, 2023

    work page 2023

  4. [4]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei- Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conf. on computer vision and pattern recognition , pages 248–255. Ieee, 2009

    work page 2009

  5. [5]

    Gonzalez and J

    S. Gonzalez and J. E. Tapia. Forged presentation attack de- tection for ID cards on remote verification systems. Pattern Recognition, 162:111352, 2025

    work page 2025

  6. [6]

    A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Effi- cient convolutional neural networks for mobile vision appli- cations, 2017

    work page 2017

  7. [7]

    D. H. Lee, S. B. Yoo, M. Choi, J. B. Ra, and J. Kim. Block poisson method and its application to large scale image edit- ing. In 19th IEEE International Conference on Image Pro- cessing, pages 2121–2124, 2012

    work page 2012

  8. [8]

    M. I. Malik, S. Ahmed, A. Marcelli, U. Pal, M. Blumen- stein, L. Alewijns, and M. Liwicki. ICDAR2015 competi- tion on signature verification and writer identification for on- and off-line skilled forgeries. In 13th International Confer- ence on Document Analysis and Recognition (ICDAR), pages 1186–1190, 2015

    work page 2015

Show all 18 references
  1. [9]

    M. I. Malik, M. Liwicki, L. Alewijnse, W. Ohyama, M. Blu- menstein, and B. Found. ICDAR competitions on signature verification and writer identification for on and offline skilled forgeries. In 12th International Conference on Document Analysis and Recognition, pages 1477–1483, 2013

    2013

  2. [10]

    R. P. Markham, J. M. E. L ´opez, M. Nieto-Hidalgo, and J. E. Tapia. Open-set: ID card presentation attack detection using neural style transfer. IEEE Access, 12:68573–68585, 2024

    2024

  3. [11]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. V . V o, M. Szafraniec, et al. DINOv2: Learning robust visual fea- tures without supervision. Transactions on Machine Learn- ing Research, 2024. Featured Certification

    2024

  4. [12]

    Strohminger, K

    N. Strohminger, K. Gray, V . Chituc, J. Heffner, C. Schein, and T. Heagins. The MR2: A multi-racial, mega-resolution database of facial stimuli. Behavior research methods, 48, 08 2015

    2015

  5. [13]

    Tan and Q

    M. Tan and Q. Le. Efficientnetv2: Smaller models and faster training. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research, pages 10096–10106. PMLR, 2021

    2021

  6. [14]

    J. E. Tapia, N. Damer, C. Busch, J. M. Espin, J. Barrachina, A. S. Rocamora, K. Ocvirk, L. Alessio, B. Batagelj, S. Pat- wardhan, R. Ramachandra, R. Mudgalgundurao, K. Raja, D. Schulz, and C. Aravena. First competition on presenta- tion attack detection on ID Card. In2024 IEEE...

    2024

  7. [15]

    Varghese and S

    R. Varghese and S. M. YOLOv8: A novel object detec- tion algorithm with enhanced performance and robustness. In International Conference on Advances in Data Engineer- ing and Intelligent Computing Systems (ADICS), pages 1–6, 2024

    2024

  8. [16]

    B. P. Veasey and A. A. Amini. Low-rank adaptation of pre- trained large vision models for improved lung nodule malig- nancy classification. IEEE Open Journal of Engineering in Medicine and Biology, pages 1–9, 2025

    2025

  9. [17]

    Vieira, A

    T. Vieira, A. Bottino, A. Laurentini, and M. Simone. Detect- ing siblings in image pairs. The Visual Computer, 30:1–13, 12 2013

    2013

  10. [18]

    Yadav, R

    D. Yadav, R. Jain, H. Agrawal, P. Chattopadhyay, T. Singh, A. Jain, S. B. Singh, S. Lee, and D. Batra. EvalAI: To- wards better evaluation systems for AI agents. ArXiv, arXiv:1902.03570, 2019

    1902 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.