REVIEW 1 major objections 6 minor 3 cited by
FantasyID: A dataset for detecting digital manipulations of ID-documents
T0 review · 1 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The FantasyID benchmark shows that current forgery detectors fail to spot face-only manipulations of identity documents.
desk verdict A genuinely useful new ID-forgery dataset whose headline evaluation table has a load-bearing flaw: FNR is pooled across attacks, so the per-attack HTERs, including the face-only numbers, don't measure what they claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the FantasyID dataset itself. Its construction pipeline has three stages: 13 multilingual card templates with real faces are generated; 362 cards are printed on a plastic-card printer; and each printed card is captured by an iPhone 15 Pro, a Huawei Mate 30, and an office scanner, yielding 1,086 bonafide images. The forged class is produced by face swapping with InSwapper and Facedancer and by text inpainting with DiffSTE and Textdiffuser2, organized into a train-val split and a held-out test split with three attack protocols: Attack-1 (text-only, built from train-val card identities), Attack-2 (face-only, on held-out cards), and Attack-3 (text-only, on held-out cards). The evaluation machinery is a binary detection protocol in which the decision threshold is fixed on the validation set at 10% false positives.
What would settle it
Train a detector on the provided split, then remove from training all pristine captures that were used to create Attack-1 forgeries and retrain; if Attack-1 performance drops substantially, the reported text-manipulation results depend on train/test image overlap rather than on a general ability to detect document tampering.
Extended reading notes
Core claim
On its own terms, the paper establishes that FantasyID can serve as a challenging evaluation benchmark for ID-document manipulation detection. The test set contains 300 bonafide images and 1,085 manipulated images spanning text-only edits (Attack-1 and Attack-3) and a face-only swap attack (Attack-2). Using the threshold set on the validation split at a false positive rate of 10%, the four baseline detectors—TruFor, MMFusion, UniFD, and FatFormer—achieve false negative rates from 47.7% to 92.7% and HTERs of 25.8% to 58.2%. On Attack-2, every baseline is near chance: MMFusion's HTER of 37.8% is the best, and TruFor, UniFD, and FatFormer land at 48.3%, 50.0%, and 58.2%. The authors interpret this as evidence that current algorithms do not generalize to realistic KYC-style forgeries, and that face-only manipulations are a particular blind spot.
Load-bearing premise
The benchmark's validity rests on the test forgeries not being recognizable from their pristine originals in the training data; Attack-1 is produced from the very images that make up the train-val split, so a detector that memorizes card layouts could inflate the text-manipulation scores.
Editorial extensions
If this is right
- A detector that cannot beat chance on face-only swaps should not be considered ready for KYC document verification.
- FantasyID gives the community a public, commercially usable testbed whose bonafide images are not tampered specimens, removing a known source of bias from earlier ID datasets.
- Text-region forgeries are comparatively easy for noise-based detectors (AUC above 99% for MMFusion and TruFor on Attack-1/3), so progress should concentrate on blending-aware and face-region cues.
- The fixed test set and train-val split let future methods be compared directly, and the provided manipulations can support localization as well as binary detection.
Reading between the lines
- If face-only swaps are as undetectable as these results suggest, KYC systems are vulnerable to an attack that changes only the photo and leaves all printed text untouched, since detectors tuned for text edits would see nothing wrong.
- The Attack-1 protocol inherits a possible shortcut: the pristine versions of its forged cards belong to the train-val split, so a model trained on FantasyID could memorize those cards; re-running Attack-1 on the held-out 300 cards would cleanly test generalization.
- The near-random scores of the two full-image detectors imply that global generation artifacts, not local tampering traces, drive their decisions; adapting them to local edits would require patch-level training or explicit localization supervision.
- The paper itself notes that localizing small text edits, such as a single expired-date digit, is a harder remaining challenge that the current binary protocol does not yet capture.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FantasyID, a new publicly available dataset for detecting digital manipulations of ID documents. The dataset consists of 1,086 bonafide images of printed fantasy ID cards captured with three devices, plus 1,085 digitally forged images covering text inpainting (Attack-1 and Attack-3) and face swapping (Attack-2). The authors evaluate four state-of-the-art image-manipulation detectors (TruFor, MMFusion, UniFD, FatFormer) using pretrained models, reporting that text manipulation is detected with moderate success by TruFor and MMFusion while all baselines perform near random on face-only manipulation. The central claim is that FantasyID is a challenging, publicly available benchmark for ID forgery detection.
Significance. If the benchmark construction is sound, FantasyID fills a genuine gap: existing ID datasets are either unsuitable for manipulation detection, contain tampered 'specimen' bonafide images, or use generated faces. The dataset's strengths include pristine bonafide cards, real faces, diverse languages and card designs, physical printing and multi-device capture, and multiple attack types. The baseline evaluation is reproducible because the authors use public pretrained models and clearly describe the evaluation protocol. The paper's most interesting finding, that face-only manipulations are missed by all tested detectors, is a useful empirical observation for the forgery-detection community, even though it uses generic detectors rather than ID-specific methods.
major comments (1)
- [Section 3.2, 'Test' paragraph and Table 2] The Attack-1 test set is not a cleanly held-out test set, and this compromises the benchmark use case that the paper advertises. The text states that Attack-1 'consists of 786 images with text-only modification created by finetuning Textdiffuser2' applied to 'the bonafide of train-val set', while the bonafide class for the test set is a separate set of 300 cards distinct from train-val. Consequently, in the Attack-1 protocol the forged images come from the train-val card distribution and the bonafide images come from a different distribution. A model fine-tuned on the train-val split could achieve high accuracy by classifying old-card images as forged without learning to detect manipulation at all, for instance by exploiting card template, face identity, or capture-device characteristics. This is not merely a subtle leakage: Attack-1 accounts for 786 of the 1,085 test forged images, so the 'all' protocol inherits the same confound. The current baseline numbers are not invalidated, because the baselines are pretrained and not fine-tuned on FantasyID, but the paper's stated purpose—that the train-val set can be used to tune models and the test set can be used as a benchmark—is undermined for Attack-1. The authors should either exclude Attack-1 from the held-out benchmark claims, clearly warn users about the distribution shift, or restructure the split so that all test attacks are derived from held-out bonafide cards.
minor comments (6)
- [Abstract] The sentence 'It especially evident, in the evaluation conditions...' is missing a verb; it should read 'This is especially evident in the evaluation conditions...'.
- [Section 4.1 and Table 3] The identical FNR values for each model across the 'all', Attack-1, Attack-2, and Attack-3 rows are expected if the same 300 bonafide test images are used as the positive set for every protocol. The paper should state this explicitly in the protocol description to avoid the appearance of a pooled or copied FNR, and to make the per-attack HTER definition precise.
- [Section 4.1] The definitions of FPR and FNR are ambiguous: 'false positive rate (FPR), where positives are bonafide images' could be read as the fraction of bonafide images rejected, whereas the reported values (e.g., 34.7% for Attack-2 TruFor) suggest FPR is the fraction of forged images accepted as bonafide. Please define both quantities explicitly in terms of the positive and negative classes.
- [Table 1] The 'Use Case' column for MIDV-2020 says 'ID Rec.', but the text in Section 2 states that MIDV-2020 also proposes altered text and face images for forgery detection. The table should reflect that dual use.
- [Section 4.2] There are several typographical issues: 'HT ER= 23.9%' and 'AU Cabove 99%' should read 'HTER = 23.9%' and 'AUC above 99%'.
- [Figure 6] The KDE plots have no y-axis label; adding a label such as 'estimated density' would improve readability.
Circularity Check
No circular reasoning identified; the dataset and its evaluation are self-contained empirical contributions.
full rationale
FantasyID is an empirical dataset paper rather than a derivation. The bonafide class is created by designing, printing, and recapturing physical cards; the attack classes are generated with third-party tools (InSwapper, Facedancer, Textdiffuser2, DiffSTE). There is no equation or model in the paper whose output is defined in terms of its own input. The baselines (TruFor, MMFusion, UniFD, FatFormer) are pretrained models released by their respective authors and used without fine-tuning on FantasyID, so the evaluation does not assume the result it claims; the operational threshold is set on the validation set at FPR=10% and metrics are then computed on the held-out test set. No load-bearing argument is justified solely by a self-citation: the authors' own HQ-WMCA dataset is used only as a face source for card generation, and their deepfake papers are background citations, not premises of the benchmark conclusion. The main caveats are experimental-validity issues, not circularity: Attack-1 is created by manipulating the same 786 bonafide captures that appear in the train-val split, and Table 3 repeats the same FNR value across all protocol rows, which may indicate a pooled rather than per-attack FNR. Even if those caveats weaken the reported Attack-2 HTERs, they are not cases of a result reducing by definition to its input. Therefore no circular step is identified.
Assumptions & free parameters
assumptions (4)
- domain assumption Printed and recaptured cards simulate real KYC onboarding captures.
- domain assumption The four manipulation methods represent the range of realistic forgery attacks.
- domain assumption Using real faces from AMFD, Face London, HQ-WMCA and Flickr does not bias the dataset in a way that affects manipulation detection.
- domain assumption The selected baseline detectors are representative state-of-the-art methods for this task.
Cite this review
Pith. "Pith review of FantasyID: A dataset for detecting digital manipulations of ID-documents." pith.science (2026). https://pith.science/paper/ZIC345XN
@misc{pith2026250720808,
author = {Pith},
title = {Pith review of: FantasyID: A dataset for detecting digital manipulations of ID-documents},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZIC345XN}},
note = {Machine review of arXiv:2507.20808}
}
read the original abstract
Advancements in image generation led to the availability of easy-to-use tools for malicious actors to create forged images. These tools pose a serious threat to the widespread Know Your Customer (KYC) applications, requiring robust systems for detection of the forged Identity Documents (IDs). To facilitate the development of the detection algorithms, in this paper, we propose a novel publicly available (including commercial use) dataset, FantasyID, which mimics real-world IDs but without tampering with legal documents and, compared to previous public datasets, it does not contain generated faces or specimen watermarks. FantasyID contains ID cards with diverse design styles, languages, and faces of real people. To simulate a realistic KYC scenario, the cards from FantasyID were printed and captured with three different devices, constituting the bonafide class. We have emulated digital forgery/injection attacks that could be performed by a malicious actor to tamper the IDs using the existing generative tools. The current state-of-the-art forgery detection algorithms, such as TruFor, MMFusion, UniFD, and FatFormer, are challenged by FantasyID dataset. It especially evident, in the evaluation conditions close to practical, with the operational threshold set on validation set so that false positive rate is at 10%, leading to false negative rates close to 50% across the board on the test set. The evaluation experiments demonstrate that FantasyID dataset is complex enough to be used as an evaluation benchmark for detection algorithms.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 3 Pith papers
-
From Forgeries to Foundation Models: A Systematic Survey of Identity Document Attack and Detection
A systematic survey unifies presentation, digital injection, and GenAI synthesis attacks on identity documents, audits datasets for a reality gap, identifies SDGI in multimodal models, and reports APCER above 25% for ...
-
Field-Localized Forgery Detection for Digital Identity Documents
FLiD is a field-localized forgery detection method for identity documents that outperforms full-document baselines and general detectors with significantly fewer parameters.
-
Detecting Text Manipulation in Images using Vision Language Models
In zero-shot benchmarks, GPT-4o outperforms open-source VLMs and specialized manipulation detectors on text tampering detection in scene images and fantasy ID documents.
Reference graph
Works this paper leans on
-
[1]
M. Al-Ghadi, Z. Ming, P. Gomez-Krämer, J.-C. Burie, M. Coustaty, and N. Sidere. Guilloche detection for id au- thentication: A dataset and baselines. In IEEE International Workshop on Multimedia Signal Processing (MMSP), pages 1–6, Poitiers, France, Sept. 2023. 2, 3, 4
work page 2023
-
[2]
V . V . Arlazarov, K. B. Bulatov, T. S. Chernov, and V . L. Ar- lazarov. MIDV-500: a dataset for identity document analysis and recognition on mobile devices in video stream. Com- puter Optics, 43(5):818–824, 2019. 2, 4
work page 2019
-
[3]
Synthetic dataset of ID and Travel Document
C. Boned, M. Talarmain, N. Ghanmi, G. Chiron, S. Biswas, A. M. Awal, and O. R. Terrades. Synthetic dataset of ID and Travel Document, Jan. 2024. arXiv:2401.01858 [cs.CV]. 2, 3, 4
work page Pith review arXiv 2024
-
[4]
K. Bulatov, E. Emelianova, D. Tropin, N. Skoryukina, Y . Chernyshova, A. Sheshkus, S. Usilin, Z. Ming, J.-C. Burie, M. M. Luqman, and V . V . Arlazarov. MIDV-2020: A comprehensive benchmark dataset for identity document analysis. Computer Optics, 46(2), Apr. 2022. 2, 3, 4
work page 2020
-
[5]
K. Bulatov, D. Matalov, and V . V . Arlazarov. MIDV-2019: challenges of the modern mobile-based document OCR. In International Conference on Machine Vision (ICMV) , vol- ume 11433, pages 717–722. SPIE, 2019. 2, 3, 4
work page 2019
-
[6]
J. Chen, Y . Huang, T. Lv, L. Cui, Q. Chen, and F. Wei. Textdiffuser-2: Unleashing the power of language models for text rendering. In European Conference on Computer Vision (ECCV), pages 386–402. Springer, 2024. 2, 5
work page 2024
-
[7]
J. M. Chen, J. B. Norman, and Y . Nam. Broadening the stim- ulus set: introducing the american multiracial faces database. Behavior Research Methods, 53:371–389, 2021. 4
work page 2021
-
[8]
X. Chen, B. Ni, Y . Liu, N. Liu, Z. Zeng, and H. Wang. Sim- Swap++: Towards Faster and High-Quality Identity Swap- ping. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(1):576–592, Jan. 2024. 2
work page 2024
Show all 26 references
-
[9]
de Sá Soares, R
A. de Sá Soares, R. B. das Neves Junior, and B. L. D. Bez- erra. BID Dataset: a challenge dataset for document pro- cessing tasks. In Anais Estendidos do XXXIII Conference on Graphics, Patterns and Images, pages 143–146. SBC, 2020. 3, 4
2020
-
[10]
DeBruine and B
L. DeBruine and B. Jones. Face Research Lab London Set, 5 2017. 4
2017
-
[11]
Guillaro, D
F. Guillaro, D. Cozzolino, A. Sud, N. Dufour, and L. Verdo- liva. TruFor: Leveraging all-round clues for trustworthy im- age forgery detection and localization. InIEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 20606–20615, 2023. 6, 7
2023
-
[12]
I. C. A. O. (ICAO). Doc 9303: Machine readable travel doc- uments, 2021. 2
2021
-
[13]
A. Jain, P. Korshunov, and S. Marcel. Improving gener- alization of deepfake detection by training for attribution. In IEEE International Workshop on Multimedia Signal Pro- cessing (MMSP), pages 1–6, 2021. 6
2021
-
[14]
J. Ji, G. Zhang, Z. Wang, B. Hou, Z. Zhang, B. Price, and S. Chang. Improving diffusion models for scene text editing with dual encoders, Apr. 2023. arXiv:2304.05568 [cs]. 2, 5
2023 arXiv
-
[15]
Kirillov, E
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4015– 4026, 2023. 5
2023
-
[16]
Korshunov and S
P. Korshunov and S. Marcel. Improving generalization of deepfake detection with data farming and few-shot learn- ing. IEEE Transactions on Biometrics, Behavior, and Iden- tity Science, Jan. 2022. 6
2022
-
[17]
H. Liu, Z. Tan, C. Tan, Y . Wei, J. Wang, and Y . Zhao. Forgery-aware adaptive transformer for generalizable syn- thetic image detection. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 10770– 10780, 2024. 6
2024
-
[18]
Mostaani, A
Z. Mostaani, A. George, G. Heusch, D. Geissbuhler, and S. Marcel. The high-quality wide multi-channel attack (HQ- WMCA) database. arXiv:2009.09703, 2020. 4
2009 arXiv
-
[19]
U. Ojha, Y . Li, and Y . J. Lee. Towards universal fake image detectors that generalize across generative models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24480–24489, 2023. 6
2023
-
[20]
Park, S.-Y
E.-J. Park, S.-Y . Back, J. Kim, and S. S. Woo. KID34K: A dataset for online identity card fraud detection. In ACM In- ternational Conference on Information and Knowledge Man- agement (CIKM), pages 5381–5385, New York, NY , USA, Oct. 2023. Association for Computing Machinery. 2, 3, 4
2023
-
[21]
Radford, J
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, and J. Clark. Learning transferable visual models from natural language supervision. In International conference on machine learn- ing (ICML), pages 8748–8763. PMLR, 2021. 6
2021
-
[22]
Rosberg, E
F. Rosberg, E. E. Aksoy, F. Alonso-Fernandez, and C. En- glund. FaceDancer: Pose- and occlusion-aware high fidelity face swapping. In IEEE/CVF Winter Conference on Applica- tions of Computer Vision (WACV), pages 3454–3463, 2023. 2
2023
-
[23]
Rosberg, E
F. Rosberg, E. E. Aksoy, F. Alonso-Fernandez, and C. En- glund. Facedancer: Pose-and occlusion-aware high fidelity face swapping. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3454– 3463, 2023. 2, 5, 7
2023
-
[24]
Suvorov, E
R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, K. Park, and V . Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. arXiv preprint arXiv:2109.07161,
-
[25]
J. E. Tapia, N. Damer, C. Busch, J. M. Espin, J. Bar- rachina, A. S. Rocamora, K. Ocvirk, L. Alessio, B. Batagelj, S. Patwardhan, R. Ramachandra, R. Mudgalgundurao, K. Raja, D. Schulz, and C. Aravena. First Competition on Presentation Attack Detection on ID Card, Aug. 2024. ar...
2024 arXiv
-
[26]
Triaridis and V
K. Triaridis and V . Mezaris. Exploring multi-modal fusion for image manipulation detection and localization. In Mul- tiMedia Modeling, volume 14556, pages 198–211. Springer Nature Switzerland, 2024. 6, 7
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.