{"id":"8f28e5dc-ce2f-4853-b378-456c01f045ed","arxiv_id":"2607.26641","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A new benchmark (FakeIDet3-DB) of digital attacks on real government-issued IDs shows SOTA detectors struggle (32.45% EER), and a new PACE algorithm extracts privacy-safe patches around redaction masks.","lead":"This paper introduces a new benchmark of digitally manipulated real government-issued ID images, along with a privacy-preserving patch extraction algorithm. It shows current forgery detectors fail badly on these realistic attacks, highlighting the gap between synthetic training data and real-world ID fraud.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Confounded comparison undermines claim that real ID substrate, not attack refinement, drives the observed difficulty gap.","rationale":"The reader's verdict (CONDITIONAL) already points to weaknesses in the benchmark comparison, including lack of error bars and post-hoc protocol choices, but its stated weakest assumption is the privacy-preservation claim. My concern is different and more directly load-bearing for the central scientific claim: the comparison between FakeIDet3-DB and FantasyID in Table V is confounded by multiple factors, most notably the refinement pipeline. This pipeline is a deliberate contribution of the paper, but its effect is not separated from the effect of using real IDs. Without such separation, the paper cannot support its headline conclusion that real-ID data are inherently more challenging. This does not invalidate the dataset or PACE, but it means the difficulty claim should be conditioned on additional experiments. Therefore, the verdict remains CONDITIONAL: the paper should be accepted only after the authors provide a controlled comparison (or substantially soften the causal claim). My agreement with the reader is partial because we both question the benchmark numbers, but the reader emphasized privacy while I emphasize the confounded difficulty comparison.","tokens_in":19271,"tokens_out":5073,"duration_ms":53539,"concrete_test":"Using the released attack-generation code, apply the same refined manipulation pipeline (Section III-B) to synthetic ID templates (e.g., from FantasyID or IDNet) to create a matched 'synthetic-refined' dataset. Then evaluate TruFor and Re-MTKD with the same frozen weights and protocol as Table V on this synthetic-refined dataset. If the resulting EER approaches that of FakeIDet3-DB (≈32% for TruFor), the difficulty gap is driven by the refinement pipeline, not ID reality. If the EER remains near the original FantasyID level (≈16%), the real-ID substrate is the dominant factor.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that FakeIDet3-DB reveals a domain gap between synthetic and real government-issued IDs is primarily supported by Table V, where TruFor's EER doubles from 16.02% on FantasyID to 32.45% on FakeIDet3-DB, and Re-MTKD's rises from 33.67% to 40.28%. This comparison is confounded: the two databases differ not only in ID provenance (synthetic vs. real) but also in attack typology (8 vs. 3 categories), the presence of a refinement post-processing pipeline (Section III-B), image resolutions, and sample sizes. The refinement pipeline is specifically designed to suppress artifacts and degrade detection; indeed, the paper notes that refined attacks worsen detection in 83.33% of evaluated cases versus cheapfakes (Section V-A). Thus, the observed difficulty increase could be entirely attributable to attack sophistication rather than the real-ID substrate. No control experiment isolates the effect of ID reality. Consequently, the strongest claim—that current detectors are far less reliable on real government IDs—is not established by the presented evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"FakeIDet3-DB introduces a new benchmark for digital manipulation detection on real, government-issued IDs, comprising 6,436 images from 250 real ID photos and 8 attack typologies, with a post-processing refinement pipeline for both classical and GenAI attacks. The paper also proposes PACE, a pseudo-anonymized contextual patch extraction algorithm based on integral images, distance transforms, and greedy NMS, and reports evaluations of several state-of-the-art detectors on full-image and patch-based benchmarks. The central claim is that FakeIDet3-DB is harder than existing synthetic proxies such as FantasyID, with TruFor's EER rising from 16.02% to 32.45%.","tokens_in":19585,"tokens_out":5326,"duration_ms":51068,"significance":"If the claims are supported, FakeIDet3-DB would be a valuable community resource: it addresses a real gap in the literature by providing publicly available, privacy-aware digital manipulations on genuine government IDs, and it ships benchmark code. PACE is a concrete algorithmic contribution to privacy-preserving patch extraction. The paper also provides reproducible evaluation protocols and makes the database available, which are strengths. However, the main evidence for the domain-gap claim is currently confounded, and the privacy guarantee is asserted rather than demonstrated. The resource is useful regardless, but the paper needs to either strengthen the causal evidence or substantially soften the headline claims.","major_comments":[{"comment":"The central claim that real government-issued IDs pose a harder detection challenge than synthetic proxies rests on Table V, where TruFor's EER rises from 16.02% on FantasyID to 32.45% on FakeIDet3-DB. This comparison is confounded: the two databases differ simultaneously in ID provenance, number and type of attack categories (8 vs 3), the presence of the refinement post-processing pipeline (Section III-B), image resolution, and sample size. The paper itself reports that refinement degrades detection in 83.33% of cases (§V-A), so the observed difference could be entirely attributable to attack sophistication rather than the real-ID substrate. A control experiment that applies the same attack pipeline and refinement to a synthetic template set, or restricts FakeIDet3-DB to the attack types present in FantasyID, is needed to isolate the effect of real ID structure. Without this, the headli","section":"§V-A, Table V"},{"comment":"The privacy-compliance claim is load-bearing. The paper states that pseudo-anonymization leaves 'small, residual sections of the sensitive data uncovered which do not compromise the owners' PII,' but no formal leakage analysis, re-identification experiment, or quantitative privacy metric is provided. Since the entire distribution model and the legal validity of the benchmark depend on this assertion, the authors should provide evidence (e.g., information-theoretic guarantees, empirical re-identification risk on extracted patches, or compliance with a recognized anonymization standard) and specify what the residual sections contain. As written, the claim 'privacy-compliant benchmark' is unsubstantiated.","section":"§IV, first paragraph"},{"comment":"PACE is presented with 'optimal evaluation stride (s=2)' chosen from Table II, which is the same table used to claim superiority over baselines. This is a post-hoc model selection on the evaluation set. The difference between s=2 and s=1 is small (73.64 vs 73.16), and s=8 also outperforms baselines, so the point is not fatal, but the selection procedure should be described and ideally validated on a separate set or with a sensitivity analysis.","section":"§IV-B, Table II"},{"comment":"The abstract's '32.45% EER in detection' is presented as a summary for 'they all struggle,' but this value is TruFor's detection EER on FakeIDet3-DB in Table V; Re-MTKD obtains 40.28% EER, while the fine-tuned Re-MTKD_ICCV25 attains much lower EER on GenAI attacks (12.18%, Table IV). Please qualify which model and which setting the headline numbers refer to. In addition, all evaluations are single-run with no confidence intervals or significance tests; with frozen weights this may be acceptable for many detectors, but the paper should state whether any stochasticity exists and report error bars where relevant.","section":"Abstract; §V-A, Tables III–V"}],"minor_comments":[{"comment":"Typo: 'avaible' should be 'available'.","section":"Abstract"},{"comment":"The text cites 'Kochkarev et al. [42]' where the reference number should be [13]; [42] is Chai et al.","section":"§IV-B"},{"comment":"The statement 'we use the models' pre-trained weights frozen' appears only before Table V; it should be stated at the beginning of Section V for all evaluations.","section":"§V-A"},{"comment":"The caption says 'comparison ... in terms of difficulty'; since the comparison is confounded, 'difficulty' should be described as database-level difficulty, not substrate-level difficulty.","section":"Table V"},{"comment":"The figure legend is dense and difficult to parse; consider separating the attack-generation and patch-extraction pipelines into two figures or enlarging the labels.","section":"Fig. 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The database and PACE are potentially valuable, and the manuscript includes code release and a reproducible protocol. My main concern is that the headline domain-gap claim is not supported by the current experimental design. The privacy claim also needs concrete support before the benchmark can be described as GDPR-compliant. I would not reject: the resource is useful and the issues are addressable with additional experiments and cautious claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious look if you do ID forensics. The core asset is FakeIDet3-DB: as far as the cited literature goes, it is the first collection of digital manipulations on real government-issued IDs, with eight attack families, a refinement pipeline that suppresses artifacts, and a patch-based benchmark. The PACE extraction algorithm is a clean combination of integral images, distance transform, and greedy NMS; Table II shows it beats a naive grid and stochastic samplers on ROI coverage and semantic density. That part is solid work.\n\nThe soft spots are real but fixable. The biggest is the privacy claim. The paper asserts that pseudo-anonymization 'leaving small, residual sections of the sensitive data uncovered' does not compromise PII, but no leakage analysis or empirical test is given. If residual patches can be reassembled to identify a person, the distribution model collapses. This needs a formal or empirical treatment before the benchmark can be called privacy-compliant.\n\nSecond, the comparison with FantasyID in Table V is confounded. The two databases differ in attack typology (8 vs 3), resolution, and the presence of the refinement pipeline, so the doubling in TruFor's EER cannot be attributed specifically to the real-ID substrate. The stress-test is right: no control isolates the effect of ID reality. The paper's 'domain gap' narrative is plausible but not proven. A control experiment using the same attack pipeline on synthetic IDs would settle it.\n\nMinor issues: no error bars or multiple runs anywhere; the stride s=2 is chosen post hoc from Table II; and the abstract reports 32.45% EER as if it were an aggregate across detectors, when it is TruFor's number. Those are easy to fix.\n\nOverall, this is a genuinely useful resource and the detector evaluations are honest about failures. The dataset itself is the contribution, not the causal story. I'd send it to peer review with a request for a privacy analysis, a controlled comparison, and error bars. The reading group might enjoy discussing the privacy-vs-utility trade-off.","headline":"Valuable real-ID digital attack benchmark, but the privacy guarantee is unproven and the domain-gap claim rests on a confounded comparison.","tokens_in":20061,"tokens_out":4062,"would_cite":true,"duration_ms":41605,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current fake-ID detectors are far less reliable on real government-issued documents than on synthetic proxies, and this paper builds the first privacy-compliant benchmark to prove it.","keywords":["fake identity documents","digital forgery detection","generative AI attacks","privacy-preserving benchmark","patch extraction","integral image","non-maximum suppression","domain gap"],"falsifier":"Run a re-identification attack on the released patches: attempt to match residual face fragments, text snippets, or ID numbers to public records. If any patch can be reliably linked to a specific person, the privacy guarantee fails and the database cannot be legally shared. Alternatively, evaluate the same detectors on a separate collection of real government IDs not touched by any preprocessing; if their error rates match synthetic-data performance, the domain-gap claim is refuted.","tokens_in":19213,"feed_emoji":"🪪","tokens_out":6355,"duration_ms":56569,"temperature":0.7,"pith_summary":"FakeIDet3-DB is the first database of digitally manipulated real, government-issued IDs, with over 6,400 images spanning classical and generative-AI attacks. The paper shows that state-of-the-art forgery detectors perform far worse on these real IDs than on synthetic templates: a leading detector's error roughly doubles. It also introduces PACE, a privacy-preserving algorithm that extracts patches from pseudo-anonymized IDs while concentrating on the residual information around redacted areas. The result is a public benchmark that lets researchers measure and improve detection without exposing personal data.","feed_headline":"Real-ID forgeries double detector failure rates versus synthetic","feed_subtitle":"A 6,400-image benchmark of manipulated government IDs shows AI forensics are far less reliable on real documents.","key_machinery":"The core mechanism is PACE (Pseudo-Anonymized Contextual patch Extraction), a three-stage algorithm that treats patch extraction as a constrained spatial packing problem. It uses an integral image (summed-area table) to verify in O(1) time whether a candidate patch touches any redacted pixel; a distance transform to rank valid patches by their proximity to anonymization-mask boundaries; and greedy spatial non-maximum suppression to place non-overlapping patches that 'crystallize' around redactions. This maximizes capture of residual sensitive data and manipulation artifacts while guaranteeing zero overlap with censored regions. The paper also relies on a refined attack-generation pipeline th","core_discovery":"On its own terms, the paper claims that current ID-forensics models are substantially less reliable on real government-issued documents than on synthetic proxies, and that this gap has been hidden by the lack of a legal, real-ID benchmark. The paper builds that benchmark and demonstrates the gap: one prominent detector's equal error rate jumps from about 16% on synthetic data to about 32% on real-ID attacks, while localization accuracy stays comparatively high. It argues that refined attacks—those post-processed with semantic alpha-blending and background inpainting—are much harder to detect than naive 'cheapfakes,' and that patch-based, privacy-compliant evaluation poses a major new challen","pith_inferences":["If the domain gap is as large as measured, regulatory and auditing standards for AI-based ID verification may need to require real-ID test sets before deployment.","PACE's redaction-boundary prioritization could generalize to other privacy-sensitive document types (passports, financial records, medical scans) where censored regions must remain inviolable.","The observation that localization outperforms detection on real-ID attacks hints that local forensic traces are present but global context masks them; training schemes that amplify local cues might close part of the gap.","A direct testable extension: fine-tuning detectors on FakeIDet3-DB patches should improve real-world ID forgery detection more than fine-tuning on synthetic data; the paper's own benchmark split enables this comparison."],"forward_implications":["Detector rankings based on synthetic ID benchmarks may not transfer to real-world ID verification; real-ID evaluation is necessary.","The doubling of error rates on real IDs suggests current verification systems are more vulnerable to digital injection attacks than previously acknowledged.","Refined attacks degrade detector performance in over 80% of evaluated cases, so evaluation protocols should include refined forgeries rather than only cheapfakes.","Patch-based privacy-preserving benchmarks are viable, but existing detectors and naive score fusion are insufficient at small patch sizes; new architectures are needed.","The database enables quantitative study of domain shift between synthetic and real identity documents for forensics."],"fun_headline_variants":["Real ID attacks expose AI forensics: 32% error rate","AI forgery detectors fail 2x more on real ID forgeries","New real-ID forgery DB stumps detection models","Privacy-aware patches reveal real ID detection gap","State-of-the-art ID detectors worse on real vs synthetic forgeries"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that pseudo-anonymization—leaving small residual sections of sensitive data uncovered—protects the ID owner's personally identifiable information; the paper provides no formal privacy analysis or empirical re-identification test.","fun_headline_variants_meta":{"raw":{"variants":["Real ID attacks expose AI forensics: 32% error rate","AI forgery detectors fail 2x more on real ID forgeries","New real-ID forgery DB stumps detection models","Privacy-aware patches reveal real ID detection gap","State-of-the-art ID detectors worse on real vs synthetic forgeries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001051,"raw_usage":{"total_tokens":4304,"prompt_tokens":851,"completion_tokens":3453,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":3383}},"tokens_in":595,"tokens_out":3453,"duration_ms":24767,"temperature":1.0,"reasoning_tokens":3383,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:41:13.387949+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a re-identification attack on the released patches: attempt to match residual face fragments, text snippets, or ID numbers to public records. If any patch can be reliably linked to a specific person, the privacy guarantee fails and the database cannot be legally shared. Alternatively, evaluate the same detectors on a separate collection of real government IDs not touched by any preprocessing; if their error rates match synthetic-data performance, the domain-gap claim is refuted.","supporting_citations":[],"review_version":1}