{"id":"734a8f45-e518-4fe6-ab1d-7f9f9af6b49c","arxiv_id":"2505.24214","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A benchmark of 41 foundation models shows CLIP/OpenCLIP/BLIP2 embeddings reach near-90% zero-shot face verification and DINO reaches 97.55% on IITD-R iris without fine-tuning.","lead":"The paper tests 41 pretrained vision-language models on six biometric tasks, with no fine-tuning for face and iris verification. It reports strong zero-shot results, such as a 96.77% true match rate on LFW, and finds self-supervised models like DINO excel on iris.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot numbers hinge on test-set absence from pretraining, which the paper never audits; for web-trained OpenCLIP and DINO the 'zero-shot' label is not yet supported.","rationale":"The strongest claim is about zero-shot capability without fine-tuning. For that claim to be true, the test data must be absent from pretraining. The reader identified this as the weakest assumption; I agree. The paper's own Section VII-5 defers the data audit, so the gap is acknowledged but unfilled. I considered a second concern: the 'Acc' column in Table IV has no specified zero-shot decision rule, which is a separate reproducibility problem. But it does not threaten the headline face/iris verification numbers, whereas leakage would. Because no leakage is demonstrated, the paper is not rejected; because the central label is unverified, it should remain conditional on an audit. The concrete test above is feasible for OpenCLIP/LAION and at least partially for DINOv2. The reader already assigned CONDITIONAL, and this concern does not change that verdict.","tokens_in":1148,"tokens_out":961,"duration_ms":69408,"concrete_test":"For OpenCLIP-H/14 and other LAION-trained models, search LAION-5B metadata (URLs and hashes) for exact matches to the test images (LFW, CFP, AgeDB) and run a feature-space membership probe: embed each test image and find its nearest neighbor among a sample of LAION images; compare the median cosine similarity against a control set of face images known to be absent from LAION. If exact/near-duplicate overlap is nontrivial (e.g., >1% of test images or control-relative similarity elevated), re-run verification on a private held-out face dataset and report both numbers. For DINO/DINOv2, whose LVD-142M training set is not public, release the per-image feature distances to a constructed control set and state whether any overlap check is possible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that off-the-shelf VLM embeddings perform zero-shot biometric verification. The load-bearing condition for that claim is that the benchmark images were not seen during pretraining, so the reported TMR values reflect transferable discrimination rather than memorized identities. This condition is not checked. Several top models are trained on web-scale corpora: OpenCLIP-H/14 on LAION-5B, DINO/DINOv2 on a large web-curated image set, and DeepSeek-VL2 on mixed web data. LFW is a public collection of internet photos of well-known individuals, and LAION contains many public face images; iris datasets may also appear in web-scraped corpora. The paper itself, in Section V-D, speculates about 'implicit exposure to ocular patterns during pretraining' and lists auditing pretraining data as future work (Section VII-5), but no leakage analysis is performed. Without such an audit, the zero-shot label is not established: the headline LFW number (96.77% TMR@1%FMR for OpenCLIP-H/14) could in principle be inflated by near-duplicate retrieval from the training set. This is a correctness risk for the central claim, not merely a presentation gap, because the same numbers would be obtained whether the model generalizes or memorizes. The concern is not that leakage is demonstrated; it is that the paper's most important interpretive claim is currently unfalsifiable from the released evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a broad benchmark of 41 publicly available vision-language and multimodal models on six biometric tasks in the face and iris domains: face verification, gender and ethnicity classification, iris recognition, iris presentation attack detection (PAD), deepfake detection, and morph attack detection. Using frozen image encoders, the authors compute cosine-similarity verification scores or train lightweight classifiers on top of extracted embeddings. Headline results include 96.77% TMR@1%FMR on LFW with OpenCLIP-H/14, 97.55% on IITD-R-Full with DINO-ViT-B/16, near-99% gender classification with several models, and strong PAD accuracy with DINO/DINOv2 embeddings. The paper also reports negative results, such as chance-level morph detection with Chameleon.","tokens_in":32697,"tokens_out":6322,"duration_ms":78953,"significance":"If the zero-shot interpretation is established, this is a useful reference benchmark for an important question: whether generic foundation-model embeddings carry transferable biometric discriminability. The breadth across 41 models and six tasks is a genuine contribution, and the consistency of the main verification tables makes the raw measurements credible. The paper also gives credit where due: it uses standard public datasets and protocols, reports both strengths and failures across model families, and makes no parameter-free derivation claims that could be circular. The main value is as an empirical baseline; the main risk is that the zero-shot label is not yet supported by a leakage audit.","major_comments":[{"comment":"The central 'zero-shot' interpretation is not yet supported because the manuscript never audits whether benchmark images appeared in pretraining corpora. OpenCLIP-H/14 is trained on LAION-5B, DINO and DINOv2 on large web-curated image sets, and LFW, CFP, and AgeDB contain public internet photographs of identifiable individuals; near-duplicate or membership information could plausibly inflate the reported TMR@1%FMR values such as 96.77% on LFW. The paper itself notes 'implicit exposure to ocular patterns during pretraining' in Section V-D and lists understanding the training data as future work in Section VII-5, so the issue is acknowledged but not resolved. Please add a contamination analysis: retrieval-based near-duplicate checks against accessible training corpora, membership or rank statistics, or at minimum a quantitative argument for each high-performing model. Without this, the headline numbers do not distinguish generalization from memorization.","section":"V-D and VII-5"},{"comment":"The zero-shot PAD result ('Acc' column) is not reproducible because no decision rule is specified. Table IV reports binary classification accuracy between Patterned and Normal iris images under 'zero-shot inference,' but the paper never states how a class label is derived from the embeddings: threshold on cosine similarity to class prototypes, nearest-centroid assignment, text-prompt similarity, or another rule. Please specify the protocol, including threshold selection and class priors, or relabel the column so that it does not imply an unsupervised decision rule.","section":"III-D and Table IV"},{"comment":"All attribute-classification results are reported without variance: Section III-B1 states 'Standard deviations are omitted for brevity,' and Section IV-B collapses 5-fold cross-validation to a single mean. Many models differ by less than 0.2% in gender accuracy (values cluster near 99.9%), so the ranking claims in Section V-C are not statistically supported. Please report standard deviations, fold-level extrema, or confidence intervals for Table II.","section":"III-B1, IV-B, and Table II"}],"minor_comments":[{"comment":"The text refers to a '40-year protocol' and assigns the best 30-year result (21.23%) to LLaVA-1.5; Table I has only 5/10/20/30-year columns. Also, the claim that 'BLIP [7] outperformed all others' on CPLFW contradicts Table I, where BLIP-Base scores 7.60 and BLIP-Large 6.03 while BLIP2-t5-xxl scores 43.03.","section":"V-A"},{"comment":"The best CFP-FP result is attributed to 'CLIP-L/14 [28],' but [28] is a prompt-tuning paper and Table I shows the 87.63% value under CLIP-L-32, not a CLIP-L/14 row. Please correct the model name and citation.","section":"V-A"},{"comment":"The LLaVA architecture figure cites reference [8] but should cite reference [10]; the caption also contains the typo 'We use only use.'","section":"Figure 5"},{"comment":"The caption contains the typo 'socres' and should read 'scores.'","section":"Figure 10"},{"comment":"The morph attack detection setup says a decision tree classifier is trained for each model, but Table VI does not report decision-tree specifics; please state the tree hyperparameters, whether results are averaged over runs, and how BPCER@10%APCER thresholds were selected.","section":"IV-C"}],"recommendation":"major_revision","confidential_remarks":"The main unresolved risk is dataset leakage; the authors should either provide contamination evidence or explicitly qualify the zero-shot claim in the title and abstract. The missing PAD decision rule is also a reproducibility blocker. If the leakage audit cannot be performed for some models, the manuscript should separate 'no fine-tuning' from 'zero-shot' and state the caveat prominently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, genuinely useful benchmark paper that should go to review, but the central 'zero-shot' framing needs a leakage audit before the headline numbers mean what they claim.\n\nWhat is actually new: the breadth. The paper evaluates 41 publicly available VLMs/MLLMs across six biometric tasks in one study. Individual evaluations exist—Froundation for face recognition, the same group's earlier iris work—but this combined multi-task, multi-model table is the contribution. The main numbers look internally consistent and the paper is honest about its limitations, listing training-data auditing as future work.\n\nThe soft spots are real but mostly fixable. The most important is the missing leakage check. Several top models (OpenCLIP on LAION-5B, DINO, DeepSeek) were trained on web-scale data that plausibly includes LFW and other public face/iris images. Without an audit, 'zero-shot' is not fully established: the 96.77% TMR on LFW could in principle be inflated by near-duplicates from pretraining. The paper itself speculates about 'implicit exposure to ocular patterns' in Section V-D, so it is aware of the issue, but it only lists auditing as future work. That is a fair condition, not a fatal flaw. The measurements themselves are still valid as baseline numbers for these model-dataset pairs.\n\nOther gaps: standard deviations are omitted for attribute classification, which matters less because the numbers are high but still should be reported; the zero-shot PAD protocol does not specify the decision rule or threshold; and there are text errors such as calling AgeDB-30 '40-year' and misciting DeepSeek-VL2. Also, no code or evaluation artifacts are released, which is a reproducibility barrier for a benchmark paper.\n\nThe citation pattern looks fine. The self-citations are used for context or methodology, not to certify the benchmark result. No evident circularity.\n\nWho this is for: anyone working on face/iris biometrics who wants a baseline for whether off-the-shelf embeddings are usable. The paper delivers that.\n\nRecommendation: send to peer review. The issues are verification gaps and presentation problems, not a demonstrated error in the measurements. A serious referee should ask for the leakage discussion, error bars, and a precise PAD protocol, then this becomes a useful reference point.","headline":"A broad and useful zero-shot biometric benchmark that needs a leakage audit before its headline 'zero-shot' claims are fully supported.","tokens_in":33281,"tokens_out":2762,"would_cite":true,"duration_ms":33618,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark of 41 vision-language models finds that frozen embeddings verify faces and irises at up to 97% without fine-tuning.","keywords":["zero-shot biometrics","vision-language models","face verification","iris recognition","presentation attack detection","deepfake detection","morph attack detection","frozen embeddings"],"falsifier":"An audit that checks LFW, IITD iris, FaceForensics++, and VMER images against the pretraining corpora of the top models: if near-duplicates are found and removing them changes the reported TMR@1%FMR by more than a few points, then the zero-shot result is substantially memorization rather than transfer.","tokens_in":32254,"feed_emoji":"👁️","tokens_out":6360,"duration_ms":68601,"temperature":0.7,"pith_summary":"This paper benchmarks 41 publicly available vision-language and multimodal models on six biometric tasks without any task-specific training, and it finds that several of them already work well enough to be useful. Frozen image encoders, compared by cosine similarity, reach 96.77% true match rate at 1% false match rate on LFW face verification, and 97.55% on IITD-R-Full iris verification. The same frozen features, topped with a simple classifier, reach above 99% accuracy for gender classification and iris presentation attack detection. The paper also shows that deepfake detection is feasible with a small MLP head, while morph attack detection remains close to chance for many models, so the zero-shot success is real but uneven.","feed_headline":"No fine-tuning needed: face and iris checks hit 97%","feed_subtitle":"Frozen vision model embeddings verify faces and irides, detect attacks, and classify gender with simple classifiers.","key_machinery":"The central object is the frozen vision encoder of each foundation model acting as a generic feature extractor. The paper takes the vision-tower embeddings from models such as CLIP, OpenCLIP, BLIP-2, DINO, DINOv2, and InternVL3, then either compares pairs by cosine similarity for verification or trains a shallow head such as logistic regression, SVM, LDA, KNN, or a two-layer MLP on the frozen features for classification and attack detection. Since no model weights are updated, the benchmark isolates what the pretrained representations alone carry over to biometric tasks.","core_discovery":"The central claim is that the image encoders of pretrained VLMs and MLLMs, used without any fine-tuning, already encode discriminative biometric information. For verification, the paper extracts the frozen embeddings of each image and computes cosine similarity between pairs; this yields the strong LFW and IITD-R results. For attribute prediction and attack detection, a lightweight classifier head is trained on the frozen features, producing near-perfect gender and iris PAD accuracies and roughly 90% accuracy for deepfake detection with InternVL3-78B on FaceForensics++. The paper interprets these results as evidence that language-supervised and self-supervised pretraining can substitute for task-specific metric learning in at least some biometric pipelines, with performance depending strongly on model family and task difficulty.","pith_inferences":["My inference: the text encoders and chat interfaces of these models are largely unexploited here, so a natural extension is to prompt MLLMs directly with identity and attribute questions and compare those answers against the embedding-only scores.","My inference: the large performance gap between IITD and UND iris datasets, and between LFW and CPLFW, points to cross-sensor and cross-pose shifts as the main remaining bottleneck, which could be tested by measuring how frozen-embedding verification degrades on unseen sensors and demographics.","My inference: because the pretraining corpora are web-scale, a direct deduplication audit of benchmark images against training data would settle whether the top scores reflect transferable discrimination or memorization, and such an audit is easy to run.","My inference: the fact that different model families rank similarly across tasks suggests that pretraining objective and data distribution, rather than parameter count, drive biometric transferability, which a controlled pretraining ablation could confirm."],"forward_implications":["Face verification can be deployed without biometric-specific training: OpenCLIP-H/14 reaches 96.77% TMR@1%FMR on LFW using only cosine similarity on frozen embeddings.","Iris recognition transfers too: DINO-ViT-B/16 reaches 97.55% TMR@1%FMR on IITD-R-Full, with cropped and full iris images performing similarly for many models.","Lightweight biometric pipelines are possible: frozen embeddings plus simple classifiers exceed 99% accuracy for gender classification and iris presentation attack detection with the best models.","Deepfake detection benefits from large MLLM embeddings, with InternVL3-78B reaching nearly 90% accuracy with a simple MLP head on FaceForensics++.","Morph attack detection is the outlier: several models stay at or near chance level, indicating that fine-grained morph artifacts are not captured by current frozen embeddings."],"supporting_citations":[{"why":"Supplies the CLIP image encoder and contrastive image-text pretraining recipe that several top face-verification results build on.","marker":"[5]"},{"why":"Source of OpenCLIP-H/14, the model with the best LFW face verification score.","marker":"[59]"},{"why":"Source of DINOv2 encoders that lead iris presentation attack detection.","marker":"[31]"},{"why":"Source of DINO-ViT-B/16, the best iris recognition model on IITD-R.","marker":"[63]"},{"why":"Supplies BLIP-2 embeddings that rank among the best for face verification on several protocols.","marker":"[8]"},{"why":"Source of LLaVA-1.5, the best model on AgeDB protocols.","marker":"[10]"},{"why":"Source of InternVL3, which gives the best deepfake detection with a simple MLP head.","marker":"[67]"},{"why":"Provides the IITD-R iris recognition benchmark used for the iris TMR results.","marker":"[83]"},{"why":"Provides the IITD-P contact-lens presentation attack benchmark used for PAD.","marker":"[85]"},{"why":"Provides FaceForensics++, the deepfake detection benchmark.","marker":"[87]"}],"fun_headline_variants":["Frozen vision models verify faces and irises at 97%","Zero-shot biometrics: VLM embeddings hit 97% accuracy","No fine-tuning: Vision-language models ace face and iris checks","Foundation model embeddings power zero-shot face and iris ID"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the benchmark images were not part of the models' pretraining data, so the reported zero-shot scores reflect transferable discrimination rather than memorized identities; the paper does not audit for this overlap.","fun_headline_variants_meta":{"raw":{"variants":["Frozen vision models verify faces and irises at 97%","Zero-shot biometrics: VLM embeddings hit 97% accuracy","No fine-tuning: Vision-language models ace face and iris checks","Foundation model embeddings power zero-shot face and iris ID"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1427,"prompt_tokens":1006,"completion_tokens":421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":350}},"tokens_in":622,"tokens_out":421,"duration_ms":4976,"temperature":1.0,"reasoning_tokens":350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:29:09.312415+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An audit that checks LFW, IITD iris, FaceForensics++, and VMER images against the pretraining corpora of the top models: if near-duplicates are found and removing them changes the reported TMR@1%FMR by more than a few points, then the zero-shot result is substantially memorization rather than transfer.","supporting_citations":[{"cited_title":"Reproducible scaling laws for contrastive language-image learning,","cited_arxiv_id":null,"evidence_quote":"Source of OpenCLIP-H/14, the model with the best LFW face verification score."},{"cited_title":"Emerging properties in self-supervised vision trans- formers,","cited_arxiv_id":null,"evidence_quote":"Source of DINO-ViT-B/16, the best iris recognition model on IITD-R."},{"cited_title":"Comparison and Combination of Iris Matchers for Reliable Personal Authentication,","cited_arxiv_id":null,"evidence_quote":"Provides the IITD-R iris recognition benchmark used for the iris TMR results."},{"cited_title":"Ghclnet: A generalized hierarchically tuned contact lens detection network,","cited_arxiv_id":null,"evidence_quote":"Provides the IITD-P contact-lens presentation attack benchmark used for PAD."},{"cited_title":"FaceForensics++: Learning to detect manipulated facial images,","cited_arxiv_id":null,"evidence_quote":"Provides FaceForensics++, the deepfake detection benchmark."}],"review_version":1}