{"id":"25c3d46d-252f-4c83-8d02-2adc8d6cb89e","arxiv_id":"2501.08799","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"With two labeled examples per class, GPT-4o reaches 2.7% ACER on face presentation attack detection on a small SOTERIA subset, close to a specialized model's 2.0%.","lead":"This paper tests whether GPT-4o can spot face-spoofing attacks (printed photos or screen replays) when shown zero, one, or two labeled examples before each test image. On a 300-image privacy-cleared subset of the SOTERIA dataset, GPT-4o's few-shot accuracy approaches a dedicated trained model and beats two commercial systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Few-shot ACER gain is not interpretable until reference/probe disjointness by identity or session is established; the paper leaves this unspecified.","rationale":"The reader's conditional verdict already captures the load-bearing concern: the central quantitative claim depends on few-shot ACER numbers, and those numbers can be inflated if the reference images come from the same 10-individual subset as the probes. Section 3.2 and Section 3.4.2 leave the reference/probe assignment unspecified, and the code link plus consent constraints do not allow an independent check from the text. My stress pass did not find a separate fatal flaw. The 'emergent attack-type' claim is indeed undercut by the explicit print/replay labels in the reference prompt, but that claim is secondary to the headline performance comparison. The correct response is not rejection but a protocol fix: report subject- or session-disjoint reference selection and the ACER threshold. With that fix, the 1-shot/2-shot results would be meaningful even if preliminary. Therefore the verdict should remain CONDITIONAL, not move.","tokens_in":11796,"tokens_out":2498,"duration_ms":29326,"concrete_test":"Re-run the 1-shot and 2-shot evaluations with references selected from a held-out set of SOTERIA subjects or sessions (or, at minimum, with no probe image ever used as a reference), and pre-specify the score threshold used to compute ACER. If ACER remains near 4.0%/2.7% under disjoint references, the concern is resolved; if it regresses toward the 15% zero-shot level, the few-shot gain is attributable to test-time label access rather than to genuine in-context generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The few-shot performance gain from 15.1% to 4.0%/2.7% ACER is only interpretable as generalization if the reference images and probe images are independent. The paper does not state this. Section 3.2 reports a consented subset of 10 SOTERIA individuals and 300 samples; Section 3.4.2 defines references as labeled images of bonafide/print/replay but never says references are disjoint from probes by identity, session, or image. With only 10 individuals, it is likely the same people appear in both roles. If so, GPT-4o's 1-shot and 2-shot prompts contain test-time labels for the very distribution (or exact images) being scored, while DeepPixBis and the COTS systems receive no such per-probe labels. The headline comparison is then not a fair comparison, and the claimed improvement is not evidence of PAD generalization. This is a protocol gap rather than an internal contradiction, and it is fixable by reporting the reference/probe split explicitly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates GPT-4o as a face presentation attack detection (PAD) system under zero-shot and few-shot in-context learning on a 300-sample, 10-subject consented subset of the SOTERIA dataset. The authors report that detailed prompts eliminate most failure-to-acquire behavior, that scores are consistent across repeated runs, that few-shot reference examples improve ACER from 15.1% (0-shot) to 4.0% (1-shot) and 2.7% (2-shot), and that GPT-4o approaches the 2.0% ACER of DeepPixBis trained on SOTERIA while outperforming two commercial systems. The paper also claims that GPT-4o predicts attack types (print vs. replay) with high accuracy in few-shot settings without being explicitly instructed to do so. The central claim is that a multimodal LLM can be a competitive PAD alternative in low-resource scenarios.","tokens_in":12043,"tokens_out":3548,"duration_ms":36020,"significance":"If the central claim held, the paper would be a valuable early quantitative study of MLLMs for face PAD, especially because it addresses data-consent issues, open-sources code, and reports consistency and failure-to-acquire behavior rather than only aggregate accuracy. The prompt-design analysis and the observation that detailed prompts prevent refusal behavior are useful and reproducible. However, the headline few-shot results rest on protocol details that are not reported, and the evaluation scale is too small to support the strong comparative claims. The paper's main contribution is therefore preliminary evidence rather than a demonstrated competitive alternative; the current protocol gaps make the few-shot comparison uninterpretable as a generalization result.","major_comments":[{"comment":"The few-shot protocol never states whether the reference images in the prompts are disjoint from the probe images by identity, acquisition session, or exact image. Section 3.2 reports only 10 consenting individuals and 300 samples; Section 3.4.2 defines references as labeled examples of bonafide, print, and replay attacks, but does not specify the split. With only 10 subjects, the same individuals very likely appear in both reference and probe roles. If so, the 1-shot and 2-shot prompts leak test-distribution labels (and possibly the exact test images) to GPT-4o, while DeepPixBis and the commercial systems receive no such per-probe labels. The ACER drop from 15.1% to 4.0%/2.7% can then reflect label leakage rather than PAD generalization. The authors must report an explicit identity-disjoint or session-disjoint reference/probe split, and should also compare against baselines given the same reference labels.","section":"§3.4.2 and §3.2"},{"comment":"The ACER metric is defined as (APCER + BPCER)/2, but the paper never specifies the decision threshold applied to GPT-4o's numeric scores, nor how that threshold was selected. If the threshold is chosen on the same test set to minimize ACER, the reported values are optimistically biased. All ACER comparisons in Table 6 and the text depend on this unspecified threshold. The authors should state the threshold selection procedure (e.g., fixed a priori, cross-validation, or threshold-free reporting such as BPCER at a fixed APCER), and ideally report score distributions or ROC curves.","section":"Footnote 2 and Table 6"},{"comment":"The evaluation uses 300 samples from 10 subjects, which is too small to support the strong claims that GPT-4o 'outperforms commercial solutions' and 'approaches DeepPixBis' with ACER values of 2.7% versus 2.0%. With roughly 150 attack samples and 150 bonafide samples, the difference between 2.7% and 2.0% corresponds to a handful of errors, and no confidence intervals, subject-level variability, or statistical tests are reported. The paper should either temper the comparative claims to observational findings or provide uncertainty quantification and a subject-disjoint evaluation with a larger consented subset.","section":"§3.2 and Table 6"},{"comment":"The claim of 'emergent reasoning' for attack-type prediction is not supported. In the few-shot scenarios, the reference images are explicitly labeled as 'print attack' and 'replay attack' (Section 3.4.2), so the model is given the attack-type vocabulary and examples in the prompt; the high attack-type accuracy is then a direct reading of the provided labels, not an emergent inference. The zero-shot attack-type accuracy is only 28.79% and 33.33%, which is near or below chance, confirming that the model does not spontaneously infer attack types. The authors should reframe this result as label-conditioned classification in few-shot settings, not emergent reasoning.","section":"Table 5 and §3.5"},{"comment":"The handling of Failure-to-Acquire (FTA) samples is unclear and affects the 0-shot ACER. Table 2 reports an 80.34% FTA rate for the short prompt, and the text states that 'for these FTA cases, the calculated ACER using the detailed prompt was 38.46%'. This sentence is confusing: are the FTA samples from the short-prompt run re-scored with the detailed prompt, or are FTA samples excluded from the ACER computation? Table 4 reports a 0-shot ACER of 15.1% using the detailed prompt, which has only a 0.67% FTA rate, so the two prompt conditions are not directly comparable. The authors should clarify whether FTA samples are excluded or imputed, and how the short-prompt condition would be scored under a consistent protocol.","section":"§3.4.1 and Table 2"}],"minor_comments":[{"comment":"Table 6 reports only the GPT-4o 2-shot ACER (2.7%), while the text quotes 0-shot (15.1%) and 1-shot (4.0%) values; the table should include all three GPT-4o rows so that the comparison is complete.","section":"Table 6"},{"comment":"The attack-type accuracy percentages in Table 5 are reported without denominators or confidence intervals; the number of print and replay attack samples should be stated so the reader can assess the reliability of 87.88% and 85.71% figures.","section":"Table 5"},{"comment":"The code URL in Footnote 1 contains a space ('wacv2025 chatgpt face pad') and appears malformed; the repository link should be corrected to a properly percent-encoded URL.","section":"Abstract and Footnote 1"},{"comment":"Tables 3 and 7 use color-coded responses (red/green) to indicate correct/incorrect; the color coding is not accessible in grayscale printing. Replace or supplement the colors with textual markers such as 'wrong' and 'correct'.","section":"Tables 3 and 7"},{"comment":"The consistency analysis reports the percentage of pairwise score differences below 0.1, but does not report the mean absolute difference or the standard deviation; adding these statistics would make the consistency claim more interpretable.","section":"§3.3 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a preliminary study, and the authors are transparent about the consented-subset constraint, which is commendable given the data-privacy context. However, the few-shot comparison is currently not interpretable because the reference/probe disjointness is unspecified, and the ACER threshold is not defined. These are load-bearing issues, not cosmetic ones. I would be willing to review a revised version that provides an explicit identity-disjoint split, a thresholding protocol, and more cautious comparative claims. I also suggest the authors disclose the exact GPT-4o API version and snapshot date, since MLLM behavior can change over time and reproducibility is already a concern in such studies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for the zero-shot numbers and the consistency data, not for the few-shot claims. The paper's real contribution is that it puts a number on something people had only talked about: GPT-4o can actually output usable liveness scores for face PAD, and with a detailed prompt it does so reliably (FTA drops from 80% to 0.67%). The five-run consistency check is a nice touch, and the code is linked. That part is solid.\n\nThe soft spot is exactly where the authors would like the paper to matter. The 1-shot and 2-shot ACER improvements (15.1% to 4.0% to 2.7%) are only evidence of generalization if the reference images shown in the prompt are independent of the probe images being scored. The paper never says they are. Section 3.2 describes a consented subset of 10 SOTERIA individuals and 300 samples; Section 3.4.2 defines references as labeled images of bonafide/print/replay but never states that references and probes are disjoint by identity or acquisition session. With 10 people, it is entirely plausible the same subjects appear in both roles. If that is true, GPT-4o receives per-probe distribution labels at test time that DeepPixBis and the COTS systems do not get, and the comparison is not fair. This is a protocol gap, fixable by reporting the split, but it currently undermines the central claim.\n\nTwo smaller issues: the ACER threshold is never specified, so it may be tuned on the same 300 images; and the 'emergent' attack-type prediction is not that surprising, because the few-shot prompt explicitly labels the references as print or replay. The 0-shot attack-type accuracy is near chance, which is consistent with the model just picking up the category names from the prompt.\n\nThe dataset is tiny (10 subjects, 300 images) and no confidence intervals are given, so the 2.7% vs DeepPixBis's 2.0% gap is not meaningful. But the paper calls itself preliminary, and the zero-shot numbers alone (15.1% vs 40.7% for COTS1) are a legitimate data point.\n\nWho gets value: researchers working on MLLMs for biometrics or low-resource PAD. It deserves peer review because the question is real and the protocol is fixable, but I would send it back for a major revision with explicit requests for the reference/probe split, a pre-specified threshold, and confidence intervals.","headline":"First quantitative ACER numbers for GPT-4o on face PAD, but the few-shot gain is uninterpretable until the reference/probe split is specified.","tokens_in":12520,"tokens_out":3160,"would_cite":false,"duration_ms":29280,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4o with two labeled examples per attack type detects face spoofs nearly as well as a specialized detector.","keywords":["face presentation attack detection","in-context learning","GPT-4o","zero-shot learning","few-shot learning","biometric security","prompt engineering","SOTERIA dataset"],"falsifier":"Re-run the 1-shot and 2-shot experiments with reference and probe sets disjoint by identity and by acquisition session, keeping prompts and scoring identical; if ACER rises toward the zero-shot level, the few-shot gain comes from overlap rather than in-context learning. A second check is to flip the labels attached to the reference images: if scores barely change, the reference images are not actually driving the decision.","tokens_in":11537,"feed_emoji":"🎭","tokens_out":6429,"duration_ms":57197,"temperature":0.7,"pith_summary":"The paper asks whether a general-purpose multimodal language model, GPT-4o, can detect face presentation attacks—spoofs made with printed photos or screen replays—without any training on the task. It reports that adding two labeled example images per class to the prompt cuts the average classification error rate (ACER) from 15.1% in the zero-shot setting to 2.7%, close to the 2.0% of DeepPixBis trained on the same SOTERIA subset and better than two commercial systems. The result matters because prompt-based adaptation could let a PAD system handle a new dataset or new attack types by editing text and images instead of retraining a model. The paper also reports that the model learns to name the attack type—print or replay—with high accuracy in few-shot settings even though it was never asked to classify the attack type.","feed_headline":"Two examples let GPT-4o rival specialized face-spoof detectors","feed_subtitle":"Zero-shot lags at 15.1% ACER; two labeled references cut errors to 2.7%.","key_machinery":"The carrying mechanism is in-context learning: the prompt itself is the training signal. A system message defines the scoring task, and the user message embeds labeled reference images (bonafide, print, replay) followed by the probe image; GPT-4o returns an authenticity score in $[0,1]$ and no weights are updated. The reference images act as anchors that reduce ambiguity and cognitive load, which explains why few-shot beats zero-shot, and the prompt wording acts as a gate on compliance: a short prompt produced failure-to-acquire on 80.34% of zero-shot probes while a detailed prompt reduced this to 0.67%.","core_discovery":"On the paper's own terms, the discovery is that in-context learning converts GPT-4o into a competitive face PAD scorer. With no examples the model reaches 15.1% ACER on the consented 300-sample SOTERIA subset; one labeled example per class lowers this to 4.0%; two examples per class lower it to 2.7%, versus 2.0% for DeepPixBis trained on SOTERIA, 10.7% for one commercial system, and 40.7% for another. The paper attributes the few-shot gains to reference images that anchor the model's notion of bonafide, print, and replay, and reports that prediction consistency improves as references are added. It further claims that short prompts cause the model to refuse most zero-shot probes (80.34% failure-to-acquire), that a detailed prompt cuts refusal to 0.67%, and that asking for explanations slightly improves accuracy (2.70% versus 2.92% ACER in the two-shot case).","pith_inferences":["The paper does not state whether reference and probe images come from disjoint identities or sessions; if they overlap, a portion of the few-shot gain may be the model recognizing particular faces. A direct test is to split by identity and rerun the two-shot protocol.","Because the model names attack types accurately without being asked, those labels could be harvested as pseudo-labels to train or fine-tune a conventional detector, turning the LLM's reasoning into reusable training data.","The same prompting recipe could be applied to attack types outside the printed-photo and replay families, such as 3D masks or silicone overlays, to see whether the few-shot gain generalizes to novel presentation attacks or only to categories represented in the reference set."],"forward_implications":["If the reported ACER holds, a PAD system could be configured for a new deployment site by writing a prompt and supplying a handful of labeled images, with no gradient training.","The few-shot numbers imply that the gap to specialized detectors is small (2.7% versus 2.0%), so prompt-based PAD is a plausible fallback where training data or compute is unavailable.","The attack-type prediction result suggests that the model's internal reasoning can be read out for free, yielding a diagnostic label alongside the authenticity score.","Prompt design becomes a first-order engineering variable: short prompts cause refusals, detailed prompts elicit scores, and explanation requests slightly improve accuracy at the cost of more refusals in zero-shot."],"supporting_citations":[{"why":"Specifies GPT-4o, the multimodal model whose zero- and few-shot behavior is under test.","marker":"[3]"},{"why":"Provides the in-context learning basis for expecting performance to improve when labeled examples are placed in the prompt.","marker":"[7]"},{"why":"Defines DeepPixBis, the specialized detector whose SOTERIA-trained ACER is the reference point for the few-shot comparisons.","marker":"[14]"},{"why":"Introduces the SOTERIA database from which the consented 300-sample subset is drawn.","marker":"[27]"},{"why":"Earlier qualitative study of multimodal LLMs for face spoofing that this paper extends by adding scores, baselines, and consent handling.","marker":"[29]"},{"why":"Prior language-guided face anti-spoofing work using a vision-language model, the closest related approach to prompting-based PAD.","marker":"[31]"}],"fun_headline_variants":["ChatGPT few-shot tricks beat commercial face-spoof detectors","Two examples drop GPT-4o face-spoof error to 2.7%","In-context learning turns GPT-4o into a face-spoof expert","Few-shot GPT-4o rivals specialized face attack detection","Zero-shot flops, few-shot shines: GPT-4o for face PAD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reference images shown in the few-shot prompts are independent enough from the probe images that the score improvement reflects generalization, not the model recognizing the same people or capture sessions it was already shown.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT few-shot tricks beat commercial face-spoof detectors","Two examples drop GPT-4o face-spoof error to 2.7%","In-context learning turns GPT-4o into a face-spoof expert","Few-shot GPT-4o rivals specialized face attack detection","Zero-shot flops, few-shot shines: GPT-4o for face PAD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1363,"prompt_tokens":1009,"completion_tokens":354,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":257}},"tokens_in":625,"tokens_out":354,"duration_ms":3602,"temperature":1.0,"reasoning_tokens":257,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:17:11.967359+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 1-shot and 2-shot experiments with reference and probe sets disjoint by identity and by acquisition session, keeping prompts and scoring identical; if ACER rises toward the zero-shot level, the few-shot gain comes from overlap rather than in-context learning. A second check is to flip the labels attached to the reference images: if scores barely change, the reference images are not actually driving the decision.","supporting_citations":[{"cited_title":"Gpt-4 technical report, 2024","cited_arxiv_id":null,"evidence_quote":"Specifies GPT-4o, the multimodal model whose zero- and few-shot behavior is under test."},{"cited_title":"Deep pixel-wise bi- nary supervision for face presentation attack detection","cited_arxiv_id":null,"evidence_quote":"Defines DeepPixBis, the specialized detector whose SOTERIA-trained ACER is the reference point for the few-shot comparisons."},{"cited_title":"A novel and responsible dataset for face presentation attack de- tection on mobile devices","cited_arxiv_id":null,"evidence_quote":"Introduces the SOTERIA database from which the consented 300-sample subset is drawn."},{"cited_title":"Flip: Cross-domain face anti-spoofing with lan- guage guidance, 2023","cited_arxiv_id":null,"evidence_quote":"Prior language-guided face anti-spoofing work using a vision-language model, the closest related approach to prompting-based PAD."}],"review_version":1}