{"id":"2c32707c-4a90-4132-82e0-fd69eb9800a9","arxiv_id":"2607.03562","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A million-scale deepfake benchmark with Edit-Check filtering, dual expert/lay explanations, and EntityScore/EvidenceScore shows fine-tuned detectors collapse under generator shift while surface fluency remains.","lead":"XPlainVerse is a one-million-image benchmark that pairs real photos with edits from twelve generators and supplies dual-level natural-language explanations plus intent-aware metrics. It makes whether a detector’s explanation names the right manipulated entity and visual evidence a measurable target, not just classification accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"VLM silver labels + VLM judges for Entity/EvidenceScore risk measuring self-consistency more than human-grounded visual evidence, undercutting the claim that grounded quality is now reliably measurable.","rationale":"The reader's weakest assumption is exactly the load-bearing soft spot: sparse human gold, low IAA on a tiny overlap, and VLM-as-judge metrics create a circularity risk for any claim that 'grounded quality' is now a trustworthy evaluation axis. The paper's multi-method human studies (realism, edit fidelity, preference vs SIDA/human, readability) and the independent spectral/probe diagnosis of generator-specific shortcuts are real strengths and keep the contribution accept-shaped under a CONDITIONAL reading. No stronger internal inconsistency appears; commercial-API/EULA reproducibility is a practical limit already noted by the reader, not a logical flaw in the argument. Expanding human calibration of the metrics (the concrete test) would either solidify or qualify the strongest claim without requiring rejection of the benchmark itself. Thus the reader's CONDITIONAL verdict and HIGH confidence stand; no adjustment is warranted.","tokens_in":43443,"tokens_out":679,"duration_ms":15826,"concrete_test":"On the 2,000 human-annotated images, recompute EntityScore and EvidenceScore for the same model outputs (fine-tuned LLaVA/Qwen and zero-shot Gemini) twice: once against silver complex refs and once against human gold refs. Separately collect human coverage ratings (entity match + evidence match, 1–5 or binary) on a stratified 200-pair subsample. If Spearman correlation of auto-vs-silver scores with human ratings (or with auto-vs-human scores) is <0.4, or if model rankings reverse, the metrics do not yet establish human-grounded measurability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that XPlainVerse makes grounded explanation quality a measurable dimension, and that EntityScore/EvidenceScore reveal fluent-but-ungrounded failures under generator shift—rests on silver complex explanations and automatic coverage judges both produced by VLMs (Gemini/GPT/GLM/Qwen families for generation; Qwen3.5-4B for entity/evidence extraction and bidirectional coverage in §4 and Appendix I). Edit-Check itself is also VLM-driven (difference extraction + text judge). Human gold covers only 2,000 of ~530K fakes; inter-annotator EntityScore/EvidenceScore on a 10-image overlap are 0.5 and 0.44. Preference studies (Table 4, Figs. 4–5) show style and overall usefulness advantages, but do not calibrate whether the auto metrics recover the same diagnostic entities and visual claims humans would mark. If silver targets and the metric judge systematically favor VLM-preferred cue phrasing or miss human-salient artifacts, high ID Entity/Evidence (and their sharper OOD drop relative to BERTScore) can reflect distribution shift in VLM-style rationales rather than true loss of visual grounding. The spectral/probe analyses remain independent, but the explanation-quality half of the strongest claim is only weakly anchored.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces XPlainVerse, a ~1M-image benchmark for joint deepfake detection and natural-language explanation. Real images from five public sources are paired with manipulations from twelve editing/synthesis models; a multi-stage Edit-Check pipeline (VLM difference extraction + text-only alignment judge) retains only edits that match intended instructions and discards off-target rewrites (~56.5% discard). The release provides dual-level explanations (complex forensic and simple lay explanations) plus authenticity explanations for reals, totaling ~1.53M image–explanation pairs, with human gold explanations on 2,000 fakes. The authors propose EntityScore and EvidenceScore—bidirectional VLM-judged coverage of diagnostic entities and visual evidence claims—alongside BERTScore and SLE, and report that fine-tuned VLMs reach high ID detection/explanation scores but drop sharply under generator-held-out OOD, while Entity/Evidence degrade more than surface similarity. Linear probes and spectral analyses suggest reliance on generator-specific low-level artifacts rather than stable visual evidence.","tokens_in":43824,"tokens_out":1025,"duration_ms":14337,"significance":"If the resource and metrics hold up, this is a substantial contribution to explainable deepfake detection: scale far exceeds prior explanation-oriented sets (Table 1), Edit-Check addresses a real source of weakly grounded supervision, dual-level explanations operationalize human-centered XAI needs, and Entity/Evidence scores target a genuine evaluation gap beyond fluency. The ID–OOD collapse and spectral/probe analyses are useful negative results for the community. Strengths include concrete inventories (Tables 8–10), kept-vs-discarded fidelity (Table 2), multi-stage human preference/quality studies (Figures 4–5, Tables 4 and 18), and a detailed appendix of prompts and protocols. The main significance risk is that explanation targets and automatic fidelity metrics are themselves largely VLM-produced, so the claim that grounded quality is now reliably measurable depends on how well those silver labels and judges track human visual evidence.","major_comments":[{"comment":"§3.1 Step 5 and §4 / Appendix I: Complex silver explanations are generated with privileged auxiliaries (paired real image, edit instruction, and difference JSON), while EntityScore/EvidenceScore extraction and bidirectional coverage are judged by Qwen3.5-4B. Human gold covers only 2,000 of ~530K fakes; inter-annotator EntityScore/EvidenceScore on a 10-image overlap are 0.5 and 0.44 (§3.2). Preference wins vs SIDA/human (Table 4) do not calibrate whether auto metrics recover the same diagnostic entities and evidence claims humans would mark. Without a reported correlation (or human re-annotation of metric components) on a larger held-out set, the central claim that Entity/Evidence make grounded quality measurable—and that their sharper OOD drop vs BERTScore proves loss of visual grounding rather than shift in VLM-style rationales—is only weakly supported. Please add metric–human agreement","section":null},{"comment":"Table 9 and §5 / Table 5: Training fakes are extremely imbalanced—Gemini-2.0-flash accounts for 354,256 of 360,000 train fakes (~98.4%), with other generators almost entirely deferred to val/test. The reported >50-point F1 OOD drops for fine-tuned models are therefore largely “train-on-Gemini, test-on-other-generators.” That is still a valid robustness finding, but it confounds “generator-specific artifact memorization” with simple single-source overfitting. Please report per-generator ID/OOD breakdowns, retrain or ablate with more balanced multi-generator training subsets, and clarify how much of the collapse remains under multi-source training before attributing failure primarily to low-level spectral shortcuts (Figure 6).","section":null},{"comment":"§3.1 Edit-Check and Appendix E: Filtering and KEEP/DISCARD decisions are fully VLM-driven (difference extractors + DeepSeek/GPT-OSS judges; KEEP if final_score ≥ 4). Table 2 and human edit-fidelity (Figure 4) support that retained pairs are cleaner, but there is no independent non-VLM audit of false KEEP/false DISCARD rates stratified by edit type (Table 11) or generator. Because explanation supervision is only as grounded as retained edit compliance, a quantified error analysis of the judge (e.g., human audit of a few thousand KEEP/DISCARD decisions, agreement with BORDERLINE handling) is load-bearing for the claim that Edit-Check enables reliable reasoning supervision at scale.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a carefully built benchmark paper, not a method paper. The useful package is scale (~1M images, ~1.53M explanation pairs), Edit-Check so explanations are tied to verified edits (~56% discard), dual complex/simple explanations, and EntityScore/EvidenceScore that try to score grounding rather than BERT-style fluency. The ID→OOD story is the empirical core: fine-tuned VLMs look strong in-domain then collapse under held-out generators, while Entity/Evidence fall harder than BERTScore, and the probe/spectral analysis independently shows modern editors have washed out low-level shortcuts.\n\nWhat is actually new is the combination, not any single piece. Smaller explanation sets already exist (SIDA, FakeClue, TRACE, etc.). What they do well is inventory and multi-method validation: kept vs discarded fidelity (Table 2), diversity (Table 3), human realism/edit/preference studies (Figs. 4–5, Tables 4 and 18), and a coherent failure diagnosis rather than just leaderboard numbers. Dual user styles are also earned—preference splits by audience are large enough that a single explanation style would leave half the users poorly served.\n\nSoft spots, in proportion: the stress-test is right that silver explanations and Entity/Evidence judges are mostly VLM-driven, with only 2K human golds and weak inter-annotator Entity/Evidence (0.5 / 0.44 on a 10-image overlap). That means “grounded quality is now measurable” is partly “measurable under a VLM judge,” and high ID scores can partly reflect style match. It does not sink the paper: edit compliance is checked against image pairs, human preference favors their complex explanations over SIDA and even human text on several axes, and the spectral/probe results do not depend on the explanation metrics. Reproducibility under EULA plus commercial APIs is a real practical limit, not a conceptual one. Citation pattern looks normal for the subfield.\n\nWho this is for: people building or evaluating multimodal deepfake detectors and human-facing XAI, not general CV. I would bring it to reading group, send it to peer review, and cite the dataset/metrics if I work on explainable detection. Treat silver labels and auto grounding scores as provisional until broader human calibration—but engage the work.","headline":"Solid million-scale explainable-deepfake benchmark with real engineering and a clear OOD diagnosis; the VLM-on-VLM scoring loop is the main soft spot, not a reason to ignore the work.","tokens_in":44489,"tokens_out":593,"would_cite":true,"duration_ms":9427,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"XPlainVerse makes grounded explanation quality a measurable part of deepfake detection at million-image scale.","keywords":["deepfake detection","explainable AI","vision-language models","image manipulation","benchmark dataset","grounded explanations","EntityScore","EvidenceScore"],"falsifier":"Train and score the same models using only the 2,000 human gold explanations as targets and as the sole Entity/Evidence reference, then check whether the reported ID-to-OOD collapse and the ranking of methods reverse or disappear relative to the silver-label results.","tokens_in":44309,"feed_emoji":"🔍","tokens_out":925,"duration_ms":10993,"temperature":0.7,"pith_summary":"Deepfake detectors increasingly output natural-language reasons, but those reasons often read as fluent stories rather than evidence tied to what was actually changed in the image. Existing benchmarks mostly score real-versus-fake accuracy and leave that gap unmeasured, so progress on trustworthy, user-facing systems is hard to track. XPlainVerse is a one-million-image benchmark built to close it: real photos from five public sources are paired with forgeries from twelve editing and synthesis models, then filtered by Edit-Check so retained fakes really show the intended edit. The release supplies both technical and simplified explanations, plus EntityScore and EvidenceScore that test whether an explanation names the right manipulated parts and the right visual cues. Human checks on two thousand pairs support the claim that this supervision is usable at scale, and experiments show high in-domain scores collapse under generator shift—pointing to shortcut learning rather than stable visual reasoning.","feed_headline":"Million-image deepfake benchmark scores grounded reasons","feed_subtitle":"Edit-checked fakes and entity/evidence metrics show fluent explanations often miss the real artifacts.","key_machinery":"Edit-Check: a multi-stage filter that extracts visible differences between real and generated images, then judges whether those differences match the intended edit and discards off-target rewrites; EntityScore and EvidenceScore then score whether predicted explanations recover the same manipulated entities and visual evidence claims as the reference.","core_discovery":"The paper’s central claim is that explainable deepfake detection becomes a well-posed research problem only when manipulations are edit-verified, explanations are dual-level (expert and lay), and evaluation measures entity and evidence fidelity rather than surface text similarity. Under that regime, fine-tuned vision–language models look strong in distribution but fail under held-out generators, while EntityScore and EvidenceScore drop more than BERTScore—exposing fluent but ungrounded reasoning.","pith_inferences":["If silver labels and VLM judges share the same blind spots, reported gains on Entity/Evidence may partly measure agreement with that judge rather than human-grounded forensics.","The same edit-instruction-plus-difference protocol could transfer to video and audio-visual deepfakes once temporal consistency checks replace still-image difference JSON.","Deployment systems may need to expose both explanation styles and let users pick, because preference splits by role are large enough that a single default leaves many users underserved.","Spectral convergence across newer generators implies that pure embedding probes will keep failing; methods that localize entities before explaining may be the practical next step."],"forward_implications":["Grounded explanation quality becomes a reported metric alongside detection F1, not an optional narrative.","Generator-shift evaluation is required; strong in-domain fine-tuning no longer counts as success if Entity/Evidence collapse.","Dual-level (technical vs simple) explanations become standard so systems can serve experts and non-experts without one style for all.","Edit-aware filtering is treated as necessary for scalable reasoning supervision, not optional data cleaning.","Detectors are pushed toward content-level cues (geometry, lighting, placement) as low-level generator artifacts fade."],"fun_headline_variants":["XPlainVerse: Million images pair edit-checked fakes with dual-level explanations","EntityScore and EvidenceScore expose fluent but ungrounded deepfake reasons","Million-scale deepfake set verifies manipulations then scores entity fidelity","Held-out generators drop EntityScore more than BERTScore on deepfake reasons","Edit-Check pipeline enables reliable supervision for explainable deepfake models"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"The load-bearing premise is that large-scale VLM-written silver explanations, and VLM judges used for filtering and Entity/Evidence scoring, stay faithful enough to human-visible manipulation evidence to train and rank models.","fun_headline_variants_meta":{"raw":{"variants":["XPlainVerse: Million images pair edit-checked fakes with dual-level explanations","EntityScore and EvidenceScore expose fluent but ungrounded deepfake reasons","Million-scale deepfake set verifies manipulations then scores entity fidelity","Held-out generators drop EntityScore more than BERTScore on deepfake reasons","Edit-Check pipeline enables reliable supervision for explainable deepfake models"]},"model":"grok-4.5","effort":"low","cost_usd":0.005268,"raw_usage":{"total_tokens":1474,"prompt_tokens":798,"num_sources_used":0,"completion_tokens":99,"cost_in_usd_ticks":52680000,"prompt_tokens_details":{"text_tokens":798,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":577,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":798,"tokens_out":99,"duration_ms":5077,"temperature":1.0,"reasoning_tokens":577,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T01:34:27.388690+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train and score the same models using only the 2,000 human gold explanations as targets and as the sole Entity/Evidence reference, then check whether the reported ID-to-OOD collapse and the ranking of methods reverse or disappear relative to the silver-label results.","supporting_citations":[],"review_version":1}