{"id":"94f62118-6d93-41ea-93fd-8ebdc2785adc","arxiv_id":"2505.01008","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A masked-region recovery score, computed with an open-source surrogate fine-tuned on under 1,000 target samples, separates images generated by a black-box diffusion model from real photos.","lead":"This paper presents a black-box method to detect AI-generated images by masking part of an image and checking how well a generative model can recover the missing region. It claims strong detection with fewer than 1,000 API samples and less than 2 GPU hours of compute, which matters for policing fake images from closed commercial models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 86.61 mAP may be inflated by overlap between LoRA alignment samples and fake test images; the paper never states the split. This should be checked before accepting the central claim.","rationale":"I read the paper in good faith and identify the central claim as the empirical black-box detection result: recovery-based scoring with a LoRA-aligned surrogate reaches 86.61 mAP across eight diffusion variants. The most load-bearing condition for that claim is that the evaluation measures generalization, not memorization. The paper describes collecting fewer than 1,000 alignment samples from the target model and then fine-tuning the surrogate with LoRA, but it never states that the test fake images are disjoint from these alignment samples. Without a clean split, the reported AP could reflect the surrogate's ability to reconstruct images it was fine-tuned on, which would not generalize to new outputs from the same model. The reader's weakest_assumption focuses on the Likelihood-Gap Hypothesis, which is a legitimate theoretical concern, but it is less decisive: even if the hypothesis is not rigorously derived, the method could still be validated empirically. The missing train/test separation is more concrete and directly threatens the headline numbers. I therefore recommend keeping the reader's conditional verdict, pending a check of the code and a re-evaluation with a disjoint split. I give partial agreement with the reader because they mention the disjointness issue in the rationale but did not elevate it to the weakest_assumption field.","tokens_in":15563,"tokens_out":3949,"duration_ms":42979,"concrete_test":"Inspect the released code at https://github.com/Anonymous/ZJU-genai-detect (or the current repository) to determine whether any fake test images used in Table 1 also appear in the LoRA alignment set S. If a split exists, recompute Table 1 using a held-out disjoint set of target-model images, e.g., 500 alignment images and 500 distinct test images. If no explicit split exists, rerun the full pipeline with the 1,000 API samples split 50/50 into alignment and test and report AP for each target model. If the mAP drops below 80%, the headline claim of 86.61% needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—86.61 mAP across eight diffusion variants using a Stable Diffusion surrogate plus LoRA—requires that the LoRA alignment set S from Section 3.2 (N < 1,000 images per target model) be disjoint from the fake test images used in Table 1. The manuscript nowhere specifies this split. If S overlaps or contains the test fake images, the reported AP can be inflated by memorization: the LoRA-tuned surrogate is evaluated on images it was fine-tuned on, so 'recovery' becomes near-retrieval. This is not a minor experimental detail; it determines whether the black-box detection numbers measure generalization to new generated images or only in-sample fit. The Likelihood-Gap Hypothesis in Section 3.3 is indeed assumed rather than derived, but the paper's practical contribution rests on the empirical numbers, and those numbers are only meaningful if the evaluation split is clean. The reader's rationale already flags this as a missing detail, and it is the most load-bearing unresolved issue in the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a black-box detector for AI-generated images based on a corrupt-and-recover principle: a surrogate generative model, aligned to the target model via LoRA fine-tuning on a small set of API-generated images, reconstructs masked regions, and a PSNR-based score separates real from generated images. Experiments report 86.61% mean average precision across eight diffusion-model variants using a Stable Diffusion surrogate, with less than 1,000 alignment samples and under two GPU hours. The paper also introduces a paired DALL-E 3 benchmark and a human-evaluation study, and releases code.","tokens_in":15748,"tokens_out":2400,"duration_ms":24614,"significance":"If the empirical claims hold with a clean evaluation protocol, the method is practically valuable: it removes the need for model weights and large real-image corpora, and it offers a resource-efficient black-box detection route. The idea of aligning a surrogate by parameter-efficient fine-tuning is sound and the reported gains over the included baselines are substantial. The paper also ships code and a new paired benchmark, which are useful community resources. However, the central empirical claim currently rests on an unstated train/test split for the alignment set, and the most relevant baseline, DIRE, is missing; both issues must be resolved before the stated performance numbers can be taken at face value.","major_comments":[{"comment":"The manuscript never states whether the alignment dataset S, collected from the target-model API in Section 3.2, is disjoint from the fake test images used to compute the detection scores in Table 1. This is load-bearing: if S contains or overlaps the test generated images, the LoRA-fine-tuned surrogate is evaluated on images it was fine-tuned on, so the reported 86.61 mAP and the Table 2 improvement from 86.74 to 92.97 AP after fine-tuning could reflect memorization rather than generalization. Please specify the exact split (e.g., generate a fresh set of target-model images for testing after collecting S) and rerun Table 1 and Table 2 on a truly held-out set of target-generated images, reporting the numbers for both the overlapped and disjoint settings.","section":"§3.2 and §4.1, Table 1"},{"comment":"DIRE [43] is cited in Related Work as observing that diffusion-generated images can be approximately reconstructed by a diffusion model while real images cannot, which is the same recovery-based principle used here. DIRE is therefore the most directly relevant baseline, yet it is not included in Table 1 or anywhere in the experiments. Without a DIRE comparison, the claim of outperforming baseline methods by 4.31% mAP is not fully supported. Please add DIRE to Table 1 and to the new DALL-E 3 benchmark, using the same evaluation protocol.","section":"§2.2 and §4.2, Table 1"},{"comment":"The Likelihood-Gap Hypothesis assumes a positive gap Delta between the expected log-likelihood of the machine generation process G and the real process H, but Delta is not derived and H is not defined as a concrete process for real photographs. Appendix A.1 then uses this hypothesis to conclude that detection is always possible, but the conclusion is conditional on the assumed gap, not a guarantee. Please clarify the status of the hypothesis: state explicitly that it is an assumption, provide a definition of H, and either prove or empirically test the existence of the gap for the models considered, or weaken the theoretical claims accordingly.","section":"§3.3 and Appendix A.1"},{"comment":"The paper introduces a new paired DALL-E 3 benchmark, but no detector results are reported on it. The benchmark is only used for the human-preference study in Section 4.4. Since the DALL-E 3 setting is described as especially challenging and the paper claims a practical detector, please report the proposed method's AP/AUROC/FPR95 on this benchmark and compare with at least the strongest baselines from Table 1, or explicitly state that the benchmark is released for future work only.","section":"§4.2, DALL-E 3 benchmark"}],"minor_comments":[{"comment":"The word 'numerious' should be 'numerous'.","section":"§3, Preliminary on Diffusion Models"},{"comment":"The word 'obervations' should be 'observations' in the visualization paragraph.","section":"§4.3, Visualization"},{"comment":"The threshold definition 'selected based on the fake data when 95% of fake test data points are declared as fake' is ambiguous; please clarify whether this is FPR at 95% recall on the fake class or another convention, and state the threshold-selection protocol used for the other metrics.","section":"§4.1, Metrics"},{"comment":"The manuscript alternates between 'AP' and 'mAP' for the same column; please use one consistent notation and define it in the caption.","section":"Table 1 and Table 2"},{"comment":"The paper does not specify how many image pairs are in the new DALL-E 3 benchmark or how the LLaVA-generated captions were filtered; please add dataset statistics and a brief filtering description.","section":"§4.2, New dataset for DALL-E 3"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is compelling but currently unverifiable without the alignment/test split disclosure and the DIRE baseline. Both are fixable within the manuscript's scope, so I recommend major revision rather than rejection. The theoretical section should also be repositioned as conditional on an assumption; otherwise it overstates the guarantee."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper proposes recovery-based detection of AI-generated images: corrupt a candidate image with a mask, ask a generative model to inpaint it, and use PSNR between original and recovered region as score. The genuinely new piece is for black-box APIs: instead of using the target model, take Stable Diffusion as surrogate, LoRA-tune it on fewer than 1,000 images sampled from the target API, and use that as inpainter. That is a clean, cheap, practical contribution. The reported 86.61 mAP across eight diffusion variants, achieved with under 2 GPU hours, makes it an appealing baseline. Code is public; ablations across masks, scores, and surrogate choices are useful. Credit where due: the idea is not brand-new—DIRE and DNA-GPT already use corrupt-and-recover—but the LoRA distribution alignment for black-box settings is a legitimate new application.\n\nSoft spots, in order of severity.\n\nFirst, the paper never states whether the LoRA alignment images S are disjoint from the fake images used in Table 1. If S overlaps the test set, the surrogate has seen the exact images it is \"recovering,\" and the 86.61 mAP no longer measures generalization to newly generated images. This is the biggest unresolved issue. It is easily fixable by reporting the split, but the current manuscript is silent.\n\nSecond, DIRE, the most relevant recovery-based baseline, is not compared. The paper cites it but omits it from Table 1. That makes the \"outperforms baselines\" claim weaker than the abstract implies.\n\nThird, the DALL-E 3 section promises a benchmark but reports no detector numbers—only human accuracy. The paper's own argument that DALL-E 3 is the hard case is therefore untested.\n\nFourth, the Likelihood-Gap Hypothesis in Sec 3.3 is assumed, not derived. The Appendix uses it to argue detection is always possible, but the argument is conditional on a positive gap Δ. If the gap is small for photorealistic models, the theoretical basis collapses. That is okay if the paper treats it as a stated assumption, but it should stop short of calling the theory a guarantee.\n\nThese are all addressable. The core pipeline is sensible, cheap, and reproducible from the public repo, assuming the code confirms the split. I think this deserves a serious peer review—conditional on the authors clarifying the alignment/test split and adding DIRE as a baseline. For a reading group, it would generate a good discussion about what counts as a fair black-box evaluation.","headline":"A practical LoRA-aligned inpainting detector for black-box AI images, but the main result is not trustworthy until the alignment/test split is clarified and DIRE is added as a baseline.","tokens_in":16302,"tokens_out":2163,"would_cite":false,"duration_ms":22860,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A black-box detector can tell AI-generated images from real photos by masking part of an image and measuring how well the suspected generative model repairs it, reaching 86.61% mean average precision across eight diffusion-model variants…","keywords":["AI-generated image detection","black-box detection","diffusion models","corrupt-and-recover","LoRA fine-tuning","likelihood gap","PSNR recovery score","DALL-E 3 benchmark"],"falsifier":"Take the paper's paired DALL-E 3 benchmark, mask half of each image, compute PSNR recovery scores with a surrogate aligned to DALL-E 3, and estimate the likelihood-gap Δ between fake and real images; if the AUROC of the recovery score approaches 0.5 (or Δ is statistically indistinguishable from zero) once image resolution and photorealism are high enough, the central claim is falsified for that model.","tokens_in":15342,"feed_emoji":"🖼️","tokens_out":9313,"duration_ms":83453,"temperature":0.7,"pith_summary":"This paper tries to establish that AI-generated images can be reliably detected without any access to model weights or large training datasets: you only need the suspect model's API. The central trick is a corrupt-and-recover test: mask part of an image, ask the model to fill in the missing region, and measure how faithfully it reconstructs it. Because a generative model should recognize its own outputs, its repairs to its own images are closer to the originals than its repairs to real photographs, so the recovery score separates fake from real. For closed APIs that only generate from scratch, the paper fine-tunes an open surrogate model with LoRA on fewer than 1,000 samples from the target API to align the two distributions. The result is 86.61% mean average precision across eight diffusion-model variants, beating supervised baselines by 4.31%.","feed_headline":"Mask an image, let AI repair it, catch fakes at 86.6% precision","feed_subtitle":"It catches images from eight diffusion models using only API access and under 2 GPU hours.","key_machinery":"The load-bearing object is the corrupt-and-recover discrepancy score: for an image x, a binary mask m splits it into known pixels X=(1−m)⊙x and unknown pixels Y0=m⊙x; the model samples K reconstructions Y′∼G(·|X), and the PSNR (Peak Signal-to-Noise Ratio) between Y0 and the recovered region is the detection statistic. It is paired with the Likelihood-Gap Hypothesis, which asserts that E_{Y∼G(·|X)}[log p(Y|X)]−E_{Y∼H(·|X)}[log p(Y|X)]>Δ, the formal reason a model should repair its own outputs better. The third component is distribution alignment: LoRA fine-tuning of an open surrogate on fewer than 1,000 target-API samples synchronizes the surrogate's distribution with the closed target model, letting the recovery test work when the API only generates from scratch. The theory also fixes the number of recovery samples at K=Ω(σ log(1/δ)/$Δ^{2}$) via a Hoeffding and sub-Gaussian argument.","core_discovery":"On the paper's own terms, the discovery is that a generative model should more easily recover its own generated images when corrupted than it would with real images. The detector masks a region, samples K recoveries from the model conditioned on the unmasked pixels, and scores the discrepancy with PSNR; a high recovery score marks the image as machine-generated. For black-box APIs that do not accept masked inputs, the paper aligns an open-source surrogate (Stable Diffusion) to the target distribution by LoRA fine-tuning on a small set of API samples, then runs the same recovery test with the surrogate. The formal statement is the Likelihood-Gap Hypothesis: conditioned on the known pixels, the expected log-likelihood of the machine generation process exceeds that of the real-image process by a positive gap $\\Delta$, and the paper argues this gap makes reliable detection possible even when the total-variation distance is small. Empirically the framework reports 86.61% mean average precision across eight diffusion-model variants (Guided, LDM, Glide, DALL-E, DALL-E 3), including a new paired DALL-E 3 benchmark on which human accuracy is about 72%.","pith_inferences":["If the Likelihood-Gap Hypothesis generalizes, the same mask-and-repair principle could be applied to video, audio, and text by substituting a domain-appropriate recovery model and scoring function; the paper only demonstrates images.","The paper's own DALL-E 3 numbers suggest the gap Δ shrinks as generators approach the real-image manifold; a testable prediction is that for future photorealistic models, the PSNR separation will narrow and LoRA alignment will become the dominant factor.","The recovery score could be repurposed as a model-attribution or membership test: among several candidate APIs, the model that repairs an image best is likely its source, which would extend binary fake/real detection to source identification.","An adversary aware of the detector could potentially craft images whose masked regions are hard to repair, so robustness of the recovery score under adversarial generation is an open question the paper does not address."],"forward_implications":["AI-content detection no longer requires white-box access: any model reachable through an API can be tested by its own repair behavior, so detectors scale to closed commercial generators.","A single open surrogate (Stable Diffusion) aligned with a small API sample set detects images from many unseen diffusion architectures, so per-model retraining is unnecessary.","The choice of discrepancy metric matters materially: PSNR outperforms SSIM, L1, and L2 by large margins, so future detectors should treat the scoring function as a first-class design choice.","In the white-box setting, using the target model itself for recovery gives the strongest separation (e.g., FPR 10.80% and AUROC 97.18% on Guided Diffusion), so the same framework covers both settings.","Detection accuracy falls on photorealistic DALL-E 3 images (75.98% AP) even though humans only reach about 72% accuracy, indicating the recovery signal survives but weakens as generation quality improves."],"supporting_citations":[{"why":"Supplies the core intuition that a model recovers its own generated content more easily than real content, which the paper adopts as its starting premise.","marker":"[46]"},{"why":"Provides Stable Diffusion, the open-source surrogate model that the framework LoRA-aligns to the target model's distribution.","marker":"[35]"},{"why":"Supplies the parameter-efficient low-rank adaptation method used to align the surrogate with the black-box target.","marker":"[19]"},{"why":"Establishes the dataset construction and evaluation setup for diffusion-generated fake images that this paper follows and compares against.","marker":"[28]"},{"why":"Demonstrates the reconstruction-based detection idea for diffusion images that this work adapts to masked black-box recovery.","marker":"[43]"},{"why":"Serves as a primary supervised baseline whose transfer to diffusion outputs fails, motivating the need for recovery-based detection.","marker":"[42]"},{"why":"Provides the patch-level classifier baseline used in the comparison table.","marker":"[6]"},{"why":"Supplies the AUROC impossibility bound that the Likelihood-Gap argument is intended to overcome.","marker":"[36]"}],"fun_headline_variants":["Ask the AI to repair a masked fake, and it snitches on itself","Black-box AI detector: mask, repair, and the gap reveals the fake","Recovery test catches AI images using only API access","AI images outed by their own repair job: 86.6% precision","Mask-and-repair test: a black-box way to spot diffusion fakes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire detector depends on the assumption that a machine generation process assigns measurably higher conditional likelihood to its own masked images than the real-photo process does; for a photorealistic model such as DALL-E 3, if this likelihood gap shrinks to zero, the recovery score no longer separates fake from real.","fun_headline_variants_meta":{"raw":{"variants":["Ask the AI to repair a masked fake, and it snitches on itself","Black-box AI detector: mask, repair, and the gap reveals the fake","Recovery test catches AI images using only API access","AI images outed by their own repair job: 86.6% precision","Mask-and-repair test: a black-box way to spot diffusion fakes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000686,"raw_usage":{"total_tokens":3130,"prompt_tokens":981,"completion_tokens":2149,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":2051}},"tokens_in":597,"tokens_out":2149,"duration_ms":14769,"temperature":1.0,"reasoning_tokens":2051,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:29:21.292828+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the paper's paired DALL-E 3 benchmark, mask half of each image, compute PSNR recovery scores with a surrogate aligned to DALL-E 3, and estimate the likelihood-gap Δ between fake and real images; if the AUROC of the recovery score approaches 0.5 (or Δ is statistically indistinguishable from zero) once image resolution and photorealism are high enough, the central claim is falsified for that model.","supporting_citations":[{"cited_title":"Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text","cited_arxiv_id":null,"evidence_quote":"Supplies the core intuition that a model recovers its own generated content more easily than real content, which the paper adopts as its starting premise."},{"cited_title":"Towards uni- versal fake image detectors that generalize across genera- tive models","cited_arxiv_id":null,"evidence_quote":"Establishes the dataset construction and evaluation setup for diffusion-generated fake images that this paper follows and compares against."},{"cited_title":"Cnn-generated images are surprisingly easy to spot","cited_arxiv_id":null,"evidence_quote":"Serves as a primary supervised baseline whose transfer to diffusion outputs fails, motivating the need for recovery-based detection."},{"cited_title":"What makes fake images detectable? understanding proper- ties that generalize","cited_arxiv_id":null,"evidence_quote":"Provides the patch-level classifier baseline used in the comparison table."}],"review_version":1}