{"id":"424444fb-64ac-4ebc-8c3d-0345a63a15ac","arxiv_id":"2505.23283","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new one-million-image benchmark shows that diffusion-generated satellite forgeries are hard for existing detectors and that training on this data improves generalization.","lead":"The paper introduces RSFAKE-1M, a dataset of one million satellite images, half real and half fake, with forgeries created by ten diffusion models. It shows that current forgery detectors struggle on these images and that training on this dataset improves their generalization and robustness.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Source-domain confound between CLIP-RS-generated fakes and fMoW reals may make RSFAKE-1M reward dataset-of-origin detection rather than manipulation detection.","rationale":"The paper's main contribution is a large-scale benchmark whose usefulness depends on the premise that the fake/real distinction teaches detectors to recognize diffusion-generated forgeries, not merely to recognize which remote sensing source dataset an image came from. The construction in Sections 3.1 and 3.2 makes this premise insecure: fakes originate from CLIP-RS (with prompts and conditioning signals derived from CLIP-RS images), while reals are cropped from fMoW. Because the two sources differ in multiple low-level properties, a classifier can exploit these properties without learning anything about manipulation. This is the reader's weakest assumption, and I agree it is the load-bearing risk. The experiments do not rule it out: Table 3 shows that pretrained detectors perform poorly on RSFAKE-1M, which is consistent with the confound (they have not seen either source domain in combination); Table 4 shows that RSFAKE-trained baselines generalize to other datasets, which could indicate genuine cues but could also indicate that the model learned to flag any image unlike fMoW as fake. A negative-control experiment using original CLIP-RS images as 'fakes' would directly separate the source-domain explanation from the manipulation explanation. The paper otherwise has reasonable scale, a clear generation pipeline, and a sensible 1:1 real/fake balance; the issue is not internal inconsistency but an unvalidated assumption about what is being learned. This supports the reader's CONDITIONAL verdict rather than prompting acceptance or rejection. I recommend keeping the verdict unchanged pending the control experiment and release of the promised supplementary materials.","tokens_in":11307,"tokens_out":4912,"duration_ms":52289,"concrete_test":"Build a negative-control training set with the same fMoW real crops as 'real' and, as 'fake', the original pre-generation CLIP-RS source images (before any diffusion processing) that were used to produce the RSFAKE-1M fakes. Train the same ResNet-50 baseline with identical hyperparameters and evaluate on (a) the RSFAKE-1M test split and (b) the cross-dataset test sets of Table 4. If the control model achieves accuracy substantially above chance on RSFAKE-1M or transfers to other datasets, the benchmark's fake-real signal is confounded by source domain; if the control model stays near chance while models trained on actual fakes succeed, the confound is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RSFAKE-1M is a valid benchmark for diffusion-based remote sensing forgery detection requires that the fake/real label aligns with manipulation, not with source dataset. In Section 3.1, fakes are generated from the top 450K CLIP-RS image-text pairs, with CLIP-RS captions used as text prompts and CLIP-RS-derived edge, segmentation, and vector maps used for image-guided generation. In Section 3.2, real images are collected by cropping the fMoW dataset. The two classes therefore differ systematically in sensor type, geographic coverage, color processing, and curation (the CLIP-Score filter), independent of any diffusion artifact. A detector trained on RSFAKE-1M could learn to separate 'CLIP-RS-like' content from 'fMoW-like' content rather than real from forged. This confound affects both headline claims: the low accuracy of pretrained detectors in Table 3 may reflect domain novelty rather than forgery difficulty, and the cross-dataset generalization gains in Table 4 may come from generic source-domain cues. The phrase 'not explicitly paired' in Section 3.2 acknowledges non-pairing but does not remove the distribution gap; no control experiment isolates the CLIP-RS-versus-fMoW factor. The reported FID of 32.26 quantifies a distribution gap but does not attribute it to manipulation cues.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RSFAKE-1M, a large-scale dataset for detecting diffusion-generated remote sensing forgeries, containing 500K fake images produced by ten diffusion models under six generation conditions (text, segmentation masks, edge maps, vector maps, and inpainting) and 500K real images cropped from the fMoW dataset. The authors evaluate existing forgery detectors on RSFAKE-1M, train unified ResNet-50 baselines for cross-dataset generalization, analyze intra-dataset generalization across generators and conditions, and study the effect of training-set scale on robustness. They conclude that RSFAKE-1M is challenging for current detectors and improves generalization and robustness when used for training.","tokens_in":11704,"tokens_out":4355,"duration_ms":42049,"significance":"If valid, RSFAKE-1M is a substantial resource: it is significantly larger and more diverse than existing remote sensing forgery benchmarks, focuses on diffusion generators, and is publicly released. The multi-generator, multi-condition design is a strength, and the paper provides a systematic comparison that can inform future detector development. However, the central empirical claims rest on the assumption that the fake/real label reflects manipulation rather than source-dataset differences, and the cross-dataset advantage claimed for RSFAKE-1M is less clear when self-test performance is excluded. The dataset itself could not be inspected at review time, so reproducibility could not be independently verified.","major_comments":[{"comment":"The fake images are generated from the top 450K CLIP-RS image-text pairs (Section 3.1), while the real images are cropped from fMoW (Section 3.2). These sources differ systematically in sensor type, geographic coverage, and curation (including the CLIP-Score filter), so the fake/real label may be confounded with source dataset rather than manipulation. The low accuracy of pretrained detectors on RSFAKE-1M in Table 3, and the cross-dataset gains in Table 4, could therefore reflect detection of 'CLIP-RS-like versus fMoW-like' content instead of genuine forgery cues. No control experiment isolates this factor. I recommend adding an experiment that includes real CLIP-RS images (or fakes generated from fMoW-derived prompts) to assess the magnitude of the source-domain confound, or at minimum a classifier trained to distinguish CLIP-RS from fMoW to quantify how separable the two source domains are.","section":"Section 3.1 vs. 3.2; Tables 3 and 4"},{"comment":"The claim that models trained on RSFAKE-1M 'generally outperform' models trained on other datasets in cross-dataset evaluation is not supported when the self-test is excluded. The reported average of 72.97 for RSFAKE-1M is driven by its 99.88 on the RSFAKE-1M test set; the leave-one-out average over the other four datasets is 66.24, lower than the FLDCF-trained model's leave-one-out average of 72.81. RSFAKE-1M-trained models are also weaker on FSI, Geo-DefakeHop, and FLDCF test sets. Please report per-dataset results with the self-test excluded and apply a statistical test over multiple training runs.","section":"Table 4"},{"comment":"All experiments report single-run accuracies without error bars or significance tests. Many cross-generator and cross-condition differences in Tables 5 and 6 are a few percentage points, and claims such as 'text-only forgeries provide stronger generalization signals' (Section 4.3) or 'models trained on larger subsets consistently achieve higher accuracy' (Section 4.4) would be more credible with multiple seeds and confidence intervals or paired significance tests.","section":"Tables 3-6 and Figure 3"}],"minor_comments":[{"comment":"The sentence 'The development of forgery detectors typically depends on the availability of suitable datasets' is repeated verbatim twice in the introduction; remove the duplicate.","section":"Section 1"},{"comment":"The name of the DiffusionSat model is inconsistently capitalized ('DiffusionSat' vs. 'DiffusionSAT'); please standardize.","section":"Throughout"},{"comment":"Table 2 states that 'All models are trained on remote sensing data,' but MapSat is trained on OpenStreetMap data from Central Belt Scotland; clarify the distinction between full remote sensing training corpora and region- or task-specific training sets.","section":"Table 2"},{"comment":"The figure caption contains a typo: 'Diffuison Model' should be 'Diffusion Model'.","section":"Figure 1"},{"comment":"The FID values reported in Table 1 are not accompanied by details of how they were computed (e.g., which feature extractor and which reference set were used); please specify this in the text or a footnote.","section":"Table 1"},{"comment":"The phrase 'not explicitly paired' in the real-image collection description only rules out exact image-level pairing; it does not rule out distributional non-independence. Please discuss the distributional relationship between CLIP-RS and fMoW more explicitly.","section":"Section 3.2"},{"comment":"The dataset URL could not be verified at review time. Please ensure the dataset, generation code, and a data card (including license and intended uses) are publicly accessible and stable at the time of publication.","section":"Data availability"}],"recommendation":"major_revision","confidential_remarks":"The source-domain confound is the key risk. If the authors can add a control experiment showing that the fake/real distinction is not reducible to CLIP-RS-versus-fMoW, the dataset contribution would be much stronger. The cross-dataset generalization claim also needs re-analysis with self-test excluded. I was unable to access the HuggingFace link during review; the editor may wish to verify dataset availability."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about RSFAKE-1M. First, it is a real resource: 500K diffusion-generated remote sensing forgeries from ten public models across six generation conditions, balanced with 500K fMoW real images. That fills an actual gap. Second, the fake and real sets are not just 'manipulated vs pristine'—they come from different universes. Fakes are generated from CLIP-RS image-text pairs filtered by CLIP-Score; reals are cropped from fMoW. So any detector trained on this data can succeed by recognizing 'CLIP-RS-ness' rather than manipulation, and the paper's headline results inherit that ambiguity.\n\nThe construction is careful in most respects. Ten RS-trained diffusion models is a meaningful step beyond the single-model GAN benchmarks, and the six conditions (text, segmentation, edge, vector, inpainting) give the dataset real coverage. The experimental suite is thorough: pretrained detectors, cross-dataset generalization, intra-dataset generalization across models and conditions, and scale/robustness curves. The finding that current detectors struggle on diffusion RS forgeries is plausible and well supported. Credit where due.\n\nThe soft spot is the confound. Section 3.1 says prompts come from CLIP-RS, and Section 3.2 says reals come from fMoW. The paper notes the sets are 'not explicitly paired,' but that doesn't remove the distribution gap—it just makes it unmeasured. The FID of 32.26 tells you there is a gap, not whether it is manipulation or source. The cross-dataset gains in Table 4 could come from generic source cues, and the poor showings in Table 3 could be domain novelty rather than forgery difficulty. No control experiment isolates the CLIP-RS factor. The single ResNet-50 baseline and absence of error bars are minor by comparison; I'd still trust the ordinal patterns, but not the exact numbers.\n\nThis paper deserves a serious referee. The dataset is large, public, and likely to be used regardless, so the field needs a published benchmark that is honest about its bounds. I would send it to review with one required addition: a control experiment where real images are drawn from the same CLIP-RS source (or fakes are generated from fMoW prompts) to show that the detector isn't just separating provenance. If that control holds, the paper becomes a standard resource. If it doesn't, the authors need to reframe the claims.\n\nFor your reading group: worth a slot, because the confound is a textbook example of a benchmark validity problem. I'd cite it only with the caveat, but I'd engage.","headline":"A genuinely useful large-scale diffusion forgery benchmark for remote sensing, but the fake real gap is confounded with source dataset (CLIP-RS vs fMoW), which muddies both headline claims.","tokens_in":12110,"tokens_out":1780,"would_cite":true,"duration_ms":21445,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RSFAKE-1M is a one-million-image benchmark for detecting diffusion-generated remote sensing forgeries; the paper shows that current detectors struggle on it, while training on it improves cross-dataset generalization.","keywords":["remote sensing forgery detection","diffusion models","synthetic image detection","benchmark dataset","cross-domain generalization","satellite imagery","deepfake geography"],"falsifier":"Train a baseline on RSFAKE-1M and test it on forgeries from a diffusion model not among the ten, generated with prompts derived from fMoW captions; if accuracy falls to near chance, the generalization claim is tied to the covered generators rather than to diffusion forgeries in general. A second check: if the same baseline can separate fMoW real crops from unmodified CLIP-RS images at high accuracy, then the benchmark's fake-real gap is partly dataset-of-origin bias.","tokens_in":11124,"feed_emoji":"🛰️","tokens_out":16711,"duration_ms":146191,"temperature":0.7,"pith_summary":"The paper sets out to show that forged satellite images made by modern diffusion models are a real and unsolved detection problem, and that a large, diverse training set is what detectors need to become robust. To that end it builds RSFAKE-1M, a million-image benchmark pairing 500,000 diffusion-generated fakes with 500,000 real remote sensing images, and runs a controlled comparison of existing detectors against uniform baselines. If the dataset holds up, it gives the community a shared test bed that is harder than earlier GAN-focused benchmarks and better at teaching cross-domain generalization. The stakes are practical: forged satellite imagery can mislead urban planning, environmental monitoring, and security decisions.","feed_headline":"Million-image fake satellite benchmark still fools top detectors","feed_subtitle":"Ten diffusion models, six forgery styles; detectors peak near 67%, training on the set lifts cross-dataset scores.","key_machinery":"The machinery is the dataset itself, RSFAKE-1M: 500,000 fake images produced by ten publicly available diffusion models (DiffusionSat-256/512, GeoRSSD, SDFRS, GeoSynth-text/sam/canny, CRSDiff, MapSat, RSPaint) fine-tuned on remote sensing data and covering six generation conditions—text prompts, segmentation masks, HED and Canny edge maps, vector maps, and inpainting masks—paired with 500,000 real images cropped from fMoW under a resolution-matching protocol that mirrors the fake set's size distribution. To make results comparable, the paper fixes a uniform standard ResNet-50 convolutional backbone with a two-layer MLP head and identical training hyperparameters across all datasets and subsets, so observed differences can be attributed to training data rather than architecture or tuning.","core_discovery":"The central claim is that diffusion-generated remote sensing forgeries are a distinct and still-open detection problem, and that a large, diverse training corpus is the key to making detectors robust. On RSFAKE-1M's test split, the best pretrained general-purpose detector (UniversalFakeDetect) reaches about 61% accuracy and an area under the ROC curve of 0.67, while remote-sensing-specific detectors such as Geo-DefakeHop and two FLDCF variants fall to roughly 52-56% accuracy, close to chance. A uniform standard ResNet-50 convolutional backbone trained on RSFAKE-1M reaches 99.88% in-domain accuracy and 72.97% average accuracy across five datasets, outperforming the same baseline trained on FSI, Geo-DefakeHop, DM-AER, or FLDCF. Within the dataset, text-only training transfers best across generation conditions, and scaling the training set from 10% to 100% monotonically improves generalization and robustness to downsampling, JPEG compression, and blur.","pith_inferences":["A stronger stress test the paper does not run: hold out an entire modern diffusion family, retrain on RSFAKE-1M, and check whether accuracy survives; the cross-generator tables suggest it may drop sharply.","If a detector trained on RSFAKE-1M can also separate fMoW crops from raw CLIP-RS images, part of the measured difficulty is source-dataset bias rather than genuine forgery signatures.","The text-only advantage hints that adding more caption-driven generators, and adding conditions like layout or editing tools, would stretch the benchmark's difficulty in the directions real forgers use.","The scaling curves imply RSFAKE-1M could serve as a pretraining corpus for a universal satellite-forgery detector, with fine-tuning on smaller application-specific sets as the final step."],"forward_implications":["Existing remote sensing and general-purpose AIGC detectors cannot be deployed directly on diffusion-generated satellite forgeries; the best pretrained detector reaches only about 61% accuracy and 0.67 AUC on RSFAKE-1M.","Training on RSFAKE-1M transfers across domains: the unified baseline reaches 72.97% average accuracy over five datasets, beating models trained on FSI, Geo-DefakeHop, DM-AER, or FLDCF.","Text-only forged images are the most transferable training signal across generation conditions, so future datasets should keep caption-driven generation at the core.","Dataset scale is a direct lever: increasing training data from 10% to 100% of RSFAKE-1M improves accuracy on out-of-domain data and under downsampling, JPEG compression, and Gaussian blur.","The dataset's ten-model, six-condition splits expose that generation condition changes can matter as much as architecture changes, since GeoSynth variants transfer unevenly to each other."],"supporting_citations":[{"why":"It supplies two of the ten generators (DiffusionSat-256/512) that create text-prompt fakes at 256 and 512 resolutions.","marker":"[33]"},{"why":"It supplies the GeoRSSD text-prompt generator that creates 768-resolution fakes.","marker":"[37]"},{"why":"It supplies the SDFRS text-prompt generator used for 512-resolution fakes.","marker":"[36]"},{"why":"It supplies three GeoSynth generators (text, segmentation, Canny) covering the text-only and image-guided conditions.","marker":"[32]"},{"why":"It supplies the CRSDiff generator that produces fakes from HED edge maps.","marker":"[31]"},{"why":"It supplies the MapSat vector-map generator and the OpenStreetMap extraction protocol.","marker":"[34]"},{"why":"It supplies the RSPaint inpainting generator used for the masked-image condition.","marker":"[35]"},{"why":"It supplies the 500,000 real images via the fMoW dataset, with a resolution-matching crop protocol.","marker":"[42]"},{"why":"It supplies the CLIP-RS image-text pairs whose captions drive the text-prompt and guided generation paths.","marker":"[38]"},{"why":"It provides FLDCF, the most recent diffusion-based remote sensing forgery dataset and detector used for cross-dataset comparisons.","marker":"[4]"}],"fun_headline_variants":["New 1M image dataset exposes satellite forgery detection gap","Diffusion fakes in space: 1M remote sensing images challenge detectors","500K diffusion fakes in RSFAKE-1M stump even top satellite detectors","Satellite forgery detection: 1M diffusion-generated images expose weak spots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's value depends on the ten diffusion models and six generation conditions being representative of the forgeries a real detector would face, and on the fake-real differences coming from actual forgery cues rather than from the different source datasets behind the fake and real images.","fun_headline_variants_meta":{"raw":{"variants":["New 1M image dataset exposes satellite forgery detection gap","Diffusion fakes in space: 1M remote sensing images challenge detectors","500K diffusion fakes in RSFAKE-1M stump even top satellite detectors","Satellite forgery detection: 1M diffusion-generated images expose weak spots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001001,"raw_usage":{"total_tokens":4267,"prompt_tokens":1005,"completion_tokens":3262,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":3192}},"tokens_in":621,"tokens_out":3262,"duration_ms":25158,"temperature":1.0,"reasoning_tokens":3192,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:47:37.545291+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a baseline on RSFAKE-1M and test it on forgeries from a diffusion model not among the ten, generated with prompts derived from fMoW captions; if accuracy falls to near chance, the generalization claim is tied to the covered generators rather than to diffusion forgeries in general. A second check: if the same baseline can separate fMoW real crops from unmodified CLIP-RS images at high accuracy, then the benchmark's fake-real gap is partly dataset-of-origin bias.","supporting_citations":[{"cited_title":"Diffusionsat: A generative foundation model for satellite imagery","cited_arxiv_id":null,"evidence_quote":"It supplies two of the ten generators (DiffusionSat-256/512) that create text-prompt fakes at 256 and 512 resolutions."},{"cited_title":"Rs5m and georsclip: A large scale vision-language dataset and a large vision-language model for remote sens- ing.IEEE Transactions on Geoscience and Remote Sens- ing, 62:1–23, 2024","cited_arxiv_id":null,"evidence_quote":"It supplies the GeoRSSD text-prompt generator that creates 768-resolution fakes."},{"cited_title":"Efficient and controllable remote sensing fake sample generation based on diffusion model.IEEE Transactions on Geoscience and Remote Sensing, 61:1–12, 2023.7","cited_arxiv_id":null,"evidence_quote":"It supplies the SDFRS text-prompt generator used for 512-resolution fakes."},{"cited_title":"Geosynth: Contextually-aware high- resolution satellite image synthesis","cited_arxiv_id":null,"evidence_quote":"It supplies three GeoSynth generators (text, segmentation, Canny) covering the text-only and image-guided conditions."},{"cited_title":"Crs-diff: Controllable remote sensing image generation with diffusion model","cited_arxiv_id":null,"evidence_quote":"It supplies the CRSDiff generator that produces fakes from HED edge maps."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the MapSat vector-map generator and the OpenStreetMap extraction protocol."},{"cited_title":"Tackling few-shot segmentation in re- mote sensing via inpainting diffusion model","cited_arxiv_id":null,"evidence_quote":"It supplies the RSPaint inpainting generator used for the masked-image condition."},{"cited_title":"Remote sensing semantic segmentation quality assessment based on vision language model","cited_arxiv_id":null,"evidence_quote":"It supplies the CLIP-RS image-text pairs whose captions drive the text-prompt and guided generation paths."},{"cited_title":"Fldcf: A collaborative framework for forgery localization and detection in satellite imagery.IEEE Transactions on Geo- science and Remote Sensing, 2024","cited_arxiv_id":null,"evidence_quote":"It provides FLDCF, the most recent diffusion-based remote sensing forgery dataset and detector used for cross-dataset comparisons."}],"review_version":1}