{"id":"52b78cc5-1b94-42b1-8eb2-2632aaddd371","arxiv_id":"2510.07135","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A first benchmark of few-shot adaptation for remote-sensing VLMs shows zero-shot accuracy does not predict few-shot gains, with no single method dominating.","lead":"This paper builds and releases a benchmark that tests how well remote-sensing vision-language models adapt when given very few labeled examples. It compares three remote-sensing models and five adaptation methods across ten scene-classification datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified pretraining-disjointness (§II-A) is load-bearing; if benchmark images appear in RS5M/RET-3/SkyScript, GeoRSCLIP's consistent advantage and zero-shot/few-shot conclusions are contaminated.","rationale":"The paper's two headline findings are empirical comparisons between models. Both require the test sets to be unseen. The stated disjointness is the only thing preventing the numbers from measuring memorization rather than generalization. The manuscript provides no evidence for it, and the pretraining sources are exactly the kinds of RS corpora that could include existing benchmark images. This is not a disagreement with consensus; it is a missing factual check on which the central claim depends. By contrast, missing confidence intervals and fixed hyperparameters are important but secondary: they would weaken the precision of rankings but not invalidate them categorically. The reader's conditional verdict is appropriate; our concern reinforces that condition rather than changing it. The open-source code is real evidence for reproducibility, but it does not address the data-leakage premise.","tokens_in":9219,"tokens_out":4754,"duration_ms":41246,"concrete_test":"Run a dataset-overlap audit: for each of the 10 benchmark datasets, compare test images against RS5M, RET-3/SEG-4/DET-10, and SkyScript using perceptual hashing and/or a pretrained ViT feature nearest-neighbor search. If any benchmark image has a near-duplicate in a pretraining corpus (e.g., similarity above the 99th percentile of a no-overlap control), remove or re-collect the contaminated dataset(s) and re-run the experiments to see whether GeoRSCLIP still ranks first and whether the zero-shot vs few-shot decoupling persists. If no overlap is found, the current conclusions are substantially strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — GeoRSCLIP consistently outperforms the other models under few-shot adaptation, and zero-shot performance is not a reliable indicator of few-shot performance — rests on the §II-A assertion that 'none of the selected evaluation datasets were used during the pretraining of the implemented RSVLMs.' This premise is asserted but never audited. GeoRSCLIP is fine-tuned on RS5M (~5M image-text pairs) and RemoteCLIP on RET-3/SEG-4/DET-10; these corpora are large, public, and often assembled from existing RS scene datasets, so it is plausible that images from EuroSAT, RESISC45, AID, PatternNet, or MLRSNet appear directly or as near-duplicates. If any benchmark test images were seen during pretraining, the zero-shot and few-shot accuracies — especially for the model with the largest pretraining corpus — are inflated, and the cross-model ranking that dominates the paper's conclusions is contaminated. No overlap check, URL/ID filter, or nearest-neighbor audit is reported anywhere. The promise of open-source code is a reproducibility plus, but it does not verify this factual premise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a benchmark for few-shot adaptation of remote sensing vision-language models (RSVLMs). It evaluates four models (GeoRSCLIP, RemoteCLIP, SkyCLIP, and CLIP) on ten RS scene-classification datasets, using five adaptation methods (CoOp, MaPLe, TaskRes, Tip-Adapter, CLIP-LoRA). Results are reported for 0, 1, 2, 4, 8, and 16 shots, mostly as averages over three seeds, with detailed tables for GeoRSCLIP and additional scaling experiments on larger backbones. The central claims are that GeoRSCLIP consistently outperforms the other models under all adaptation methods; that zero-shot performance is not a reliable predictor of few-shot adaptation performance; and that no single adaptation method dominates, with CLIP-LoRA strongest at low shots and Tip-Adapter strongest at 16 shots. The paper also promises open-source code.","tokens_in":9543,"tokens_out":4876,"duration_ms":43823,"significance":"If the empirical findings hold, this is a useful first structured benchmark for few-shot adaptation of RSVLMs. The multi-dataset scope, the inclusion of several adaptation families, and the promise of reproducible code are strengths. The finding that method rankings are shot- and dataset-dependent would be practically valuable. However, the main conclusions currently rest on an unverified pretraining-disjointness claim and on three-seed averages without any noise characterization, so the significance of the specific rankings is not yet established.","major_comments":[{"comment":"The load-bearing premise that 'none of the selected evaluation datasets were used during the pretraining of the implemented RSVLMs' is asserted but never audited. GeoRSCLIP is fine-tuned on RS5M, RemoteCLIP on RET-3/SEG-4/DET-10, and SkyCLIP on SkyScript; these public corpora are often assembled from existing RS scene datasets, so overlap with EuroSAT, RESISC45, AID, PatternNet, MLRSNet, and similar is plausible. If any benchmark test images or near-duplicates appeared during pretraining, the zero-shot and few-shot accuracies in Figure 1 and Tables III–V are inflated for the affected models, directly contaminating the paper's central cross-model ranking. Please provide an overlap audit (e.g., URL/ID filtering, duplicate/near-duplicate detection, or dataset-construction documentation) and re-run or qualify the results accordingly.","section":"II-A"},{"comment":"All quantitative claims rest on averages over three random seeds, yet no standard deviations, confidence intervals, or significance tests are reported anywhere. For example, Table III shows differences of 1–3 points between methods (e.g., 4-shot TaskRes average 85.4 vs. CLIP-LoRA 87.8; 1-shot TaskRes 79.7 vs. CLIP-LoRA 79.4), and Figure 1 has no error bars. Without per-seed variability, the assertions that 'GeoRSCLIP consistently outperforms' and that 'zero-shot performance is not always a reliable indicator' are not statistically supported. Please report per-seed results or standard deviations, and state the exact support-set generation procedure: the seed values, whether support sets are class-balanced, and whether the same support sets are used across methods and models.","section":"III (Fig. 1, Tables III–V)"},{"comment":"The paper's central methodological claim is that it provides a reproducible benchmark, but the description of the few-shot protocol is incomplete. Section II-C defines the support set formally but does not specify how the C×K examples are sampled per seed, what the query set is, or whether validation splits are used for early stopping or hyperparameter selection. The split '50/25/25 with a fixed random seed' is mentioned in II-A, but the relationship between that seed and the 'three random seeds' used in the experiments is unclear. Without these details, the benchmark is not fully reproducible and the fairness of the method comparison is difficult to assess. Please specify the protocol precisely and, ideally, release the exact splits and support-set indices.","section":"II-C / IV"}],"minor_comments":[{"comment":"The GitHub URL in the full text and code-accessibility section contains a space ('fewshot RSVLMs') and should be a proper hyperlink with the underscore; the abstract version is correct.","section":"Abstract/Sec. IV"},{"comment":"The axis labels contain repeated placeholder text '(averaged out on 10 datasets)' and lack error bars; this makes the figure hard to parse and overstates certainty.","section":"Fig. 1"},{"comment":"The 'Vision Encoder Backbones' row is difficult to read because column boundaries are not clear. Please format the backbone lists explicitly for each model.","section":"Table II"},{"comment":"For unbalanced datasets (AID, MLRSNet, RSICB128, RSICB256) the 'Available Training Samples per Class (avg.)' is an average over classes and could be misleading; consider giving the per-class range or median.","section":"Table I"},{"comment":"The name 'Tip-Adapter' is inconsistently capitalized as 'TIP-Adapter' in several places (e.g., Table IV, Table V). Please standardize.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The pretraining-overlap issue is the most serious concern and should be resolved before acceptance; without an overlap audit, the main ranking claim is fragile. The lack of statistical grounding is also important for a benchmark that draws comparative conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the first structured few-shot adaptation benchmark for RSVLMs, and it is genuinely useful: ten public datasets, five adaptation methods, four models, code on GitHub. The finding that zero-shot accuracy does not predict few-shot adaptation is a real, nontrivial result for the community. Second, the paper's central comparison rests on an unverified factual premise: Section II-A asserts that none of the benchmark datasets were used in pretraining, but no overlap audit is given. GeoRSCLIP was fine-tuned on RS5M, a large aggregated corpus that very likely contains images from EuroSAT, RESISC45, AID, and similar public scene datasets. If that is true, the model with the largest pretraining set gets an unfair advantage and the headline result that GeoRSCLIP 'consistently outperforms' would be contaminated.\n\nWhat the paper does well: the experimental design is mostly sensible. They fix the backbone (ViT-B/32) for the main comparison, use five established adaptation methods, and report averages over ten datasets at shots 1, 2, 4, 8, 16. They also run a scaling study to larger backbones, which strengthens the claim that low-rank tuning is robust. The code release is a real asset; the community can re-run and extend.\n\nSoft spots, in order of severity:\n\n1. The overlap issue. This is not a modeling choice, it is a factual claim that needs evidence. A nearest-neighbor or URL/ID audit against RS5M, RET-3, SkyScript and the public benchmark images is cheap to run and should be mandatory. Without it, the cross-model ranking is not trustworthy.\n\n2. No variance reporting. The paper says results are over three random seeds, but Tables III-V give only point averages, with no standard deviations or significance tests. Given that they also claim 'variability of performance' as a finding, this is a gap. It is fixable.\n\n3. Minor: support-set sampling procedure is underspecified. Fixed random seed for splits, but not clear how the few-shot support sets are drawn per seed. That matters for reproducibility, though less serious than the overlap issue.\n\n4. The co-author relationship with CLIP-LoRA is fine—it is evaluated against external baselines—but worth noting for the record.\n\nVerdict: This paper is worth engaging with and deserves a proper peer review, but the empirical conclusions should be treated as provisional until the overlap audit and error bars are provided. If the audit comes back clean, this becomes a solid reference benchmark for the RS few-shot community.","headline":"Useful first few-shot adaptation benchmark for RSVLMs, but the unverified pretraining-disjointness premise could flip the main ranking.","tokens_in":9956,"tokens_out":1980,"would_cite":true,"duration_ms":17371,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that zero-shot accuracy does not predict few-shot adaptation for remote sensing vision-language models, and establishes the first structured benchmark that exposes the gap.","keywords":["few-shot learning","vision-language models","remote sensing","scene classification","benchmark","CLIP","prompt tuning","low-rank adaptation"],"falsifier":"A concrete test: audit the public pretraining corpora (RS5M, RET-3, SkyScript) for image or class overlap with the ten benchmark datasets, or re-run the benchmark on held-out datasets collected after the models' release; if GeoRSCLIP's margin shrinks or vanishes on uncontaminated data, the claim that it is inherently more amenable to few-shot adaptation would be weakened.","tokens_in":9183,"feed_emoji":"🛰️","tokens_out":4278,"duration_ms":33784,"temperature":0.7,"pith_summary":"This paper argues that the remote sensing community cannot infer few-shot adaptation ability from zero-shot accuracy, and backs that claim with the first structured benchmark dedicated to low-data adaptation of remote sensing vision-language models. Across ten scene-classification datasets, three RS-specialized models, and five adaptation methods, it finds that GeoRSCLIP consistently adapts best, while the best adaptation method shifts with shot count: CLIP-LoRA leads at two to four shots, Tip-Adapter at sixteen. The paper also shows that low-rank finetuning scales better to large backbones than prompt-tuning or text-residual methods. A reproducible, extensible codebase accompanies the benchmark, so future models and methods can be compared under identical conditions.","feed_headline":"Zero-shot scores fail to predict few-shot adaptation in remote sensing","feed_subtitle":"A 10-dataset benchmark shows GeoRSCLIP adapts best, while CLIP-LoRA and Tip-Adapter trade leads as shots grow.","key_machinery":"The central object is the benchmark suite itself: ten remote sensing scene-classification datasets, three RS-specialized vision-language models (RemoteCLIP, GeoRSCLIP, SkyCLIP) plus the original CLIP, and five few-shot adaptation methods (CoOp, MaPLe, TaskRes, Tip-Adapter, CLIP-LoRA). The comparisons are carried by a controlled protocol: fixed random train/validation/test splits, three seeds, a shared ViT-B/32 backbone for the main model comparison, and the original hyperparameters for each adaptation method. This setup isolates the influence of pretraining data and adaptation strategy from architectural differences, making the observed ranking and shot-dependent trends interpretable.","core_discovery":"The central finding is that zero-shot performance is not a reliable proxy for few-shot adaptation in remote sensing scene classification. Models with nearly equal zero-shot accuracy, such as GeoRSCLIP and SkyCLIP, diverge once trained on a handful of labeled examples, with GeoRSCLIP ahead across all five adaptation methods on a matched backbone. At the method level, no single strategy dominates across datasets and shot counts: CLIP-LoRA is strongest on average at 2-4 shots, Tip-Adapter becomes best at 16 shots, TaskRes excels on high-class-count datasets, and MaPLe lags. Scaling experiments reinforce the low-rank result: CLIP-LoRA remains accurate on ViT-L/14 and ViT-H/14 backbones while CoO","pith_inferences":["If zero-shot and few-shot rankings diverge as broadly as this benchmark suggests, zero-shot leaderboards for RSVLMs should be accompanied by a few-shot evaluation before deployment decisions are made; the paper demonstrates the gap but stops short of prescribing a standard few-shot reporting protocol.","The dataset-dependent method rankings hint that the optimal tuning location—text embeddings, prompts, intermediate weights, or cached features—depends on class granularity and class count; TaskRes's strength on MLRSNet and RESISC45 suggests a testable hypothesis that text-side residual tuning helps when many fine-grained classes share visual structure.","A natural extension is to measure whether the GeoRSCLIP advantage persists under distribution shift or on other RS tasks such as segmentation and object detection, not just scene classification; the paper's fixed-seed, three-run protocol makes such extensions directly comparable.","The lack of an overlap audit between the RSVLMs' pretraining corpora and the benchmark datasets means the reported margins should be re-examined if any overlap is found; the paper's own numbers would remain internally comparable but their absolute size could be inflated."],"forward_implications":["GeoRSCLIP should be the default RSVLM for few-shot scene classification when labels are scarce, since it led across all five adaptation methods on the shared backbone.","Ranking adaptation methods by a single shot count is unreliable; any practical recommendation should cite the supervision level, because CLIP-LoRA leads at low shots while Tip-Adapter wins at 16.","Low-rank finetuning is the safest choice when scaling to large backbones: CLIP-LoRA kept improving with backbone size, while CoOp, MaPLe, and TaskRes lost accuracy on ViT-H/14.","Even one labeled example per class produces a significant improvement over zero-shot evaluation across all models and methods, so minimal supervision is worth exploiting.","The benchmark's open codebase allows future RSVLMs and adaptation methods to be inserted and compared under the same protocol, turning few-shot performance into a checkable quantity rather than an assumption."],"fun_headline_variants":["Zero-shot accuracy won't tell you who wins in few-shot","For remote sensing VLMs, zero-shot ≠ few-shot readiness","Benchmark: zero-shot scores mislead on few-shot adaptation","Few-shot adaptation in RSVLMs: zero-shot is no guide","New benchmark shows zero-shot can't predict few-shot gains"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that none of the ten evaluation datasets were used in pretraining the implemented RSVLMs; the paper asserts this in Section II-A but does not audit the pretraining corpora, and if any benchmark dataset leaked into training the zero-shot and few-shot numbers would be inflated and the cross-model comparison contaminated.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot accuracy won't tell you who wins in few-shot","For remote sensing VLMs, zero-shot ≠ few-shot readiness","Benchmark: zero-shot scores mislead on few-shot adaptation","Few-shot adaptation in RSVLMs: zero-shot is no guide","New benchmark shows zero-shot can't predict few-shot gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000779,"raw_usage":{"total_tokens":3284,"prompt_tokens":754,"completion_tokens":2530,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2444}},"tokens_in":498,"tokens_out":2530,"duration_ms":10010,"temperature":1.0,"reasoning_tokens":2444,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:59:32.279821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: audit the public pretraining corpora (RS5M, RET-3, SkyScript) for image or class overlap with the ten benchmark datasets, or re-run the benchmark on held-out datasets collected after the models' release; if GeoRSCLIP's margin shrinks or vanishes on uncontaminated data, the claim that it is inherently more amenable to few-shot adaptation would be weakened.","supporting_citations":[],"review_version":1}