{"id":"0d7a97b2-b26d-4961-9d25-ee623e878700","arxiv_id":"2607.24810","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Current vision-language models score only moderately on a new 10,738-image military remote-sensing benchmark and collapse on visual grounding and referring segmentation.","lead":"This paper introduces RRS-10K, a benchmark of 10,738 military-related satellite images with multiple-choice, captioning, box, and mask tasks to test how well vision-language AI models understand rare scenes. It evaluates 52 models and finds they handle coarse scene description but often fail at precise object location, pixel segmentation, and complex spatial reasoning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The all-or-nothing VG scores in Table II are the signature of an output-format/parsing artifact rather than a capability gradient; if confirmed, the headline claim of a 'clear weakness in visual grounding' is unsupported.","rationale":"The reader's weakest assumption is precisely the VG parsing/scoring artifact, and my independent reading converges on the same point. The bimodal distribution in Table II is the clearest internal signal of a flawed measurement rather than a true capability pattern: a zero score for Gemini-3-Flash, which otherwise ranks first overall, is implausible as a genuine inability to locate objects. The paper provides no description of the output-normalization step, no example of accepted/rejected model outputs, and no release of evaluation code, so the concern cannot be dismissed. I also considered other potential issues—missing dataset release, commercial imagery redistribution, internal table arithmetic inconsistencies, and potential leakage from using GPT-5.4 during annotation—but each is either secondary or addressable without threatening the central conclusion. The VG parsing issue directly determines whether the paper's headline claim of 'clear weaknesses in visual grounding' is true. Because the reader already set CONDITIONAL, my recommendation is UNCHANGED, with the concrete test serving as the condition that must be satisfied before ACCEPT. The check is inexpensive and would settle the question definitively.","tokens_in":35231,"tokens_out":3975,"duration_ms":38919,"concrete_test":"Select the 200 VG samples from the benchmark and re-score the raw outputs of all 43 models with a lenient parser: (1) extract all floating-point numbers in order via regex, (2) accept optional parentheses, brackets, or labels, (3) interpret the first four numbers as normalized coordinates (if all ≤1) or convert pixel values by image width/height, (4) evaluate IoU for every ordering of the two corners and take the maximum. Compare the resulting Acc@0.5 with Table II. If any model that previously scored 0.00 (e.g., Qwen3-VL-235B-Instruct, Gemini-3-Flash) rises above ~20 points, the original scores are parsing artifacts and the benchmark's grounding conclusion must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that current VLMs show 'clear weaknesses in visual grounding' rests on the VG column of Table II, where nearly all open-source models score 0.00–0.86, GPT-5.4 scores 49.89, Claude 25.80, and GPT-4o 14.93. This is not a smooth capability gradient but a bimodal cliff. The paper defines VG accuracy via Eq. (9) (IoU threshold) but never specifies how raw model outputs are parsed and normalized into (x1,y1,x2,y2) coordinates. The only protocol hint is the prompt example in Fig. 6, which requests 'the format (x1,y1, x2,y2)'. Many VLMs naturally emit coordinates in other layouts (JSON lists, prose, pixel coordinates, different corner order, extra text). A strict regex expecting exactly '(num,num),(num,num)' would return 0 for any deviation, producing exactly the observed pattern. The fact that even a frontier model like Gemini-3-Flash scores 0.00 on VG while leading other tasks strongly suggests a parsing failure, not an inability to ground. The paper's own construction pipeline (Section III-B.2.4) uses GPT-5.4 to generate referring expressions and Grounding DINO to produce boxes; no evidence is provided that the evaluation harness handles the same model's output formats. Until the parsing protocol is released and validated, the VG column cannot be interpreted as measuring capability, and the headline conclusion loses its strongest quantitative support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RRS-10K introduces a benchmark of 10,738 military-related remote sensing images for evaluating vision-language models (VLMs) on rare-scene interpretation. The benchmark is organized into 3 capability dimensions, 6 sub-dimensions, and 20 leaf tasks spanning perception, reasoning, and robustness, using multiple answer formats (MCQ, open-ended, bounding boxes, masks). Construction combines expert knowledge, open-source textual augmentation, GPT-5.4-assisted annotation, a CLIP-based similarity distractor filtering strategy (SDFS), and human review. The authors evaluate 43 VLMs and 9 referring-segmentation models and report that current models perform only moderately, with clear weaknesses in visual grounding, referring segmentation, and complex semantic reasoning, while leading open-source models are becoming competitive with proprietary systems.","tokens_in":35579,"tokens_out":8536,"duration_ms":77653,"significance":"RRS-10K addresses a real gap: rare and military remote sensing scenes are underrepresented in current VLM benchmarks. The paper's strengths are the scale (10,738 images, 24 countries), the hierarchical 20-task taxonomy, the 52-model evaluation, the SDFS distractor-filtering idea, and the multi-format annotation pipeline. If the measurement issues are resolved, the benchmark would be a useful diagnostic resource, and the finding that open-source models are competitive on many tasks would be informative. At present, however, several load-bearing evaluation choices are not validated, so the headline claims are not yet established at the level the paper asserts.","major_comments":[{"comment":"The VG column in Table II is bimodal: GPT-5.4 scores 49.89, Claude Sonnet 4.6 25.80, GPT-4o 14.93, while nearly all open-source models score 0.00–0.86 and Gemini-3-Flash scores 0.00 despite leading most other tasks. The manuscript defines Acc@τ but never states which τ is used for Table II and, more importantly, never specifies how raw model outputs are parsed and normalized into (x1,y1,x2,y2) coordinates. A strict regex parser expecting one exact layout would produce exactly the observed pattern. The prompt example in Fig. 6 requests the format '(x1,y1, x2,y2)', but no evidence is provided that all evaluated models' output formats are handled. Because the abstract and Section V identify visual grounding as a clear weakness, this is a load-bearing measurement question. Please release the parser, report unparseable rates, re-run the evaluations, or otherwise validate that the zeros reflec","section":"§IV-B(3), Eq. (9); §III-B.2.4; Fig. 6; Table II"},{"comment":"GPT-5.4 generated the preliminary question-answer-reasoning annotations, the referring expressions for VG and segmentation, and the hallucination-detection questions, and GPT-5.4 is then evaluated on this same benchmark. The manuscript asserts human review and correction but provides no quantitative edit rate, inter-annotator agreement, or analysis of residual model bias. The hallucination task is especially circular because GPT-5.4 wrote the deceptive prompts it is later tested against. This is not necessarily intentional bias, but it prevents the GPT-5.4 scores from being interpreted as an independent measurement. Please provide a human-authored held-out subset, exclude GPT-5.4 from headline rankings, or report per-item provenance and revision statistics.","section":"§III-B.2.3, §III-B.2.4, §III-B.3; Table II"},{"comment":"All comparisons are point estimates; there are no confidence intervals, bootstrap replicates, or significance tests across the 20 tasks or 52 models. Several headline differences are small (e.g., Qwen3-VL-235B-Instruct 69.63 vs GPT-4o 69.60 in Table II; Qwen3-VL-4B 62.82 vs Claude Sonnet 4.6 62.58). The claims that proprietary and open-source models are 'broadly competitive' and that scaling is 'not uniformly decisive' depend on rankings that may be within sampling noise. Add uncertainty quantification for the aggregated and task-level scores.","section":"§IV-C, Tables II–IV, Fig. 9"},{"comment":"SDFS is described only qualitatively: distractors are selected from the 'middle portion' of a CLIP similarity ranking, but no cutoffs, sampling rule, or number of candidates are specified. This makes the MCQ construction non-reproducible and leaves open the possibility that the benchmark's difficulty is coupled to CLIP's similarity space. Please release the similarity thresholds, per-item scores, and the list of rejected/kept options, and report human revision statistics. Without this, the MCQ Acc values in Tables II–VI cannot be independently reconstructed.","section":"§III-B.3.2"},{"comment":"Distance Calculation and Area Calculation require converting pixel coordinates to physical meters, but the manuscript does not describe how the metric-scale ground-truth labels were derived, which images contain usable scale cues, or how the 'weak-supervision metric-scale information' was extracted. The high DC/AC scores (e.g., Qwen3-VL-235B-Instruct 85.41 and 70.00 in Table II) are not interpretable without this protocol. Specify the conversion rule and exclude or flag samples without reliable scale information.","section":"§III-C.2 and Fig. 7"}],"minor_comments":[{"comment":"Typographical errors: 'PA VEPAW radar' appears in the scene-classification description and the VG example in Fig. 6 contains '0.635.0.213' instead of '0.635,0.213'.","section":"§III-C.1 and Fig. 6"},{"comment":"The abstract states images are 'collected from first-hand sources,' but Section III-B.1 says data come from 'publicly available sources and commercial satellite imagery providers, including Google Earth, Maxar, Airbus, and Planet Labs.' Please clarify what 'first-hand' means and specify data licenses and redistribution terms.","section":"§III-B.1 vs. Abstract"},{"comment":"Image Description is scored with BERTScore-F1, and the reported scores cluster tightly around 88–91 even for OCR-specialized models that fail badly on other tasks. BERTScore-F1 appears to saturate for these generic descriptions. Consider adding a more discriminative metric (e.g., CIDEr or human ratings) or reporting breakdowns by semantic content.","section":"§IV-B(2)"},{"comment":"The paper criticizes existing benchmarks for possible pretraining overlap, but it does not provide a leakage or near-duplicate analysis for RRS-10K images sourced from Google Earth, Maxar, Airbus, and Planet Labs. Please add a decontamination check or at least discuss the residual risk of overlap with VLM training corpora.","section":"§III-B.1"},{"comment":"The column label 'Geomatic & Spatial Inference' should be 'Geometric-Spatial Inference' for consistency with Section III-C.2.","section":"Table III"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope and the resource is potentially valuable. The gating issues are measurement validity rather than scope: the VG parsing protocol, the GPT-5.4 annotation/evaluation overlap, and the absence of uncertainty quantification. If the authors can address these in revision, I would be willing to review again."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know about RRS-10K, but don't trust the visual-grounding column. The benchmark—10,738 rare/military remote sensing images, a 3/6/20 task taxonomy, a CLIP-based distractor filter, and a 52-model evaluation—is a genuinely useful resource for the remote-sensing VLM community. The taxonomy and SDFS are well-motivated, and the broad finding that VLMs are only moderately good at rare scenes is plausible and consistent with the field's experience. The paper earns credit for organizing a fragmented area and for running a wide set of models.\n\nThe soft spots are real, though. The stress-test is right: the all-or-nothing VG scores (0.00–0.86 for nearly all open-source models, 49.89 for GPT-5.4, 0.00 for Gemini-3-Flash) have the signature of an output-parsing artifact, not a capability gradient. The paper never specifies how raw outputs are converted to coordinates, and the only prompt example demands a strict '(x1,y1),(x2,y2)' format. A regex that fails on any deviation would produce exactly this bimodal pattern. Until the parsing protocol is released and validated, the headline 'clear weaknesses in visual grounding' is not supported by the data. This is a load-bearing issue, but it's fixable: release the parser, show qualitative output samples, and report accuracy as a function of format tolerance.\n\nTwo more concerns. First, GPT-5.4 generated the preliminary QA/reasoning annotations, the referring expressions, and the hallucination questions, and is then evaluated on the same benchmark. Human review is asserted but not detailed; that circularity should be addressed with a visible human-audit sample and inter-annotator agreement. Second, the dataset is not released and includes commercial imagery (Maxar, Airbus, Planet), so redistribution rights are unclear; a license statement and commit hash are needed. Minor issues: no confidence intervals or leakage analysis, and a few table averages don't reconcile.\n\nNone of this sinks the paper. The taxonomy, SDFS, and the idea of a rare-scene benchmark are worth taking seriously. But a serious referee should require the VG parsing evidence before the central claims are accepted.\n\nI'd send it to review. The reading group might enjoy the parsing-artifact discussion as a cautionary tale.","headline":"RRS-10K is a needed rare-scene benchmark, but its visual-grounding column looks like a parsing artifact — don't cite the VG numbers until the parser is released.","tokens_in":36152,"tokens_out":2116,"would_cite":true,"duration_ms":20070,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current vision-language models are only moderately capable on rare remote sensing scenes, and RRS-10K is designed to measure exactly where they fail.","keywords":["rare remote sensing image interpretation","vision-language models","benchmark","visual grounding","referring segmentation","semantic reasoning","robustness","military scenes"],"falsifier":"Re-score the visual grounding subset after normalizing model outputs into several accepted coordinate formats (bracketed, unbracketed, decimal vs. percent, JSON vs. plain text) and compare the resulting Acc@0.5 scores. If open-source models jump from near zero to the level of the best proprietary model, the claimed grounding deficit is a scoring artifact. A complementary check is to audit a random sample of the ground-truth boxes and masks produced by the automatic detection and segmentation tools, since annotation errors would be inherited by every model score.","tokens_in":35060,"feed_emoji":"🛰️","tokens_out":6121,"duration_ms":54672,"temperature":0.7,"pith_summary":"The paper aims to establish that today's vision-language models, which look strong on common urban and rural remote sensing imagery, are only moderately capable on rare scenes such as military installations, and that the gap is concentrated in visual grounding, referring segmentation, and complex semantic reasoning. To make this claim testable, the authors built RRS-10K, a benchmark of 10,738 first-hand images organized into three capability dimensions, six sub-dimensions, and twenty leaf tasks, covering perception, reasoning, and robustness. They also introduce a distractor-filtering strategy to keep multiple-choice questions neither too easy nor too ambiguous, and they evaluate 52 models. If the benchmark is valid, it provides a diagnostic decomposition of rare-scene capability gaps rather than just a ranking.","feed_headline":"52 AI models score weak on rare satellite scenes","feed_subtitle":"New 10,738-image benchmark isolates the failures: visual grounding, segmentation, and complex spatial reasoning.","key_machinery":"The central object is RRS-10K itself: a hierarchical benchmark built from 10,738 rare-scene remote sensing images with multiple answer formats, organized into perception, reasoning, and robustness dimensions that decompose into 20 leaf tasks. The construction machinery that carries the argument is two-part: a human-centered hybrid annotation pipeline that combines expert descriptions, open-source references, and model-generated candidates later corrected by humans; and the similarity-based distractor filtering strategy (SDFS), which ranks a predefined option pool with a contrastive image-text encoder and keeps only distractors in a moderate-similarity band, so that multiple-choice questions","core_discovery":"The central claim is that a systematic evaluation of 52 models on rare remote sensing imagery reveals a reproducible capability hierarchy: coarse scene understanding and image description are comparatively mature, while visual grounding, referring segmentation, distance and area reasoning, and robustness to noise or occlusion remain major bottlenecks. The paper argues that this pattern reflects weak coupling between global scene semantics and fine-grained local evidence, and that leading open-source models are now broadly competitive with proprietary ones, with model scale alone insufficient to explain performance. RRS-10K is presented as the instrument that makes these bottlenecks visible a","pith_inferences":["The near-zero visual grounding scores for almost all open-source models, against a single proprietary score near 50 percent, look like a coordinate-format or parsing artifact; re-scoring with flexible output parsing could change the reported grounding ranking.","Because the same model used to draft annotations and referring expressions is also among the evaluated systems, the benchmark may implicitly favor that model's phrasing; a human-generated control set would test this.","The hierarchical taxonomy could be ported to other long-tail domains — disaster response, marine surveillance, or critical infrastructure — by replacing the scene categories while keeping the capability dimensions.","The paper evaluates zero-shot only; a natural next test is whether fine-tuning on RRS-10K subtasks closes the identified bottlenecks or merely overfits the benchmark."],"forward_implications":["If RRS-10K measures what it claims, then even the strongest evaluated models, averaging around 71 percent overall, are far from reliable on rare-scene interpretation, with visual grounding and referring segmentation near zero for most models.","The 20-task taxonomy turns the benchmark into a diagnostic tool: future work can target specific bottlenecks, such as the coupling of global scene semantics with local evidence, instead of optimizing a single aggregate score.","The reported competitiveness of leading open-source models implies that rare-scene remote sensing research can proceed with open baselines rather than depending on proprietary APIs.","The robustness results imply that hallucination suppression is largely solved relative to noise and occlusion robustness, so reliability work should concentrate on degraded and partially occluded imagery.","Because the benchmark uses first-hand imagery and a difficulty-aware distractor filter, score gaps are more likely to reflect genuine capability differences than leakage from pretraining corpora or trivially easy options."],"fun_headline_variants":["52 AI models flunk rare satellite scene tests","Rare satellite imagery: 52 AI models show weak reasoning","New benchmark: AI struggles with rare satellite scenes","10,738 rare images expose AI vision model limits","Rare satellite scenes defeat 52 vision-language models"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the evaluation harness correctly parses and scores visual-grounding and segmentation outputs; in the task-level results table, the Visual Grounding column collapses to 0.00 for nearly all open-source models while a single proprietary model reaches about 50, and if those zeros are a format or parsing artifact, the paper's headline 'clear weakness' claim is substantially weakened.","fun_headline_variants_meta":{"raw":{"variants":["52 AI models flunk rare satellite scene tests","Rare satellite imagery: 52 AI models show weak reasoning","New benchmark: AI struggles with rare satellite scenes","10,738 rare images expose AI vision model limits","Rare satellite scenes defeat 52 vision-language models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0005,"raw_usage":{"total_tokens":2260,"prompt_tokens":698,"completion_tokens":1562,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":442,"completion_tokens_details":{"reasoning_tokens":1486}},"tokens_in":442,"tokens_out":1562,"duration_ms":10472,"temperature":1.0,"reasoning_tokens":1486,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T06:57:32.998606+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the visual grounding subset after normalizing model outputs into several accepted coordinate formats (bracketed, unbracketed, decimal vs. percent, JSON vs. plain text) and compare the resulting Acc@0.5 scores. If open-source models jump from near zero to the level of the best proprietary model, the claimed grounding deficit is a scoring artifact. A complementary check is to audit a random sample of the ground-truth boxes and masks produced by the automatic detection and segmentation tools, since annotation errors would be inherited by every model score.","supporting_citations":[],"review_version":1}