{"id":"03724709-0292-4e3f-8138-75c55efee6f6","arxiv_id":"2611.16413","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"RICapBench is a benchmark of 245,466 annotated remote sensing instances testing vision-language models on instance localization, attribute description, and spatial relation captioning.","lead":"This paper introduces RICapBench, a dataset and benchmark that evaluates how well vision-language models can localize and describe individual objects and their spatial relations in remote sensing images. It finds that current models, including state-of-the-art ones, struggle with precise instance localization, especially in scenes with many small objects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Spatial relation parser accuracy is unquantified; RelCap-F1 and derived RICScore comparisons may reflect parser artifacts rather than model capability.","rationale":"The reader's weakest assumption exactly identifies the parser accuracy issue. This is indeed the most load-bearing concern because the paper's composite metric and spatial relation analyses depend on it. However, the paper's central claim about instance localization being a bottleneck is supported by DetA, which is less dependent on the parser. The dataset contribution and the instance localization findings are robust even if the parser has moderate errors. The paper could improve by quantifying parser accuracy, but the current lack of quantification does not invalidate the core contributions. Hence, the verdict should remain ACCEPT (UNCHANGED). The reader's moderate confidence is appropriate.","tokens_in":15160,"tokens_out":2067,"duration_ms":16809,"concrete_test":"Manually annotate spatial relation triples for a random subset of 200 evaluation images (covering all three data sources) with at least two independent annotators, then compute the parser's precision and recall against this ground truth. Report inter-annotator agreement (e.g., Cohen's kappa). If precision < 85% or recall < 80%, the parser is insufficient for reliable metric-based comparisons. If both exceed 90%, the concern is largely mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that existing VLMs struggle with fine-grained understanding rests partly on spatial relation captioning results measured by RelCap-F1, which relies entirely on the custom spatial relation parser (Section 3.2). The parser is built via two rounds of manual validation, but its accuracy is never quantitatively assessed. No precision, recall, or F1 of the parser against a held-out human-annotated set is reported. Without this, differences in RelCap-F1 across models (e.g., LLaVA-OneVision-2-8B highest at 30.76% vs. LocateAnything at 23.40%) could be driven by the parser’s sensitivity to linguistic variation or systematic biases in rule coverage, not by genuine differences in spatial relation understanding. The parser’s rules are tailored to the manually verified captions, which themselves may have unknown quality. Thus, the spatial relation component of the analysis, including the finding that 'strong instance localization does not necessarily coincide with accurate spatial relation understanding,' is not rigorously grounded. This does not undermine the instance localization bottleneck claim (supported by DetA), but it weakens the conclusions about spatial relation captioning, which is a significant dimension of the benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces RICapBench, a dataset and benchmark that unifies instance localization, attribute captioning, and spatial relation captioning for remote sensing imagery. The dataset contains 245,466 instance-level annotations (oriented bounding boxes paired with fine-grained captions) and 498,005 spatial relation tuples sourced from DIOR-R, CODrone, and FAIR1Mv2.0. The authors propose a custom spatial relation parser and a composite metric RICScore (geometric mean of DetA, CapA, and RelCap-F1). They evaluate 15 vision–language models, reporting that instance localization is the primary bottleneck and that even remote-sensing-fine-tuned models achieve low detection accuracy. The work aims to fill a gap in assessing VLMs' ability to localize and describe instances without target-specific text queries.","tokens_in":15503,"tokens_out":2309,"duration_ms":18683,"significance":"If the proposed parser and metrics are reliable, RICapBench fills a clear gap in remote sensing VLM evaluation by requiring models to autonomously localize instances and generate spatially grounded captions. The finding that instance localization is the dominant bottleneck is valuable for guiding model design, and the public availability of the dataset supports future work. However, the significance hinges on the validity of the spatial relation parser and the reliability of the ground-truth captions, both of which lack quantitative quality assessment.","major_comments":[{"comment":"The spatial relation parser is central to computing RelCap-F1 and RICScore, yet its accuracy is never quantitatively evaluated. The construction process (two rounds of manual validation with LLM assistance) is described, but no precision, recall, or F1 score of the parser against a held-out human-annotated set is reported. Without this, differences in RelCap-F1 across models (e.g., LLaVA-OneVision-2-8B at 30.76% vs. LocateAnything at 23.40%) may reflect parser artifacts (e.g., sensitivity to linguistic variation, rule coverage gaps) rather than genuine differences in spatial relation understanding. This undermines the spatial relation captioning conclusions, which are a significant dimension of the benchmark. The authors should report parser performance on a representative test set and discuss its impact on metric reliability.","section":"Section 3.2, Spatial Relationship Parser","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The manual verification of instance captions is described but not quantified. No inter-annotator agreement (e.g., Cohen's kappa or Fleiss' kappa) is reported for the human reviews that check visual grounding and spatial relation accuracy. While manual verification is standard, the reliability of the ground truth is unknown. Given that the dataset serves as a benchmark, the degree of annotator consistency should be documented to assure users of label quality. The authors should provide an estimate of agreement, even on a subset, to strengthen trust in the annotations.","section":"Section 3.2, Construction and Quality Control","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The 'Samples' column for RICapBench lists 245.5K, which corresponds to instance-level annotations, not image count (which is 26,854). The column heading is ambiguous; consider renaming it to 'Instance Annotations' or adding a note to clarify what is being counted, as other rows (e.g., DIOR-RSVG, 38.3K) likely refer to image-level samples.","section":"Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The last sentence of the abstract states that LocateAnything 'scores lower on spatial relation captioning than other task-fine-tuned open-source VLMs.' This is vague; specify which models are being compared (e.g., all other fine-tuned models, or the highest-scoring one) for clarity.","section":"Abstract and Section 5.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The term 'CIDEr-D(ˆcp, Sg)/10' is unusual because standard CIDEr-D is typically on a 0–10 scale; scaling by 1/10 is fine but should be briefly justified or referenced. Additionally, the composite CapA combines METEOR, CIDEr-D/10, and SPICE-OA, but the relative weighting is equally weighted; consider commenting on the sensitivity to this choice.","section":"Section 4.2, Eq. (2)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The heatmaps share a common color scale but the figure is grayscale in the PDF; ensure the final version uses color or distinguishable patterns to convey the scores clearly.","section":"Section 5.2, Figure 4 caption","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper builds a large, careful dataset (245K instance annotations, 498K spatial relation tuples) combining OBBs with fine-grained captions, and evaluates 15 VLMs in a no-text-cue setting. The central finding—that instance localization is the main bottleneck—is convincingly supported by DetA scores and holds regardless of the parser question. That part is useful and likely reproducible.\n\nThe soft spot is the spatial relation parser. The stress test is right: parser accuracy is never quantified. The paper describes two rounds of manual validation, which is more than many benchmarks do, but absence of precision/recall/F1 on a held-out set means we cannot judge whether RelCap-F1 gaps are real or artifacts of rule coverage. The authors acknowledge the parser's construction but do not add a limitation statement about unmeasured accuracy. This matters because the claim that strong localization does not imply good spatial understanding (LLaVA-OneVision vs. LocateAnything) relies on RelCap-F1. That gap is plausible—different model designs could trade off—but the metric is the only evidence and it is unvalidated.\n\nThe detection and attribute metrics (DetA, CapA) are on firmer ground. The composite RICScore is thus only as trustworthy as its weakest component. For a benchmark paper, this is a fixable issue: a small human-annotated test set of spatial relation triples plus parser F1 would settle it.\n\nWho this is for: anyone working on remote sensing VLMs or fine-grained vision-language evaluation. The dataset is the main asset; the evaluation protocol is a second asset with a fixable weakness.\n\nRecommendation: send to review. The core contribution (dataset, instance-localization bottleneck) is solid. The spatial relation component needs stronger evidence before the paper can claim that RelCap-F1 is a reliable metric. Ask the authors to validate the parser and report inter-annotator agreement for the manual caption verification.","headline":"Solid benchmark with a clear instance-localization result, but the spatial relation metric rests on an unvalidated parser.","tokens_in":15899,"tokens_out":922,"duration_ms":8797},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"RICapBench is a benchmark that unifies instance localization, attribute captioning, and spatial relation captioning in remote sensing without target-specific text prompts.","keywords":["remote sensing","vision-language models","fine-grained understanding","benchmark dataset","spatial relations","instance localization","attribute captioning","RelCap-F1"],"falsifier":"Have three independent experts manually annotate the spatial relation triples for a random subset of 500 captions. Compute the parser's precision, recall, and F1 against the majority vote of experts. If the parser F1 falls below 0.9, then RelCap-F1 is not a reliable measure of model spatial relation captioning ability, and the benchmark's conclusions about spatial understanding are suspect.","tokens_in":16701,"feed_emoji":"🛰️","tokens_out":4259,"duration_ms":33705,"temperature":0.7,"pith_summary":"The paper introduces RICapBench, a dataset and benchmark designed to evaluate vision-language models on fine-grained understanding of remote sensing instances without relying on target-specific textual cues. It contains 245,466 instance-level annotations with oriented bounding boxes and natural-language descriptions covering appearance, location, and spatial relations to surrounding objects. The authors propose a spatial relation parser and the RelCap-F1 metric to measure spatial relation captioning accuracy, and combine them with existing metrics into a composite RICScore. Evaluating 15 open-source and proprietary VLMs, they find instance localization to be the primary bottleneck, with the best model achieving only 44.5% detection accuracy. The work aims to drive progress in end-to-end instance-level visual understanding for remote sensing applications such as disaster monitoring and UAV reconnaissance.","feed_headline":"New benchmark isolates instance localization as VLM bottleneck in remote sensing","feed_subtitle":"RICapBench tests 15 models and finds that even the best fails to localize 55% of objects without textual cues.","key_machinery":"The central mechanism is the RICapBench dataset (26,854 images, 245,466 instance annotations, 498,005 spatial relation tuples) built from three source datasets (DIOR-R, CODrone, FAIR1Mv2.0) covering general, UAV, and dense small-object remote sensing scenarios. Each instance annotation includes an oriented bounding box, a category label, and a natural-language description covering visible attributes, image location, and spatial relations to surrounding objects. Evaluation uses a custom spatial relation parser that converts captions into structured triples (subject, predicate, object), and the RelCap-F1 metric computes the F1 score over parsed triples. The composite RICScore is the geometric mean of DetA (detection accuracy), CapA (captioning accuracy for attributes and object descriptions), and RelCap-F1.","core_discovery":"The paper claims that existing general-purpose and remote-sensing-specific vision-language models (VLMs) struggle with fine-grained instance-level understanding in remote sensing imagery, particularly in autonomously localizing instances without text queries. Using the new RICapBench benchmark, the authors show that instance localization (measured by DetA) is the dominant weakness across all 15 tested models, with scores below 25% for most models and below 6% for remote-sensing fine-tuned models. They further discover a dissociation between localization and spatial relation captioning: LocateAnything achieves the highest DetA (44.50%) but low RelCap-F1 (23.40%), while LLaVA-OneVision-2-8B achieves the highest RelCap-F1 (30.76%) but low DetA (9.16%). This suggests that both capabilities need separate improvement.","pith_inferences":["The spatial relation parser's accuracy is not quantified against human annotation; if the parser introduces systematic errors, the RelCap-F1 and RICScore metrics may misrepresent model capability. Reporting parser precision/recall on a held-out set would strengthen trust in the benchmark.","Because RelCap-F1 is computed only on successfully localized instances, a model with high localization recall (like LocateAnything) may have its spatial captioning unfairly penalized on many easy instances, while a model with low recall (like GeoChat) can appear better on RelCap-F1 by focusing only on a tiny subset. This compositional effect could mislead comparisons if not interpreted carefully.","The benchmark may inspire a new family of tasks that require an agent to 'find, describe, and relate' instances jointly, moving beyond isolated VQA or grounding tasks. This could lead to more holistic evaluation of VLMs for environmental monitoring."],"forward_implications":["Model developers can use RICapBench as a standardized test to drive progress in joint instance localization and fine-grained description, a capability needed for autonomous geospatial analysis.","The identified bottleneck implies that improving visual encoders to retain small-object details and spatial layout during compression is a concrete research direction.","Applications like UAV disaster inspection and remote sensing emergency monitoring, where users cannot provide target-specific text cues, could directly benefit from models that score well on RICapBench.","The dissociation between localization and spatial relation captioning suggests that future work should design training pipelines that explicitly optimize both objectives, rather than relying on a single strong capability."],"supporting_citations":[{"why":"Provides the DIOR-R dataset (general remote sensing) as one of the three image and annotation sources for RICapBench.","marker":"[49]"},{"why":"Provides the CODrone dataset (UAV remote sensing) as a source of UAV imagery and oriented bounding boxes.","marker":"[50]"},{"why":"Provides the FAIR1Mv2.0 dataset (dense small-object remote sensing) as a source for dense small-object scenarios.","marker":"[51]"},{"why":"Provides the CHOTA evaluation framework that defines detection accuracy (DetA) and the composite metric structure used by RICScore.","marker":"[46]"},{"why":"Provides the SPICE metric whose relation parsing and structure are adapted but found insufficient, motivating the new spatial relation parser.","marker":"[23]"},{"why":"Provides the METEOR metric used in computing CapA (captioning accuracy for attributes and object descriptions).","marker":"[52]"},{"why":"Provides the COCO criteria for categorizing object sizes, used in dataset statistics.","marker":"[47]"}],"fun_headline_variants":["RICapBench: instance localization is the VLM bottleneck","Top VLM still fails to localize 55% of remote objects","New benchmark isolates localization as VLM weakness","VLMs miss 55% of objects without textual cues in RS","LocateAnything leads in detection but lags in spatial captioning"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The spatial relation parser correctly and consistently converts natural language captions into structured spatial relation triples; the paper does not report the parser's agreement with human annotations, so any systematic parsing error would propagate directly into RelCap-F1 and RICScore, undermining the validity of the composite evaluation.","fun_headline_variants_meta":{"raw":{"variants":["RICapBench: instance localization is the VLM bottleneck","Top VLM still fails to localize 55% of remote objects","New benchmark isolates localization as VLM weakness","VLMs miss 55% of objects without textual cues in RS","LocateAnything leads in detection but lags in spatial captioning"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"cost":0.000257068,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000257068,"upstream_inference_prompt_cost":0.000113988,"upstream_inference_completions_cost":0.00014308},"total_tokens":1530,"prompt_tokens":1019,"completion_tokens":511,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":425}},"tokens_in":1019,"tokens_out":511,"cost_source":"openrouter_usage","duration_ms":4554,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":425,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T15:11:20.195851+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Have three independent experts manually annotate the spatial relation triples for a random subset of 500 captions. Compute the parser's precision, recall, and F1 against the majority vote of experts. If the parser F1 falls below 0.9, then RelCap-F1 is not a reliable measure of model spatial relation captioning ability, and the benchmark's conclusions about spatial understanding are suspect.","supporting_citations":[{"cited_title":"Anchor-Free Oriented Proposal Generator for Object Detection","cited_arxiv_id":null,"evidence_quote":"Provides the DIOR-R dataset (general remote sensing) as one of the three image and annotation sources for RICapBench."},{"cited_title":"More Clear, More Flexible, More Precise: A Comprehensive Oriented Object Detection benchmark for UAV","cited_arxiv_id":"2504.20032","evidence_quote":"Provides the CODrone dataset (UAV remote sensing) as a source of UAV imagery and oriented bounding boxes."},{"cited_title":"FAIR1M: A benchmark dataset for fine-grained object recognition in high-resolution remote sensing imagery","cited_arxiv_id":null,"evidence_quote":"Provides the FAIR1Mv2.0 dataset (dense small-object remote sensing) as a source for dense small-object scenarios."},{"cited_title":"Dense Video Object Captioning from Disjoint Supervision","cited_arxiv_id":null,"evidence_quote":"Provides the CHOTA evaluation framework that defines detection accuracy (DetA) and the composite metric structure used by RICScore."},{"cited_title":"SPICE: Semantic Propositional Image Caption Evaluation","cited_arxiv_id":null,"evidence_quote":"Provides the SPICE metric whose relation parsing and structure are adapted but found insufficient, motivating the new spatial relation parser."},{"cited_title":"METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments","cited_arxiv_id":null,"evidence_quote":"Provides the METEOR metric used in computing CapA (captioning accuracy for attributes and object descriptions)."},{"cited_title":"Microsoft COCO: Common Objects in Context","cited_arxiv_id":null,"evidence_quote":"Provides the COCO criteria for categorizing object sizes, used in dataset statistics."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}