{"paper":{"title":"RICapBench: A Fine-Grained Visual Understanding Benchmark Dataset for Vision-Language Models in Remote Sensing","arxiv_id":"2611.16413","primary_cat":"cs.CV","submitted_at":"2026-11-12T07:24:49+00:00","version_compiled":"v1"},"notice":"Every quoted sentence is the authors' text from the arXiv listing abstract. Labels and probabilities are the outputs of classifier-1.13.0 on the question bank abstract-p1-20260923, exactly as returned. No sentence in this record was written by a language model.","schema":"pith.compiled-record/v0","status":"compiled","abstract":{"source":"arXiv listing abstract v1","sentences":[{"i":0,"p":1.0,"id":"s0","own":false,"role":"background","text":"Advances in vision-language models (VLMs) have improved remote sensing image analysis and supported geospatial decision-making.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.04,"main_claim_p":0.03,"quantitative":false,"quantitative_p":0.03,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":1.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":1,"p":0.58,"id":"s1","own":false,"role":"gap","text":"However, fine-grained understanding of remote sensing imagery remains challenging for VLMs.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.08,"main_claim_p":0.04,"quantitative":false,"quantitative_p":0.05,"role_probabilities":{"gap":0.58,"method":0.0,"result":0.0,"background":0.42,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":2,"p":0.86,"id":"s2","own":false,"role":"background","text":"Large variations in object scale, complex geospatial relations among instances, and limited visual detail hinder precise instance localization, visible attribute identification, and spatial relation understanding, potentially leading to descriptions that lack visual grounding.","hedge":{"p":0.69,"label":"hedged_possibility"},"own_p":0.07,"main_claim_p":0.04,"quantitative":false,"quantitative_p":0.04,"role_probabilities":{"gap":0.14,"method":0.0,"result":0.0,"background":0.86,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":3,"p":0.99,"id":"s3","own":false,"role":"gap","text":"Existing datasets and benchmarks insufficiently assess end-to-end instance localization and captioning without target-specific textual cues.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.06,"main_claim_p":0.04,"quantitative":false,"quantitative_p":0.06,"role_probabilities":{"gap":0.99,"method":0.0,"result":0.0,"background":0.01,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":4,"p":1.0,"id":"s4","own":true,"role":"contribution","text":"To support this evaluation, we introduce RICapBench, a dataset and benchmark that unifies instance localization and fine-grained captioning in remote sensing.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.94,"main_claim_p":0.08,"quantitative":false,"quantitative_p":0.04,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":1.0,"interpretation":0.0}},{"i":5,"p":0.67,"id":"s5","own":true,"role":"contribution","text":"RICapBench contains 245,466 instance-level annotations and 498,005 spatial relation tuples spanning general remote sensing, UAV remote sensing, and dense small-object scenarios.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.86,"main_claim_p":0.04,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.33,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.67,"interpretation":0.0}},{"i":6,"p":0.75,"id":"s6","own":true,"role":"method","text":"Each annotation pairs an object's oriented bounding box with a natural language description of its visible attributes and spatial relations to surrounding instances and environmental elements.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.79,"main_claim_p":0.03,"quantitative":false,"quantitative_p":0.05,"role_probabilities":{"gap":0.0,"method":0.75,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.25,"interpretation":0.0}},{"i":7,"p":0.83,"id":"s7","own":true,"role":"method","text":"To evaluate the accuracy and completeness of spatial relation captioning, we propose a remote sensing spatial relation parser and the RelCap-F1 metric.","hedge":{"p":0.99,"label":"asserted"},"own_p":0.96,"main_claim_p":0.04,"quantitative":false,"quantitative_p":0.1,"role_probabilities":{"gap":0.0,"method":0.83,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.17,"interpretation":0.0}},{"i":8,"p":0.8,"id":"s8","own":true,"role":"method","text":"We combine RelCap-F1 with instance localization and attribute captioning metrics into a composite metric, RICScore.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.96,"main_claim_p":0.05,"quantitative":false,"quantitative_p":0.07,"role_probabilities":{"gap":0.0,"method":0.8,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.2,"interpretation":0.0}},{"i":9,"p":0.99,"id":"s9","own":true,"role":"result","text":"Our evaluation of 15 advanced open-source and closed-source VLMs identifies instance localization as a major bottleneck for these models.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.97,"main_claim_p":0.44,"quantitative":true,"quantitative_p":0.8,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.99,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.01}},{"i":10,"p":1.0,"id":"s10","own":true,"role":"result","text":"LocateAnything achieves the highest RICScore of 31.68, but scores lower on spatial relation captioning than other task-fine-tuned open-source VLMs.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.92,"main_claim_p":0.17,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":11,"p":0.82,"id":"s11","own":false,"role":"contribution","text":"Code and dataset are available at https://github.com/Xldln/RICapBench.","hedge":{"p":0.98,"label":"asserted"},"own_p":0.41,"main_claim_p":0.02,"quantitative":false,"quantitative_p":0.04,"role_probabilities":{"gap":0.0,"method":0.01,"result":0.01,"background":0.01,"limitation":0.0,"future_work":0.15,"contribution":0.82,"interpretation":0.0}}]},"compiled":{"main_claim":{"p":0.99,"text":"Our evaluation of 15 advanced open-source and closed-source VLMs identifies instance localization as a major bottleneck for these models.","location":"abstract s9","hedge_strength":{"p":1.0,"label":"asserted"}},"provenance":{"route":{"via":"TypeSafe API","model":"classifier-1.13.0","endpoint":"https://api.typesafe.ai/v1/systemone","provider":"TypeSafe"},"packing":"one request per paper","requests":1,"input_tokens":8904,"cost_usd_reported":0.0},"compiled_at":"2026-11-13","bank_version":"abstract-p1-20260923","model_version":"classifier-1.13.0","abstract_states":{"claims_first":0.1,"states_limitation":0.25,"human_participants":0.07,"controlled_comparison":0.18,"randomized_assignment":0.01,"states_quantitative_result":0.98,"compares_with_prior_results":0.87,"mentions_code_or_data_availability":0.99},"evidential_basis":{"p":0.98,"value":"observation or experiment","probabilities":{"argument":0.0,"literature":0.0,"proof_or_derivation":0.0,"analysis_of_existing_data":0.01,"observation_or_experiment":0.98,"simulation_or_computation":0.01}},"contribution_type":{"p":1.0,"value":"dataset or benchmark","probabilities":{"review":0.0,"method_or_tool":0.0,"empirical_study":0.0,"simulation_study":0.0,"theoretical_result":0.0,"dataset_or_benchmark":1.0,"position_or_commentary":0.0}},"contribution_statement":{"p":1.0,"text":"To support this evaluation, we introduce RICapBench, a dataset and benchmark that unifies instance localization and fine-grained captioning in remote sensing.","location":"abstract s4"}},"coverage":{"tier":"abstract","levels_run":["P abstract"],"not_examined":["full text","figures and tables","references and citing sentences","relations to other papers"],"sources_read":["arXiv listing abstract v1"],"levels_not_run":["S sections","T body sentences","C citations","R cross-paper relations"],"sentences_examined":12,"sentences_in_abstract":12},"measured_accuracy":{"url":"https://pith.science/accuracy","measured_on":"2026-10-02","method":"two independent model families labelled a fresh random sample blind; the figure is how often the published label matches their consensus","sentence_role_agreement":0.92,"main_claim_agreement":0.908,"typed_citation_agreement":0.933,"typed_citation_agreement_95_interval":[0.917,0.945],"review_comments_accurate":0.758,"review_comments_accurate_95_interval":[0.684,0.82],"review_caught_admitted_error_found_or_related":0.448,"review_comments_judged":{"judged_by":["gpt-5.6-luna","grok-4.7"],"judged_against":"Pith's own text copy of each paper: text extracted from the PDF, the first 90,000 characters","measured_on":"2026-10-04","accurate":113,"decided":149,"sampled_reviews":200,"split_by_lane":false,"by_lane":"https://pith.science/accuracy#by-lane"}}}