{"id":"7343ab42-2e64-4ce5-b571-235c0da73bc4","arxiv_id":"2608.00877","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Filtering retrieved maps to image-unverifiable attributes preserves most of the RAG accuracy gain and removes the conflict failure mode in three remote-sensing MLLMs.","lead":"This paper tests a simple rule for giving maps to satellite-image AI models: include only facts the image cannot verify, like a facility's function, and leave out facts the image can check, like roads and runways. Across three open models, this filtering keeps most of the accuracy gain of full map retrieval while making answers robust to wrong map records.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline robustness and retained-gain claims rest on a key-level verifiability map whose boundary-error prevalence is unmeasured; the paper's own exceptions and the missing filtered BigEarthNet condition leave this load-bearing assumption untested.","rationale":"The reader correctly identified the key-level verifiability map as the weakest assumption. My stress-test agrees and sharpens the concern: the map's boundary errors are documented but their prevalence is not quantified, and the external benchmark omits the very condition (rag-filtered) needed to bound the map's behavior outside fMoW. This does not invalidate the fMoW results, which are well-controlled and internally consistent; it limits the generality of the central claim that verifiability-guided content selection is a broadly effective mechanism. The existing conditional verdict is therefore appropriate, and the missing filtered BigEarthNet run plus a map-sensitivity analysis would settle whether the caveat is benign or consequential.","tokens_in":15394,"tokens_out":17686,"duration_ms":167298,"concrete_test":"Run the complete GeoArbiter pipeline (rag-filtered) on the 2,000-image BigEarthNet land-cover QA benchmark and report the retained gain relative to full injection, alongside the conflict-probe balanced accuracy if feasible. If the filter discards the landuse/natural keys that drive the land-cover gain and retained gain drops below roughly 50% or becomes negative, the central mechanism is task-specific rather than a general grounding principle. As a complementary check, perform a map-sensitivity analysis on fMoW: move the documented boundary cases (amenity=parking to v=1; functional tags such as hangar or office under v=1 keys to v=0) and recompute Table 1; if headline retained-gain and conflict-cell figures move beyond the reported bootstrap intervals, the conclusions are an artifact of the exact key cut.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is the binary key-level verifiability map of §3.3/Appendix D: all records under amenity/shop/tourism/military/office are injected, all under building/highway/aeroway/natural/landuse are withheld. The paper itself documents two classes of boundary error: visible amenity=parking survives and fools models in 83–92% of parking conflict cells, and functional evidence expressed through structural keys (hangars under aeroway, offices under building) is discarded, causing 12.85–15.31% of the retained-gain loss (§6.4). These are value-level exceptions to a primary-key map, and the paper provides no prevalence estimate for such exceptions in other OSM tagging cultures, regions, or scales. The external BigEarthNet evaluation (Table F.4) reports only full injection; the filtered condition is absent, so we cannot assess whether GeoArbiter retains any of the 4.10–7.95-point gain on a task where the useful keys (landuse, natural) are exactly the ones withheld. If boundary-key prevalence is higher in other distributions, the aggregate benefit—85% of QA gain retained plus conflict robustness—could reverse. This makes the map's approximation the load-bearing assumption of the central claim, and it is currently supported only by fMoW-specific aggregate results, not by a sensitivity analysis or an external filtered-condition run.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes that cross-modal verifiability—whether the RS image can in principle adjudicate a retrieved attribute—should govern which OSM records are exposed to a frozen remote-sensing MLLM. GeoArbiter injects only image-unverifiable records (amenity, shop, tourism, military, office, etc.) and withholds image-verifiable physical-structure keys (building, highway, aeroway, natural, landuse). On the full fMoW functional validation split, full OSM injection raises four-way land-use QA accuracy by 12.06–17.19 points across three open MLLMs; stratified injection retains 84.69–87.15% of that gain while remaining robust under fabricated-record conflicts (90.50–93.25% vs. 36.75–72.00% for full injection). The paper further reports that six instruction-level arbitration prompts fail to enforce the policy, that budget-matched random/frequency/inverted filters retain less accuracy, and that claim-level hallucination drops under both a standard and a source-blinded judge.","tokens_in":15680,"tokens_out":6800,"duration_ms":57204,"significance":"The paper has several strengths: full-scale evaluation (n=28,087 per condition), paired bootstrap with location clustering, budget-matched controls, modality-isolation ablations, a controlled existence probe with human-verified ground truth, human adjudication of the judge, and released code and caches for reproducibility. The main evaluation uses external labels (fMoW functional labels, BigEarthNet land cover) that are independent of the injected OSM knowledge, and the paper explicitly avoids OSM-derived QA. If the results hold, verifiability-guided content selection is a simple, model-agnostic, training-free mechanism for grounding RS MLLMs in fallible structured knowledge, with a clear argument for why content filtering outperforms instruction-level arbitration. The direction of selection, not the reduced context, is supported by the budget-matched controls. However, the operationalization of verifiability as a binary key-level map is load-bearing and currently tested only on fMoW aggregate results.","major_comments":[{"comment":"The central claim that verifiability is the mechanism rests on the binary key-level map: all amenity/shop/tourism/military/office records are injected and all building/highway/aeroway/natural/landuse records are withheld. The paper itself documents two classes of boundary errors: visible amenity=parking survives the filter and fools models in 83–92% of parking conflict cells (§5.5), and functional evidence expressed under structural keys (hangars under aeroway, offices under building) is discarded, accounting for 12.85–15.31% of the retained-gain loss (§6.4). No prevalence estimate for such boundary cases is provided, so we cannot assess whether the aggregate benefit would persist in other OSM tagging cultures, regions, or at different scales. I request a sensitivity analysis that varies the map (e.g., moving amenity=parking to v=1, or adding building=office/industrial subtypes to v=0) and re-measures retained QA gain and conflict robustness, or at least a quantification of boundary-key prevalence in the fMoW corpus.","section":"§6.4 / Appendix D"},{"comment":"The external BigEarthNet evaluation is reported only for the full-injection condition; the rag-filtered condition is absent. On land-cover QA, the discriminative keys (landuse, natural) are exactly the keys withheld by GeoArbiter, so this missing condition is the direct test of whether verifiability-guided selection generalizes beyond fMoW or simply disappears when the useful evidence is filtered out. Without it, Eq. (1) is not actually tested on the external benchmark, and the retained-gain claim may be corpus-specific. Please report the rag-filtered BigEarthNet accuracy (and if feasible the budget-matched controls) for all three models.","section":"§5.8 / Table F.4"}],"minor_comments":[{"comment":"The caption states n=1,069 for the existence probe, but the three columns (conf, abs, pres) likely have different denominators; please report per-cell n or clarify the shared n.","section":"§5.5 / Table 2"},{"comment":"The abstract reports the source-blinded hallucination reduction (9.58–26.34%), while §5.2 first presents the standard-judge values (19.28–45.19%) and only later points to the blinded figures. Consider reporting both consistently at the first occurrence.","section":"§5.2 / Appendix I"},{"comment":"No confidence intervals are given for the BigEarthNet results; adding paired bootstrap CIs, as in Table 1, would aid comparison with the fMoW results.","section":"Table F.4"},{"comment":"The legend entry 'prior retrieval baselines' is vague; the caption or legend should list RS-RAG, GeoNames, and Wikipedia GeoSearch explicitly.","section":"Figure 1"},{"comment":"The automatic judge is a frozen Qwen2.5-7B-Instruct whose agreement with adjudicated human labels is moderate (κ=0.61); the paper discloses this as a limitation, but explicitly noting that the same judge is used across all conditions mitigates bias in paired comparisons would be useful.","section":"§4"},{"comment":"The reproducibility table lists 'all unparsable counts wrong' for answer parsing; the frequency of unparsable answers is not reported. Please report the parse rate.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The empirical work is extensive and the release of code and caches is commendable. My main reservation is that the two missing pieces—a sensitivity analysis of the key-level verifiability map and a filtered-condition run on BigEarthNet—are exactly what is needed to support the general claim that verifiability, rather than corpus-specific key groupings, is the mechanism. I believe these are feasible additions rather than fundamental flaws. The instruction-level arbitration contrast is convincing when viewed through the full-probe balanced accuracy, though the QA pilot sample is small. Overall this is a solid paper with a clearly articulated hypothesis, but the load-bearing approximation needs direct testing before the central claim is fully established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you work on retrieval-augmented multimodal models. The core idea is simple: when injecting OSM knowledge, expose only facts the image can't verify. The paper shows this content-level filter keeps about 85% of the QA gain while making models robust to fabricated records, and that prompt-level arbitration fails. The evaluation is genuinely careful: full fMoW validation split, paired bootstrap, budget-matched controls that separate selection from context size, modality isolation, a source-blinded judge, and human adjudication. The paper also discloses its weak spots honestly.\n\nThe soft spots are real but not fatal. The key-level verifiability map is a binary approximation, and the paper itself documents two boundary failures: visible amenity=parking fools models in most conflict cells, and functional evidence under structural keys gets discarded, causing most of the retained-gain loss. There is no prevalence estimate for such boundary errors in other regions or tagging cultures, and the external BigEarthNet evaluation only reports full injection, so we can't verify that the filter preserves any gain on a task where the useful keys are the withheld ones. These are fixable in revision, and they don't overturn the central finding: on fMoW, selection by verifiability beats random and inverted selection at matched budgets, and the controlled probe isolates the mechanism.\n\nOne reproducibility gripe: the claimed code release has no URL. That should be fixed. The hallucination judge's moderate agreement is disclosed, and the authors sensibly report source-blinded numbers.\n\nThis paper deserves peer review. It's a solid, defensible empirical contribution. I'd cite it if writing about inference-time grounding for VLMs, and it's a good reading-group example of honest evaluation.","headline":"A well-controlled empirical paper showing that content-level filtering by cross-modal verifiability is a simple, effective fix for grounding RS MLLMs; the key-level map is the main soft spot.","tokens_in":16170,"tokens_out":2910,"would_cite":true,"duration_ms":26465,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Withholding image-verifiable geographic records from remote-sensing language models keeps 85% of retrieval's accuracy gain while making answers robust to wrong database entries.","keywords":["remote sensing","multimodal large language model","retrieval-augmented generation","cross-modal verifiability","OpenStreetMap","hallucination","grounding","conflict robustness"],"falsifier":"On a new or held-out region where image-verifiable attributes are predominantly tagged with keys the filter injects (e.g., amenity=parking, building=office), run the 1,069-item existence probe; if conflict-cell accuracy falls to the full-injection range (36.75–72.00%) rather than the 90.50–93.25% reported here, the key-level verifiability map is mis-calibrated and the central claim would not hold there.","tokens_in":15185,"feed_emoji":"🛰️","tokens_out":11515,"duration_ms":86188,"temperature":0.7,"pith_summary":"The paper claims that the correct way to ground a remote-sensing multimodal language model in geographic data is to give it only the facts the image cannot itself verify. Coordinate-keyed retrieval from OpenStreetMap improves four-way land-use accuracy by 12–17 points across three open models, but when a retrieved record contradicts what is plainly visible, the models side with the record and flip to a wrong answer. GeoArbiter applies a deterministic tag-level filter that removes physical-structure records (buildings, roads, runways, land cover) and keeps function and identity records (amenities, shops, offices, names). This filtering retains 84.69–87.15% of the retrieval gain, reduces hallucinations even when the judge cannot see the injected records, and is the only condition that stays robust when a fabricated record is injected. The mechanism is the content selection itself, not instruction-level prompting.","feed_headline":"Withhold image-verifiable records, keep 85% of retrieval gain","feed_subtitle":"A training-free filter injects only unverifiable facts, so fabricated records no longer flip answers.","key_machinery":"The central object is the key-level verifiability map, which assigns each OpenStreetMap primary tag key a binary label according to whether the image could in principle adjudicate that attribute. Keys encoding function or identity (amenity, shop, tourism, military, office, name, and related keys) are labeled image-unverifiable and injected; keys encoding visible physical structure (building, highway, aeroway, natural, landuse) are labeled image-verifiable and withheld. Stratified injection is the deterministic subset $S^- = \\{f \\in R : v(\\kappa(f)) = 0\\}$, which keeps about 11% of retrieved features. This map converts an arbitration problem into a content-filtering problem: it requires no detector, no extra model call, and no per-image classifier, and it leaves the frozen model to answer from the image and the filtered text.","core_discovery":"Cross-modal verifiability—whether the image can in principle confirm or refute an attribute—is the criterion that separates useful geographic retrieval from harmful retrieval. On the 28,087-image fMoW functional validation split, injecting coordinate-matched OpenStreetMap records raises land-use QA accuracy by 12.06–17.19 points across three frozen open models, but fabricated image-verifiable records (for example, a planted helipad) cause losses of 13.50–51.00 points; none of six arbitration prompts recovers the image-only baseline. GeoArbiter injects only the image-unverifiable subset of records and retains 84.69–87.15% of the full-retrieval accuracy gain, cuts claim-level hallucination by 9.58–26.34% under a source-blinded judge, and exceeds the no-knowledge conflict-cell accuracy on all three models (90.50–93.25% versus 36.75–72.00% for full injection). Robustness comes from withholding, not from reasoning: at the same record budget, keeping the image-verifiable records instead collapses conflict accuracy to 20.5–52.8%, below full injection.","pith_inferences":["A natural refinement the paper leaves untested: an attribute-level filter over tag values (e.g., keep building=office, drop amenity=parking) could recover the 12.85–15.31% of lost QA gain while maintaining conflict robustness, since the paper identifies exactly those boundary cases as the source of the loss.","Verifiability-guided content selection may transfer to other image-plus-knowledge settings, such as medical imaging or disaster response, where a database record can claim what the image cannot show; the paper only demonstrates remote-sensing benchmarks, so this is an extension.","The source-blinded hallucination result (9.58–26.34% reduction) suggests the filter's main safety benefit is conflict robustness rather than hallucination reduction per se; combining content filtering with retrieval-quality estimation might handle naturally occurring wrong records, something the paper does not test.","Because the filter is deterministic and training-free, it could be applied as a safety layer on top of any deployed remote-sensing multimodal model without retraining, adding only 4.9 ms cache assembly and 413 prompt tokens per image."],"forward_implications":["Coordinate-keyed structured retrieval is a training-free way to supply the knowledge a remote-sensing multimodal language model lacks, adding 12.06–17.19 accuracy points on fMoW land-use QA.","The decision to expose a retrieved record should be based on whether the image can verify the disputed attribute, not on which source is usually reliable.","Instruction-level arbitration cannot replace content-level filtering: all six tested prompts leaked across attribute types and biased yes/no answers, so the policy must be compiled into the prompt content.","Full-injection retrieval can look best on clean accuracy while collapsing under fabricated records, so robustness evaluations must include conflict conditions as well as clean ones."],"supporting_citations":[{"why":"Supplies the fMoW functional validation split (28,087 images with coordinates) on which all main accuracy numbers are measured.","marker":"(Christie et al., 2018)"},{"why":"Defines Qwen2.5-VL-7B, one of the three frozen multimodal language models the method is evaluated on.","marker":"(Qwen Team, 2025)"},{"why":"Defines InternVL3-8B, the second frozen multimodal language model tested.","marker":"(Zhu et al., 2025)"},{"why":"Defines LLaVA-OneVision-7B, the third frozen multimodal language model tested.","marker":"(Li et al., 2024)"},{"why":"Introduces OpenStreetMap, the coordinate-keyed knowledge source that GeoArbiter filters.","marker":"(Haklay and Weber, 2008)"},{"why":"Provides RS-RAG, the visual-similarity retrieval baseline reproduced and outperformed by coordinate-keyed retrieval.","marker":"(Wen et al., 2025)"},{"why":"Supplies BigEarthNet, used for the external land-cover QA validity test.","marker":"(Sumbul et al., 2019)"},{"why":"Supplies ESA WorldCover, used as an alternative knowledge source and in the source ablations.","marker":"(Zanaga et al., 2022)"}],"fun_headline_variants":["Inject only unverifiable facts to keep 85% of retrieval gain","GeoArbiter: verifiability-guided grounding keeps 85% gain","Cross-modal verifiability: filter records, keep 85% gain","Withhold image-decisive records, retain 85% of retrieval gain","GeoArbiter: keep 85% gain, cut hallucination by 26%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The binary key-level verifiability map—every key either fully image-verifiable or fully unverifiable—must correctly approximate the attribute-level boundary; the paper's own results show boundary cases such as a visibly present parking lot tagged as an amenity survive the filter and fool models in 83–92% of conflict cells, so if such cases dominate in other regions the aggregate benefit could reverse.","fun_headline_variants_meta":{"raw":{"variants":["Inject only unverifiable facts to keep 85% of retrieval gain","GeoArbiter: verifiability-guided grounding keeps 85% gain","Cross-modal verifiability: filter records, keep 85% gain","Withhold image-decisive records, retain 85% of retrieval gain","GeoArbiter: keep 85% gain, cut hallucination by 26%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00094,"raw_usage":{"total_tokens":4058,"prompt_tokens":1025,"completion_tokens":3033,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":2931}},"tokens_in":641,"tokens_out":3033,"duration_ms":20436,"temperature":1.0,"reasoning_tokens":2931,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:16:05.931213+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a new or held-out region where image-verifiable attributes are predominantly tagged with keys the filter injects (e.g., amenity=parking, building=office), run the 1,069-item existence probe; if conflict-cell accuracy falls to the full-injection range (36.75–72.00%) rather than the 90.50–93.25% reported here, the key-level verifiability map is mis-calibrated and the central claim would not hold there.","supporting_citations":[],"review_version":2}