{"id":"d895a21e-2b5f-4912-a67e-fcf75f54204b","arxiv_id":"2608.03882","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Even with gold geographic facts and computation tools, LLMs cap below two-thirds exact match on MultiGlobeQA, with grid indexing and shape computation as the weakest families.","lead":"MultiGlobeQA is a new benchmark with 46,060 geospatial questions in 17 languages, built by executing queries over three knowledge graphs with worldwide income and density stratification. It shows that current language models fail at spatial computation even when given the right facts and tools, with grid indexing and shape questions as the hardest families.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Oracle 'gold triples' omit closed-world/negative facts needed for exhaustive set, count, and nearest-entity questions, so the 61.6 EM ceiling may understate true computational ability.","rationale":"The paper is carefully built: execution-based ground truth, released code and KG snapshots, transparent appendices, and per-family analyses. I take the central claim to be that with perfect retrieval (oracle), accuracy plateaus below two-thirds because models cannot perform the required spatial computation. The weakest link is the oracle's semantics. The limitation statement explicitly concedes that set/count/nearest answers are only as complete as the KG snapshot, but the oracle prompts (Figs. 29/33) never tell the model that the injected facts are exhaustive. On such questions, an oracle is not equivalent to perfect retrieval: the model cannot distinguish 'this is the nearest entity' from 'this is one candidate.' Because refusals are scored incorrect, models that demand closure are penalized, lowering EM. This is a genuine confound for the headline number, even though per-family grid/shape results independently support the computation-bottleneck story. The reader's weakest_assumption already points to completeness of gold triples; I mark partial agreement because the reader routes it mainly to KG corruption/ground-truth drift and the income-gap overclaim, whereas the load-bearing issue for the central claim is the missing closed-world assertion in the oracle prompt itself. The proposed test—adding an explicit exhaustiveness sentence—is cheap and decisive. If it moves the affected subset materially, the paper should either restrict the headline to computation-only families or acknowledge that the oracle includes an implicit closed-world assumption. If it does not, the current wording stands. The reader's CONDITIONAL verdict remains appropriate; no change is needed.","tokens_in":37672,"tokens_out":8662,"duration_ms":84489,"concrete_test":"Run the T3-o and T1-o evaluations on the small split with an added closure assertion in the oracle prompt, e.g., 'The following triples are exhaustive: no other entity in the knowledge graph satisfies the conditions of the question.' Compare EM/coverage/false-refusal on the set-enumeration, count, and nearest/optimization sub-templates, and overall. If EM on this subset rises by more than about 10 points, or overall EM crosses the two-thirds threshold, then missing closed-world information explains a material share of the 'computation bottleneck.' If EM moves within seed noise (roughly ±1 point), the concern is refuted. As a secondary check, re-run on templates that do not require exhaustiveness (distance, direction, coordinates, geohash, topology) to confirm the plateau is genuinely computational.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'computation, not access to knowledge, is the bottleneck' rests on the oracle tiers (T1-o, T3-o) in §4, which inject 'the gold KG triples supporting each answer.' These are positive facts only. For questions whose gold answer is defined by an exhaustive search—set enumerations (759 questions in the small split), counts (289), and nearest/optimization templates under SFIDs A/F/G—correctness requires knowing that no unlisted entity satisfies the query. The oracle prompts (Figs. 29/33) assert 'all the information you need is already provided' but never state a closure condition (e.g., that the listed candidates are exhaustive). Models that recognize this and abstain are scored wrong because refusals are counted incorrect (§4). The paper's own Limitations admit 'incompleteness affects questions whose answer ranges over a set of entities, such as counts, set enumerations, and nearest-entity questions,' yet the oracle design does not neutralize this. Consequently the headline result—'given gold triples and compute tools, no model exceeds 61.6 EM'—conflates computational failure with under-specification: for roughly 20% of the small-split true-premise questions (set/count plus nearest/optimization templates), the oracle is not actually perfect retrieval. Per-family results for grid indexing (13 EM) and shape (25 EM) are likely unaffected, so the qualitative conclusion may survive, but the quantitative 'plateaus below two thirds' wording is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MultiGlobeQA, a multilingual benchmark of 46,060 geospatial question-answer pairs derived from three knowledge graphs via 65 templates and 14 spatial-function families, with execution-based ground truth, stratified sampling over income and density tiers, 17 language versions, and false-premise and multimodal slices. The authors evaluate four LLMs under parametric (T1), reasoning (T2), agentic retrieval (T3a/T3b/T3c), and oracle conditions with injected gold triples (T1-o/T3-o). The central claims are that retrieval and tools help substantially but that performance plateaus below two thirds even under gold facts, indicating a computation bottleneck, and that model accuracy is worse for low-income regions, a gap that gold facts widen. The construction is careful in many respects, but the headline oracle and income-gap results are currently entangled with confounds that the manuscript itself acknowledges.","tokens_in":38033,"tokens_out":5927,"duration_ms":61413,"significance":"If the central claim is established, the paper makes a timely and important contribution: it provides a large, multilingual, execution-grounded benchmark and shows that current LLMs cannot reliably perform geospatial computation even under near-perfect retrieval with code execution. The benchmark construction has notable strengths: execution-verified ground truth, human post-edited translations with inter-annotator agreement reported, released gold triples for auditability, per-seed variance, and honest reporting of non-replications such as the density-tier result. The per-family findings—especially the collapse on grid indexing and shape computation—are likely robust and valuable. However, the headline quantitative ceiling ('no model exceeds 61.6 EM', 'plateaus below two thirds') and the income-gap claim are not cleanly supported as stated, because the oracle is not closed-world for exhaustive-answer templates, and the income comparison is confounded by triple-count imbalance. These are fixable with additional analysis or rephrasing, so the paper merits major revision rather than rejection.","major_comments":[{"comment":"The oracle tiers inject only positive gold triples that 'support' each answer. For set-enumeration (759 items in the small split), count (289), and nearest/optimization/ranking templates under SFIDs A/F/G, correctness requires knowing that no unlisted entity satisfies the query. The prompts in Figs. 29/33 assert 'all the information you need is already provided' but never state a closure or exhaustiveness condition, and refusals are scored incorrect (§4). The Limitations explicitly concede that incompleteness affects counts, set enumerations, and nearest-entity questions. Consequently, the 61.6 EM ceiling conflates computational failure with under-specification for roughly a fifth of the small-split questions. Please either add explicit closure statements to the oracle contexts (e.g., 'the listed candidates are exhaustive') or recompute the ceiling on the subset of templates for which th","section":"§4 (oracle conditions) and Limitations"},{"comment":"The abstract's claim that gold facts widen the low-income gap is not identifiable from the presented comparison. §5.3 states that the injected oracle contexts carry on average about twice as many gold triples for high-income as for low-income questions. Since T1-o/T3-o accuracy can depend on evidence volume and context length, the widening under oracle conditions may be an artifact of triple-count imbalance rather than of computation or regional familiarity. Please control for the number and length of injected triples (e.g., matched triple-count subsets or per-triple accuracy) or explicitly rephrase the headline to acknowledge this confound. As written, a central advertising result is undermined by the paper's own analysis.","section":"§5.3 (Fig. 3) and Abstract"},{"comment":"The statement that 'no model exceeds 61.6 EM' and the abstract's 'plateaus below two thirds' are contradicted within the same section: excluding context-window overflow, Qwen3.5-35B reaches 68.4 at T3-o. Because the oracle condition overflows the context window on 8.9–19.6% of open-weight questions, the intent-to-treat ceiling partly measures context capacity, not computational ability. The text should clearly separate intent-to-treat from per-protocol results, or the headline should be re-estimated on questions whose contexts fit. The qualitative conclusion may survive, but the specific 61.6 / two-thirds bound does not follow without this caveat.","section":"§5.1 and Table 14"}],"minor_comments":[{"comment":"The negative Krippendorff alpha for French is explained only in a table footnote; given its relevance to translation reliability, a one-sentence note in the main text would be appropriate.","section":"§3.5 / Appendix B.2 (Table 9)"},{"comment":"The caption contains 'hatched-obars', which appears to be a typo for 'hatched oracle bars'; please clarify the oracle-condition symbology.","section":"Fig. 1 caption"},{"comment":"The note says 'EM above baseline is in bold', but the table as rendered does not show bold formatting. Please ensure the final PDF/HTML preserves the intended emphasis.","section":"Table 4"},{"comment":"The phrase 'the highest is a middle tier' is vague; please name the specific tier(s) per model or refer more precisely to Fig. 3 so readers can verify the ordering.","section":"§5.3"},{"comment":"The tolerance floors (e.g., δ = 0.5/5/10 km for distances, δ = 5 km/200 km for coordinates) are central to EM, but their motivation is not discussed. A sentence or a small sensitivity check would strengthen the metric's credibility.","section":"Table 12"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong benchmark contribution and the authors are unusually transparent about limitations and confounds. The main risk is that the abstract and conclusion overstate the oracle result and the income-gap result relative to what the data currently support. I would not reject; the benchmark itself and the per-family error analyses are valuable, and the issues identified are addressable within the manuscript's scope. I would ask the editor to ensure that the revision explicitly resolves the closed-world oracle issue and the triple-count confound before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read MultiGlobeQA. Bottom line: this is a real contribution, worth refereeing. The benchmark is broad, executable, and carefully constructed: 46k QA pairs, 14 spatial-function families, 15 answer formats, execution-verified ground truth over three KGs, stratified sampling across 201 countries, 17 human-verified languages, false-premise and multimodal slices, and oracle conditions that separate knowledge access from computation. They also do the boring things right: per-seed variance, honest reporting of context overflows and budget exhaustion, released code and data, and a Limitations section that admits several confounds most authors would bury. Credit where it is due.\n\nThe soft spots are real but not fatal. The 61.6 EM ceiling and 'computation, not access to knowledge, is the bottleneck' claim rest on the oracle tiers. For set enumerations, counts, and nearest-entity questions, the gold triples are positive facts only; there is no closure condition telling the model the list is exhaustive. The oracle prompt says 'all the information you need is already provided' but never states that. Refusals are scored incorrect, so a model that recognizes the under-specification gets punished. The paper's own Limitations admit incompleteness affects exactly these question types, but the oracle design does not neutralize it. That is roughly 20% of the true-premise small split, so the precise 'plateaus below two thirds' wording is not fully supported. The per-family results for grid indexing (13 EM) and shape (25 EM) are likely safe, so the qualitative conclusion survives, but the quantitative ceiling is soft.\n\nThe income-gap claim in the abstract is overclaimed too. They say gold facts 'widen rather than close' the low-income gap, but they also report that oracle contexts carry about twice as many gold triples for high-income questions. You cannot separate causal widening from evidence-volume imbalance with that design. The honest fix is to remove the causal wording or control for triple count.\n\nWho is this for? Benchmark builders and anyone studying LLM geospatial reasoning. It is a substantial new resource, and the execution-grounded construction raises the bar. It deserves a serious referee. I would send it out with a request for major revision, focusing on the two overclaims above. Desk rejection would be wrong.","headline":"MultiGlobeQA is a genuinely useful, well-built benchmark, but the headline 'computation is the bottleneck' claim is stronger than the oracle design and the income-gap analysis actually support.","tokens_in":38507,"tokens_out":1767,"would_cite":true,"duration_ms":19802,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Given gold triples and compute tools, no evaluated large language model exceeds 61.6% exact match on MultiGlobeQA's geospatial questions, and grid indexing stays below 13%.","keywords":["geospatial reasoning","benchmark","multilingual evaluation","knowledge graphs","spatial indexing","execution-based ground truth","LLM evaluation","agentic reasoning"],"falsifier":"Re-run MultiGlobeQA's count, set-enumeration, and nearest-entity queries against the same knowledge graphs updated by one year, and score models that retrieve live data against the new gold. If accuracy rises substantially on those families, the plateau partly reflects stale or incomplete snapshots rather than pure computational failure.","tokens_in":37593,"feed_emoji":"🧭","tokens_out":5654,"duration_ms":56907,"temperature":0.7,"pith_summary":"MultiGlobeQA is a multilingual benchmark of 46,060 question-answer pairs whose answers are computed by executing query templates over three knowledge graphs, so correctness is verified by construction rather than judged by a model. The paper uses it to ask whether large language models fail geospatial questions because they lack geographic knowledge or because they cannot compute over it. Across four models in parametric, reasoning, and agentic settings, the answer is computation: injecting the exact gold triples and giving the model a Python interpreter caps exact match at 61.6%, with grid-indexing questions below 13%. Retrieval and tools produce large gains, but they cannot close the gap, and low-income regions still lag even under perfect retrieval.","feed_headline":"Geospatial LLMs plateau below 62% even with gold facts","feed_subtitle":"MultiGlobeQA's 46,060 multilingual questions show missing knowledge is not the failure point—computation is.","key_machinery":"The load-bearing mechanism is the template-executor pipeline: 65 typed templates, expanded into 129 sub-templates and 315 natural-language paraphrases, are each paired with a hand-written query executed over WorldKG, KnowWhereGraph, or OSMH3KG, so every gold answer is produced by a query engine and every QA pair is execution-verified. The second mechanism is the oracle condition—injecting the supporting gold triples into the prompt or into a tool-using agent's context—which cleanly separates whether a failure comes from missing knowledge or from computation over the evidence.","core_discovery":"The paper's central finding is that access to the right facts is not the limiting factor in geospatial reasoning for current LLMs. In oracle conditions where the supporting gold triples are injected directly, no evaluated model exceeds 61.6% exact match even when it can write and execute Python; most models fall well short, and H3/S2 grid-indexing questions stay below 13%. Because these conditions eliminate retrieval failure, the residual errors are attributed to executing geometric and topological computation over the evidence—distances, containment, shape, and spatial indexing—rather than to missing parametric knowledge. The benchmark itself is constructed so that every gold answer is prod","pith_inferences":["A testable extension of the computation-bottleneck claim: replace raw Python with a fixed GIS function library (pre-bound distance, geohash, and H3 APIs) and measure whether the 61.6% ceiling rises; if it does, the bottleneck is partly code generation, not arithmetic.","The sharp contrast between geohash (54.5) and H3/S2 (16.2) under gold facts suggests a predictable ordering of failure by spatial-index complexity, which could be probed with a smaller set of discrete encodings.","Because the injected oracle contexts carry about twice as many gold triples for high-income questions, the widening low-income gap may partly reflect evidence volume rather than regional knowledge; controlling triple count per question would test this.","The 61.6% ceiling under perfect retrieval is a natural calibration target: if models trained on synthetic spatial-computation traces exceed it, that would confirm the bottleneck is trainable rather than architectural."],"forward_implications":["Geospatial QA systems should stop expecting token prediction to perform spatial arithmetic and instead delegate to deterministic geographic libraries, treating the LLM as planner and parser.","Agentic geospatial evaluation should report oracle-condition accuracy alongside live retrieval, or gains from better search will be conflated with gains from better computation.","Grid-indexing and shape families are the diagnostic floor: models that cannot handle H3/S2 cell arithmetic should not be assumed ready for navigation, logistics, or disaster-response deployment.","Perfect retrieval does not fix regional equity: lower-income regions remain harder even with gold facts, so benchmark scores need per-region reporting rather than aggregate claims.","Accuracy is stable across the 17 languages, indicating that for this benchmark cross-lingual performance is mostly a function of English tooling and entity names, not of the question language itself."],"supporting_citations":[{"why":"Supplies WorldKG, one of the three source knowledge graphs, contributing named places and amenities as points with the broadest country coverage.","marker":"Dsouza et al., 2021"},{"why":"Supplies KnowWhereGraph, the source of hazard, health, and administrative-hierarchy entities used in the benchmark.","marker":"Janowicz et al., 2022"},{"why":"Supplies OSMH3KG, the H3-indexed knowledge graph covering the full range of OpenStreetMap geometries.","marker":"Böckling et al., 2024"},{"why":"GeoQuestions1089 is the earlier manually authored geospatial QA benchmark with executable GeoSPARQL queries that this paper's broader execution-based construction extends.","marker":"Kefalidis et al., 2023"},{"why":"MapQA is the precedent for execution-based ground truth computed via spatial SQL over map data.","marker":"Li et al., 2025b"},{"why":"GS-QA is a similar execution-based geospatial QA benchmark, used as a comparison point for automatic generation and computational evaluation.","marker":"Saeedan et al., 2026"},{"why":"Cited for the claim that tool-augmented evaluation conflates tool assistance with reasoning ability, motivating the benchmark's oracle separation.","marker":"Bao et al., 2026"},{"why":"Documents OpenStreetMap completeness and inequality, motivating the benchmark's income- and density-stratified sampling.","marker":"Herfort et al., 2023"},{"why":"Supplies the WorldPop population raster used to assign density tiers in the sampling strata.","marker":"Tatem, 2017"}],"fun_headline_variants":["LLMs fail geospatial reasoning even when given the right facts","Computation, not knowledge, is the LLM geospatial bottleneck","Geospatial LLMs cap below 62% even with gold facts injected","Give LLMs the facts, they still can't compute geospatial answers","MultiGlobeQA: 46k questions prove LLMs need more than facts"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The result assumes the gold triples injected in oracle conditions are complete and correct enough to compute each answer, yet the paper notes snapshots can miss entities or contain wrong geometries, so some answers marked wrong may actually be right about the real world.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fail geospatial reasoning even when given the right facts","Computation, not knowledge, is the LLM geospatial bottleneck","Geospatial LLMs cap below 62% even with gold facts injected","Give LLMs the facts, they still can't compute geospatial answers","MultiGlobeQA: 46k questions prove LLMs need more than facts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1520,"prompt_tokens":735,"completion_tokens":785,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":690}},"tokens_in":479,"tokens_out":785,"duration_ms":7237,"temperature":1.0,"reasoning_tokens":690,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:29:05.535480+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run MultiGlobeQA's count, set-enumeration, and nearest-entity queries against the same knowledge graphs updated by one year, and score models that retrieve live data against the new gold. If accuracy rises substantially on those families, the plateau partly reflects stale or incomplete snapshots rather than pure computational failure.","supporting_citations":[],"review_version":1}