{"id":"0ff6f3e0-6dbc-4f99-995a-109d17eca182","arxiv_id":"2505.10714","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GeoGrid-Bench evaluates 11 foundation models on 3,200 expert-curated questions about gridded climate data across 16 variables, finding vision-language models strongest and code generation weakest.","lead":"GeoGrid-Bench is a new benchmark that asks AI models to answer thousands of multiple-choice questions about gridded climate data, shown as tables or heatmaps. It measures whether today's foundation models can actually reason over geo-spatial data, which matters for building trustworthy climate and disaster-response tools.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's validity rests on oracle-generated ground truth, but no validation procedure or human audit is reported; a systematic oracle bug or degenerate options would invalidate every reported accuracy.","rationale":"The reader's weakest-assumption analysis identifies precisely the most load-bearing point: ground truth correctness is asserted but never demonstrated. The paper's own wording makes this explicit: quality is guaranteed only after template and oracle validation, yet no such validation is described. This is not a minor implementation detail because every downstream conclusion, including the headline VLM-over-text result, is computed against these oracle answers. If the oracle is wrong, the benchmark is not merely noisy; it is invalid. The public code and data release are positive evidence, and the template design is thoughtful, but they do not substitute for a reported audit. A CONDITIONAL verdict is therefore appropriate: the benchmark is promising and the evaluation is extensive, but the central validity claim should not be accepted as established until the oracle and option-generation machinery are independently checked. No change to the reader's verdict is needed.","tokens_in":13622,"tokens_out":4982,"duration_ms":55826,"concrete_test":"Run the released oracle code on all 3,200 instances; independently recompute answers from raw ClimRR data with a second implementation written solely from Table 1 and the Section 3.1 specifications, then compare. In addition, have two annotators independently audit a stratified sample (e.g., 25 per template) for exactly-one-correct options and plausible distractors; report Cohen's kappa and every disagreement. If all outputs match and annotator agreement is high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GeoGrid-Bench is a valid instrument and that the VLM-over-text finding is a real property rests on the assertion in Section 3.2 that 'once a template and its oracle are validated, we ensure the quality of every generated instance.' No validation procedure, human audit, inter-annotator agreement, or error analysis is reported anywhere in the paper or appendix. The oracle is the sole source of ground truth and of all multiple-choice options, and the templates include subjective discretizations whose thresholds are not specified (e.g., correlation strength in Template 3, 'largest increase' in Template 1, 'greater change' in Template 7). If any oracle routine has a systematic bug, or if option construction yields cases with zero or two correct answers, then all 3,200 instances and all accuracy numbers in Figures 4-6 are affected. This is a correctness risk, not a disagreement with external consensus; it is directly about whether the benchmark measures what it claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GeoGrid-Bench introduces a benchmark for evaluating foundation models on gridded geo-spatial data. Built from the ClimRR climate dataset, it spans 16 climate variables, 150 North American locations, and roughly 3,200 multiple-choice question-answer pairs generated from 8 domain-expert-curated templates. Each question is rendered in tabular and three image formats (heatmap, annotated heatmap, and map-overlaid heatmap), and answers are produced deterministically by oracle code targeting four answer aspects: overall trend, spatial reference, coordinate reference, and label reference. The paper evaluates 11 models (OpenAI and open-source) in language-only, code, and vision settings, reporting that vision-language models perform best overall, code-based performance is inferior, and fine-grained spatial/label reasoning is harder than trend identification. The central claims are that GeoGrid-Bench is a valid, reusable benchmark and that the VLM-over-text advantage is a real property of current foundation models.","tokens_in":13829,"tokens_out":2719,"duration_ms":28057,"significance":"If the benchmark construction is sound, GeoGrid-Bench fills a genuine gap: existing geo-spatial benchmarks focus on remote-sensing imagery or GIS/SQL retrieval, whereas gridded climate data with dense numerical values and multimodal presentations is underexplored. The paper's strengths include the use of real-world ClimRR data, public release of code and data, deterministic oracle-based answer generation, a broad model sweep (11 models), and fine-grained evaluation across answer aspects and visualization formats. The central VLM-over-text finding, if statistically robust, would be a practically useful result for designing AI-assisted climate analysis tools. However, the validity of every reported accuracy number depends on the correctness and uniqueness of oracle-generated ground truth, and the manuscript does not yet provide the validation evidence needed to underwrite that dependency.","major_comments":[{"comment":"The validity of the entire benchmark rests on the oracle code, which is the sole source of ground truth and of all four multiple-choice options. The paper states 'once a template and its oracle are validated, we ensure the quality of every generated instance,' but no validation procedure, human audit, inter-annotator agreement, or error analysis is reported anywhere in the paper or appendix. If any oracle routine has a systematic bug, or if option construction yields cases with zero or two correct answers, then all 3,200 instances and every accuracy number in Figures 4-6 are affected. This is a load-bearing correctness risk, not a stylistic issue. Please add a concrete validation protocol, such as a human audit on a stratified sample of instances per template, automated checks that exactly one option is correct, and an error analysis of any generated instances that fail these checks.","section":"Section 3.2"},{"comment":"The evaluation reports accuracy numbers from a single decoding run per model/setting with default temperature, and no variance estimates, confidence intervals, or statistical tests are provided. Given that each template has only 100 instances and that fine-grained columns (e.g., coordinate or label references) split the data further, the observed differences—such as the claimed VLM advantage over language-only or the differences among image formats—could plausibly arise from sampling noise. Please report multiple runs (or bootstrap resamples) with confidence intervals, or apply a significance test, so that the central empirical claims are quantifiably supported.","section":"Section 4.2, Figures 4-6"},{"comment":"Several templates rely on qualitative terms whose operational definitions in the oracle are not specified: 'largest increase' (Template 1), 'greater change' (Template 7), and 'correlation' thresholds (Template 3) such as what counts as strong versus weak correlation. Without precise operationalizations, another group cannot reproduce the oracle rules or verify that the generated answers match the intent of the domain experts. Please state the exact oracle decision rules for each template, including all thresholds and criteria used to select the correct option and to construct distractors.","section":"Table 1 and Section 3.2"}],"minor_comments":[{"comment":"The abstract and the Figure 1 caption both use 'find-grained'; this should be 'fine-grained'.","section":"Abstract and Figure 1 caption"},{"comment":"In the first paragraph of the introduction, 'Most existing efforts docus on' should read 'Most existing efforts focus on'.","section":"Section 1"},{"comment":"The appendix captions repeatedly refer to 'maximum annual temperate' and 'minimum annual temperate'; these should be 'temperature'.","section":"Appendix B, Figures 7-13"},{"comment":"The caption for Figure 12 says 'time_frame1 = historical period, and time_frame1 = mid-century period (RCP4.5)'; the second should be time_frame2.","section":"Figure 12 caption"},{"comment":"The data statistics say 50 locations are associated with each of 16 climate variables, which would imply 800 location-variable pairs, while the text also says there are 150 distinct locations. Please clarify how locations are shared across variables and how the 50-per-variable count relates to the 150 unique locations.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the benchmark has clear potential value, but the missing oracle validation is a correctness risk that must be addressed before the accuracy numbers can be trusted. I would be willing to accept a revision that includes a documented validation protocol (even a modest human audit with error analysis) and basic statistical evidence for the main comparisons. The heavy self-citation in the related-work section is noticeable but not disqualifying; it does not appear to misrepresent the prior literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers something the field actually needs: a benchmark for gridded, numerical geo-spatial data in the forms scientists use—tables, heatmaps, annotated heatmaps, map overlays—rather than another remote-sensing or GIS-retrieval task. The gap is real, the templates come from domain experts, and the evaluation spans 11 reasonably current models across three input modes. The public code and data are a plus. The central finding, that vision-language models beat language-only and code modes on these tasks, is plausible and individually useful for model selection.\n\nThe main soft spot is the oracle. The paper says in Section 3.2 that 'once a template and its oracle are validated, we ensure the quality of every generated instance,' but no validation procedure, human audit, or error analysis is reported. Since the oracle is the sole source of ground truth and all four options, a systematic bug—or a template with subjective thresholds like 'largest increase' or 'greater change'—would propagate through all 3,200 instances and every accuracy figure. This is a real risk, not a manufactured one. It is also fixable: a human audit on a few hundred instances, checks that each question has exactly one correct answer, and a short appendix describing the oracle logic would largely address it.\n\nSmaller issues: the evaluation uses one decoding run per setting with no variance estimates or significance tests, so some reported differences may be noise. The location sampling (50 'most prominent' locations per variable) could bias results toward easy cases. Both are worth mentioning in revision but they are secondary.\n\nI disagree with any reading that treats this as a throwaway or a pure artifact paper. The benchmark construction is careful, the writing is clear, and the limitations section is honest about geographic scope. The oracle gap is the kind of thing peer review is for. I would send this to review rather than desk reject, ask the authors to open the black box, and then let the community use the artifact.","headline":"A genuinely useful benchmark for gridded climate data, but the oracle-generated ground truth needs visible validation before the accuracy numbers can be trusted.","tokens_in":14315,"tokens_out":1276,"would_cite":true,"duration_ms":14926,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GeoGrid-Bench, a roughly 3,200-question benchmark built on real climate projections, shows that foundation models understand gridded geo-spatial data best when it is shown as heatmaps, not tables or code.","keywords":["geospatial data","gridded data","benchmark","foundation models","vision-language models","climate data","multiple-choice generation","heatmaps"],"falsifier":"Take a random sample of about 200 GeoGrid-Bench instances, have climate-domain experts independently derive the correct multiple-choice option from the raw grid data, then compare their labels with the oracle's; if expert agreement falls below the reported model accuracies, the ground truths do not support the paper's conclusions. A second check is to re-present the vision questions with the four options in different orders and see whether accuracy moves, which would indicate position bias rather than genuine spatial understanding.","tokens_in":13458,"feed_emoji":"🗺️","tokens_out":7477,"duration_ms":64941,"temperature":0.7,"pith_summary":"GeoGrid-Bench is a benchmark for testing whether foundation models can make sense of gridded geospatial data, the kind of dense numerical climate output that scientists actually work with. It compiles about 3,200 multiple-choice questions from eight templates designed with domain experts, with ground-truth answers produced by deterministic oracle code, and presents the underlying data either as tables or as heatmap images. The paper's central finding is that vision-language models answer these questions most accurately, while models that receive the data as text tables or that must write Python code perform worse. If the benchmark is valid, it gives the community a reusable instrument for measuring and improving AI assistance on gridded climate data.","feed_headline":"Heatmaps beat tables and code for climate-data AI","feed_subtitle":"A new 3,200-question benchmark finds foundation models read gridded climate data best as images.","key_machinery":"The benchmark's working parts are the eight expert-curated question templates and the oracle code paired with each one. Given a template, sampled climate variables, locations, and time frames from a real-world gridded climate dataset, the oracle deterministically computes the correct answer and generates four multiple-choice options, so a single validated template yields many reliable instances. Each instance is rendered in four modalities: a table as text, a bare heatmap, a heatmap with per-cell numerical annotations, and a heatmap overlaid on a geographic base map. Answer options are classified by which aspect they test: overall pattern, spatial region, coordinate reference, or map-label reference. The evaluation protocol compares language-only, language-plus-code, and language-plus-vision input settings.","core_discovery":"On its own terms, the paper claims that foundation models understand gridded geo-spatial data best when the data is presented as a heatmap image rather than as a dense numerical table or as a prompt to generate analysis code. Across the evaluated commercial and open-source models, vision-language input consistently outperformed language-only input, and code-based performance was inferior, with many generated scripts failing to execute in a single pass. The paper also finds that all models are better at identifying overall trends than at fine-grained references to spatial regions, row/column coordinates, or map labels, and that adding numerical annotations to heatmaps helps while overlaying heatmaps on geographic base maps hurts. These results are presented as evidence about how to build practical AI-assisted tools for geo-spatial and climate analysis.","pith_inferences":["A natural test of the central finding is to apply the same template-oracle pipeline to non-climate gridded data, such as demographic or geophysical grids; if heatmaps still win, the vision advantage is about grids generally, while a failure would point to map-specific color conventions.","An independent human audit of a random sample of oracle-generated answers would directly test whether the reported accuracies reflect model skill or artifacts of the answer-generation code.","Because annotated heatmaps helped and map overlays hurt, ablating visual clutter (label density, color scale, grid-line width) could isolate exactly which visual features drive the vision-language advantage."],"forward_implications":["For current foundation models, presenting gridded climate data as heatmaps is a more effective interface than tables or one-shot code generation.","The template-plus-oracle pipeline lets the benchmark scale beyond its current 3,200 instances without new human annotation.","Fine-grained spatial, coordinate, and label reasoning is a distinct weakness of current models that trend-level accuracy alone does not expose.","Code-based geo-spatial analysis by foundation models is not yet reliable enough for single-shot use by domain scientists, pointing to agentic models as the next step."],"supporting_citations":[{"why":"Supplies the ClimRR gridded climate projection data that all GeoGrid-Bench instances are sampled from.","marker":"Argonne National Laboratory, 2023"},{"why":"Defines the GPT-4o model family evaluated across the three input modalities.","marker":"Hurst et al., 2024"},{"why":"Documents the o-series reasoning models such as o4-mini that achieve the highest overall accuracy.","marker":"Jaech et al., 2024"},{"why":"Defines the Llama-3 and Llama-4 open-source model families evaluated in the study.","marker":"Grattafiori et al., 2024"},{"why":"Provides the Qwen2.5-VL open-source vision-language model that anchors the open VLM baseline.","marker":"Bai et al., 2025"},{"why":"Supplies the sentence-embedding model used to match model responses that do not output an explicit option letter.","marker":"Reimers & Gurevych, 2019"},{"why":"Provides the geographic base maps used to render the most realistic heatmap-over-map image modality.","marker":"OpenStreetMap contributors, 2024"}],"fun_headline_variants":["AI reads climate data best as heatmap images","Vision-language models top climate geo-spatial benchmark","Heatmap images beat tables and code for geo-AI","Gridded climate data: VLMs favor visual input","Benchmark: VLMs outperform on grid climate tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on the oracle code producing a correct and unambiguous answer for every generated instance, and the paper reports no independent human validation of that code.","fun_headline_variants_meta":{"raw":{"variants":["AI reads climate data best as heatmap images","Vision-language models top climate geo-spatial benchmark","Heatmap images beat tables and code for geo-AI","Gridded climate data: VLMs favor visual input","Benchmark: VLMs outperform on grid climate tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1404,"prompt_tokens":888,"completion_tokens":516,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":441}},"tokens_in":504,"tokens_out":516,"duration_ms":4757,"temperature":1.0,"reasoning_tokens":441,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:04:40.379318+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of about 200 GeoGrid-Bench instances, have climate-domain experts independently derive the correct multiple-choice option from the raw grid data, then compare their labels with the oracle's; if expert agreement falls below the reported model accuracies, the ground truths do not support the paper's conclusions. A second check is to re-present the vision questions with the four options in different orders and see whether accuracy moves, which would indicate position bias rather than genuine spatial understanding.","supporting_citations":[],"review_version":1}