{"id":"21035558-6ae5-4492-beb1-94f2bf67f337","arxiv_id":"2501.00316","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MapEval, a 700-question benchmark spanning textual, API, and visual map tasks, finds no foundation model exceeds 67% accuracy while humans score at least 82%.","lead":"MapEval is a new benchmark that tests AI models on map-based reasoning with 700 multiple-choice questions across text, map API calls, and visual snapshots. In tests of 30 AI models, the best scored 66%, while humans scored above 82%, showing AI still lags in navigation and spatial reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The human ceiling used for the >20% gap is not an apples-to-apples model condition: author-affiliated raters, highest-attempt scoring, and reuse of the textual human score for the API task may inflate the reported model-human gap.","rationale":"The reader's weakest-assumption analysis already points to the human baseline; I agree and sharpen it. The headline conclusion is an inequality between model scores and human scores, so the human scores are load-bearing. What would have to be true for the central claim is that the human numbers are a fair, single-try ceiling under the same information available to each model. The paper's Section 3.3 procedure does not establish that: it explicitly uses author-affiliated raters, best-of-multiple attempts, and reuses the textual human score for the API task. The visual protocol fails to specify whether humans see only the snapshot or can interact with the map. These are internal inconsistencies with the evaluation protocol, not just disagreements with external consensus. The paper has genuine strengths--released resources, a controlled pseudo-API cache, and a broad model sweep--so the appropriate response is not rejection but a conditional acceptance: the model-vs-human gap must be re-measured under parity conditions. If the external parity test reproduces roughly 85% human accuracy, the concern is resolved and the current verdict stands; if not, the over-20% claim needs quantitative revision.","tokens_in":35246,"tokens_out":7083,"duration_ms":72748,"concrete_test":"Recruit at least 10 independent participants with no prior exposure to MapEval and run a preregistered parity evaluation on a stratified sample (or all 700 items): for Textual, provide only the textual context; for Visual, provide only the static snapshot, forbidding live maps, zoom, or pan; for API, provide only the cached pseudo-API tools, with no textual context. Allow exactly one attempt per question, and do not select the best of several attempts. Compare the resulting human accuracy to the reported 86.67% (Textual), 82.23% (Visual), and to the API condition separately. If the parity-respecting human accuracy is materially below the reported values, the abstract's '>20% gap' claim should be revised and the human baseline replaced with the parity-respecting numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim--'all models lagging over 20% behind human performance'--rests on the human baselines in Section 3.3. Those baselines are not measured under the same conditions as the models. (1) The human raters are two team members and their reported score is their 'highest-scoring attempts,' so multiple tries are allowed; each model gets a single deterministic pass. Retrying inflates human accuracy relative to the model protocol. (2) For MapEval-API, Section 3.3 says the human score is the same as MapEval-Textual, yet the agent in MapEval-API receives no textual context and must query pseudo-APIs. A valid human baseline for the API task must be measured with the same tool-only interface, not with pre-fetched textual contexts. (3) For MapEval-Visual, the protocol does not state that humans are restricted to the static snapshot; if they use the live or zoomable map at the saved URL, they have strictly more information than the VLM, which sees one image. The best models score 61-66%, and humans are reported at 82-87%; an inflation of even 5-10 points in the human numbers would erase or substantially shrink the headline >20% gap. The benchmark may remain useful, but the model-human comparison is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MapEval, a benchmark of 700 multiple-choice questions for evaluating geo-spatial reasoning in foundation models across three tasks: textual (MapEval-Textual), API-based (MapEval-API), and visual (MapEval-Visual). The dataset spans 180 cities and 54 countries and covers categories such as place info, nearby, routing, trip planning, counting, and unanswerable queries. The authors evaluate 30 proprietary and open-source models and report that none surpasses 67% accuracy, with all models lagging over 20% behind human performance. They also provide qualitative error analyses and experiments with tool integration (calculator, Chameleon) that improve performance on specific subskills.","tokens_in":35570,"tokens_out":4524,"duration_ms":44079,"significance":"If the benchmark and its evaluation protocol are sound, MapEval is a valuable contribution to an under-explored area of geospatial reasoning, offering broad geographic coverage, multiple modalities, and a reproducible pseudo-API environment that supports controlled agent evaluation. The release of data and code is a strength. However, the headline claim of a >20% model-human gap depends on a human baseline that is not measured under conditions matched to the model protocols, and the accuracy tables report only point estimates without uncertainty quantification. These issues are load-bearing because the gap is the paper's central quantitative claim; they are fixable with additional data collection and analysis, so the manuscript should be revised rather than rejected.","major_comments":[{"comment":"The human baseline is not an apples-to-apples comparison with the model evaluation protocols. First, the two raters are author-affiliated team members and their reported score is their 'highest-scoring attempts,' allowing multiple tries per question, whereas each model receives a single deterministic pass. Second, for MapEval-API, the paper states that the same human score as MapEval-Textual is used, yet the API agents receive no textual context and must interact with tools; a valid human baseline must be measured with the same tool-only interface, not with pre-fetched textual contexts. Third, for MapEval-Visual, the protocol does not specify whether human raters are restricted to the static snapshot or can access the live or zoomable Google Maps URL that accompanies each image; if the latter, they have strictly more information than the VLM. Because the abstract and conclusion assert a >20% gap, these protocol mismatches could materially inflate the reported human performance and shrink or erase the headline gap. I recommend re-measuring human performance with independent, non-author raters under conditions strictly matched to each task (static image for visual, tool-only for API), reporting per-rater scores and all attempts rather than the maximum.","section":"Tables 3, 4, and 5"},{"comment":"All accuracy results are reported as point estimates without confidence intervals or significance tests. With 300–400 items per task, the standard error of a proportion is roughly 2–3 percentage points, so differences of a few points (e.g., Claude-3.5-Sonnet 61.65 vs. Qwen2.5-VL-72B 60.35 in Table 5) are not meaningful. The central claim that 'none surpasses 67%' rests on top scores of 66.33 and 64.00, which are within sampling error of 67. Where the paper makes comparative claims such as 'outperforming GPT-4o and Gemini-1.5-Pro by 16% and 21%' (Section 4.2.2), the authors should provide confidence intervals or significance tests, or at minimum discuss the precision of these differences. I recommend adding Wilson intervals to Tables 3–5 and avoiding over-interpretation of small gaps.","section":"Appendix B.2, Filtering via LLMs"},{"comment":"The dataset construction removes questions that 'the majority of the LLMs could easily provide the correct answer.' This design choice is defensible for building a challenging benchmark, but it directly affects the interpretation of the model-human gap: since human raters are evaluated on the filtered set, the reported human accuracy (86.67% textual, 82.23% visual) is measured on a set specifically selected to be hard for LLMs. The paper should quantify how many questions were removed, analyze the difficulty distribution before and after filtering, and discuss whether the >20% gap reflects a fundamental limitation of models rather than a benchmark calibration artifact. This is not a fatal flaw, but it is necessary to support the 'real-world' framing and to allow readers to judge the magnitude of the gap.","section":"Section 3.3"}],"minor_comments":[{"comment":"The model name 'Claude-3-5-Sonnet' appears with a hyphen in Table 5 and elsewhere, while the rest of the paper uses 'Claude-3.5-Sonnet'; please standardize.","section":"Table 5"},{"comment":"The phrase 'most comprehensive evaluation framework for geospatial AI' is a strong claim that goes beyond the evidence presented, especially given the restriction to five Google Maps APIs and the filtering of easy questions; consider tempering the wording.","section":"Abstract"},{"comment":"The sentence 'For MapEval-API, as the questions are identical to MapEval-Textual, we report the same human performance for both' is the source of the major concern above, but even if the human score is reused by design, this should be explicitly flagged as a limitation and discussed.","section":"Section 3.3"},{"comment":"The open-ended evaluation uses an automated grader (o3-mini) that the paper itself notes is unreliable for the Unanswerable category; this is a useful caveat, but the open-ended results are not used in the main claims, so the observation does not affect the main conclusions.","section":"Appendix G.2"},{"comment":"The percentages in the two pies each sum to 100%, but the visual and textual/API subcategories are not directly comparable; a note that the two pies represent disjoint subsets of 400 and 300 questions would help readability.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and its resources are likely to be useful to the community, and the broad multi-model, multi-task evaluation is a strength. However, the human-baseline protocol is the load-bearing weakness of the paper's central claim. The authors should be required to re-collect human performance under matched conditions with independent raters before the claim of a >20% gap can be accepted. Given the benchmark's other merits, this is a fixable issue rather than a fundamental invalidation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers a genuinely useful artifact: 700 multiple-choice questions spanning textual, API, and visual map reasoning, with geographic diversity and a broad model evaluation. The tasks go beyond simple location lookup—trip planning, counting, unanswerability, and map-image interpretation are all represented—and the dataset fills a real gap in geospatial evaluation. The human quality-control process (multi-annotator agreement, dispute resolution) and the supplementary experiments (calculator integration, fine-tuning, open-ended evaluation) show care. The promised code and data will make this a reusable resource.\n\nThe weak point is the human baseline, and the stress-test note gets this right. Two team members who did not annotate the data are still inside the project, their highest-scoring attempts are used, and for the API condition the same textual human score is reported even though the models have no textual context. For the visual task, nothing in the protocol says the human raters were limited to the static snapshot, so they may have exploited the live map at the saved URL. Each of these choices inflates human accuracy relative to the model protocol, and the headline 'over 20% behind human performance' is load-bearing on those numbers. Even a 5-point inflation would shrink the gap substantially. The benchmark can still be useful without that headline claim, but the model-human comparison is not yet supported as stated.\n\nLesser issues: the accuracy tables show point estimates with no confidence intervals, so differences like 61.65 vs. 60.35 on 400 items are not meaningful. The pseudo-API environment is clearly described and defensible as a controlled evaluation, though it weakens claims about real-world tool use. The LLM-based filtering of 'too easy' questions is transparent but does shape the observed difficulty. The self-citation to MapQaTor is legitimate—it is their own tool and they document its role.\n\nThis is a serious, competent paper. The benchmark deserves to exist and be used, but the central quantitative claim needs re-measurement or careful scoping before it can be taken at face value. I would send it to peer review and require the authors to either fix the human baseline (independent raters, single attempt, matched interface) or soften the claim to 'models perform far below a favorable human estimate.' The benchmark itself is a good contribution; the gap claim is not yet earned.","headline":"A useful new geospatial reasoning benchmark, but the headline model-human gap rests on a human baseline that was measured under easier conditions than the models faced.","tokens_in":36073,"tokens_out":1542,"would_cite":true,"duration_ms":17842,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 700-question map benchmark finds every tested AI model below 67% accuracy, over 20 points behind humans.","keywords":["map-based reasoning","geospatial AI benchmark","foundation models","spatial reasoning","API-based agents","visual question answering","trip planning","unanswerable questions"],"falsifier":"Run the same 700 questions with a fresh, geographically diverse group of about 50 people who have not seen the dataset and who answer each question once; if their mean accuracy comes out below roughly 80% on textual and 75% on visual, the claimed over-20% model-human gap does not hold at that scale.","tokens_in":35085,"feed_emoji":"🗺️","tokens_out":3993,"duration_ms":39544,"temperature":0.7,"pith_summary":"MapEval is a benchmark of 700 multiple-choice questions drawn from real map data across 180 cities and 54 countries, testing foundation models in three modes: answering from pre-fetched map text, acting as an agent that calls map APIs, and reading map screenshots. Across 30 models, none surpasses 67% accuracy, while human performance is reported at 86.67% on the textual task and 82.23% on the visual task, implying a gap of more than 20 percentage points. The paper argues this gap concentrates in tasks that combine facts—multi-stop trip planning, straight-line distances, cardinal directions, counting points of interest, and recognizing unanswerable queries—rather than simple place recall. A sympathetic reading is that the benchmark now provides a concrete, reproducible yardstick for a largely untested capability that matters for navigation, logistics, and real-world tool use.","feed_headline":"30 AI models trail humans by 20+ points on map reasoning","feed_subtitle":"A 700-question benchmark covering 180 cities shows all models below 67% accuracy; distances and routes trip them up.","key_machinery":"The MapEval benchmark itself: 700 multiple-choice questions with ground truth derived from Google Maps, delivered in three coordinated modes—pre-fetched textual context, ReAct agents that call a controlled pseudo-Google Maps environment with cached API responses, and map snapshots for vision-language models. A second load-bearing mechanism is the tool-augmentation probe: a calculator implementing the Haversine formula and bearing computation, which isolates numeric geospatial computation from the language and reasoning components of the task.","core_discovery":"The paper reports that current foundation models have not yet learned to reason over geospatial information the way map users do: the best model, Claude-3.5-Sonnet, reaches 66.33% on MapEval-Textual, 64.00% on MapEval-API, and 61.65% on MapEval-Visual, and every one of the 30 evaluated models trails the reported human scores by more than 20 points. The deficits are systematic: straight-line distance questions are answered correctly at only 51.06% by the best model, cardinal-direction accuracy varies wildly (16.67% to 91.67%), Trip planning tops out near 49%, and visual counting at a 47.73% best rate. The paper also shows that giving models a calculator for Haversine distance and bearing jumps straight-line distance accuracy from 51.06% to 85.11% and cardinal-direction accuracy to 95.83%, and that fine-tuning on 97 MCQs yields less than 5% improvement, indicating the difficulty is not merely missing task exposure.","pith_inferences":["If the human baseline is confirmed by an independent, non-author participant pool, MapEval becomes a stable yardstick for geospatial AI; if not, the claimed over-20% gap may shrink.","The calculator result implies part of 'geospatial reasoning' is arithmetic disguised as reasoning; agents that invoke computational tools may close much of the gap without deeper spatial understanding.","MapEval's API and Unanswerable categories reward agents that know when to stop querying, a property that transfers beyond maps to robust tool use.","The MCQ format likely bounds reported accuracy relative to open-ended interaction, and the paper's own open-ended experiment shows evaluation is noisier, so absolute numbers should be read as MCQ-specific."],"forward_implications":["No current foundation model can reliably plan multi-stop trips: the best textual Trip accuracy is about 49%, far below the reported human 88%.","Models can read map text but cannot compute from coordinates: even the best straight-line distance accuracy is 51.06% without external tools.","API-based agents lose ground on Nearby and Routing relative to textual context (e.g., Claude-3.5-Sonnet drops from 73.49% to 55.42% on Nearby), so tool orchestration adds its own error.","Visual map reasoning degrades at zoom levels beyond 14, meaning map-specific visual complexity is an active bottleneck for VLMs.","Fine-tuning on 97 MapEval questions yields gains below 5%, suggesting the gap is not simply a lack of task exposure."],"supporting_citations":[{"why":"Supplies MapQaTor, the annotation interface that auto-collects textual map context from Google Maps APIs.","marker":"Dihan et al., 2024"},{"why":"Defines ReAct agents used for the MapEval-API task.","marker":"Yao et al., 2023"},{"why":"Chameleon framework, adapted to improve GPT-3.5-Turbo's API performance from 27.33% to 49.33%.","marker":"Lu et al., 2024"},{"why":"MMLU-style multiple-choice format, invoked to justify accuracy-based evaluation.","marker":"Hendrycks et al."},{"why":"Travel Planner benchmark, the closest prior work using Google Maps API for itinerary planning, which MapEval extends.","marker":"Xie et al."},{"why":"VisualWebArena, one of the map-usage agent environments MapEval positions against.","marker":"Koh et al., 2024"},{"why":"Establishes the broader GeoAI foundation-model challenges that MapEval is designed to test.","marker":"Mai et al., 2024"}],"fun_headline_variants":["AI map IQ: best model scores 66%, humans 20+ ahead","MapEval: 30 AI models all under 67% on map reasoning","Maps stump AI: no model beats 67% on new benchmark","Geospatial gap: AI trails humans by 20 points on maps","Claude-3.5 tops AI map test at 66% — still far behind"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The human performance ceiling (86.67% textual, 82.23% visual) was measured from two team members' best attempts, so if those scores are higher than a typical unscreened user would achieve, the reported model-human gap shrinks.","fun_headline_variants_meta":{"raw":{"variants":["AI map IQ: best model scores 66%, humans 20+ ahead","MapEval: 30 AI models all under 67% on map reasoning","Maps stump AI: no model beats 67% on new benchmark","Geospatial gap: AI trails humans by 20 points on maps","Claude-3.5 tops AI map test at 66% — still far behind"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1527,"prompt_tokens":996,"completion_tokens":531,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":429}},"tokens_in":612,"tokens_out":531,"duration_ms":6271,"temperature":1.0,"reasoning_tokens":429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:52:50.762020+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 700 questions with a fresh, geographically diverse group of about 50 people who have not seen the dataset and who answer each question once; if their mean accuracy comes out below roughly 80% on textual and 75% on visual, the claimed over-20% model-human gap does not hold at that scale.","supporting_citations":[],"review_version":1}