OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents

Bin Fu; Botian Shi; Minghao Liu; Qunshu Lin; Tianyu Zheng; Yang Chen; Yufan Shen; Yunwen Li; Zhi Yu

REVIEW 3 major objections 3 minor 2 cited by

A map-document benchmark forces LVLMs to do visual reasoning that text alone cannot replace, and even the best model tops out at 75%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 00:34 UTC pith:LV546BVG

load-bearing objection Useful map-document VQA suite with a simple visual-dependency metric; abstract-only, so VDI construction and annotation quality stay uncheckable. the 3 major comments →

arxiv 2607.09068 v1 pith:LV546BVG submitted 2026-07-10 cs.CV cs.AI

OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents

Yang Chen , Yunwen Li , Yufan Shen , Minghao Liu , Tianyu Zheng , Bin Fu , Qunshu Lin , Zhi Yu

show 1 more author

Botian Shi

This is my paper

classification cs.CV cs.AI

keywords OmniMapBenchvisual-centric reasoninglarge vision-language modelsmap documentsVisual Dependency Indexdocument understandingbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Many document-understanding benchmarks let large vision-language models (LVLMs) succeed by reading text that is already present in the image or recoverable from a caption; the visual layout itself is optional. OmniMapBench is built to close that loophole. It supplies 2,096 human-written questions on 1,603 real maps spanning nine categories, ranging from simple perception to multi-step visual inference. The authors introduce the Visual Dependency Index (VDI): the accuracy drop that occurs when every map image is replaced by a short, question-agnostic textual description. OmniMapBench yields a higher VDI than prior document suites, which the authors take as quantitative evidence that its questions truly require looking at the map. Across 25 leading LVLMs the strongest system reaches only 75.03 percent accuracy, showing that current models still lack reliable visual-centric reasoning on cartographic documents.

Core claim

OmniMapBench is a map-document QA suite whose questions are substantially more dependent on the actual visual content of the maps than the questions in existing document benchmarks; this property is measured by a higher Visual Dependency Index, and it leaves the best of 25 evaluated LVLMs at only 75.03 percent accuracy.

What carries the argument

The Visual Dependency Index (VDI): the measured drop in model accuracy when every map image is swapped for a fixed, question-agnostic textual description of the same document. Higher VDI is used as direct evidence that the benchmark rewards irreducible visual reasoning rather than text recovery.

Load-bearing premise

That swapping map images for short, question-agnostic textual descriptions cleanly isolates visual dependency and is not confounded by description quality, map-domain priors, or annotation artifacts.

What would settle it

Re-run the same 25 models after replacing the question-agnostic descriptions with richer, question-aware map captions; if the accuracy gap (VDI) shrinks to the level of ordinary document benchmarks, the claim that OmniMapBench uniquely demands irreducible visual reasoning is undermined.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful map-document VQA suite with a simple visual-dependency metric; abstract-only, so VDI construction and annotation quality stay uncheckable.

read the letter

This is a specialized document-VQA benchmark for maps plus a simple ablation metric they call VDI (accuracy drop when you swap the image for a question-agnostic caption). That is the whole package. It is not a new architecture or a theory paper; it is an evaluation resource aimed at labs that care about spatial grounding and irreducible visual reasoning.

What is actually new is the domain focus and the measurement. They put together 2,096 manually annotated QA pairs over 1,603 maps in nine categories, with a skill hierarchy from perception up to multi-step visual reasoning, and they report that the best of 25 LVLMs only hits 75.03%. They also claim OmniMapBench shows a higher VDI than the usual document benchmarks, which is their quantitative argument that the questions cannot be answered from text alone. The public GitHub pointer is a real plus; if the data and code ship cleanly, that is the main value.

The soft spots are exactly the ones you cannot close from the abstract. VDI is only as good as the captions and the annotation process. If the descriptions are incomplete or the questions were written with the captions in mind, the drop can be inflated. There is no inter-annotator agreement, no caption-construction detail, no table of the comparison numbers, and no stats in what we have. That does not make the claim false; it makes it unverified. Circularity risk is moderate, not fatal: the metric itself is not circular by definition, but same-team design of questions and descriptions can bias it. Soundness score should stay provisional until the full paper is read.

Who it is for: people building or evaluating multimodal document models, especially anyone working on maps, GIS-adjacent VQA, or visual grounding. A reading group that already does DocVQA / ChartQA / spatial reasoning will get something concrete out of the numbers and the failure modes. It does not reorganize the field, but it is a legitimate harder test set in a real niche.

I would send it to peer review. The contribution is clear enough, the scale is non-trivial, and the metric is simple enough to be useful if the construction holds up. Referees should demand the VDI construction details, IAA, and the actual comparison table. Until then I would not cite it myself, but I would not desk-reject it either.

Referee Report

3 major / 3 minor

Summary. The manuscript introduces OmniMapBench, a benchmark of 2,096 manually annotated question–answer pairs over 1,603 map documents from nine categories, intended to evaluate visual-centric reasoning in large vision–language models (LVLMs) on map documents. It proposes the Visual Dependency Index (VDI), defined as the accuracy drop when map images are replaced by question-agnostic textual descriptions, and claims that OmniMapBench exhibits higher VDI than established document benchmarks, thereby validating a focus on irreducible visual reasoning. The authors report a comprehensive evaluation of 25 leading LVLMs, with the best model reaching only 75.03% accuracy, and release dataset and code publicly.

Significance. If the construction of VDI and the annotation quality hold under scrutiny, OmniMapBench would address a real gap: many document-understanding benchmarks admit high performance via text-reducible cues, so a map-centric suite that forces visual grounding is useful. Strengths visible from the abstract include the public release of data and code, the scale of the model survey (25 LVLMs), and the proposal of a simple, falsifiable benchmark-level metric (VDI). A demonstrated higher VDI relative to prior suites would strengthen the claim that the benchmark isolates visual-centric skills rather than OCR or text retrieval, and could catalyze progress on map and spatial document understanding.

major comments (3)

[Abstract (VDI definition)] The central claim that OmniMapBench “quantitatively validates” irreducible visual-centric reasoning rests on VDI (accuracy drop under image→description substitution). The abstract does not specify how the question-agnostic descriptions are written (authoring protocol, independence from the QA pairs, length/completeness relative to map content, or any human validation). Without those details, the measured drop could be driven by incomplete or low-quality captions rather than genuine visual irreducibility. This operationalization is load-bearing for the higher-VDI claim and must be fully specified and stress-tested in the manuscript.
[Abstract (VDI comparison claim)] The abstract asserts that OmniMapBench has higher VDI than “established benchmarks” but supplies neither the comparison suite, the numerical VDI values for those baselines, nor any statistical test of the gap. These comparisons are load-bearing for the claim that the benchmark is more visually dependent than existing document suites; they must appear with clear methodology (same models, same description protocol) in the full paper.
[Abstract (dataset construction)] The 2,096 pairs are described as “manually annotated” and as probing a hierarchy from perception to multi-step visual reasoning, yet the abstract reports no inter-annotator agreement, quality-control protocol, or checks against map-domain priors and data leakage. Annotation artifacts or question design that inadvertently encodes non-visual shortcuts could inflate both difficulty and VDI; these controls are load-bearing for the claim that the benchmark cleanly isolates visual reasoning.

minor comments (3)

[Abstract] The nine map categories are mentioned but not listed; a brief enumeration in the abstract (or a pointer to a table) would orient readers.
[Abstract] The 75.03% top accuracy should name the model (and preferably the evaluation protocol: zero-shot, few-shot, or fine-tuned) so the result is interpretable in isolation.
[Abstract] “Question-agnostic descriptions” is a key term; a short parenthetical definition in the abstract would reduce ambiguity before the full method section.

Circularity Check

0 steps flagged

No circularity detectable from abstract; VDI and accuracy claims are empirical, not definitional reductions.

full rationale

Only the abstract is available, so the derivation chain cannot be fully walked. From the abstract alone, OmniMapBench is a new dataset of 2,096 manually annotated QA pairs; VDI is defined as the measured accuracy drop when images are replaced by question-agnostic descriptions and is reported as higher than established benchmarks; top LVLM accuracy is reported as 75.03%. These are empirical measurements and comparisons, not self-definitional identities, fitted parameters renamed as predictions, uniqueness theorems imported from the same authors, or ansatzes smuggled via self-citation. The abstract does not reduce the headline claims to inputs by construction. Residual risks (annotation quality, description completeness, same-author design of questions and captions) are methodological validity concerns, not circularity under the stated criteria. Score 0 with empty steps is therefore the warranted finding for an abstract-only review.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 2 invented entities

As a benchmark paper the central claims rest on construction choices and evaluation protocol rather than free physical parameters. The load-bearing pieces are the manual annotation process, the nine map categories, the hierarchy of skills, and the operational definition of VDI via question-agnostic descriptions. No new physical entities are postulated; the invented construct is the VDI metric itself.

axioms (3)

ad hoc to paper Question-agnostic textual descriptions of maps are a valid substitute that isolates non-visual information for measuring visual dependency.
VDI is defined in the abstract as the accuracy drop under this substitution; validity of that operationalization is assumed rather than derived from prior theory.
domain assumption Manually annotated map QA pairs across nine categories form a representative probe of perception-to-multi-step visual reasoning for LVLMs.
Scale and category coverage are stated as design facts; representativeness of real-world map understanding is taken as given.
domain assumption Accuracy on multiple-choice or free-form QA is an adequate proxy for visual-centric document understanding.
Standard evaluation assumption in LVLM benchmarking; used throughout the reported 25-model comparison.

invented entities (2)

Visual Dependency Index (VDI) no independent evidence
purpose: Quantify how much a benchmark requires irreducible visual input by measuring accuracy drop when images are replaced by question-agnostic descriptions.
Introduced in this work as a benchmark-level metric; independent evidence would be third-party replications on other datasets, which the abstract does not provide.
OmniMapBench dataset no independent evidence
purpose: Provide 2,096 QA pairs on 1,603 map documents to stress visual-centric reasoning in LVLMs.
New annotated resource; falsifiable only via public release quality and external re-evaluation, claimed but not inspectable from the abstract alone.

pith-pipeline@v1.1.0-grok45 · 6160 in / 2323 out tokens · 26299 ms · 2026-07-13T00:34:11.039134+00:00 · methodology

0 comments

read the original abstract

Recent advancements in LVLMs necessitate robust benchmarks for complex, visually grounded reasoning. A critical limitation is identified in many document understanding benchmarks: visual content is often reducible to text, enabling high performance without genuine visual grounding. To address this limitation, OmniMapBench is introduced to foster visual-centric reasoning for map documents. The benchmark comprises 2,096 manually annotated question-answer pairs across 1,603 map documents from nine categories. It is designed to probe a hierarchy of skills, ranging from perception to multi-step visual reasoning. To quantify benchmark properties, a simple yet effective benchmark-level metric is proposed: the Visual Dependency Index (VDI), defined as the accuracy drop when images are replaced with question-agnostic descriptions. OmniMapBench exhibits higher VDI than established benchmarks, which quantitatively validates its focus on irreducible visual reasoning. Comprehensive evaluations of 25 leading LVLMs are conducted on OmniMapBench. A significant performance gap is observed, with the top-performing model achieving only 75.03\% accuracy. This result underscores the challenges posed by OmniMapBench to current LVLMs. This work aims to catalyze progress in visual-centric reasoning for document understanding of LVLMs. The dataset and code are publicly available at https://github.com/SIGMME/OmniMapBench.

discussion (0)

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Visual Credit Audit for Multimodal Spatial Reasoning
cs.CV 2026-07 accept novelty 6.0

VCA finds 12.73–26.25% of spatial decisions are correct yet uncredited by the image, and separates marginal image support from relation-specific visual response.
Visual Credit Audit for Multimodal Spatial Reasoning
cs.CV 2026-07 conditional novelty 5.0

Between 12.7% and 26.3% of correct spatial yes/no answers from four open MLLMs receive no support advantage from the benchmark image over text-only or blank controls.