The reviewed record of science sign in
Pith

arxiv: 2606.02747 · v1 · pith:XLQQVJCU · submitted 2026-06-01 · cs.CV · cs.AI

Plan2Map: A Multimodal Benchmark for Document-Grounded Geospatial Boundary Reconstruction from Planning Records

Fabian Degen , Oishi Deb , Jindong Gu , Junchi Yu , Samuele Marro , Philip Torr , Jialin Yu This is my paper

Reviewed by Pith2026-06-28 14:53 UTCgrok-4.3pith:XLQQVJCUopen to challenge →

classification cs.CV cs.AI
keywords Plan2Mapgeospatial boundary reconstructionmultimodal benchmarkplanning recordsdocument-grounded reconstructionGeoPlanAgentIoU evaluationVLM baselines
0
0 comments X

The pith

GeoPlanAgent reconstructs geospatial boundaries from UK planning documents at 0.736 mean IoU by chaining evidence extraction with map tools.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Plan2Map, a benchmark of 208 cases from UK planning records that requires systems to derive valid geospatial polygons solely from notice text, map plates, labels, and annotations. It proposes GeoPlanAgent as a system that breaks the task into sequential stages of evidence extraction, localisation, registration, segmentation, projection, and verification. The agent reaches 0.736 mean IoU and 0.904 median IoU on held-out references, with most predictions above 0.8 IoU, while direct vision-language model outputs fall short. A sympathetic reader would care because planning records encode legal spatial restrictions that remain hard to convert into machine-readable form. If the decomposition approach holds, public document archives could be turned into usable geospatial data without manual digitization.

Core claim

The paper claims that a document-grounded agent called GeoPlanAgent, which integrates multimodal evidence from planning records with geospatial tools in a sequential loop, produces boundary polygons whose intersection-over-union with held-out reference GeoJSON reaches 0.736 on average and exceeds 0.8 in two-thirds of cases, substantially above direct VLM-to-GeoJSON baselines; diagnostic breakdowns locate remaining shortfalls mainly in localisation and map registration while showing that supervised segmentation lifts mask quality.

What carries the argument

GeoPlanAgent, a tool-in-the-loop system that decomposes reconstruction into evidence extraction, localisation, map registration, boundary segmentation, projection, and verification stages.

If this is right

  • Direct vision-language model prediction stays unreliable for extracting precise geospatial boundaries from mixed text-and-map documents.
  • The largest error sources lie in the localisation and map registration stages rather than in final polygon output.
  • Adding supervised boundary segmentation measurably raises pixel-level mask quality over end-to-end prediction.
  • Plan2Map functions as a reusable testbed for evaluating multimodal systems on public planning archives.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The staged decomposition may transfer to other document-to-map tasks such as zoning or environmental permit digitization.
  • Better map registration methods could close the remaining performance gap without changing the overall agent structure.
  • If scaled, the approach would allow automated monitoring of how planning restrictions change over time across jurisdictions.

Load-bearing premise

The held-out reference GeoJSON polygons constitute accurate ground-truth boundaries that can be validly derived from the multimodal evidence in the source planning documents.

What would settle it

An independent audit that compares a random subset of the benchmark's reference GeoJSON polygons against the original legal planning notices and finds frequent mismatches in boundary location or area.

Figures

Figures reproduced from arXiv: 2606.02747 by Fabian Degen, Jialin Yu, Jindong Gu, Junchi Yu, Oishi Deb, Philip Torr, Samuele Marro.

Figure 1
Figure 1. Figure 1: Plan2Map overview. Plan2Map pairs source planning documents with verified GeoJSON boundaries and includes cases spanning different boundary shapes, document formats, and scan qualities. Given only the source document as input, a system must reconstruct the corresponding geospatial boundary. planning datasets, and audit or update records over time. The source documents usually provide only indirect spatial … view at source ↗
Figure 2
Figure 2. Figure 2: GeoPlanAgent workflow. GeoPlanAgent decomposes Plan2Map into document evidence extraction, geographic localisation, map-to-basemap registration, boundary segmentation, and GeoJSON projection. The Reader converts the source PDF into structured spatial evidence; the Worker calls the Locate sub-agent, runs map matching and boundary extraction tools, and commits a candidate GeoJSON. The optional Critic reviews… view at source ↗
Figure 3
Figure 3. Figure 3: Boundary segmentation, scored against hand [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Mean IoU (filled) and fraction of cases at IoU [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Example of the visual input the Critic receives, [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Style-transfer augmentation applied to one [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Reader system prompt. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Worker system prompt. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Locate sub-agent system prompt. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Critic system prompt. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: VLM end-to-end baseline prompt. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
read the original abstract

Planning records define restrictions over geographic areas, but their source documents often provide only indirect spatial evidence rather than machine-readable boundaries. We introduce Plan2Map, a 208-case multimodal benchmark for document-grounded geospatial boundary reconstruction from UK planning records. Given only a source planning document, systems must reconstruct a valid geospatial boundary from notice text, schedules, map plates, map labels, and boundary annotations; the reference GeoJSON is held out for scoring. We propose GeoPlanAgent, a document-grounded, geospatial-tool-in-the-loop system that decomposes the task into evidence extraction, localisation, map registration, boundary segmentation, projection, and verification. On Plan2Map, GeoPlanAgent achieves 0.736 mean IoU and 0.904 median IoU, with 67.8\% of predictions at or above 0.8 IoU, substantially outperforming direct VLM-to-GeoJSON baselines. Diagnostic analysis shows that direct VLM prediction remains unreliable, while remaining errors are concentrated in localisation and map registration, and supervised boundary segmentation substantially improves pixel-level mask quality. Plan2Map provides a concrete testbed for multimodal geospatial reconstruction from public planning records. Project page: https://odeb1.github.io/Plan2Map_Project_Page/.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces Plan2Map, a 208-case multimodal benchmark for document-grounded geospatial boundary reconstruction from UK planning records. Given only source documents containing text, schedules, map plates, labels and annotations, systems must output valid GeoJSON boundaries; held-out reference polygons are used for evaluation. The authors propose GeoPlanAgent, a tool-in-the-loop agent that decomposes the task into evidence extraction, localisation, map registration, boundary segmentation, projection and verification, reporting 0.736 mean IoU, 0.904 median IoU and 67.8% of predictions ≥0.8 IoU, substantially above direct VLM-to-GeoJSON baselines.

Significance. If the reference GeoJSON polygons are verifiably derivable from the supplied multimodal document evidence alone, Plan2Map would constitute a useful public testbed for multimodal geospatial reasoning tasks. The diagnostic breakdown (localisation and registration as primary error sources, benefit of supervised segmentation) and the concrete performance gap versus direct VLM baselines would provide actionable guidance for future work on document-grounded mapping.

major comments (2)
  1. [Abstract / dataset section] Abstract and dataset construction section: No protocol is described for creating the held-out reference GeoJSON polygons, including inter-annotator agreement, resolution of ambiguous boundaries, or confirmation that every coordinate is derivable solely from the provided text, schedules, map plates and annotations without external GIS layers or author knowledge. This is load-bearing for the central claim that the reported IoU gap measures document-grounded reconstruction performance.
  2. [Evaluation / results section] Evaluation protocol (implied in abstract): The manuscript provides no information on how the 208 cases were selected, whether the split is stratified, or any statistical testing (confidence intervals, significance of the gap versus baselines). Without these details the headline metrics (0.736 mean IoU, 67.8% ≥0.8 IoU) cannot be assessed for robustness.
minor comments (1)
  1. [Abstract] The project page URL is given but no statement appears on data licensing or release of the benchmark itself.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful and constructive review. The two major comments highlight important gaps in transparency around dataset construction and evaluation details. We address each point below and will revise the manuscript accordingly to strengthen the work.

read point-by-point responses

  1. Referee: [Abstract / dataset section] Abstract and dataset construction section: No protocol is described for creating the held-out reference GeoJSON polygons, including inter-annotator agreement, resolution of ambiguous boundaries, or confirmation that every coordinate is derivable solely from the provided text, schedules, map plates and annotations without external GIS layers or author knowledge. This is load-bearing for the central claim that the reported IoU gap measures document-grounded reconstruction performance.

    Authors: We agree this protocol description is essential. The reference GeoJSON polygons were produced by geospatial experts using only the supplied multimodal planning documents (text, schedules, map plates, labels and annotations) with no external GIS layers or author prior knowledge. In the revised manuscript we will add a dedicated subsection describing the full annotation workflow, inter-annotator agreement statistics, the consensus procedure for resolving ambiguous boundaries, and explicit confirmation that all coordinates derive solely from the provided evidence. This will directly support the central claim. revision: yes

  2. Referee: [Evaluation / results section] Evaluation protocol (implied in abstract): The manuscript provides no information on how the 208 cases were selected, whether the split is stratified, or any statistical testing (confidence intervals, significance of the gap versus baselines). Without these details the headline metrics (0.736 mean IoU, 67.8% ≥0.8 IoU) cannot be assessed for robustness.

    Authors: We acknowledge the need for these details. The 208 cases were chosen for diversity across document types and UK regions. In revision we will describe the selection criteria, confirm the evaluation set is stratified by document complexity and geography, add bootstrap 95% confidence intervals for all metrics, and include paired statistical significance tests (e.g., Wilcoxon) comparing GeoPlanAgent against the VLM baselines. These additions will allow proper assessment of robustness. revision: yes

Circularity Check

0 steps flagged

No circularity: benchmark metrics rely on held-out external references

full rationale

The paper presents an empirical benchmark (Plan2Map) and an agent system (GeoPlanAgent) evaluated via IoU against held-out GeoJSON references created from planning documents. No equations, parameter fits, self-citation chains, or ansatzes are described that would reduce the reported performance numbers to quantities defined by the authors' own prior outputs or by construction. The evaluation uses independent geospatial tools and held-out data, so the central claims remain self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper rests on the domain assumption that source documents contain reconstructible boundary information and on standard computer-vision evaluation practices; no free parameters, invented entities, or ad-hoc axioms are introduced beyond those implicit in multimodal agent design.

axioms (1)
  • domain assumption Source planning documents contain sufficient multimodal evidence (text, schedules, map plates, labels, annotations) to reconstruct valid geospatial boundaries.
    This premise is required for the benchmark task to be well-posed and is invoked in the problem definition.

pith-pipeline@v0.9.1-grok · 5778 in / 1350 out tokens · 28379 ms · 2026-06-28T14:53:17.386797+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 1 canonical work pages

  1. [1]

    Extract: Using ai to unlock historic planning data. Samuel Colvin, Eric Jolibois, Hasan Ramezani, Adrian Garcia Badaracco, Terrence Dorsey, David Mon- tague, Serge Matveenko, Marcelo Trylesinski, Syd- ney Runkle, David Hewitt, Alex Hall, and Victorien Plot. 2026. Pydantic Validation. Martin A. Fischler and Robert C. Bolles. 1981. Ran- dom sample consensus...

    2026

  2. [2]

    Shape Matches

    Legalbench: A collaboratively built bench- mark for measuring legal reasoning in large language models. InAdvances in Neural Information Process- ing Systems. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778. ...

    work page arXiv 2016

  3. [3]

    planning boundary

    This diversity-bucketed top-K scheme prevents one strong configuration from saturating every slot with near-duplicate windows. Composite re-rank.The surviving top- K can- didates are re-ranked in two passes. First, each candidate’s RANSAC inlier countV is multiplied by a quadrant-coverage factor Q/4, where Q∈ {0,1,2,3,4} is the number of map quadrants con...

    2025

  4. [4]

    propose_centers() -- get one ranked candidate (lat/lon/sigma_m/source)

  5. [5]

    no OS roads within radius

    match_at(page=N, name, lat, lon). Returns candidate_id, area_group, page, n_inliers, scale_consistency, road_name_agreement (+ road_name_verdict), committed_groups, budget_remaining. THREE SIGNALS -- explicit tiers: n_inliers (RANSAC match strength): >= 100 STRONG -- commit on this attempt unless another signal disagrees. 50-99 OK -- commit ONLY after try...

  6. [6]

    STRONG n_inliers + GOOD scale + STRONG/NEUTRAL roads -> commit

  7. [7]

    Otherwise try another propose_centers candidate for this group BEFORE committing anything below STRONG

  8. [8]

    After 2+ attempts: highest n_inliers wins; tie-break on scale_consistency closer to 1.0, then on road_name_agreement

  9. [9]

    Re-calling overwrites the slot

    commit_match(candidate_id) -- commits ONE candidate for its area_group. Re-calling overwrites the slot

  10. [10]

    accepted

    Return BoundaryOutcome with status="accepted" (or "district_lookup" if you took the lookup_district path). BUDGET: max 5 match_at calls per case. NO INVENTED COORDINATES -- every (lat, lon) must come from propose_centers. To add a missing place call propose_centers(extra_terms=["..."]). After a weak match, call propose_centers(match_context="...") describ...

  11. [11]

    Look for labels, landmarks, distinctive features, road junctions, named,→ buildings, hatched site polygon, neighbouring features

    VIEW the map image carefully. Look for labels, landmarks, distinctive features, road junctions, named,→ buildings, hatched site polygon, neighbouring features. Note ANYTHING that's on the map but missing from,→ pdf_info

  12. [12]

    Priority of signals (most specific first): - house_number + named road in site_address - Named place / landmark from pdf_info OR from the map image - Parish name

    SCAN pdf_info. Priority of signals (most specific first): - house_number + named road in site_address - Named place / landmark from pdf_info OR from the map image - Parish name

  13. [13]

    Aim for 2-4 candidates from different signal types

    BUILD POOL via tool calls. Aim for 2-4 candidates from different signal types. Augment with terms FROM,→ THE MAP IMAGE (do not limit yourself to pdf_info)

  14. [14]

    - Single ambiguous (common place) -> sigma=800-1500m, confidence='med'

    CLUSTER & PICK: - 2+ candidates within 500m -> tight consensus, sigma=200m, confidence='high'. - Single ambiguous (common place) -> sigma=800-1500m, confidence='med'

  15. [15]

    CANDIDATE {id} [COMMITTED] group {g} page {p}

    EMIT the LocatePick to terminate. Once you have your pick, output the LocatePick directly as your final,→ response -- do NOT make further tool calls. Be meticulous: copy the lat/lon EXACTLY from your strongest,→ tool result -- do not paraphrase, do not round prematurely. Common bugs: (a) dropping a minus sign that,→ should be there (-0.14 emitted as 0.14)...

  16. [16]

    * Shape (rectangular, L-shaped, multiple disjoint parcels, elongated strip along the river, etc.)

    Note:,→ * Line style (red solid outline, hatched red, dashed blue, filled pink, black dot-dash). * Shape (rectangular, L-shaped, multiple disjoint parcels, elongated strip along the river, etc.). * If a printed scale is available, it can be useful for estimating the boundary's real-world size. ==============================================================...