{"id":"ec0faca0-d767-4dc4-82c1-c50810653978","arxiv_id":"2606.02747","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Plan2Map is a 208-case multimodal benchmark for document-grounded geospatial boundary reconstruction from UK planning records, where the proposed GeoPlanAgent system reaches 0.736 mean IoU.","lead":"The paper introduces Plan2Map, a benchmark of 208 UK planning documents requiring reconstruction of geospatial boundaries from text, schedules, map plates, labels, and annotations. GeoPlanAgent, a tool-in-the-loop agent, achieves 0.736 mean IoU and outperforms direct VLM baselines on this task.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Ground-truth GeoJSON fidelity to document-only evidence not independently validated","rationale":"Reader's weakest assumption matches the load-bearing precondition for any benchmark claim. Full-text methods and data sections would need to contain explicit validation of reference provenance for the claim to be secure; absent that, the numeric results remain provisional regardless of system architecture.","tokens_in":1744,"tokens_out":293,"duration_ms":12161,"concrete_test":"Sample 30 Plan2Map cases; have two independent geospatial experts each produce a GeoJSON boundary using only the released document images/text and the same coordinate system; compute mean IoU between their outputs and the held-out references. If expert-expert or expert-reference mean IoU falls below 0.75, the benchmark validity claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline metrics (0.736 mean IoU, 67.8% ≥0.8 IoU) treat the held-out reference polygons as the unique correct boundaries reconstructible from the supplied multimodal evidence (text, schedules, map plates, labels, annotations). No section describes the reference-creation protocol, inter-annotator agreement, or an audit confirming that every reference coordinate is derivable without external GIS layers or author knowledge. If references resolve ambiguities or incorporate information absent from the documents, the reported gap versus direct VLM baselines is not guaranteed to measure document-grounded reconstruction.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces Plan2Map, a 208-case multimodal benchmark for document-grounded geospatial boundary reconstruction from UK planning records. Given only source documents containing text, schedules, map plates, labels and annotations, systems must output valid GeoJSON boundaries; held-out reference polygons are used for evaluation. The authors propose GeoPlanAgent, a tool-in-the-loop agent that decomposes the task into evidence extraction, localisation, map registration, boundary segmentation, projection and verification, reporting 0.736 mean IoU, 0.904 median IoU and 67.8% of predictions ≥0.8 IoU, substantially above direct VLM-to-GeoJSON baselines.","tokens_in":1866,"tokens_out":487,"duration_ms":15165,"significance":"If the reference GeoJSON polygons are verifiably derivable from the supplied multimodal document evidence alone, Plan2Map would constitute a useful public testbed for multimodal geospatial reasoning tasks. The diagnostic breakdown (localisation and registration as primary error sources, benefit of supervised segmentation) and the concrete performance gap versus direct VLM baselines would provide actionable guidance for future work on document-grounded mapping.","major_comments":[{"comment":"Abstract and dataset construction section: No protocol is described for creating the held-out reference GeoJSON polygons, including inter-annotator agreement, resolution of ambiguous boundaries, or confirmation that every coordinate is derivable solely from the provided text, schedules, map plates and annotations without external GIS layers or author knowledge. This is load-bearing for the central claim that the reported IoU gap measures document-grounded reconstruction performance.","section":"Abstract / dataset section"},{"comment":"Evaluation protocol (implied in abstract): The manuscript provides no information on how the 208 cases were selected, whether the split is stratified, or any statistical testing (confidence intervals, significance of the gap versus baselines). Without these details the headline metrics (0.736 mean IoU, 67.8% ≥0.8 IoU) cannot be assessed for robustness.","section":"Evaluation / results section"}],"minor_comments":[{"comment":"The project page URL is given but no statement appears on data licensing or release of the benchmark itself.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful and constructive review. The two major comments highlight important gaps in transparency around dataset construction and evaluation details. We address each point below and will revise the manuscript accordingly to strengthen the work.","responses":[{"response":"We agree this protocol description is essential. The reference GeoJSON polygons were produced by geospatial experts using only the supplied multimodal planning documents (text, schedules, map plates, labels and annotations) with no external GIS layers or author prior knowledge. In the revised manuscript we will add a dedicated subsection describing the full annotation workflow, inter-annotator agreement statistics, the consensus procedure for resolving ambiguous boundaries, and explicit confirmation that all coordinates derive solely from the provided evidence. This will directly support the central claim.","revision_made":"yes","referee_comment":"[Abstract / dataset section] Abstract and dataset construction section: No protocol is described for creating the held-out reference GeoJSON polygons, including inter-annotator agreement, resolution of ambiguous boundaries, or confirmation that every coordinate is derivable solely from the provided text, schedules, map plates and annotations without external GIS layers or author knowledge. This is load-bearing for the central claim that the reported IoU gap measures document-grounded reconstruction performance."},{"response":"We acknowledge the need for these details. The 208 cases were chosen for diversity across document types and UK regions. In revision we will describe the selection criteria, confirm the evaluation set is stratified by document complexity and geography, add bootstrap 95% confidence intervals for all metrics, and include paired statistical significance tests (e.g., Wilcoxon) comparing GeoPlanAgent against the VLM baselines. These additions will allow proper assessment of robustness.","revision_made":"yes","referee_comment":"[Evaluation / results section] Evaluation protocol (implied in abstract): The manuscript provides no information on how the 208 cases were selected, whether the split is stratified, or any statistical testing (confidence intervals, significance of the gap versus baselines). Without these details the headline metrics (0.736 mean IoU, 67.8% ≥0.8 IoU) cannot be assessed for robustness."}],"tokens_in":1455,"tokens_out":465,"duration_ms":24371,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core of this paper is a fresh benchmark of 208 UK planning records that bundles notice text, schedules, map plates, labels, and annotations, with held-out GeoJSON for scoring boundary reconstruction. GeoPlanAgent breaks the job into evidence extraction, localisation, map registration, segmentation, projection, and verification, then reports 0.736 mean IoU and 0.904 median IoU while beating direct VLM-to-GeoJSON runs, with 67.8% of outputs at or above 0.8 IoU.\n\nThe work does a few things cleanly. The step-wise decomposition and the diagnostics that point to localisation and registration as main error sources are useful, and the note that supervised boundary segmentation lifts pixel masks is a concrete takeaway. The task itself is practical for regulatory document pipelines, and the held-out reference setup avoids some obvious circularity.\n\nThe soft spot is the ground-truth protocol. Nothing in the abstract or the stress-test note describes how the reference polygons were drawn, whether multiple annotators agreed, or whether every coordinate can be derived strictly from the supplied documents without external GIS data or author knowledge. If the references resolve ambiguities using information outside the multimodal evidence, the gap versus baselines no longer cleanly measures document-grounded performance. Dataset construction details, statistical tests, and baseline implementation notes are also absent, so the evaluation cannot be fully assessed yet.\n\nThis is aimed at people working on multimodal document understanding or applied geospatial AI. It is worth a serious referee because the benchmark is new and the agent results give something to test against, provided the methods section supplies the missing validation steps for the references.","headline":"Plan2Map gives a new 208-case benchmark for boundary reconstruction from UK planning docs and shows a tool-using agent beating direct VLMs on IoU, but ground-truth creation details are missing.","tokens_in":2375,"tokens_out":414,"would_cite":false,"duration_ms":18839,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"GeoPlanAgent reconstructs geospatial boundaries from UK planning documents at 0.736 mean IoU by chaining evidence extraction with map tools.","keywords":["Plan2Map","geospatial boundary reconstruction","multimodal benchmark","planning records","document-grounded reconstruction","GeoPlanAgent","IoU evaluation","VLM baselines"],"falsifier":"An independent audit that compares a random subset of the benchmark's reference GeoJSON polygons against the original legal planning notices and finds frequent mismatches in boundary location or area.","tokens_in":2649,"feed_emoji":"🗺️","tokens_out":702,"duration_ms":18055,"temperature":0.7,"pith_summary":"The paper introduces Plan2Map, a benchmark of 208 cases from UK planning records that requires systems to derive valid geospatial polygons solely from notice text, map plates, labels, and annotations. It proposes GeoPlanAgent as a system that breaks the task into sequential stages of evidence extraction, localisation, registration, segmentation, projection, and verification. The agent reaches 0.736 mean IoU and 0.904 median IoU on held-out references, with most predictions above 0.8 IoU, while direct vision-language model outputs fall short. A sympathetic reader would care because planning records encode legal spatial restrictions that remain hard to convert into machine-readable form. If the decomposition approach holds, public document archives could be turned into usable geospatial data without manual digitization.","feed_headline":"Agent reaches 0.9 median IoU reconstructing planning boundaries","feed_subtitle":"Plan2Map benchmark shows tool-in-the-loop extraction and registration beats direct VLM outputs on 208 UK cases","key_machinery":"GeoPlanAgent, a tool-in-the-loop system that decomposes reconstruction into evidence extraction, localisation, map registration, boundary segmentation, projection, and verification stages.","core_discovery":"The paper claims that a document-grounded agent called GeoPlanAgent, which integrates multimodal evidence from planning records with geospatial tools in a sequential loop, produces boundary polygons whose intersection-over-union with held-out reference GeoJSON reaches 0.736 on average and exceeds 0.8 in two-thirds of cases, substantially above direct VLM-to-GeoJSON baselines; diagnostic breakdowns locate remaining shortfalls mainly in localisation and map registration while showing that supervised segmentation lifts mask quality.","pith_inferences":["The staged decomposition may transfer to other document-to-map tasks such as zoning or environmental permit digitization.","Better map registration methods could close the remaining performance gap without changing the overall agent structure.","If scaled, the approach would allow automated monitoring of how planning restrictions change over time across jurisdictions."],"forward_implications":["Direct vision-language model prediction stays unreliable for extracting precise geospatial boundaries from mixed text-and-map documents.","The largest error sources lie in the localisation and map registration stages rather than in final polygon output.","Adding supervised boundary segmentation measurably raises pixel-level mask quality over end-to-end prediction.","Plan2Map functions as a reusable testbed for evaluating multimodal systems on public planning archives."],"fun_headline_variants":["GeoPlanAgent achieves 0.736 mean IoU on 208 UK planning cases","Agent outperforms direct VLM with 0.904 median IoU in Plan2Map","0.736 mean IoU from tool-augmented boundary reconstruction agent","GeoPlanAgent secures 67.8 percent above 0.8 IoU on Plan2Map"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The held-out reference GeoJSON polygons constitute accurate ground-truth boundaries that can be validly derived from the multimodal evidence in the source planning documents.","fun_headline_variants_meta":{"raw":{"variants":["GeoPlanAgent achieves 0.736 mean IoU on 208 UK planning cases","Agent outperforms direct VLM with 0.904 median IoU in Plan2Map","0.736 mean IoU from tool-augmented boundary reconstruction agent","GeoPlanAgent secures 67.8 percent above 0.8 IoU on Plan2Map"]},"model":"grok-4.3","cost_usd":0.008771,"raw_usage":{"total_tokens":3965,"prompt_tokens":698,"num_sources_used":0,"completion_tokens":89,"cost_in_usd_ticks":87712000,"prompt_tokens_details":{"text_tokens":698,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3178,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":698,"tokens_out":89,"duration_ms":23070,"temperature":1.0,"reasoning_tokens":3178,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T14:53:17.386797+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An independent audit that compares a random subset of the benchmark's reference GeoJSON polygons against the original legal planning notices and finds frequent mismatches in boundary location or area.","supporting_citations":[],"review_version":1}