{"id":"42d2bdd7-fc4a-4da9-9a46-68fd6cff943d","arxiv_id":"2507.04664","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VectorLLM, a multimodal LLM that regresses building contour vertices token by token, reports gains of 5.6 to 13.6 AP over prior polygon extraction methods on WHU, WHU-Mix, and CrowdAI.","lead":"A new model called VectorLLM makes a multimodal language model draw building outlines corner by corner, like a human annotator, and reports large accuracy gains over earlier methods on three remote sensing benchmarks. The catch is that a separate detector pre-crops each building for the model, and some of the claimed zero-shot objects may have appeared in the COCO pretraining data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The detector-crop protocol confounds the headline AP gains: baselines are never run on the same single-building crops, so the 5.6/7.1/13.6 AP margins may reflect easier inputs rather than LLM contour quality.","rationale":"The reader's weakest_assumption correctly identifies the most load-bearing concern: the comparison protocol is not apples-to-apples. VectorLLM receives single-building crops from a separately trained detector, while prior contour extraction methods must operate on full images. This confound directly threatens the paper's strongest quantitative claim, because the crop protocol removes the detection and instance-separation burden, as evidenced by the large gap between oracle-bbox and detector-based results in Table 1 (90.2 vs 79.4 AP). The proposed concrete test—running the same baselines on the same crops—would settle whether the LLM's contour regression actually adds value beyond the easier input format. I agree with the reader that this makes the paper CONDITIONAL rather than a rejection: the architecture is simple, the ablations are informative, and the failure cases are honestly reported. The zero-shot concern is secondary but real; however, it does not affect the main quantitative comparison. My recommendation is UNCHANGED: the paper should remain CONDITIONAL pending the controlled baseline experiment and the release of code and weights.","tokens_in":20998,"tokens_out":3703,"duration_ms":40858,"concrete_test":"Run P2PFormer, Line2Poly, and BuildMapper with their released code on the exact detector-generated single-building crops (FCOS/Swin-L boxes, 1.3× enlargement) and compute AP with the same evaluation script used for VectorLLM on each dataset. If their crop-based AP matches or exceeds VectorLLM's detector-based numbers (79.4/68.4/79.6), the headline gain is a protocol artifact. Additionally, report the FCOS detector's recall to quantify the upper bound the detector imposes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is the SOTA AP improvements in Tables 1–3. However, VectorLLM is tested under a fundamentally easier protocol: §3.5 and §4.1 state that an external FCOS/Swin-L detector crops each building (1.3× enlargement) and VectorLLM regresses a single contour from that crop. Baselines such as P2PFormer, Line2Poly, and BuildMapper are evaluated end-to-end from full images, so they must solve detection, instance separation, and contour extraction jointly. The paper never reports the detector recall, nor does it run the baselines on the same crops. The oracle-bbox rows in Table 1 (90.2 vs 79.4 AP) show that the crop protocol alone moves AP by ~11 points, so the reported margins (5.6, 7.1, 13.6) cannot be cleanly attributed to the LLM or the point-by-point formulation. Without the controlled comparison, the 'LLM paradigm' claim is not established. (The zero-shot claim is additionally weakened by COCO pretraining in §4.1, since aircraft and ships appear in COCO.)","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"VectorLLM introduces a multimodal large language model that extracts regular building contours by autoregressively predicting corner-point coordinates from single-building image crops. The architecture combines a RADIO vision encoder, learnable positional embeddings, an MLP projector, and a Qwen3-based language model, with coordinate-specific tokens added to the vocabulary. Training proceeds through pretraining, supervised fine-tuning, and direct preference optimization, using the WHU, WHU-Mix, CrowdAI, and COCO datasets. The paper reports state-of-the-art AP on the three building datasets and shows qualitative zero-shot contour extraction on aircraft, ships, water bodies, and other objects. The work also includes ablations of the positional embedding, vision encoder, training stages, model size, coordinate tokenization, and data scaling.","tokens_in":21224,"tokens_out":6397,"duration_ms":68092,"significance":"If the reported comparisons were controlled, this would be a significant result: a detector-plus-autoregressive-LLM system could replace multi-stage segmentation, vectorization, and refinement pipelines, and the oracle-box results (90.2 AP on WHU) suggest that the contour regression component itself is highly accurate. The ablations are systematic and the DPO preference-data construction is standard. The main weakness is that the headline SOTA margins compare VectorLLM under a detector-crop protocol to baselines evaluated end-to-end on full images, so the margins cannot yet be cleanly attributed to the LLM or the point-by-point task formulation. The zero-shot claim is also weakened by the inclusion of COCO in pretraining. A controlled comparison under the same protocol is needed before the central claim is established.","major_comments":[{"comment":"The headline SOTA margins are not based on a controlled comparison. In §3.5 and §4.1, VectorLLM is tested on single-building crops produced by an external FCOS/Swin-L detector with 1.3× box enlargement, while prior methods such as P2PFormer, Line2Poly, and BuildMapper are evaluated end-to-end from full images. The paper does not report the detector's recall or run any baseline on the same detector-crop protocol, so the 5.6, 7.1, and 13.6 AP margins in Tables 1–3 may reflect the external detector's localization and instance separation rather than the LLM's contour quality. The oracle-bbox row in Table 1 (90.2 vs. 79.4 AP) further shows that bounding-box quality alone changes AP by about 11 points, so detector effects are material. The revision should report AP for P2PFormer, Line2Poly, and BuildMapper on the same FCOS crops, and should also report detector recall, before claiming that the LLM formulation is responsible for the gains.","section":"§3.5, §4.1, Tables 1–3"},{"comment":"The zero-shot claim is overstated because the model is pretrained on COCO. Section 4.1 states that VectorLLM is jointly trained on WHU, WHU-Mix, CrowdAI, and COCO training datasets, and Section 4.4 describes the results on aircraft, ships, water bodies, and oil tanks as objects that 'never appear in the training set.' However, COCO contains airplanes and ships, so those categories are not unseen if COCO instance masks are converted into contour sequences under the pretraining protocol described in §3.2. The paper should either exclude COCO categories from the zero-shot evaluation, report which COCO categories are actually used, or qualify the 'unseen' claim to categories absent from all training data.","section":"§4.1, §4.4, Figure 5"}],"minor_comments":[{"comment":"The title contains a typo ('vis' should be 'via'), and the abstract's 'significantly outperformed' should be rephrased as 'outperforms' or 'outperformed' with a clear subject.","section":"Title/Abstract"},{"comment":"The column headers for WHU, CrowdAI, and WHU-Mix do not indicate the metric; 'AP' should be included in each numeric column or specified in the caption.","section":"Table 7"},{"comment":"The captions state that the black bounding boxes are manually provided by users, but Section 4.1 describes a detector-based test protocol; the paper should clarify how the manual-box visualizations relate to the quantitative FCOS-based results and whether the qualitative figures use manual or detector boxes.","section":"Figures 1, 5, and 6"},{"comment":"The DPO data-generation rule 'IoU less than 0.8' should specify whether the IoU is computed between the predicted polygon and the ground-truth polygon, and in which coordinate space (image pixels or normalized coordinates).","section":"§3.4"},{"comment":"The check-mark rows in the tuning-strategy ablation are not self-explanatory; a row label such as 'Pretraining only' would help readers interpret the effect of each training stage.","section":"§4.3, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript comes from a group with strong prior work on building contour extraction, and the oracle-box results suggest that the contour regression module is genuinely capable. My main concern is evaluation fairness: the current comparison does not allow the reported SOTA margins to be attributed to the proposed method. I would ask the editor to require the same-protocol comparison with baselines on the detector-crop inputs, and to request that the authors either avoid claiming zero-shot on COCO categories or exclude those categories from the zero-shot evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is the first MLLM for fine-grained vector contour extraction in remote sensing, and the corner-by-corner coordinate-token regression formulation is a real departure from prior multi-stage pipelines. Second, the headline SOTA margins are not cleanly supported: VectorLLM is tested on single-building crops produced by an external FCOS/Swin-L detector, while the baselines are evaluated end-to-end on full images. The oracle-bbox rows in Table 1 show that the crop protocol alone moves AP by about 11 points, so the claimed 5.6/7.1/13.6 AP gains likely overstate what the LLM contributes to contour quality. The paper never reports detector recall or runs the baselines on the same crops, and that is a load-bearing gap for the central claim.\n\nWhat the paper does well: the architecture is simple, the training-stage ablations (pretraining, SFT, DPO) are sensible and clearly described, and the failure cases are discussed honestly. The DPO data construction from the model's own low-IoU outputs is a standard preference-learning setup, not circular. The zero-shot section is interesting qualitatively, but the claim is weakened because the model sees COCO during pretraining and COCO contains aircraft and ships. The zero-shot eval is only qualitative, with no metrics, so the word 'strong' is doing too much work.\n\nSoft spots, in proportion: the comparison-protocol confound is the big one and needs a controlled experiment. The zero-shot claim should be trimmed or quantified on non-COCO categories. Also, the paper promises code and weights but ships neither, which limits reproducibility. The 128x128 input is acknowledged as a limit for large buildings, which is fine.\n\nWho this is for: anyone working on remote sensing vector mapping or MLLMs for structured output tasks. The idea will likely be built on, and the ablations are useful even if the SOTA numbers are not yet trustworthy. This deserves a serious referee, but with a clear request: run the baselines on the same detector crops, report detector recall, and revise the zero-shot framing. I would not desk reject this; I would send it back for major revision.","headline":"A genuinely new MLLM-based approach to building contour extraction, but the headline AP gains are inflated by an unfair crop protocol and the zero-shot claim outruns the evidence.","tokens_in":21754,"tokens_out":2250,"would_cite":true,"duration_ms":25709,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a multimodal LLM that outputs building contours corner-point by corner-point beats multi-stage segmentation-vectorization pipelines by 5.6-13.6 AP and zero-shots to unseen objects.","keywords":["building contour extraction","multimodal large language model","remote sensing imagery","corner-point regression","zero-shot generalization","preference optimization","vectorization"],"falsifier":"Run P2PFormer, Line2Poly, and BuildMapper under the identical protocol—FCOS/Swin-L detections, 1.3x enlarged crops, single-building inputs—and compare AP on WHU, WHU-Mix, and CrowdAI; if the margins shrink to near zero, the central claim of LLM superiority for contour extraction does not hold. A complementary check would be to feed VectorLLM full scene images without any detector crops and measure the resulting drop, which would quantify how much of the system's capability depends on the external detector.","tokens_in":20776,"feed_emoji":"🏢","tokens_out":5761,"duration_ms":56515,"temperature":0.7,"pith_summary":"This paper claims that a multimodal large language model, fed a single cropped building image, can write out the building's vector outline as a sequence of corner-point coordinates, the way a human annotator might, and that this direct approach outperforms the standard multi-stage pipeline of pixel segmentation, vectorization, and polygon refinement. The proposed system, VectorLLM, is trained in three stages: pretraining, supervised fine-tuning, and preference optimization, and a single set of weights beats previous state-of-the-art methods by 5.6 AP, 7.1 AP, and 13.6 AP on the WHU, WHU-Mix, and CrowdAI datasets. The paper further reports that the same model draws plausible contours for objects it never trained on, such as aircraft, water bodies, and oil tanks. If these results hold, vector extraction from remote sensing imagery would become a single learned step rather than a fragile assembly of separate algorithms, possibly extending to roads, water, and other geographic features.","feed_headline":"MLLM draws building outlines one corner at a time","feed_subtitle":"VectorLLM beats building-extraction pipelines by 5.6-13.6 AP and draws contours it never saw in training.","key_machinery":"The load-bearing mechanism is corner-by-corner autoregressive decoding into a coordinate vocabulary: the LLM emits one special token per x- or y-value, which forces the model to commit to a spatial position at each step and lets it use the already-emitted corners as context, exactly like a human tracing a polygon. Three training stages carry the argument: pretraining aligns vision features to the LLM with plain coordinate-string targets, supervised fine-tuning imposes a fixed orientation and starting point (top-left, clockwise) so outputs are regular and stable, and direct preference optimization (DPO) on automatically constructed chosen/rejected pairs cleans up repeated outputs, missing small lines, and roof-plane interference. An upstream FCOS detector with a Swin-L backbone crops each building instance, with boxes enlarged 1.3x, so the model sees one prominent building per image. The 128x128 input resolution and one-token-per-coordinate formulation keep the design deliberately simple.","core_discovery":"VectorLLM recasts regular building contour extraction as an autoregressive language modeling task: the model predicts the next corner point's x- and y-coordinates, expressed as special tokens, conditioned on the image and the previously emitted corners. The architecture attaches a RADIO vision encoder to a Qwen3 LLM through an MLP projector, adds learnable position embeddings to the vision tokens, and trains with next-token prediction on coordinate strings, then on instruction-style data, then with direct preference optimization on chosen versus rejected contours. On the three benchmark datasets the model reports 79.4 AP, 68.4/56.4 AP (in-domain/out-of-domain), and 79.6 AP with detector-supplied boxes, and 90.2, 78.5/64.6, and 87.7 AP with oracle boxes, exceeding prior methods across the board. The authors argue this demonstrates that LLM-based topological reasoning is a qualitative leap for vector extraction, not an incremental gain.","pith_inferences":["The reported gains compare detector-cropped VectorLLM against prior methods evaluated end-to-end on full images; with the same crop protocol, the baselines might narrow the gap, so the cleanest attribution of the improvement to the LLM itself requires a matched evaluation.","Because VectorLLM cannot yet find the buildings it delineates, the system's end-to-end accuracy is capped by the external detector; making the LLM itself localize instances would be the natural next step toward a true single-model pipeline.","The fixed 128x128 input resolution is the likely cause of the reported failures on large and complex buildings; a multi-resolution or patch-based visual front end could raise the ceiling on fine detail.","The zero-shot success on objects with very different geometry suggests the model may have learned a general straight-line-polygon prior during building training; testing on non-polygonal or curved boundaries, such as meandering rivers, would show how general that prior really is."],"forward_implications":["Building contour extraction collapses from three pipeline stages to one trained model: no pixel segmentation, no separate vectorization, and no polygon refinement step.","A single co-trained model with one set of weights transfers across datasets with different annotation standards and image styles, as shown by the out-of-domain WHU-Mix test-2 score.","Zero-shot contours for aircraft, water bodies, oil tanks, roads, and general objects imply the corner-point skill generalizes beyond buildings, opening a route to unified vector extraction for multiple geographic feature types.","Scaling the LLM from 0.6B to 1.7B parameters improves AP on every dataset, so further scaling is expected to yield more gains.","Preference optimization contributes a measurable ~1.8 AP gain and specifically fixes corner cases like duplicated nodes and missed small edges, so alignment techniques transfer to geometric outputs."],"supporting_citations":[{"why":"P2PFormer is the primary prior SOTA baseline that VectorLLM must beat on all three datasets.","marker":"[Zhang et al., 2024c]"},{"why":"Line2Poly is the prior SOTA on WHU and a key contour-extraction comparison.","marker":"[Wei et al., 2024]"},{"why":"BuildMapper represents the multi-stage segmentation-to-vector pipeline paradigm the paper argues against.","marker":"[Wei et al., 2023]"},{"why":"The WHU dataset is one of the three training and evaluation benchmarks.","marker":"[Ji et al., 2018]"},{"why":"The CrowdAI dataset provides the low-resolution, noisy-annotation test setting.","marker":"[Mohanty, 2019 (accessed November 10, 2019)]"},{"why":"FCOS is the detector that supplies building crops to VectorLLM, essential to the test protocol.","marker":"[Tian et al., 2019]"},{"why":"Swin-L is the backbone of the FCOS detector used for cropping.","marker":"[Liu et al., 2021]"},{"why":"RADIO is the vision encoder that provides semantic and low-level features to the LLM.","marker":"[Ranzinger et al., 2024]"},{"why":"Qwen3 is the LLM backbone that performs autoregressive coordinate prediction.","marker":"[Yang et al., 2025]"},{"why":"Direct preference optimization is the method used to train the final preference-aligned model.","marker":"[Rafailov et al., 2024]"}],"fun_headline_variants":["Autoregressive LLM extracts building contours corner by corner","VectorLLM: corner-by-corner building mapping from images","LLM mimics human annotators to trace building outlines","Next-corner prediction for building contours in remote sensing","Multimodal LLM draws building polygons like a human labeller"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline gains assume the comparison protocol is fair: VectorLLM is evaluated on single-building crops produced by a separately trained detector, while the prior methods it beats are evaluated in their native end-to-end setting on full images, and the paper does not report those baselines under the same crop-and-recall protocol.","fun_headline_variants_meta":{"raw":{"variants":["Autoregressive LLM extracts building contours corner by corner","VectorLLM: corner-by-corner building mapping from images","LLM mimics human annotators to trace building outlines","Next-corner prediction for building contours in remote sensing","Multimodal LLM draws building polygons like a human labeller"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000343,"raw_usage":{"total_tokens":1931,"prompt_tokens":1034,"completion_tokens":897,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":815}},"tokens_in":650,"tokens_out":897,"duration_ms":10053,"temperature":1.0,"reasoning_tokens":815,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:42:20.162075+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run P2PFormer, Line2Poly, and BuildMapper under the identical protocol—FCOS/Swin-L detections, 1.3x enlarged crops, single-building inputs—and compare AP on WHU, WHU-Mix, and CrowdAI; if the margins shrink to near zero, the central claim of LLM superiority for contour extraction does not hold. A complementary check would be to feed VectorLLM full scene images without any detector crops and measure the resulting drop, which would quantify how much of the system's capability depends on the external detector.","supporting_citations":[{"cited_title":"From lines to Polygons: Polygonal building contour extraction from High-Resolution remote sensing imagery","cited_arxiv_id":null,"evidence_quote":"Line2Poly is the prior SOTA on WHU and a key contour-extraction comparison."},{"cited_title":"BuildMapper: A fully learnable framework for vectorized building contour extraction","cited_arxiv_id":null,"evidence_quote":"BuildMapper represents the multi-stage segmentation-to-vector pipeline paradigm the paper argues against."},{"cited_title":"P., 2019 (accessed November 10, 2019)","cited_arxiv_id":null,"evidence_quote":"The CrowdAI dataset provides the low-resolution, noisy-annotation test setting."},{"cited_title":"Fcos: Fully convolutional one-stage object detection","cited_arxiv_id":null,"evidence_quote":"FCOS is the detector that supplies building crops to VectorLLM, essential to the test protocol."},{"cited_title":"Swin transformer: Hierarchical vision transformer using shifted windows","cited_arxiv_id":null,"evidence_quote":"Swin-L is the backbone of the FCOS detector used for cropping."},{"cited_title":"Am-radio: Agglomerative vision foundation model reduce all domains into one","cited_arxiv_id":null,"evidence_quote":"RADIO is the vision encoder that provides semantic and low-level features to the LLM."},{"cited_title":"D., Ermon, S., Finn, C., 2024","cited_arxiv_id":null,"evidence_quote":"Direct preference optimization is the method used to train the final preference-aligned model."}],"review_version":1}