{"id":"743e92f8-e6c4-4500-aef0-1ef41579e35f","arxiv_id":"2411.16616","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"GeoFormer generates multi-polygon building footprints directly from satellite images via an autoregressive transformer, reporting state-of-the-art average precision on the Aicrowd Mapping Challenge.","lead":"A new deep learning model, GeoFormer, reads a satellite image and directly outputs the corner coordinates of every building as a sequence, instead of first making a pixel map and then converting it to polygons. On a standard building detection benchmark, it reports substantially higher accuracy than previous methods, though the comparison may not be perfectly fair because the earlier numbers were not recomputed with the same evaluation pipeline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GeoFormer's headline AP depends on an unreported detection-confidence and sampling protocol; without same-script evaluation of baselines, the 12-point SOTA gap is unverifiable.","rationale":"The reader's weakest assumption was that the evaluation protocol is consistent across methods, focusing on baseline numbers and sampling. I agree that this is a serious concern, but I locate the deepest unverified step slightly earlier: the paper does not state how GeoFormer's own AP/AR are computed from an autoregressive generative model. COCO AP requires per-detection confidence scores and a defined set of detection hypotheses; neither the scoring function nor the number of sampled sequences per image is specified in Sections 4.1 or 5. Without those definitions, the headline metric is not reproducible even with the released code, because the evaluation script would have to be reverse-engineered from the repository. This is independent of whether the baselines were rerun: even a perfectly self-consistent GeoFormer row could be protocol-dependent. I do not regard the ablation discrepancy in Section 6 as the core issue, because the reduced rows in Table 2 match corresponding rows in Table 5 and the post hoc explanation for masking is plausible. The paper does ship code and acknowledges inference-speed limitations, which are points in its favor. The right verdict is conditional: the SOTA claim should be accepted only after the authors specify the confidence/sampling protocol and provide a same-script comparison of at least HiSup (the strongest baseline) under identical matching and scoring rules.","tokens_in":30552,"tokens_out":5049,"duration_ms":52114,"concrete_test":"Open the released GeoFormer evaluation code and identify the exact score assigned to each predicted polygon when computing AP (e.g., uniform, normalized sequence log-likelihood, or special-token probability) and the number of sampled sequences per image. Then take the released HiSup or PolyWorld predictions and run them through the same evaluation function, including the same rasterization, the same 0.5-overlap matching rule, and the same scoring convention. If the gap shrinks below 5 AP points, the comparability concern lands; if it remains at the reported level, the headline claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that GeoFormer outperforms prior work by almost 12 AP points rests on Table 1, but Section 4.1 defines the metrics without specifying the confidence score used to rank GeoFormer's predicted polygons. COCO AP and AR are rank-based: AP requires a per-detection score ordering, and AR requires a defined set of hypotheses per image. Section 5 only states that results use 'conditional inference' and nucleus sampling with p=0.95; it does not state how many sequences are sampled per image, whether one sequence or many, nor whether each predicted polygon is scored by sequence likelihood, a special-token probability, a uniform score, or some other rule. Table 1 further notes that baseline values are taken from the respective papers or from HiSup's authors, so differences in rasterization, the 50%-overlap polygon-matching filter, and sampling versus deterministic decoding could all inflate the gap. With GeoFormer reporting AR=97.8 and AP=91.5, small changes in score ordering or number of sampled hypotheses can materially change the headline number. The concern is not that the result is false, but that the paper as written does not define the metric pipeline tightly enough for the state-of-the-art claim to be checked.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GeoFormer, an encoder-decoder autoregressive transformer that directly generates multiple building polygons from satellite imagery by representing polygons as flattened sequences of discretized x/y coordinate tokens with start, separator, and stop tokens, and by optimizing a single negative log-likelihood. The encoder is a SwinV2 feature-pyramid network, and the decoder combines causal attention, cross-attention, RoPE embeddings, and ALiBi biases. The authors report state-of-the-art results on the Aicrowd Mapping Challenge, with AP 91.5 versus HiSup's 79.4, and include ablations over 32 configurations plus robustness experiments under downsampling, dropout, and rotation perturbations.","tokens_in":30773,"tokens_out":7185,"duration_ms":69434,"significance":"If the reported results are reproducible, GeoFormer is a meaningful contribution: it is one of the first demonstrations that a single likelihood-based autoregressive transformer can predict multiple building polygons end-to-end, without per-loss weights or raster-to-vector post-processing. The paper ships code, reports a large set of ablations, and makes a falsifiable performance claim. The main weakness is that the central state-of-the-art claim currently rests on an underspecified evaluation protocol and on baseline numbers that were not produced with the same pipeline, so the 12-point AP improvement cannot yet be verified.","major_comments":[{"comment":"The headline state-of-the-art claim is not verifiable as written because AP and AR are rank-based metrics, but the paper never specifies how GeoFormer's stochastically sampled polygons are scored or ordered. Section 5 states only that inference uses nucleus sampling with p=0.95; it does not state how many sequences are sampled per image, whether each polygon is scored by sequence likelihood, average token log-probability, a special-token probability, or a constant score, or how duplicate or overlapping sampled polygons are removed. Please provide the full inference protocol (number of samples, scoring rule, deduplication, and rank ordering) and release the exact evaluation code used for the GeoFormer column of Table 1.","section":"Section 5, Table 1, Section 4.1"},{"comment":"The comparison with baselines is not made under a common evaluation pipeline. The table footnote says baseline values were provided by the respective paper authors or by HiSup's authors, while GeoFormer's numbers were produced by the authors' own pipeline, which, for bAP, C-IoU, and PoLiS, first filters candidate polygons by requiring more than 50% overlap with the ground truth. Because the same filter and rasterization details may or may not have been used for the baseline numbers, the 12-percentage-point AP gap cannot be taken at face value. The authors should rerun all baselines with their own evaluation script and report the results, or, failing that, restrict the SOTA claim to the GeoFormer pipeline and clearly mark which published numbers were used.","section":"Table 1, Section 4.1"},{"comment":"The claim that this is the first successful application of auto-regressive transformer models for multi-polygon predictions in remote sensing is not substantiated. The related-work discussion dismisses earlier auto-regressive studies, including Polygonizer [15], as \"limited to predicting a single object per scene,\" but no details or results for Polygonizer are given, so the reader cannot verify whether Polygonizer already addressed multi-polygon prediction or used a transformer decoder. Please state explicitly how Polygonizer and other auto-regressive methods differ in output space and architecture, and clarify whether they were evaluated on the same benchmark.","section":"Abstract, Section 2"}],"minor_comments":[{"comment":"The N-ratio is described only as comparing the cardinality of the predicted polygon to the ground truth; please provide the exact formula and explain how it is aggregated over a test set.","section":"Section 4.1"},{"comment":"The main-text ablation table consists of rows of checkmarks without a legend or row labels, so it is impossible to identify the ablated configuration from the printed table. Table 5 is dense and lacks clear row labeling; please add explicit row labels or configuration keys to both tables.","section":"Table 2 and Table 5"},{"comment":"Equation (8) uses a Sigmoid Linear Unit (SiLU) activation but cites reference [12], which is the GELU paper; please correct the citation or the activation name.","section":"Equation (8) and reference [12]"},{"comment":"The feature-map dimension list begins with H x W and the final representation is said to be 36 x 36 x C, but this is not derivable from a 224 x 224 input with patch size 4; please clarify the pyramid fusion and upsampling stages that produce the 36 x 36 feature map.","section":"Section 3.1"},{"comment":"The optimizer is specified as beta = (0.9, 0.999), but the manuscript should state which value is beta_1 and which is beta_2, or use the standard notation beta = (beta_1, beta_2).","section":"Appendix A"},{"comment":"There are spacing artifacts in the title and affiliation text (\"T echnical\", \"Pol YGON\", \"SEGMENTA TION\"); please clean the LaTeX source.","section":"First page"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper you'll want to know about: GeoFormer is an autoregressive transformer that generates whole building polygons directly from satellite images, using a SwinV2 feature pyramid and a decoder that emits x,y coordinates with start/separator/stop tokens. It's the first such model that works for multiple polygons per scene in remote sensing, and it claims a stunning jump on the Aicrowd benchmark: AP 91.5 vs 79.4 for the previous best (HiSup).\n\nThe genuinely good parts are real. They replace multi-loss segmentation-plus-vectorization pipelines with a single log-likelihood, which is elegant. The ablations on a small subset support their design choices—ALiBi, RoPE, and polygon sorting all matter. They also ship code and are unusually honest in Section 6.1 about the costs: 64x slower inference than HiSup, and the 36x36 feature map that hurts performance on downsampled inputs. That's the kind of limitation statement that many papers omit.\n\nThe soft spots are all in the evaluation protocol, and they're load-bearing for the headline. Table 1 takes baseline numbers from other papers (or from HiSup's authors), while GeoFormer's numbers come from their own pipeline, which includes a >50% overlap filter before computing bAP/C-IoU/PoLiS and uses nucleus sampling with p=0.95. COCO AP and AR need a per-prediction confidence score and a fixed hypothesis set, but the paper never says which score ranks GeoFormer's polygons (sequence likelihood? separator probability? uniform?) or how many sequences are sampled per image. Without that, the 12-point gap is not verifiable, and a large part of it could be protocol rather than method. This is fixable: rerun the baselines with the same evaluation script and publish the sampling details.\n\nTwo smaller issues. The main paper's ablation table (Table 2) is confusing—several rows are 0.00 and the real numbers are in the supplementary—and the explanation for the pyramid-features-without-mask collapse is reasonable but only post hoc. The robustness study shows GeoFormer losing to HiSup under downsampling, which they attribute to the coarse feature map; that's plausible, but the robustness ranking again relies on their own pipeline.\n\nBottom line: this deserves a serious referee, not a desk reject. The architecture is coherent, the limitations are acknowledged, and the empirical claim is important enough to spend referee time on. If the evaluation is tightened and the numbers survive, this is a real advance. For now, I'd take the SOTA claim with a grain of salt, but I'd definitely engage.","headline":"First autoregressive multi-polygon transformer for satellite buildings with a big claimed SOTA jump, but the evaluation protocol is underspecified enough that the 12-point gap is not yet verifiable.","tokens_in":31278,"tokens_out":3053,"would_cite":false,"duration_ms":27940,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GeoFormer generates building polygons directly as autoregressive token sequences and reports AP 91.5 on the Aicrowd Mapping Challenge, about 12 points above the previous best.","keywords":["GeoFormer","building footprint extraction","autoregressive transformer","multi-polygon segmentation","satellite imagery","SWINv2","Aicrowd Mapping Challenge","sequence generation"],"falsifier":"Re-run FFL, PolyWorld, and HiSup through the exact evaluation pipeline used for GeoFormer, including matching only polygons with more than 50% overlap and decoding with nucleus sampling p=0.95; if the AP margin over HiSup falls well below the reported 12 points, the central comparison is an artifact of protocol rather than the model. A second check is to set inference to greedy decoding and count how many sampled sequences per image are needed to reproduce the reported AP and AR.","tokens_in":30324,"feed_emoji":"🛰️","tokens_out":9159,"duration_ms":77377,"temperature":0.7,"pith_summary":"GeoFormer argues that building footprints in satellite imagery should be produced directly as vectors rather than as segmentation masks that later need polygonization. It presents an encoder-decoder transformer that flattens every building polygon in an image into one sequence of discretized x and y coordinates, with special tokens for the start of the sequence, the separation between buildings, and the end. The whole model is trained with a single categorical likelihood, so there are no loss-term weights to tune and no raster-to-vector post-processing. On the Aicrowd Mapping Challenge validation set the model reports an AP of 91.5, about 12 points above the previous best, and leads on boundary AP and complexity-aware IoU. The paper's claim, read sympathetically, is that scale-invariant building vectorization is a sequence-generation problem that transformers can solve end-to-end.","feed_headline":"Transformer draws building polygons directly, beats prior best by 12 AP","feed_subtitle":"One likelihood loss replaces segmentation-plus-polygonization and lifts Aicrowd polygon AP from 79.4 to 91.5.","key_machinery":"The central object is a token sequence: all vertices of all buildings in an image are flattened as $(x_0,y_0,x_1,y_1,\\dots)$ in a fixed order, separated by a separator token between buildings and bounded by start and stop tokens. The decoder computes the conditional likelihood $p_\\theta(s_t|I,s_{<t})$ over discretized coordinates plus three special tokens, using causal self-attention over the sequence and cross-attention to a $36 \\times 36$ feature map built from the four SWINv2 encoder stages. Two positional mechanisms do much of the work: RoPE encodes relative positions between coordinate tokens, and ALiBi gives a third of the attention heads a linear bias toward nearby tokens so the model can localize each object's vertices.","core_discovery":"GeoFormer establishes that multi-building vectorization works as autoregressive polygon generation: the decoder predicts each vertex coordinate conditional on the satellite image and all previously predicted tokens. Modeling keypoints as spatially dependent tokens lets a single SWINv2-based encoder-decoder transformer output closed polygons directly, with no bounding-box pretraining, no separate corner detection, and no learned or heuristic polygonization stage. The reported results on the Aicrowd Mapping Challenge are AP 91.5, AR 97.8, bAP 97.1, IoU 98.1, and C-IoU 97.4, compared with HiSup's 79.4, 81.5, 66.5, 94.3, and 89.6; the only metric where GeoFormer trails is PoLiS distance, 0.913 versus 0.726. The paper further claims this is the first successful application of autoregressive transformer models to multi-polygon prediction in remote sensing.","pith_inferences":["The same sequence parameterization should extend to other vector map objects such as roads, parcels, or building clusters, and to wireframe-like problems where shared vertices must be revisited; the paper names wireframes as future work, but the transfer to other vector layers is not explored.","A testable consequence of the categorical tokenization is that switching to a discretized mixture-of-logistics likelihood, which the paper suggests for future work, could shorten sequences and reduce sampling cost while possibly changing the reported AP.","The paper leaves unquantified how much of the 12-point gap is architectural versus procedural; re-evaluating all baselines with identical 50%-overlap matching and identical inference sampling would separate those factors.","Because the reported numbers come from nucleus sampling with p=0.95, decoding greedily or varying the number of sampled sequences per image would reveal how much of the high AP depends on stochastic inference rather than the learned distribution."],"forward_implications":["A single negative-log-likelihood objective replaces the common combination of segmentation loss, polygonization loss, and hand-tuned weights.","Multi-building scenes are handled by a separator token, so the approach is not limited to single-object prediction like earlier Polygon-RNN variants.","The model's bAP and C-IoU advantages indicate that directly predicting vertices yields cleaner edges, which matters for map products that need right angles and precise corners.","The performance gain currently costs about 1.93 seconds per image with 97.4M parameters, roughly 64 times slower than HiSup, so speed is the main practical price.","Robustness results show the model is comparatively stable under rotation and missing pixels but loses more than HiSup under spatial downsampling, consistent with the $36 \\times 36$ feature map being a resolution bottleneck."],"supporting_citations":[{"why":"Defines the Aicrowd Mapping Challenge dataset whose validation set anchors every reported comparison.","marker":"[25]"},{"why":"HiSup is the strongest prior baseline on the benchmark, supplying the AP/AR/bAP/C-IoU numbers GeoFormer is compared against.","marker":"[34]"},{"why":"PolyWorld contributes the C-IoU metric and is a leading graph-based polygon baseline.","marker":"[42]"},{"why":"Frame Field Learning is the raster-segmentation baseline that the single-likelihood approach is meant to supersede.","marker":"[10]"},{"why":"PolyMapper is the earlier autoregressive polygon method that predicted single objects and motivates the multi-polygon extension.","marker":"[19]"},{"why":"SWINv2 supplies the hierarchical encoder whose per-stage features are fused into the 36 by 36 pyramidal map.","marker":"[23]"},{"why":"ALiBi contributes the localized attention bias that the ablations identify as critical for multi-object localization.","marker":"[27]"},{"why":"RoPE encodes relative positions among coordinate tokens and is shown by ablations to be essential.","marker":"[30]"},{"why":"Nucleus sampling with p=0.95 is the decoding procedure that produces all reported GeoFormer results.","marker":"[13]"}],"fun_headline_variants":["One loss, one model: GeoFormer maps buildings as polygons end-to-end","GeoFormer beats HiSup by 12 AP with a single likelihood loss","Autoregressive transformer draws building polygons directly from satellite images","GeoFormer: first autoregressive transformer for multi-polygon building vectorization","One likelihood loss replaces polygonization: GeoFormer builds AP 91.5"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The performance comparison assumes the earlier methods' published scores were obtained under the same polygon-matching, filtering, and inference-sampling rules that the authors apply to GeoFormer, since the paper does not re-run those baselines with its own pipeline.","fun_headline_variants_meta":{"raw":{"variants":["One loss, one model: GeoFormer maps buildings as polygons end-to-end","GeoFormer beats HiSup by 12 AP with a single likelihood loss","Autoregressive transformer draws building polygons directly from satellite images","GeoFormer: first autoregressive transformer for multi-polygon building vectorization","One likelihood loss replaces polygonization: GeoFormer builds AP 91.5"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000927,"raw_usage":{"total_tokens":3944,"prompt_tokens":889,"completion_tokens":3055,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":2959}},"tokens_in":505,"tokens_out":3055,"duration_ms":18649,"temperature":1.0,"reasoning_tokens":2959,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:54:22.106386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run FFL, PolyWorld, and HiSup through the exact evaluation pipeline used for GeoFormer, including matching only polygons with more than 50% overlap and decoding with nucleus sampling p=0.95; if the AP margin over HiSup falls well below the reported 12 points, the central comparison is an artifact of protocol rather than the model. A second check is to set inference to greedy decoding and count how many sampled sequences per image are needed to reproduce the reported AP and AR.","supporting_citations":[{"cited_title":"Deep learning for understanding satellite imagery: An experimental survey","cited_arxiv_id":null,"evidence_quote":"Defines the Aicrowd Mapping Challenge dataset whose validation set anchors every reported comparison."},{"cited_title":"Hisup: Accurate polygonal mapping of buildings in satellite imagery with hierarchical supervision","cited_arxiv_id":null,"evidence_quote":"HiSup is the strongest prior baseline on the benchmark, supplying the AP/AR/bAP/C-IoU numbers GeoFormer is compared against."},{"cited_title":"Poly- world: Polygonal building extraction with graph neural networks in satellite images","cited_arxiv_id":null,"evidence_quote":"PolyWorld contributes the C-IoU metric and is a leading graph-based polygon baseline."},{"cited_title":"Polygonal Building Segmentation by Frame Field Learning","cited_arxiv_id":"2004.14875","evidence_quote":"Frame Field Learning is the raster-segmentation baseline that the single-likelihood approach is meant to supersede."}],"review_version":1}