Pith. sign in

REVIEW 3 major objections 5 minor 28 references

Visual prompts let SAM 3 lock onto top-down remote-sensing geometry, but text prompts inject ground-level semantic bias that can actively degrade detection.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 01:58 UTC pith:ETF4OIPR

load-bearing objection Solid multi-task stress test of SAM 3 on EO with a clear visual-vs-text interference diagnosis; the “avoids overfitting / high HM” claim rests on a proxy that does not measure supervised base/novel transfer. the 3 major comments →

arxiv 2607.09583 v1 pith:ETF4OIPR submitted 2026-07-10 cs.CV

Promptable Concept Segmentation from Above: Evaluating SAM 3's Zero-Shot and One-Shot Capabilities in Remote Sensing

Mohammad Dabaja , Turgay Celik This is my paper
classification cs.CV
keywords Cross-Modal AlignmentEarth ObservationGeneralized Zero-Shot LearningPromptable Concept SegmentationRemote SensingSAM 3Instance SegmentationOpen-Vocabulary Object Detection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether Segment Anything Model 3 can be used out of the box on satellite and aerial imagery without any domain training. The authors adapt its presence head into a zero-shot scene classifier and run a five-way prompt ablation on standard remote-sensing benchmarks for classification, detection, and instance segmentation. They find that a single visual box exemplar aligns the multimodal decoder to complex overhead geometry and yields strong localization, while pure text prompts—and often text mixed with boxes—import ground-level expectations that misalign with top-down shapes and can hurt coordinate regression. Using a training-free proxy that treats multimodal prompts as “base” and text-only prompts as “novel,” the frozen model avoids the closed-set overfitting common in specialized remote-sensing models and posts high harmonic-mean scores on instance segmentation. The work still flags hard limits: sub-pixel objects, shadow-distorted shapes, and overhead semantic blind spots for concepts like schools or churches. The practical upshot is a clear mandate to fine-tune only the text path and multimodal decoder rather than retrain the whole visual backbone.

Core claim

Across remote-sensing scene classification, object detection, and instance segmentation under strict zero- and one-shot constraints, SAM 3’s multimodal decoder is well aligned by visual prompts to top-down geometry, yet textual prompts inject misaligned ground-level semantic bias that can actively degrade coordinate regression; the frozen model therefore avoids the overfitting typical of domain-adapted zero-shot detectors and reaches high harmonic-mean scores on segmentation proxies, while remaining limited by resolution and overhead semantic blind spots.

What carries the argument

Five systematically isolated prompt configurations (text-only, box-only, text+box, and the same two with oracle negative filtering) plus repurposing of the decoupled binary presence head as a standalone zero-shot classifier; these isolate cross-modal interference inside the multimodal decoder and supply a training-free base/novel proxy for harmonic-mean comparison.

Load-bearing premise

That treating one-shot multimodal prompts as “base” performance and text-only prompts as “novel” performance is a fair stand-in for the usual supervised base/novel splits used by specialized remote-sensing models.

What would settle it

Re-run the DIOR five-configuration ablation after replacing every class name with an explicitly aerial description (e.g., “top-down view of a baseball diamond”); if Text-Only and Text+Box then match or exceed Box-Only mAP, the claimed ground-level text bias is refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript provides a multi-task empirical evaluation of SAM 3 on remote sensing under zero-shot and one-shot constraints, covering AID scene classification, DIOR object detection, and iSAID instance segmentation. The authors repurpose SAM 3’s decoupled presence head as a zero-shot scene classifier (Eqs. 2–5), ablate five prompt configurations (text-only, box-only, text+box, and oracle-filtered variants), and diagnose cross-modal interference: visual prompts align the multimodal decoder to top-down geometry while textual prompts inject ground-level semantic bias that can degrade localization. They further introduce a training-free proxy that maps multimodal (Conf 3/5) AP50 to “Base” and text-only (Conf 1/4) AP50 to “Novel,” then compute Harmonic Means for comparison against supervised GZSD/GZSI models (Tables V, VIII). The paper concludes that frozen SAM 3 avoids closed-set overfitting and achieves high HM on segmentation, yet remains limited by sub-pixel resolution and overhead semantic blind spots, motivating PEFT of the multimodal decoder.

Significance. Evaluating SAM 3 on standard EO benchmarks is timely and useful. The five-way prompt ablation, class-wise tables (III, VII), and qualitative figures (3, 4) give a clear, reproducible diagnosis of visual–textual interference that is of independent interest to the remote-sensing VLM community. The presence-head adaptation for scene classification and the absolute one-shot localization numbers (e.g., Box-Only 66.55 mAP on DIOR) are concrete contributions. If the proxy-based “avoids overfitting / high HM” claims can be either validated or carefully reframed, the work would serve as a solid baseline paper charting where frozen PCS models succeed and fail from above.

major comments (3)
  1. §III-D (Eqs. 7–8) and Tables V/VIII: The central comparative claim that SAM 3 “avoids the overfitting commonly seen in legacy domain-adapted models, achieving high Harmonic Mean scores” rests entirely on treating Conf 3/5 (Text+Box) AP50 as Base and Conf 1/4 (Text-Only) AP50 as Novel. This equates one-shot visual exemplar matching on every class (including community Novel classes) with a model supervised on a disjoint base subset that must generalize to held-out novel classes. The two quantities are not interchangeable: Conf 3 never tests true zero-shot transfer after base-class fine-tuning, and Conf 1 collapses precisely on the sprawling DIOR Novel zones (Airport/Baseball Field/Ground Track Field ≈ 0 mAP in Table III). The reported HMs (3.26/5.48 DIOR; 40.24/41.87 iSAID) therefore cannot be read as evidence of reduced closed-set bias relative to CoseDet, ZoRI, etc. Either validate the p
  2. §IV-B.4 and Table V: The paper itself notes that the DIOR Novel split (Windmill, Airport, Baseball Field, Ground Track Field) is an “artifact” that bottlenecks HM, yet still reports the low HM as a primary result and contrasts it with supervised models. If the proxy is retained, the manuscript must either (a) recompute HM on a geometry-balanced Novel subset where text-only is non-degenerate, or (b) present the DIOR HM only as a diagnostic of the community split’s difficulty, not as a model ranking. Leaving the table as a direct leaderboard comparison overstates the evidence.
  3. Abstract and §V: The mandate for “parameter-efficient geospatial fine-tuning of its multimodal decoder” is well motivated by the interference diagnosis, but the claim that SAM 3 already “avoids overfitting” while remaining “fundamentally constrained” is currently supported only by the contested proxy. After addressing the proxy, the conclusion should separate (i) robust visual backbone / one-shot localization from (ii) any statement about closed-set bias relative to fine-tuned GZSD/GZSI models.
minor comments (5)
  1. Table I: SAM 3’s best AID accuracy (38.70%) is far below even general-domain CLIP L/14 (69.25%). The discussion correctly attributes this to ground-level text alignment, but a short note on whether multi-prompt ensembling or template engineering was tried would help readers gauge the ceiling of the presence-head adaptation.
  2. Figure 2 caption and marker legend are dense; a small legend panel or color-only encoding (in addition to shape) would improve readability in print.
  3. Eq. (1) and the reconstruction Sc = Pc · Lc (Eq. 4) are clear, but the text sometimes switches between “Presence Score,” “Global Presence Score,” and “presence token” without a single notation table; a short symbol glossary would help.
  4. §II-C and §III-D: The proxy is described as applying to “scene classification and instance segmentation” in the abstract, yet scene classification is not evaluated under the Base/Novel HM protocol. Align the abstract wording with the actual experiments (detection + segmentation).
  5. References: SAM 3 is cited as arXiv:2511.16719 (2026); confirm the final citation once the official version is available. Minor typos: “sam 3 model” (lowercase) in §IV-C.3; “overfitting commonly seen” is repeated almost verbatim from abstract to conclusion.

Circularity Check

0 steps flagged

No significant circularity: empirical multi-task evaluation of frozen SAM 3 against public benchmarks; the Base/Novel proxy is a methodological mapping of measured APs, not a result forced by definition or fit.

full rationale

The paper is a zero/one-shot empirical benchmark of an external foundation model (SAM 3) on three public remote-sensing datasets (AID, DIOR, iSAID) with published baselines. All reported mAP/AP50/accuracy numbers are direct measurements under the five prompt configurations; none are obtained by fitting free parameters to the target metrics and then re-predicting them. The Confidence Score Sc = Pc · Lc (Eq. 4) simply reconstructs the model’s own factored match probability already stated in Eq. 1; it introduces no new free parameter. The training-free GZSD/GZSI proxy of §III-D (Conf 3/5 → Base AP50, Conf 1/4 → Novel AP50, then HM) is an explicit methodological choice that maps already-measured quantities onto the community’s standard formula; the resulting HM values are therefore not equivalent to their inputs by construction, nor do they rest on self-citation or uniqueness theorems. No load-bearing self-citations appear, and no ansatz is smuggled via prior author work. The absolute localization numbers, cross-modal interference diagnosis, and class-wise failure modes stand independently of the proxy interpretation. Consequently the derivation chain contains no circular steps.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 3 invented entities

The paper is empirical, not a first-principles derivation. Load-bearing content is mostly domain assumptions about how to query a frozen multimodal model and how to map those queries onto GZSD/GZSI metrics. There are almost no fitted free parameters; the main invented constructs are the evaluation protocol pieces used to claim comparability with supervised zero-shot literature.

free parameters (2)
  • Combined confidence fusion Sc = Pc · Lc
    Multiplicative fusion of presence and localisation is chosen as the primary scene-classification decision rule (Eqs. 4–5). It reconstructs SAM 3’s factored match probability but is still a design choice among possible fusions; reported AID accuracy (38.70%) depends on it.
  • IoU evaluation thresholds (0.50 and 0.50:0.95)
    Standard COCO-style thresholds are adopted without sensitivity analysis; macro rankings could shift under other IoU operating points, especially for small RS objects.
axioms (5)
  • domain assumption SAM 3’s published multimodal decoder, presence head, and prompt encoders behave as described in the SAM 3 reference and can be queried independently per class without fine-tuning.
    All zero/one-shot results rest on this architectural interface (§II-B, §III-B/C).
  • domain assumption A single ground-truth bounding-box crop is a fair one-shot visual exemplar for Conf 2/3/5.
    One-shot Base proxy performance is defined from these GT exemplars (§III-C); if real one-shot exemplars are noisier, Base AP and HM drop.
  • ad hoc to paper Text+Box (Conf 3/5) ≈ supervised Base and Text-Only (Conf 1/4) ≈ Novel is a valid proxy for GZSD/GZSI Harmonic Mean comparison.
    Introduced in §III-D specifically to enable Tables V and VIII without training; central to the ‘avoids overfitting / high HM’ narrative.
  • ad hoc to paper Oracle negative filtering (multiply confidence by GT presence indicator fc) isolates false-positive noise without changing the interpretation of true-positive localization quality.
    Defines Conf 4/5 and Oracle HM (§III-C/D); uses ground truth unavailable at real deployment.
  • domain assumption Public AID/DIOR/iSAID splits and class taxonomies are appropriate external benchmarks for open-vocabulary RS evaluation.
    Standard community assumption; all quantitative claims are relative to these datasets (§III-A).
invented entities (3)
  • Training-free GZSD/GZSI proxy evaluation protocol (Base←visual multimodal prompts, Novel←text-only prompts) no independent evidence
    purpose: Compute Harmonic Mean against supervised generalized zero-shot literature without any base-class fine-tuning.
    Core methodological invention of the paper (§III-D); independent evidence is limited to face validity of the mapping and consistency with qualitative ablations.
  • Five-configuration prompt modality matrix (Conf 1–5) including oracle negative filtering no independent evidence
    purpose: Isolate textual vs visual contributions and false-positive susceptibility in the multimodal decoder.
    Evaluation construct defined in §III-C; useful diagnostic, not a physical entity, and not validated outside this study’s tables.
  • Presence-head-as-standalone-zero-shot-scene-classifier adaptation no independent evidence
    purpose: Turn SAM 3’s binary presence token into a multi-class AID scene classifier via argmax over concepts.
    Structural reuse described in §III-B; performance (38.70% top-1) is measured only here relative to CLIP-family baselines.

pith-pipeline@v1.1.0-grok45 · 23210 in / 4014 out tokens · 57908 ms · 2026-07-13T01:58:13.801089+00:00 · methodology

0 comments
read the original abstract

The deployment of large-scale foundation models, such as the Segment Anything Model 3 (SAM 3), promises a transition toward open-vocabulary, training-free computer vision. However, their capacity to generalize out-of-distribution to the complex, top-down geometric structures of Earth Observation imagery remains largely unquantified. Driven by SAM 3's performance disparities in highly specialized domains, we present a comprehensive, multi-task empirical evaluation across remote sensing scene classification, object detection, and instance segmentation under strict zero-shot and one-shot constraints. To achieve this, we introduce a structural adaptation of SAM 3 by repurposing its decoupled binary presence head into a standalone zero-shot classifier. Furthermore, by systematically isolating textual and visual prompt modalities across five configurations, we explicitly diagnose the alignment mechanics within the model's multimodal decoder. Our findings reveal severe cross-modal interference: while visual prompts successfully align the decoder to complex remote sensing geometry, textual prompts inject misaligned, ground-level semantic bias, actively degrading coordinate regression. To benchmark these capabilities without resource-intensive training, we formulate a novel training-free proxy evaluation protocol for Generalized Zero-Shot tasks (scene classification and instance segmentation). Ultimately, our results demonstrate that SAM 3 avoids the overfitting commonly seen in legacy domain-adapted models, achieving high Harmonic Mean scores in segmentation tasks. However, it remains fundamentally constrained by sub-pixel resolution limits and overhead semantic blind spots, charting a definitive mandate for parameter-efficient geospatial fine-tuning of its multimodal decoder.

Figures

Figures reproduced from arXiv: 2607.09583 by Mohammad Dabaja, Turgay Celik.

Figure 1
Figure 1. Figure 1: The architectural evolution of SAM 3, highlighting the integration of the Multimodal Decoder, specialized prompt [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: 1) Scoring Head Dynamics (Presence vs. Localisation): As shown in Table I, the classification strategy guided by the standalone Presence Score (Equation 2) achieves an accuracy of 37.26%, outperforming the Conditional Localisation Score (Equation 3), which yields 28.25%. This performance gap highlights the structural decoupling of the model. The indepen￾dent presence head evaluates the global existence of … view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative zero-shot scene classification results on selected categories from the AID dataset. Each row displays three [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative object detection results on the DIOR dataset across the five prompt configurations (Configurations [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative instance segmentation results on the iSAID dataset across the prompt configurations ( [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 4 linked inside Pith

  1. [1]

    Segment anything,

    A. Kirillov, E. Mintun, N. Raviet al., “Segment anything,” inProceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 4015–4026

    2023

  2. [2]

    Sam 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Huet al., “Sam 2: Segment anything in images and videos,”arXiv preprint arXiv:2408.00714, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Sam 3: Segment anything with concepts,

    N. Carion, L. Gustafson, Y .-T. Huet al., “Sam 3: Segment anything with concepts,”arXiv preprint arXiv:2511.16719, 2026

    Pith/arXiv arXiv 2026

  4. [4]

    Remoteclip: A vision language foundation model for remote sensing,

    F. Liu, D. Chen, Z. Guanet al., “Remoteclip: A vision language foundation model for remote sensing,”IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–16, 2024

    2024

  5. [5]

    Semantic-promoted debiasing and background disambiguation for zero-shot instance segmentation,

    S. He, H. Ding, and W. Jiang, “Semantic-promoted debiasing and background disambiguation for zero-shot instance segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  6. [6]

    Zero-shot instance segmentation,

    Y . Zheng, J. Wu, Y . Qinet al., “Zero-shot instance segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 2593–2602

    2021

  7. [7]

    Aid: A benchmark data set for perfor- mance evaluation of aerial scene classification,

    G.-S. Xia, J. Hu, F. Huet al., “Aid: A benchmark data set for perfor- mance evaluation of aerial scene classification,”IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 7, pp. 3965–3981, 2017

    2017

  8. [8]

    Object detection in optical remote sensing images: A survey and a new benchmark,

    K. Li, G. Wan, G. Chenget al., “Object detection in optical remote sensing images: A survey and a new benchmark,”ISPRS Journal of Photogrammetry and Remote Sensing, vol. 159, pp. 296–307, 2020

    2020

  9. [9]

    isaid: A large-scale dataset for instance segmentation in aerial images,

    S. W. Zamir, A. Arora, A. Guptaet al., “isaid: A large-scale dataset for instance segmentation in aerial images,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 28–37

    2019

  10. [10]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacyet al., “Learning transferable visual models from natural language supervision,” inInternational Conference on Machine Learning (ICML). PMLR, 2021, pp. 8748–8763

    2021

  11. [11]

    Rs5m and georsclip: A large- scale vision-language dataset and a large vision-language model for remote sensing,

    Z. Zhang, T. Zhao, Y . Guoet al., “Rs5m and georsclip: A large- scale vision-language dataset and a large vision-language model for remote sensing,”IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–16, 2024

    2024

  12. [12]

    Rsdix: Lightweight and data- efficient vlms for remote sensing through self-distillation,

    A. Terlizzi, A. Nazzaro, L. Bernardiet al., “Rsdix: Lightweight and data- efficient vlms for remote sensing through self-distillation,”International Joint Conference on Neural Networks (IJCNN), 2025

    2025

  13. [13]

    End-to-end object detec- tion with transformers,

    N. Carion, F. Massa, G. Synnaeveet al., “End-to-end object detec- tion with transformers,” inEuropean conference on computer vision. Springer, 2020, pp. 213–229

    2020

  14. [14]

    Robust region feature synthesizer for zero-shot object detection,

    P. Huang, J. Han, D. Chenget al., “Robust region feature synthesizer for zero-shot object detection,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 7622– 7631

    2022

  15. [15]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection,

    S. Liu, Z. Zeng, T. Renet al., “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” inEuropean Conference on Computer Vision (ECCV). Springer, 2024

    2024

  16. [16]

    Cosedet: Open-vocabulary remote sensing object detection with contextual semantic information,

    J. Yao, Y . Zheng, S. Luet al., “Cosedet: Open-vocabulary remote sensing object detection with contextual semantic information,”IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 18, pp. 1120–1135, 2025

    2025

  17. [17]

    Zori: Towards discriminative zero- shot remote sensing instance segmentation,

    S. Huang, H. He, and W. Ding, “Zori: Towards discriminative zero- shot remote sensing instance segmentation,” inProceedings of the AAAI Conference on Artificial Intelligence, 2025

    2025

  18. [18]

    Skyscript: A large and semanti- cally diverse vision-language dataset for remote sensing,

    Z. Wang, R. Prabha, T. Huanget al., “Skyscript: A large and semanti- cally diverse vision-language dataset for remote sensing,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 5805–5813, 2024

    2024

  19. [19]

    Rs-clip: Zero shot remote sensing scene classification via contrastive vision-language supervision,

    X. Li, C. Wen, Y . Huet al., “Rs-clip: Zero shot remote sensing scene classification via contrastive vision-language supervision,”International Journal of Applied Earth Observation and Geoinformation, vol. 124, p. 103497, 2023

    2023

  20. [20]

    Enhancing remote sensing vision-language models through mllm and llm-based high-quality image-text dataset generation,

    Y . Heet al., “Enhancing remote sensing vision-language models through mllm and llm-based high-quality image-text dataset generation,”arXiv preprint arXiv:2507.16716, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Vk-det: Visual knowledge guided prototype learning for open-vocabulary aerial object detection,

    J. Yao, Y . Zheng, S. Luet al., “Vk-det: Visual knowledge guided prototype learning for open-vocabulary aerial object detection,” inPro- ceedings of the AAAI Conference on Artificial Intelligence, 2026

    2026

  22. [22]

    Zero-shot object detection via descriptor registration,

    X. Zhu, J. Pang, and C. Wang, “Zero-shot object detection via descriptor registration,”IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  23. [23]

    Open-vocabulary object detection via vision and language knowledge distillation,

    X. Gu, T.-Y . Lin, W. Kuoet al., “Open-vocabulary object detection via vision and language knowledge distillation,”Proceedings of the International Conference on Learning Representations (ICLR), 2022

    2022

  24. [24]

    Yolo-world: Real-time open- vocabulary object detection,

    B. Cheng, I. Misra, A. G. Schwinget al., “Yolo-world: Real-time open- vocabulary object detection,”Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  25. [25]

    Castdet: Toward open vocabulary aerial object detection with clip-activated student-teacher learning,

    Y . Li, W. Guo, X. Yanget al., “Castdet: Toward open vocabulary aerial object detection with clip-activated student-teacher learning,”European Conference on Computer Vision (ECCV), 2024

    2024

  26. [26]

    Semantics-guided contrastive network for zero-shot object detection,

    C. Yan, X. Chang, M. Luoet al., “Semantics-guided contrastive network for zero-shot object detection,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 3, pp. 1530–1547, 2024

    2024

  27. [27]

    Frustratingly simple but effective zero-shot detection and segmentation: Analysis and a strong baseline,

    S. Khandelwal, A. Nambirajan, B. Siddiquieet al., “Frustratingly simple but effective zero-shot detection and segmentation: Analysis and a strong baseline,”arXiv preprint arXiv:2302.07319, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip,

    Q. Yu, J. He, X. Denget al., “Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip,”Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023

    2023