REVIEW 2 major objections 5 minor 35 references
This paper claims that the poor performance of open-vocabulary segmentation on remote sensing imagery is largely caused by the text query—the class name is a weak address into the vision-language embedding space—and that learning a pseudo-w
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 02:02 UTC pith:JSS4EVRR
load-bearing objection A plausible, clearly-written demonstration that textual inversion can fix weak text queries for frozen open-vocabulary segmenters on overhead imagery; the central claim is probably right but two controls are missing. the 2 major comments →
Few-Shot Open-Vocabulary Remote Sensing Segmentation via Textual Inversion
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that the text query, not the segmentation model, is the bottleneck for open-vocabulary segmentation on overhead imagery. A class name lands at a point in the vision-language embedding space, and when that landing point is weak the model produces a diffuse, non-committal response. The paper demonstrates that renaming recovers categories with nominal failures, while categories like ground track field resist all tested rephrasings and are reached only by a learned address. Using textual inversion on a frozen open-vocabulary segmenter specialized for overhead imagery, the authors fit a compact pseudo-word per category from a handful of masks, then discard the support and que
What carries the argument
The central object is the learned pseudo-word embedding: a small set of input token embeddings (two by default) that replaces the class name in the frozen text encoder's input sequence. Initialized from the class name's own embeddings, these tokens are optimized by textual inversion—fitting a concept from a few examples while the model stays frozen—against a segmentation objective built from the smooth maximum of the instance and semantic readouts gated by a presence score, plus a cross-category negative term. The learned tokens become the query representation that selects a category in the shared embedding space; because every stage of the model is frozen, the learned address acts only by r
Load-bearing premise
The method assumes that for a failing category, the frozen model's image features already rank the true region above its surroundings—so the class name has high AUROC even where IoU is near zero—meaning the only missing ingredient is a text address that concentrates the response; if the features themselves fail to separate the category, a learned token cannot recover it.
What would settle it
A concrete test: pick a category on a dataset outside the paper's main benchmark whose name query has near-chance AUROC (features do not rank the region), fit a learned token from several masks, and check whether IoU rises. The paper predicts it will not, because the failure would be visual rather than nominal. Conversely, if a category with high name-AUROC and low name-AUPRC fails to improve after learning a token, the method's effectiveness is not purely about the query.
If this is right
- If the diagnosis is correct, a large part of the remote-sensing gap in open-vocabulary segmentation can be closed by repairing the text query rather than retraining the model or supplying visual prompts at inference.
- Enrolling a category is a one-time cost; once the token is learned, inference runs at the speed and memory footprint of a zero-shot text query, with no support image, mask, or feature bank retained.
- Because the learned text address and ordinary class names are scored by the same full-vocabulary assignment rule, enrolled categories compete directly with named categories without separate calibration.
- The text-only inference design isolates the contribution of the address itself: any gain over the class name must come from the query, not from visual evidence added at test time.
- The per-class diagnosis separating nominal from visual failures provides a practical guide for choosing between renaming, learned tokens, and visual grounding.
Where Pith is reading between the lines
- The 'address' framing suggests a general recipe for any domain shift where a frozen vision-language model underperforms on named categories: if the name ranks the region but lacks precision, textual inversion on a few examples could recover it without changing the inference interface.
- A testable extension: measure the name query's AUROC and AUPRC per category before enrollment; categories with high AUROC but low AUPRC should be exactly the ones the learned token rescues, giving a cheap predictor of benefit.
- The paper's own limitation (baseball diamond and soccer field) points to an ensemble strategy: route a category to a learned token when its failure is nominal and to visual prompting when it is visual, decided by the same ranking-versus-precision diagnostic.
- Since the learned token is fixed per category, conditioning it on the input image (while still entering as text) might handle appearance shifts that a single static address cannot follow.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies open-vocabulary semantic segmentation on remote sensing imagery. It argues that a substantial part of the gap to everyday-scene performance is caused by the text query rather than by the segmentation model: for many overhead categories the class name is a weak address in the shared vision-language embedding space, producing diffuse responses. The authors support this with a diagnosis on iSAID, then propose to recover the address by textual inversion on a frozen model: a small set of pseudo-word embeddings per category is optimized on a few support masks, and at inference the category is queried with the learned text token alone. The evaluation covers eight remote sensing datasets and compares against zero-shot queries, a name-ensemble renaming baseline, and two few-shot visual-prompting baselines. On five iSAID categories with near-zero zero-shot IoU, mean IoU rises from 3.9 to 39.4, and the average over eight datasets improves at 5 and 10 shots.
Significance. If the central claim holds, the paper makes a useful contribution: it shows that a frozen open-vocabulary segmenter can be adapted to remote sensing categories through the text interface alone, without carrying visual prompts into inference. The experimental protocol is careful—support masks are drawn from a disjoint pool, the official evaluation split is not used for selection or fitting, and all methods share the same frozen backbone and full-vocabulary argmax. Releasing code is a plus. The per-class separation between categories that a better name repairs (storage tank) and categories that require a learned address (ground track field) is a clean and actionable finding, and the paper is honest about categories where visual prompting remains superior.
major comments (2)
- [§3.3, Eq. (1)/(5), §4.3] The central claim needs a calibration check. Eq. (1) feeds p·max(I,S) into a full-vocabulary argmax with a fixed background threshold. Eq. (5) includes a presence term that pushes p upward and segmentation losses that push the soft score on support pixels to 1, with no constraint that the learned token's score distribution match class-name queries. A louder learned query can then claim pixels from quieter name-based categories and background, inflating mIoU independent of any address improvement. The §4.3 AUPRC analysis does not rule this out because AUPRC is rank-based and invariant to rescaling, and the assertion that a per-category name threshold cannot reproduce the gain is not tested. Please report score/presence distributions for learned vs name queries, and run a version with learned-token scores calibrated to the name-query scale before argmax, or a per-category threshold on the
- [§4.3, Table 1] The domain-level diagnosis rests on a narrow base. The AUROC/AUPRC analysis covers nine iSAID categories at five shots, and the headline '3.9→39.4' is on categories chosen for zero-shot IoU<10. The method improves the eight-dataset average, but on LoveDA and UDD5 it does not beat zero-shot at five shots, and on baseball diamond and soccer field the visual baselines are better (Table 2). This is consistent with the paper's own limitations, but it means 'much of the gap traces to the text query' is demonstrated only for a subset of iSAID. Please apply the same diagnostic on at least one other dataset (e.g., VDD or OpenEarthMap) or temper the abstract to present the query-side diagnosis as an iSAID result and the eight-dataset comparison as a method evaluation.
minor comments (5)
- [Fig. 3b] The PCA projection of token embeddings would be more informative with the percentage of variance explained by the two principal components.
- [Table 1] The fully supervised reference is listed alongside few-shot methods without a footnote; consider adding a note that it is not part of the few-shot comparison and is taken from SegEarth-OV3.
- [Eq. (4)] The text says the smooth maximum 'converges to the hard maximum as τ approaches zero'; please specify τ→0+ and define the domain of τ.
- [§4.1] The language model used to generate candidate names for Name Ensemble is not specified; stating it would improve reproducibility of the renaming baseline.
- [Fig. 5 caption] 'crop land' should be 'cropland' for consistency with OpenEarthMap labels.
Circularity Check
No circularity: the few-shot token is fit on disjoint support and evaluated on held-out splits; the only self-citation is non-load-bearing.
full rationale
The paper's central claim is that a pseudo-word learned by textual inversion on a frozen segmenter recovers categories whose class-name queries fail. This is not circular: the token v is optimized in Eq. (5) against support masks via Lseg, Lpre, and Lneg, then evaluated with the frozen inference rule of Eq. (1) on an official split disjoint from the support pool. No quantity that defines the method is also used as its prediction. On iSAID, the hard-category set is selected on a held-out selection split before support masks are drawn, so the 3.9-to-39.4 result is a forward evaluation on the official split, not a refit of the reported number. The Section 4.3 diagnostic uses the frozen model's own responses: the name's high AUROC with low AUPRC and the learned token's higher AUPRC are measured on the model's outputs, and although the AUPRC gain is plausibly correlated with the training objective, it is not treated as an input to the fit. The claim that a per-category threshold on the name cannot reproduce the gain may be under-supported, but that is a correctness/validity concern, not a by-construction equivalence. The only self-citation is reference [6], a related-work sentence about diffusion-augmented support; it is not load-bearing. The skeptic's score-scale-shift worry is a legitimate calibration threat to interpretation, but it does not make Eqs. (1)-(5) reduce to their own inputs. Therefore no circular step is established.
Axiom & Free-Parameter Ledger
free parameters (9)
- M (number of pseudo-word tokens) =
2
- learning rate =
1e-2
- optimization steps =
300
- EMA decay =
0.99
- smooth-max temperature tau =
0.1
- lambda_p (presence weight) =
0.5
- lambda_n (negative weight) =
0.5
- category selection threshold =
zero-shot IoU < 10
- minimum support mask area =
50 pixels and 1% of patch
axioms (5)
- standard math Smooth maximum m_tau converges to hard maximum as tau approaches 0 and stays on the same normalized score scale.
- domain assumption The frozen SegEarth-OV3 model's text encoder can interpret pseudo-word embeddings placed in the embedding table as ordinary words.
- domain assumption The image encoder's features already separate the target regions (high AUROC), so the failure is one of query precision.
- domain assumption Support masks are representative of the test distribution.
- domain assumption Initialization from the class-name embedding is a useful prior.
read the original abstract
Open-vocabulary segmentation labels arbitrary categories from a text query without per-class training, yet on remote sensing imagery it underperforms on categories it handles reliably elsewhere. We find that much of this gap traces to the text query rather than to the segmentation model. Because these models are not specialized for overhead imagery, the class name that serves as the query is often a weak address into the vision-language embedding space. We show that a better name repairs part of the gap, while the remaining failures call for an address that the tested natural-language rephrasings do not provide. We recover that address from a few examples through textual inversion on a frozen model, keeping inference text only. On a representative benchmark this raises the mean intersection over union on the affected categories from 3.9 to 39.4, and across eight remote sensing datasets it improves over few-shot methods that instead inject visual prompts at inference.
Figures
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2605.17630 (2026)
Boudiaf, A., Hussain, I., Javed, S.: Segrag: Training-free retrieval-augmented se- mantic segmentation. arXiv preprint arXiv:2605.17630 (2026)
Pith/arXiv arXiv 2026
-
[2]
Journal of Visual Communication and Image Repre- sentation109, 104429 (2025)
Cai, W., Jin, K., Hou, J., Guo, C., Wu, L., Yang, W.: Vdd: Varied drone dataset for semantic segmentation. Journal of Visual Communication and Image Repre- sentation109, 104429 (2025)
2025
-
[3]
In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV)
Chen, Y., Wang, Y., Lu, P., Chen, Y., Wang, G.: Large-scale structure from motion with semantic constraints of aerial images. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV). pp. 347–359. Springer (2018)
2018
-
[4]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Cho,S.,Shin,H.,Hong,S.,Arnab,A.,Seo,P.H.,Kim,S.:Cat-seg:Costaggregation for open-vocabulary semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4113–4123 (2024)
2024
-
[5]
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An image is worth one word: Personalizing text-to-image genera- tionusingtextualinversion.In:TheEleventhInternationalConferenceonLearning Representations
-
[6]
arXiv preprint arXiv:2503.03785 (2025)
Immanuel, S.A., Cho, W., Heo, J., Kwon, D.: Tackling few-shot segmentation in remote sensing via inpainting diffusion model. arXiv preprint arXiv:2503.03785 (2025)
Pith/arXiv arXiv 2025
-
[7]
In: International conference on machine learning
Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International conference on machine learning. pp. 4904–4916. PMLR (2021)
2021
-
[8]
arXiv preprint arXiv:2201.03546 (2022)
Li, B., Weinberger, K.Q., Belongie, S., Koltun, V., Ranftl, R.: Language-driven semantic segmentation. arXiv preprint arXiv:2201.03546 (2022)
Pith/arXiv arXiv 2022
-
[9]
arXiv preprint arXiv:2512.08730 (2025)
Li, K., Zhang, S., Wang, Y., Deng, Y., Wang, Z., Meng, D., Cao, X.: Segearth- ov3: Exploring sam 3 for open-vocabulary semantic segmentation in remote sensing images. arXiv preprint arXiv:2512.08730 (2025)
Pith/arXiv arXiv 2025
-
[10]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition
Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., Mar- culescu, D.: Open-vocabulary semantic segmentation with mask-adapted clip. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 7061–7070 (2023)
2023
-
[11]
Ad- vances in Neural Information Processing Systems35, 17612–17625 (2022)
Liang, V.W., Zhang, Y., Kwon, Y., Yeung, S., Zou, J.Y.: Mind the gap: Under- standing the modality gap in multi-modal contrastive representation learning. Ad- vances in Neural Information Processing Systems35, 17612–17625 (2022)
2022
-
[12]
IEEE Transactions on Geoscience and Remote Sensing62, 1–16 (2024)
Liu, F., Chen, D., Guan, Z., Zhou, X., Zhu, J., Ye, Q., Fu, L., Zhou, J.: Remoteclip: A vision language foundation model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing62, 1–16 (2024)
2024
-
[13]
arXiv preprint arXiv:2305.13310 (2023)
Liu, Y., Zhu, M., Li, H., Chen, H., Wang, X., Shen, C.: Matcher: Segment anything with one shot using all-purpose feature matching. arXiv preprint arXiv:2305.13310 (2023)
Pith/arXiv arXiv 2023
-
[14]
ISPRS journal of photogrammetry and remote sensing165, 108–119 (2020)
Lyu, Y., Vosselman, G., Xia, G.S., Yilmaz, A., Yang, M.Y.: Uavid: A semantic seg- mentation dataset for uav imagery. ISPRS journal of photogrammetry and remote sensing165, 108–119 (2020)
2020
-
[15]
In: Proceedings of the IEEE/CVF international conference on computer vision
Min, J., Kang, D., Cho, M.: Hypercorrelation squeeze for few-shot segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6941–6952 (2021)
2021
-
[16]
In: International conference on machine learning
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from Few-Shot Open-Vocabulary RS Segmentation via Textual Inversion 17 natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)
2021
-
[17]
In: Artificial Intelligence for Security and Defence Applications III
Ruis, F., Burghouts, G., Kuijf, H.: Textual inversion for efficient adaptation of open-vocabulary object detectors without forgetting. In: Artificial Intelligence for Security and Defence Applications III. vol. 13679, pp. 481–494. SPIE (2025)
2025
-
[18]
arXiv preprint arXiv:2508.10104 (2025)
Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khali- dov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., et al.: Dinov3. arXiv preprint arXiv:2508.10104 (2025)
Pith/arXiv arXiv 2025
-
[19]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Singha, M., Pal, H., Jha, A., Banerjee, B.: Ad-clip: Adapting domains in prompt space using clip. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4355–4364 (2023)
2023
-
[20]
ISPRS Ann
Sohn, G., Jung, J., Gerke, M., Baillard, C., Benitez, S., Breitkopf, U.: The isprs benchmark on urban object classification and 3d building reconstruction. ISPRS Ann. Photogramm. Remote Sens (2012)
2012
-
[21]
arXiv preprint arXiv:2509.26036 (2025)
Timmermann, C., Lee, H., Lee, W.: Semobridge: Semantic modality bridge for efficient few-shot adaptation of clip. arXiv preprint arXiv:2509.26036 (2025)
Pith/arXiv arXiv 2025
-
[22]
arXiv preprint arXiv:2604.05433 (2026)
Tsai, Y.J., Lin, Y.Y., Wang, C.Y.: Few-shot semantic segmentation meets sam3. arXiv preprint arXiv:2604.05433 (2026)
Pith/arXiv arXiv 2026
-
[23]
IEEE Transactions on Geoscience and Remote Sensing60, 1–18 (2021)
Wang, B., Wang, Z., Sun, X., Wang, H., Fu, K.: Dmml-net: Deep metametric learning for few-shot geographic object segmentation in remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing60, 1–18 (2021)
2021
-
[24]
arXiv preprint arXiv:2110.08733 (2021)
Wang, J., Zheng, Z., Ma, A., Lu, X., Zhong, Y.: Loveda: A remote sensing land-cover dataset for domain adaptive semantic segmentation. arXiv preprint arXiv:2110.08733 (2021)
Pith/arXiv arXiv 2021
-
[25]
In: proceedings of the IEEE/CVF inter- national conference on computer vision
Wang, K., Liew, J.H., Zou, Y., Zhou, D., Feng, J.: Panet: Few-shot image semantic segmentation with prototype alignment. In: proceedings of the IEEE/CVF inter- national conference on computer vision. pp. 9197–9206 (2019)
2019
-
[26]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Wang, X., Wang, W., Cao, Y., Shen, C., Huang, T.: Images speak in images: A generalist painter for in-context visual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6830–6839 (2023)
2023
-
[27]
In: Proceedings of the IEEE/CVF international conference on computer vision
Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., Huang, T.: Seggpt: Towards segmenting everything in context. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1130–1140 (2023)
2023
-
[28]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops
Waqas Zamir, S., Arora, A., Gupta, A., Khan, S., Sun, G., Shahbaz Khan, F., Zhu, F., Shao, L., Xia, G.S., Bai, X.: isaid: A large-scale dataset for instance segmentation in aerial images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 28–37 (2019)
2019
-
[29]
In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision
Xia, J., Yokoya, N., Adriano, B., Broni-Bediako, C.: Openearthmap: A bench- mark dataset for global high-resolution land cover mapping. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 6254–6264 (2023)
2023
-
[30]
Advances in neural information processing systems34, 12077–12090 (2021)
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems34, 12077–12090 (2021)
2021
-
[31]
IEEE Transactions on Geoscience and Remote Sensing60, 1–11 (2021)
Yao, X., Cao, Q., Feng, X., Cheng, G., Han, J.: Scale-aware detailed matching for few-shot aerial image semantic segmentation. IEEE Transactions on Geoscience and Remote Sensing60, 1–11 (2021)
2021
-
[32]
arXiv preprint arXiv:2405.20141 (2024) 18 J
Yilmaz, G., Peng, S., Pollefeys, M., Engelmann, F., Blum, H.: Opendas: Open- vocabulary domain adaptation for 2d and 3d segmentation. arXiv preprint arXiv:2405.20141 (2024) 18 J. Heo and J. Park
Pith/arXiv arXiv 2024
-
[33]
In: International Conference on Learning Representations
Zhang, R., Jiang, Z., Guo, Z., Yan, S., Pan, J., Dong, H., Qiao, Y., Peng, G., Li, H.: Personalize segment anything model with one shot. In: International Conference on Learning Representations. vol. 2024, pp. 18250–18279 (2024)
2024
-
[34]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision- language models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16816–16825 (2022)
2022
-
[35]
International journal of computer vision130(9), 2337–2348 (2022)
Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. International journal of computer vision130(9), 2337–2348 (2022)
2022
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.