Pith. sign in

REVIEW 1 major objections 6 minor 14 references

OVEarth-Bench, a 172-category benchmark with vocabulary, referring, and reasoning queries on freshly collected satellite imagery, shows that current open-vocabulary Earth observation models are weak, that MLLM-based methods dominate the lea

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 03:17 UTC pith:5VKIDATC

load-bearing objection A carefully built OV-EO benchmark that deserves serious refereeing; the language-annotation validity is the one real risk, but it is not fatal. the 1 major comments →

arxiv 2607.27278 v2 pith:5VKIDATC submitted 2026-07-29 cs.CV

OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation

Kaiyu Li , Zepeng Xin , Zixuan Jiang , Jing Fu , Lanxuan Xue , Lingyu Zhang , Xiangyong Cao This is my paper
classification cs.CV
keywords open-vocabulary Earth observationbenchmarkcategory breadthquery diversityreferring segmentationreasoning segmentationzero-shot evaluationmultimodal large language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces OVEarth-Bench, a zero-shot benchmark for open-vocabulary Earth observation built from 172 categories and three query types—vocabulary phrases, referring expressions, and reasoning prompts—with mask, horizontal-box, and oriented-box annotations. It argues that existing benchmarks with narrow category sets cannot reliably rank models, and demonstrates that broad category coverage makes rankings stable. On this benchmark, current methods perform poorly: the best vocabulary segmentation score is 38.75% mean IoU, and the best detection F1 at 0.5 is 24.53%. MLLM-based systems occupy nine of the top ten vocabulary results and all ten top referring and reasoning results, while EO-specific methods rarely enter the top ten. If correct, the benchmark provides a shared yardstick and redirects design toward MLLM-centred architectures with EO-specific spatial adaptation.

Core claim

The central claim is that OVEarth-Bench—with its hierarchical 172-category taxonomy, 1,346 unique vocabulary strings, 732 referring expressions, and 1,056 reasoning queries on freshly collected imagery—is a reliable and more complete zero-shot evaluation instrument for open-vocabulary Earth observation. The experiments support three findings: current open-vocabulary EO performance is limited (best ma-IoU 38.75% vocabulary, 41.32% referring, 37.96% reasoning; best detection mi-F1 0.5 at 24.53%, 35.56%, and 26.12%); MLLM-based methods lead, with 9/10 and 10/10 of the top results; and EO-specific methods rarely match the strongest general models. A category-subsampling analysis shows that ranki

What carries the argument

The benchmark itself is the carrying object: a seven-domain, 172-category hierarchy grounded in land-use standards, dataset inventories, and online map tags, with newly collected global imagery, human-verified polygon masks, derived horizontal and oriented bounding boxes, and a two-stage query-generation pipeline that combines LLM generation, automated verification, and manual review. Its evaluation protocol scores vocabulary, referring, and reasoning queries under a unified zero-shot rule for masks and boxes, and its category-subset resampling procedure quantifies ranking stability. The two-stage query pipeline is essential for preventing absent-category and spatial-reference errors; the pa

Load-bearing premise

The benchmark's conclusions rest on the correctness of its human-reviewed language queries; if those queries contain residual LLM-style phrasing or ambiguous spatial references, the measured rankings—especially the MLLM dominance in referring and reasoning—could reflect the query genre rather than genuine open-vocabulary Earth-observation capability.

What would settle it

Run an independent re-annotation of a random sample of OVEarth-Bench queries by a new set of annotators with no access to the original labels, measuring inter-annotator agreement and then re-ranking all 49 models on the re-validated subset. If agreement is low (for example, kappa below 0.6) or the re-validated ranking differs materially from the original—if MLLM dominance shrinks or the top-five recovery drops—the central claims about MLLM superiority and ranking stability are not robust.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Broad category coverage is necessary for trustworthy model comparison: evaluations on 5–20 categories can reorder leading models, so future open-vocabulary EO benchmarks should include 50 or more categories.
  • MLLM-based methods, especially grounding models followed by a promptable segmenter, set the current standard; EO-specific training alone does not close the gap.
  • Presence/absence judgment is nearly independent of localization quality (MCC is at most 0.35 for all but one model, and rank correlation with ma-IoU is 0.107), so models need explicit negative-supervision and calibrated presence prediction.
  • Neither grounding-based nor direct-segmentation paradigms dominate uniformly: the highest score comes from one large grounding model, while the second-best grounding model trails the second-best segmentation model.
  • Future progress will likely come from combining MLLM semantic breadth with EO-specific spatial competence: multiscale encoding for small targets, orientation-aware decoding, and EO instruction tuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the benchmark's language-validity assumption holds, the MLLM advantage likely reflects pretraining scale and region-language instruction data rather than a fundamental architectural superiority; a controlled study varying only the output head would settle this.
  • The 57.3% human revision rate for referring expressions implies that LLM-generated spatial language is unreliable without image-based verification; other benchmark builders may need similarly strict pipelines, and evaluations using unverified LLM queries could artificially favour or penalise certain models.
  • The benchmark's negative-query set (2,168 negatives) could be repurposed as a standalone presence-calibration leaderboard, since current MCC scores are near zero for most models.
  • Because 91.8% of boxes occupy under 1% of image area, OVEarth-Bench is a demanding small-object test; grounding-based MLLMs may improve further with high-resolution encodings specifically for such targets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper introduces OVEarth-Bench, a new open-vocabulary Earth observation benchmark built from newly collected imagery. It provides 172 hierarchical categories, 1,346 unique vocabulary strings, 732 referring expressions, and 1,056 reasoning queries, together with human polygon masks, HBBs, and OBBs. The authors evaluate 49 general and EO-specific model variants under a unified zero-shot protocol, reporting segmentation metrics (ma-P/R/IoU, mi-IoU, MCC) and detection metrics (mi-F1 at multiple IoU thresholds). Their main empirical claims are: (1) current methods achieve limited performance (best vocabulary ma-IoU 38.75%, best mi-F1_0.5 24.53%); (2) MLLM-based methods dominate the leaderboards, especially on referring and reasoning tasks; (3) EO-specific methods rarely enter the top ten; and (4) rankings computed on narrow category sets (5–20 categories) are materially unstable, with median Spearman rho = 0.84 at 5 categories and top-five recovery of 0.6. The paper includes detailed appendices covering prompt templates, metrics, stability analyses, model descriptions, and limitations.

Significance. If the benchmark and its measurements hold, OVEarth-Bench would be a valuable shared resource for open-vocabulary EO evaluation. Its strengths include: newly collected images that reduce (though do not eliminate) pretraining overlap; a broad taxonomy with explicit positive/negative vocabularies; a unified protocol across masks and boxes; a released evaluation package; and an honest reporting of query-generation revision statistics (57.3% referring rewriting, 28.6% auto-correction, 16.0% negative modification). The category-subset stability analysis is a useful methodological contribution, quantifying an effect that many benchmark papers assert informally. However, the central validity of the language ground truth is not yet established to the standard required for a benchmark whose headline finding is 'MLLMs dominate referring/reasoning'. The absence of inter-annotator agreement or an independent audit, combined with the very high human revision rates, leaves open the possibility that measured rankings partly reflect query-genre artifacts. The paper's significance is therefore conditional: the resource is potentially important, but the evidence presented does not fully secure the l

major comments (1)
  1. [Experiments; Tables 2–5] The paper's comparison of grounding-based vs. segmentation-based paradigms (Appendix D, Table 11) selects the two highest-scoring models within each paradigm per task, based on the same full-benchmark metric used in the comparison. This is acknowledged as a limitation, but it weakens the generality of the statement 'no general advantage for either paradigm.' The selection procedure makes the comparison sensitive to the specific top-2 models chosen, and the crossing pattern (Rex-Omni+SAM first, LocateAnything+SAM second) could be interpreted as showing that the top grounding model is strong but the second is not. I suggest also reporting aggregate quantiles or a head-to-head matched comparison (e.g., models with comparable training scale), so readers can calibrate how much the conclusion depends on the choice of the two representatives.
minor comments (6)
  1. [Abstract] 'best mi-F1 0.5 24.53%' should be formatted consistently (e.g., mi-F1@0.5 = 24.53%). Also, '0.5' appears to be a subscript; ensure the notation matches the body.
  2. [Table 12] The negative phrase list for 'stele' contains 'moutain' (typo for 'mountain'). Please proofread the vocabulary lists.
  3. [Appendix D, Figure 6] The shaded bands are described as 'empirical 2.5th–97.5th percentiles' and not confidence intervals. Good. Please also report the number of annotations per sampled category subset (e.g., median and range) in the caption or text, to support the sample-size concern.
  4. [Appendix C, Equation (11)] The MCC formula is correct, but when the denominator is zero the evaluator returns a null value. The paper should state explicitly which models have undefined MCC and why (e.g., always-predict-positive). Currently Table 2 shows a dash without a note; adding a footnote would improve reproducibility.
  5. [Dataset Statistics; Figure 8] The task-coverage per annotation shows 81% of annotations contain all three query types, but only 1% contain OV only. This is a useful statistic, but the 5 OV-only annotations should be described—are they cases where referring/reasoning were impossible (e.g., small indistinct objects)? A short clarification would help.
  6. [References] Several references have identifiers such as 'arXiv:2605.23144' and 'arXiv:2605.27365' that appear to be future-dated relative to the submission domain. Verify that these are correct and that all citations are to published or openly available works.

Circularity Check

0 steps flagged

No significant circularity: the benchmark findings are measurements from a newly collected eval set, with no fitted parameters, no predictions defined by construction, and no load-bearing self-citation chain.

full rationale

OVEarth-Bench is an evaluation/benchmark paper. Its headline claims are measurements of 49 model variants on a newly collected dataset, not derivations from a theory with fitted constants. The zero-shot protocol is explicit: 'All methods are evaluated zero-shot with respect to OVEarth-Bench, with no training, fine-tuning, or threshold selection on the benchmark.' The only equations define metrics (ma-IoU, mi-F1, MCC) from ground-truth masks/boxes and model outputs; no metric or ranking is defined in terms of a parameter fitted to those same outputs. The stability analysis (subset sampling vs full ranking) is a self-contained resampling statistic, not a fitted prediction. The authors do include their own prior models (SegEarth-OV/OV3/R1/R2, Pi-Seg) as baselines, but these are evaluated and generally underperform; one top-ten vocabulary entry and none in referring/reasoning, so the self-citations do not inflate the central conclusion. The manuscript's own limitation statements (Appendix G: 'this interpretation remains descriptive rather than causal', 'geographic sampling remains uneven', 'some categories contain only one or two samples') are acknowledged uncertainties, not circular steps. The most load-bearing validity risk is the language ground truth: queries were LLM-generated and human-reviewed, with high revision rates ('Human reviewers changed or removed 57.3% of referring expressions...'). That is a construct-validity/annotation-quality concern about the measurement instrument, not a circularity in the derivation chain; no conclusion is equivalent to the query-generation input by construction. Therefore no circular step is identified.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

This is an empirical benchmark paper, not a derivation, so there are no fitted constants in the physics sense. The structural hand-chosen inputs are the taxonomy, the query-generation design, and the exclusion rules; the axioms above are the unproved background premises the benchmark conclusions depend on.

free parameters (1)
  • Detection-subset exclusion rules = 12 non-instanceable categories removed; 4 annotations (heliostat×2, fish raft house, moat) removed post-hoc
    Hand-chosen rules defining the box-evaluation subset of 4,810 boxes; the post-hoc part is explicitly labeled so in Appendix E and affects which instances are scored.
axioms (5)
  • domain assumption The 172-category hierarchy derived from GB/T 21010-2017, EarthNets inventories, and OpenStreetMap tags is the reference semantic space for open-vocabulary EO.
    Taxonomy and Category Design section; representativeness of the category space is assumed, not demonstrated — the paper notes the taxonomy 'does not exhaustively cover the open world' (Appendix G).
  • domain assumption The human-reviewed LLM annotations are correct ground truth: positive phrases match masks, negatives are truly absent, and referring/reasoning queries uniquely identify the full target.
    Query Generation Pipeline; 57.3% of referring expressions were rewritten by humans, so final validity rests on three annotators' judgment; no inter-annotator agreement is reported.
  • domain assumption Default-hyperparameter zero-shot evaluation (no training or threshold selection on the benchmark) is a fair comparison across heterogeneous model families, including detection outputs converted to masks via SAM2.1-L.
    Experimental Setup / Inference Settings in Appendix F; the SAM2.1-L conversion adds a shared but extra pipeline stage whose interaction with each detector is unmeasured.
  • domain assumption Newly collected images reduce, but do not eliminate, overlap with model pretraining corpora.
    The paper itself states this in Appendix G: 'it cannot guarantee complete absence from the pretraining corpora of foundation models.'
  • domain assumption Query-level averaging is the primary aggregation; annotation- and category-macro alternates confirm robustness.
    Appendix C reports Spearman rho 0.994–0.999 between Q-, Ann-, and Cat-mIoU rankings; the robustness claim depends on equivalence of these aggregations.

pith-pipeline@v1.3.0-alltime-deepseek · 42677 in / 20520 out tokens · 844359 ms · 2026-08-04T03:17:08.591376+00:00 · methodology

0 comments
read the original abstract

Open-vocabulary Earth observation (EO) aims to localize geospatial concepts specified in natural language rather than a fixed label set. Existing benchmarks, however, usually cover narrow category vocabularies or limited query forms. To fill this gap, we introduce OVEarth-Bench, which extends existing evaluation in two directions: category breadth, through broad hierarchical category coverage with positive and negative expressions, and query diversity, through vocabulary, referring, and reasoning queries. The benchmark supports mask and box localization under a unified zero-shot protocol. We evaluate a broad set of general and EO-specific methods. The evaluation reveals that: (1) the performance of current methods remains limited, while broader category coverage yields more stable model rankings; (2) MLLM-based methods achieve the strongest overall performance; and (3) EO-specific methods generally underperform general models and rarely match the strongest methods. These findings provide guidance for future open-vocabulary EO method design and highlight the importance of developing more realistic, diverse, high-quality, and large-scale benchmarks for reliable evaluation. Our data and evaluation package are released at https://earth-insights.github.io/OVEarth-bench.

Figures

Figures reproduced from arXiv: 2607.27278 by Jing Fu, Kaiyu Li, Lanxuan Xue, Lingyu Zhang, Xiangyong Cao, Zepeng Xin, Zixuan Jiang.

Figure 1
Figure 1. Figure 1: Compared with previous EO benchmarks with [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visual examples of some object categories in the OVEarth-Bench dataset. See Figure 14 for more samples. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Construction pipeline of OVEarth-Bench: taxonomy design, POI-guided image collection, human mask annotation and [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Image width and spatial-resolution distributions. [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Annotation statistics of OVEarth-Bench. expressions with spatial context, and the reasoning task pro￾vides implicit functional descriptions without naming the category. The evaluator supports segmentation masks, HBBs, and OBBs as separate localization modalities, each scored with its corresponding metrics. For segmentation, following prior work (Lai et al. 2024), we report macro Precision, Recall, and IoU … view at source ↗
Figure 6
Figure 6. Figure 6: Sensitivity of vocabulary-segmentation model [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Detailed image statistics of OVEarth-Bench. The [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Detailed annotation statistics of OVEarth-Bench. The panels summarize task and vocabulary counts, task coverage per [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Word cloud of terms used in the language an [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Model comparisons. (a) Composition of the [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Performance of the two models selected within [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: Bidirectional category overlap between OVEarth [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Additional visual examples from OVEarth-Bench, illustrating the breadth of annotated EO concepts across [PITH_FULL_IMAGE:figures/full_fig_p034_14.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 4 linked inside Pith

  1. [1]

    Verify whether any negative candidates are actually present in the image

  2. [2]

    Check each referring expression for factual correct- ness, especially spatial relations and direction words (north/south/east/west, left/right, adjacent, near)

  3. [3]

    InProceedings of the Computer Vision and Pattern Recognition Conference, 15033–15042

    Distilling spectral graph for object-context aware open-vocabulary semantic segmentation. InProceedings of the Computer Vision and Pattern Recognition Conference, 15033–15042. Lai, X.; Tian, Z.; Chen, Y.; Li, Y.; Yuan, Y.; Liu, S.; and Jia, J. 2024. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF conference on computer...

    Pith/arXiv arXiv 2024

  4. [4]

    verification_results

    If a referring expression is correct, keep it unchanged and mark it valid. Output strictly as a JSON object with these keys: 1,000 2,000 3,000 4,000 5,000 6,000 Width (px) 0 50 100 150 200 250 300Count (a) Image Width Distribution Median 1,385 Mean 1,639 0 2 4 6 8 10 Spatial Resolution (m/pixel) 0 50 100 150 200 250Count +8 outliers (>10 m/px) (b) Spatial...

    2000

  5. [5]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 34865–34876

    Unigeoseg: Towards unified open-world segmentation for geospatial scenes. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 34865–34876. Pan,J.;Liu,Y.;Fu,Y.;Ma,M.;Li,J.;Paudel,D.P.;VanGool, L.;andHuang,X.2025. Locateanythingonearth: Advancing open-vocabulary object detection for remote sensing commu- nity. InProceedings ...

    2025

  6. [6]

    InInternational Conference on Learning Representations, volume 2025, 28085–28128

    Sam 2: Segment anything in images and videos. InInternational Conference on Learning Representations, volume 2025, 28085–28128. Rottensteiner, F.; Sohn, G.; Jung, J.; Gerke, M.; Baillard, C.; Benitez, S.; and Breitkopf, U. 2012. The ISPRS benchmark on urban object classification and 3D building reconstruction. Shabbir, A.; Zumri, M.; Bennamoun, M.; Khan, ...

    Pith/arXiv arXiv 2025

  7. [12]

    If a referring expression contains an incorrect or overly specific location claim, rewrite it into a correct, safer expression that still names the category

  8. [19]

    tower” or “greenhouse

    Thus, no query form is uniformly easier or harder across the evaluated models. The model-level changes can also be substantial. From vocabulary to referring queries, SegEarth- R2 decreases by 7.43ma-IoU points, whereas UniGeoSeg and LocateAnything+SAM increase by 7.80 and 8.12 points, respectively. Thesearedescriptivecomparisonsontheshared benchmark targe...

    2026

  9. [2019]

    InProceedingsoftheIEEE/CVFconferenceon computer vision and pattern recognition workshops, 28–37

    isaid: Alarge-scaledatasetforinstancesegmentationin aerialimages. InProceedingsoftheIEEE/CVFconferenceon computer vision and pattern recognition workshops, 28–37. Xia, G.-S.; Bai, X.; Ding, J.; Zhu, Z.; Belongie, S.; Luo, J.; Datcu, M.; Pelillo, M.; and Zhang, L. 2018. DOTA: A large-scale dataset for object detection in aerial images. In Proceedings of th...

    2018

  10. [2022]

    positive_ovs_phrases

    Earthnets: Empowering ai in earth observation.arXiv preprint arXiv:2210.04936. Yao, L.; Liu, F.; Chen, D.; Zhang, C.; Wang, Y.; Chen, Z.; Xu, W.; Di, S.; and Zheng, Y. 2025. Remotesam: Towards segment anything for earth observation. InProceedings of the 33rd ACM International Conference on Multimedia, 3027–3036. Ye, C.; Zhuge, Y.; and Zhang, P. 2025. Towa...

    Pith/arXiv arXiv 2025

  11. [2023]

    InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 6254–6264

    Openearthmap: A benchmark dataset for global high-resolution land cover mapping. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 6254–6264. Xin, Z.; Li, K.; Chen, L.; Li, W.; Yuchen, X.; Qiao, H.; Zhang, W.; Meng, D.; and Cao, X. 2026. Segearth-r2: Towards comprehensive language-guided segmentation for remote sensing im...

    2026

  12. [2024]

    Huang, Z.; Feng, Y.; Liu, Z.; Yang, S.; Liu, Q.; and Wang, Y

    Cross-modal bidirectional interaction model for re- ferring remote sensing image segmentation.arXiv preprint arXiv:2410.08613. Huang, Z.; Feng, Y.; Liu, Z.; Yang, S.; Liu, Q.; and Wang, Y

    Pith/arXiv arXiv

  13. [2025]

    InProceedings of the IEEE/CVF International Conference on Computer Vision, 8384–8394

    Openrsd: Towards open-prompts for object detection in remote sensing images. InProceedings of the IEEE/CVF International Conference on Computer Vision, 8384–8394. Jiang, Q.; Huo, J.; Chen, X.; Xiong, Y.; Zeng, Z.; Chen, Y.; Ren, T.; Yu, J.; and Zhang, L. 2026. Detect anything via next point prediction. InProceedings of the IEEE/CVF Conference on Computer ...

    2026

  14. [2026]

    Ni,S.;Wang,D.;Chen,H.;Guo,H.;Zhang,N.;andZhang,J

    Unipixel: Unifiedobjectreferringandsegmentationfor pixel-level visual reasoning.Advances in Neural Information Processing Systems, 38: 126078–126108. Ni,S.;Wang,D.;Chen,H.;Guo,H.;Zhang,N.;andZhang,J