{"id":"b234e933-9589-4d43-8a68-080fbbdeeb95","arxiv_id":"2506.03709","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"AetherVision-Bench curates existing RGB and IR segmentation datasets across three viewpoint classes and shows open-vocabulary models degrade sharply under sensor shift and viewpoint change.","lead":"A new benchmark, AetherVision-Bench, evaluates open-vocabulary semantic segmentation models on RGB and infrared images from aerial, slant-angle, and ground viewpoints, and reports that models drop sharply in performance on IR and across viewpoints. The benchmark is a curated collection of six existing datasets rather than a new dataset, and its design does not cleanly separate viewpoint effects from dataset effects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'nearly half' sensor-shift drop in Section 4.1 conflates 6-dataset RGB averages with 4-dataset IR averages; matched-pair drops are 29-68%, so the central quantitative claim is inflated.","rationale":"The reader's weakest assumption points to dataset comparability confounds (scene content, class sets, resolution) as undermining the attribution of mIoU drops to modality and viewpoint. My concern is more specific and more damaging to the paper's headline number: the reported RGB and IR mIoU averages are computed over different dataset subsets, with two high-scoring RGB-only datasets (ISPRS and UA Vid) inflating the RGB baseline. This is not an external validity issue; it is an internal analysis inconsistency. The central qualitative finding (IR performance is substantially worse) survives, but the quantitative claim 'decreases by nearly half' is an artifact of averaging over unequal dataset sets. A matched-pair reanalysis would settle it. Since the reader's verdict is already CONDITIONAL, and this concern strengthens the need for revision without overturning the benchmark's basic value, I recommend keeping the verdict at CONDITIONAL (UNCHANGED relative to the reader).","tokens_in":8125,"tokens_out":3366,"duration_ms":30484,"concrete_test":"Recompute all mIoU averages in Tables 2-4 using only the four datasets that provide both RGB and IR modalities (IndraEye, CART, MSRS, MVSeg), and also report per-dataset paired RGB/IR relative drops for the co-registered pairs (especially MSRS and CART). If the matched-average relative drop falls to roughly 30-45% or varies across models by more than 20 percentage points, the 'nearly half' claim in Section 4.1 should be revised to a model-dependent range.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 claims 'the robustness of all models to sensor shifts is poor. The overall performance decreases by nearly half,' citing Tables 2, 3, and 4. However, in every table the RGB mIoU column averages six datasets (ISPRS, UA Vid, IndraEye, CART, MSRS, MVSeg) while the IR mIoU column averages only four (IndraEye, CART, MSRS, MVSeg). The two RGB-only datasets, ISPRS and UA Vid, are also among the highest-scoring, which inflates the RGB baseline. Recomputing the Table 2 RGB average over the same four datasets that have IR, the RGB-to-IR drops are: SAN 21.88 to 15.53 (29%), Zegformer 13.55 to 4.33 (68%), OVSeg 19.83 to 10.62 (46%), CAT-Seg 25.35 to 14.30 (44%), OVSeg-L 22.80 to 14.43 (37%), CAT-Seg-L 25.83 to 18.09 (30%). Thus 'nearly half' is not a uniform finding; for SAN and CAT-Seg-L the drop is about 30%. The same averaging artifact affects Tables 3 and 4. Because the paper's central claim is specifically that sensor shift cuts performance by roughly half, this unmatched averaging is load-bearing: it is an internal arithmetic inconsistency, not merely an external confound. The direction of the effect is real, but the stated magnitude is unsupported as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AetherVision-Bench, a benchmark composed of six public semantic segmentation datasets spanning bird's-eye, slant-angle, and ground-level viewpoints, with four of the datasets also providing infrared (IR) imagery. The authors evaluate four open-vocabulary semantic segmentation models (SAN, ZegFormer, OVSeg, CAT-Seg) under three training settings (COCO-Stuff, MSRS RGB, and IndraEye RGB) and report per-dataset mIoU tables. The central findings are that current OVSS models generalize poorly from RGB to IR and from one viewpoint to another, with the paper claiming performance decreases by nearly half on sensor shift. The benchmark addresses a relevant gap in the evaluation of OVSS models for embodied AI applications.","tokens_in":8393,"tokens_out":3582,"duration_ms":35992,"significance":"If the benchmark and its conclusions are properly supported, this work would be a useful testbed for the robustness of open-vocabulary segmentation models in aerial and ground robotics, a largely underexplored direction. The paper brings together six datasets with different viewpoints and modalities and evaluates four representative model architectures under multiple training settings, which is a genuine contribution. However, the reproducibility and quantitative rigor of the benchmark are currently limited by the missing evaluation protocol and by an inconsistent computation of the headline sensor-shift claim. The direction of the findings is credible, but the specific magnitude stated in Section 4.1 is not supported as reported.","major_comments":[{"comment":"The evaluation protocol is not specified. Section 4 states only that models are evaluated using their official implementations and pretrained weights, with mIoU as the metric. The paper does not provide the class text prompts or prompt templates used for open-vocabulary evaluation, the input image resolution, the class-name mapping for each dataset, or the training hyperparameters for the three training settings (COCO-Stuff, MSRS, IndraEye). Because OVSS results are highly sensitive to prompt phrasing and resolution, the benchmark as presented is not reproducible, and the cross-dataset comparisons in Tables 2-4 mix unknown configurations. The full protocol (exact prompt strings, image sizes, number of iterations, learning rates, batch sizes, and any post-processing) must be included in the paper or a supplement.","section":"Section 4.1, Sensor Domain Shift"},{"comment":"The attribution of mIoU drops to sensor modality and viewing angle assumes the six datasets are comparable except along these two axes. In reality, ISPRS (aerial German urban scenes), UA Vid (drone over varied terrain), IndraEye (drone and ground), CART (Caltech aerial thermal), MSRS (ground-level fusion), and MVSeg (multispectral video) differ in scene content, class sets, annotation style, resolution, and sensor hardware. These confounds enter at dataset selection in Section 3.1 and affect all Section 4.1 conclusions. The paper should report the number of classes, class lists, and class overlap across datasets, and discuss how these differences are controlled or how their influence is bounded.","section":"Section 3.1, Dataset selection"},{"comment":"No variance or repeated-run statistics are reported. Each table shows a single mIoU value per model-dataset combination, so differences between models (e.g., Table 2: SAN RGB 27.79 vs. OVSeg 21.75) cannot be distinguished from training or evaluation noise. Given that the paper's purpose is to compare robustness across conditions, at least standard deviations over multiple seeds or bootstrap confidence intervals should be provided.","section":"Tables 2-4 and Section 4"}],"minor_comments":[{"comment":"There is a typo in the title: 'Open-V ocabulary' should be 'Open-Vocabulary'.","section":"Title and Abstract"},{"comment":"The phrase 'even when models are trained on the MSRS RGB dataset, their performance on co-registered IR pairs drops to less than half' is supported only for the MSRS IR column in Table 3; the statement should be clarified to specify that this refers to the MSRS subset, not the overall average.","section":"Section 4.1, Sensor Domain Shift"},{"comment":"The word 'visiable' appears in 'both visiable (RGB) and infrared (IR) sensor data'; it should be 'visible'.","section":"Section 1, Introduction"},{"comment":"'incase' should be 'in case'.","section":"Section 4.1, Evaluation under diverse viewing angles"},{"comment":"The radar chart in Figure 1 is described as illustrating the 'balanced nature' of the benchmark 'across these key dimensions,' but the axes (number of classes, resolution, total pixel volume, overall volume) are not defined in the text, and it is unclear how 'overall dataset volume' is normalized. A caption with axis definitions would improve clarity.","section":"Figure 1"},{"comment":"The conclusion restates the contributions but does not mention any limitations of the benchmark, such as the lack of class-overlap control or the small number of IR datasets; adding a limitations paragraph would strengthen the paper.","section":"Section 5, Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is timely and the dataset compilation is useful, but the evaluation protocol needs to be made fully reproducible and the sensor-shift claim needs to be recomputed with matched dataset sets. The inconsistencies are fixable within the scope of the manuscript, so I do not recommend rejection. I would also encourage the authors to make the exact prompt templates and evaluation code public, as is customary for benchmark papers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious referee, but the headline number needs fixing. The benchmark itself—six existing RGB/IR datasets organized by viewpoint, with three training settings—is a reasonable contribution, and the broad finding that OVSS models transfer poorly to IR and to unseen viewpoints is consistent with the tables and with prior work like MESS. The paper is honest about its ingredients; it does not claim new data or a new model.\n\nThe soft spots are real. The stress-test note is correct: Section 4.1's 'nearly half' drop is computed by averaging RGB over six datasets (including ISPRS and UA Vid, which have no IR counterpart) and IR over the four that do. Recomputing matched pairs gives drops of 29–68%, with SAN and CAT-Seg-L around 30%. The central quantitative claim is inflated as stated. The direction is real, but the magnitude is not supported. Also, the evaluation lacks error bars, chance-level baselines, and a fully specified protocol (prompt templates, resolution, hyperparameters). MESS already covers some of this ground, and the paper's claim of being 'first' is too strong—the novelty is in the specific viewpoint+modality combination, not in the categories.\n\nThe dataset-selection confound (different scene content, class sets, resolution) is worth noting but is partly inherent to benchmark construction; the paper should acknowledge it and, where possible, test on matched pairs. The self-citations are not problematic per se, but the writing has some typos and the related-work section is a bit thin on the 'first' claim.\n\nWho it's for: people building or evaluating OVSS models for drones and autonomous vehicles. It could be useful if the authors release code, prompts, and corrected statistics. As it stands, I'd take the qualitative trend seriously and the specific numbers with a grain of salt. I'd send it to review—the benchmark concept is useful and the measurements are mostly reproducible in principle—but the revision should be substantial.","headline":"A useful benchmark idea with a real arithmetic flaw in its central claim: the 'nearly half' sensor-shift drop is inflated by comparing 6-dataset RGB averages to 4-dataset IR averages.","tokens_in":8944,"tokens_out":1993,"would_cite":false,"duration_ms":18325,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AetherVision-Bench, a multi-angle RGB-infrared benchmark, shows that open-vocabulary segmentation models lose nearly half their mIoU when moving from RGB to infrared and generalize poorly across viewing angles.","keywords":["open-vocabulary semantic segmentation","RGB-infrared benchmark","multi-angle segmentation","sensor modality shift","viewpoint generalization","zero-shot transfer","aerial and ground perspectives","embodied AI perception"],"falsifier":"Take a single co-registered RGB-IR dataset with identical scenes and viewpoints, such as MSRS, and evaluate the same models after controlling class sets and resolution; if the RGB-to-IR mIoU drop disappears or shrinks sharply, the benchmark's attribution of the drop to sensor modality would be largely an artifact of dataset confounds rather than a general sensor-shift fragility.","tokens_in":7898,"feed_emoji":"🛰️","tokens_out":8463,"duration_ms":83776,"temperature":0.7,"pith_summary":"Open-vocabulary semantic segmentation (OVSS) assigns pixel labels from arbitrary text categories, but the paper asks whether such models survive the shifts that real drones and vehicles face: changing sensor modality from visible RGB to infrared, and changing viewing angle from bird's-eye to slant-angle to ground-level. To answer this, it introduces AetherVision-Bench, a collection of six RGB and four infrared datasets spanning those viewpoints, and evaluates four state-of-the-art OVSS models under three training regimes. The central finding is that sensor shift is a severe failure mode: mean intersection-over-union drops by nearly half when models move from RGB to IR, even on co-registered ground-level pairs. Models trained on one viewpoint also generalize poorly to the others, suggesting that downstream fine-tuning cannot replace diverse pretraining. The benchmark's contribution is a standardized protocol for measuring these two robustness axes, which the paper argues are essential for embodied-AI deployment.","feed_headline":"Sensor shifts halve open-vocabulary segmentation accuracy","feed_subtitle":"New benchmark across aerial and ground views shows mIoU drops by nearly half from RGB to infrared.","key_machinery":"The central object is AetherVision-Bench itself, a benchmark organized by a three-way taxonomy of viewing angles — bird's-eye view, slant-angle, and ground-level — crossed with RGB versus infrared modality. Its machinery is a fixed evaluation protocol: train each model on one of three source settings (COCO-Stuff, ground-level MSRS RGB, or slant-angle IndraEye RGB), then measure mIoU on all six RGB and four IR datasets in the benchmark. The cross of training viewpoint with evaluation viewpoint and the paired RGB-IR datasets make sensor and viewpoint shifts measurable; the mIoU drop between a dataset's RGB and IR columns is the operational definition of sensor-shift fragility.","core_discovery":"The paper's claim is that current open-vocabulary segmentation models are not robust to sensor modality or viewing-angle shifts, and that this brittleness is large enough to matter in practice. Across all three training settings, every evaluated model's mIoU falls by roughly half on infrared inputs relative to RGB, and the drop is worst on noisy multispectral video. Training on ground-level RGB does not help at aerial viewpoints, and training on slant-angle RGB does not help at ground level; models appear to lean on their pretraining to cover unseen viewpoints. The authors consequently conclude that sensor modality, viewing angle, and class semantics are the factors that dominate zero-shot transfer performance, and that these factors need to be measured explicitly by a dedicated benchmark.","pith_inferences":["The paper's datasets differ in scene content, class sets, resolution, and annotation style, so the measured sensor-shift and viewpoint-shift effects are likely partially confounded; a controlled sub-benchmark with matched classes and resolutions would let the field attribute the drop more precisely.","If the drop on co-registered RGB-IR pairs is a representation gap, then inserting an RGB-to-IR translation module before the segmenter and measuring how much mIoU it recovers would directly test that gap.","Viewpoint-specific text prompts, for example adding 'aerial view' or 'ground view' to class names, might recover part of the cross-view drop, since the paper finds textual similarity shapes performance; the benchmark could test this without retraining.","The near-halving result sets an upper bound on what current vision-language encoders contribute to infrared understanding, suggesting future progress may require pretraining on paired thermal-visible data rather than better segmentation decoders alone."],"forward_implications":["Deployed systems that need night or weather vision cannot currently rely on open-vocabulary segmentation over IR, since the near-halving of mIoU would produce unusable maps in low-light conditions.","Pretraining diversity matters more than fine-tuning data, because training on a single viewpoint leaves models unable to transfer to other viewpoints, so benchmark rankings should weight cross-viewpoint generalization rather than in-domain mIoU.","Co-registered RGB-IR pairs with the same scene and viewpoint still show the drop, meaning the loss is at least partly a representation gap in how models encode thermal input, not just a scene change.","The benchmark's three training settings give a standard protocol for testing fixes such as domain adaptation, RGB-to-IR translation, fusion, or viewpoint-conditioned prompts, with a clear before-and-after metric.","Performance on noisy multispectral video deteriorates even further, indicating that sensor noise compounds the modality shift."],"supporting_citations":[{"why":"The prior multi-domain benchmark establishes the multi-dataset evaluation protocol and first notes that infrared imagery is a hard sensor shift for OVSS models.","marker":"[2]"},{"why":"Supplies the COCO-Stuff training set used in the first evaluation setting, the standard zero-shot transfer source.","marker":"[3]"},{"why":"The cost-aggregation model whose pixel-wise similarity scores give the strongest COCO-Stuff-trained results and the clearest RGB-to-IR drop.","marker":"[4]"},{"why":"A two-stage mask-based OVSS model evaluated across all benchmark settings.","marker":"[5]"},{"why":"A mask-adapted CLIP segmentation model evaluated across all benchmark settings.","marker":"[15]"},{"why":"A side-adapter OVSS model evaluated across all benchmark settings.","marker":"[32]"},{"why":"Supplies the bird's-eye-view RGB test set for the aerial viewpoint condition.","marker":"[26]"},{"why":"Supplies the slant-angle RGB-IR dataset used both as a test set and as the training source for one evaluation setting.","marker":"[24]"},{"why":"Supplies the ground-level RGB-IR dataset with co-registered pairs, used both as a test set and as the training source for another setting.","marker":"[27]"},{"why":"Supplies the ground-level multispectral video test set whose noisy IR inputs produce the largest performance deterioration.","marker":"[9]"}],"fun_headline_variants":["Infrared cuts segmentation accuracy by half","Aerial vs ground views slash segmentation scores","Open-vocabulary models bleed accuracy on IR","Multispectral shifts halve zero-shot segmentation","Viewpoint change halves segmentation mIoU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's diagnosis assumes the six datasets differ mainly in viewing angle and sensor modality, but they also differ in scene content, class sets, resolution, annotation style, and sensor hardware, so part of the measured mIoU drop could come from those other differences.","fun_headline_variants_meta":{"raw":{"variants":["Infrared cuts segmentation accuracy by half","Aerial vs ground views slash segmentation scores","Open-vocabulary models bleed accuracy on IR","Multispectral shifts halve zero-shot segmentation","Viewpoint change halves segmentation mIoU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000422,"raw_usage":{"total_tokens":2110,"prompt_tokens":832,"completion_tokens":1278,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":1211}},"tokens_in":448,"tokens_out":1278,"duration_ms":8310,"temperature":1.0,"reasoning_tokens":1211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:56:49.085357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a single co-registered RGB-IR dataset with identical scenes and viewpoints, such as MSRS, and evaluate the same models after controlling class sets and resolution; if the RGB-to-IR mIoU drop disappears or shrinks sharply, the benchmark's attribution of the drop to sensor modality would be largely an artifact of dataset confounds rather than a general sensor-shift fragility.","supporting_citations":[{"cited_title":"What a mess: Multi-domain evaluation of zero-shot semantic segmentation","cited_arxiv_id":null,"evidence_quote":"The prior multi-domain benchmark establishes the multi-dataset evaluation protocol and first notes that infrared imagery is a hard sensor shift for OVSS models."},{"cited_title":"Coco- stuff: Thing and stuff classes in context","cited_arxiv_id":null,"evidence_quote":"Supplies the COCO-Stuff training set used in the first evaluation setting, the standard zero-shot transfer source."},{"cited_title":"De- coupling zero-shot semantic segmentation","cited_arxiv_id":null,"evidence_quote":"A two-stage mask-based OVSS model evaluated across all benchmark settings."},{"cited_title":"Open-vocabulary semantic segmentation with mask-adapted clip","cited_arxiv_id":null,"evidence_quote":"A mask-adapted CLIP segmentation model evaluated across all benchmark settings."},{"cited_title":"Side adapter network for open-vocabulary semantic segmentation","cited_arxiv_id":null,"evidence_quote":"A side-adapter OVSS model evaluated across all benchmark settings."},{"cited_title":"Isprs potsdam dataset within the isprs test project on urban classification, 3d building reconstruction and semantic labeling, 2012","cited_arxiv_id":null,"evidence_quote":"Supplies the bird's-eye-view RGB test set for the aerial viewpoint condition."},{"cited_title":"SAGA: Semantic-Aware Gray color Augmentation for Visible-to-Thermal Domain Adaptation across Multi-View Drone and Ground-Based Vision Systems","cited_arxiv_id":"2504.15728","evidence_quote":"Supplies the slant-angle RGB-IR dataset used both as a test set and as the training source for one evaluation setting."},{"cited_title":"Piafusion: A progressive infrared and visible im- age fusion network based on illumination aware.Information Fusion, 83:79–92, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the ground-level RGB-IR dataset with co-registered pairs, used both as a test set and as the training source for another setting."},{"cited_title":"Multispectral video se- mantic segmentation: A benchmark dataset and baseline","cited_arxiv_id":null,"evidence_quote":"Supplies the ground-level multispectral video test set whose noisy IR inputs produce the largest performance deterioration."}],"review_version":1}