{"id":"3b70499e-6340-422f-bcc9-9d631ad60b8d","arxiv_id":"2505.08589","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new publicly released drone dataset with multi-altitude urban imagery, 15-class annotations, and pose metadata enables benchmarking of altitude-dependent semantic segmentation.","lead":"This paper introduces MESSI, a dataset of 2,525 drone images taken at different altitudes over dense urban neighborhoods, with pixel-level labels for 15 object classes. It lets researchers test how well semantic segmentation models perform when the drone changes height, which matters for autonomous landing and navigation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fig. 7's altitude-transfer claim is confounded by the test mask: low-altitude tests use the full image while higher-altitude tests use shrinking central crops, so the reported degradation may reflect the evaluation region rather than training altitude.","rationale":"Good-faith reading: MESSI is a dataset contribution, and its central claim is that the 2,525-image collection with multi-altitude revisits, per-image pose, and 15-class annotations can support segmentation research and altitude studies. The existence of the dataset and the basic benchmark are largely supported by the collection protocol, the annotation workflow with double checking, the multiple models evaluated, and the per-class tables. The weakest point is the Fig. 7 experiment meant to demonstrate the dataset's unique altitude capability, because that experiment's dependent variable mixes the training-altitude manipulation with a systematically changing test mask. The reader identified this same issue, and I agree. I do not see grounds to reject the paper: the dataset itself is plausibly usable, and the benchmark tables do not depend on Fig. 7. The concern justifies the conditional status: Fig. 7 should be re-analyzed with a fixed test region or clearly reframed as an exploratory comparison, and the exact protocol (central-crop computation, resizing, seeds, number of runs) should be pinned down. I also note a minor internal inconsistency in Section 4.2, where the text says the Water category was omitted from Table 5, but Table 5 includes a Water row and reports mIoU both with and without Water; this is an editorial flaw, not a threat to the central dataset claim.","tokens_in":13041,"tokens_out":13226,"duration_ms":135905,"concrete_test":"Recompute the Fig. 7 curves with a fixed evaluation region and fixed pixel count at every test altitude: for each descend trajectory, take the central crop corresponding to the lowest-altitude footprint, resize it to one common resolution (e.g., 2048x1366) at all altitudes, and evaluate the same trained models on those normalized patches; also report per-class IoU on the same physical region. If the accuracy-versus-altitude trends, especially the 100 m training curve, flatten or reverse under this protocol, the original conclusion is an artifact of the shrinking evaluation mask rather than training altitude.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest link is the Fig. 7 experiment, which is the paper's evidence for its headline capability of studying altitude effects. The text states that accuracy was computed on the overlapping field of view shared by all altitudes in each descend trajectory: at the lowest altitude the full 5472x3684 image is scored, while at higher altitudes only progressively smaller central crops participate. This changes several quantities across the x-axis of Fig. 7 simultaneously: the number of evaluated pixels, the ground-truth class distribution, the object scale relative to the model's training crops, and the amount of surrounding context available to the network. The reported trend, for example that a model trained only at 100 m degrades as the test altitude decreases, could therefore be produced by the evaluation region itself rather than by the training altitude. A model may simply score higher on large low-altitude full images than on small high-altitude central patches for reasons unrelated to what it learned about altitude. Because the curves come from a single run with no error bars, the evaluation-mask confound cannot be separated from the model property the paper claims to isolate. The dataset itself is not in question; the specific altitude-effect conclusion in Fig. 7 is not established by the current protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MESSI, a drone-captured urban aerial image dataset of 2,525 images at 5472x3684 resolution, with 15 semantic classes, per-image pose and intrinsics, horizontal multi-altitude trajectories, and vertical descent sequences. It describes the annotation workflow with double-independent quality control, class statistics, training/validation/test splits, and reports benchmarks for several segmentation models under three class-weighting schemes. The best result is SegFormer-B3 with Sqrt weighting reaching 52.6 mIoU on the out-of-distribution test set. The paper also presents an experiment (Fig. 7) on how training altitude affects accuracy on vertical descent sequences, claiming that training at higher altitudes degrades performance at lower altitudes.","tokens_in":13260,"tokens_out":5763,"duration_ms":51250,"significance":"MESSI fills a real gap in the aerial semantic segmentation literature: publicly available datasets generally lack systematic revisits of the same area at multiple altitudes together with per-image pose and intrinsic parameters. The annotation workflow, with nearly 100% rechecking in the final QC phase, is a methodological strength, and the decision to withhold test ground truth for an online benchmark follows good practice. The multi-architecture, multi-weighting baselines are useful for the community. If the Fig. 7 altitude-transfer claim is made rigorous, the dataset would enable a genuinely new class of experiments on altitude and scale robustness of segmentation models.","major_comments":[{"comment":"The altitude-transfer experiment is confounded by an altitude-dependent evaluation mask. The text states that accuracy was calculated on the overlapping field of view, with the full image scored at the lowest altitude and progressively smaller central crops at higher altitudes. This simultaneously varies the number of evaluated pixels, the class distribution, and the object scale relative to the network's receptive field, so the reported degradation with test altitude may be an artifact of the evaluation region rather than a property of the training altitude. The curves also appear to come from a single run with no error bars. To support the claim, the authors should evaluate on a common ground-area intersection across altitudes (or a fixed central crop) and report mean and standard deviation over multiple training runs.","section":"Section 4.2, Fig. 7"},{"comment":"The text says that the Water category \"was omitted from Table 5 and from the mIoU calculations,\" yet Table 5 includes a Water row and reports both \"mIoU with water\" and \"mIoU without water.\" This internal contradiction affects the reported evaluation protocol and must be corrected, either by removing Water from the table and recomputing the mIoU values, or by revising the sentence to state that Water was included and that an additional mIoU without Water is also reported.","section":"Section 4.2, Table 5"},{"comment":"The description of the Fig. 7 training conditions is incomplete. The authors do not specify the exact sets of training altitudes used for each curve, the number of images per training condition, or whether the \"all altitudes\" model includes all Ir Yamim horizontal images at each altitude as well as the Agamim paths. Without these details, the experiment cannot be reproduced or interpreted, even after fixing the evaluation-mask issue.","section":"Section 4.1"}],"minor_comments":[{"comment":"The caption \"per attitude interval\" should read \"per altitude interval.\"","section":"Section 3.2, Fig. 3 caption"},{"comment":"The table header uses \"Agamin\" while the text and Figure 1 use \"Agamim\"; please standardize the spelling.","section":"Section 3, Table 2"},{"comment":"The abbreviation mAcc is used without being defined in Section 4.1; please define it as mean pixel accuracy alongside the mIoU definition.","section":"Section 4.2, results text"},{"comment":"The table lists 16 entries including \"void,\" but the paper refers to 15 classes; please add a note that void is an unlabeled catch-all class and not one of the 15 semantic classes.","section":"Section A.3, Table 8"},{"comment":"The openDD row contains a stray fragment \"Segmentation search sensor\" that appears to be a formatting error; please clean up the table cell.","section":"Section 2, Table 1"},{"comment":"The phrase \"double-independent checking\" is used once; please clarify in one sentence whether this means two annotators independently label each image or another protocol.","section":"Section 3.1"},{"comment":"The sample-image tables are hard to evaluate in the PDF because the images are small; consider providing higher-resolution crops or a zoomed inset for at least one row.","section":"Appendix, Tables 10-14"}],"recommendation":"major_revision","confidential_remarks":"The dataset itself is a solid and useful contribution, and the paper is within the journal's scope. The main concern is that the headline altitude-effect experiment (Fig. 7) is not currently rigorous enough to support the paper's third claimed contribution; this is fixable by evaluating on a common field of view and repeating runs with error bars. The Water-class inconsistency in the results section also needs a correction. I would support publication after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset itself is the real contribution, and it is a good one. MESSI gives the community something UAVid, UDD, and VDD do not: the same ground areas revisited at multiple altitudes, plus full vertical descent sequences, with per-image pose and intrinsics. The annotation workflow is described with unusual care, and the benchmark numbers across several model families give a solid first reference point. For anyone working on drone perception or altitude-robust segmentation, this is a resource worth having.\n\nThe soft spots are real but mostly fixable. The biggest one is Fig. 7, which the paper itself flags as the payoff for studying altitude effects. The text says accuracy is computed on the overlapping field of view, so the lowest altitude scores the full image while higher altitudes score progressively smaller central crops. That changes the evaluated pixels, class distribution, context, and effective object scale all at once. The reported degradation when training at 100 m and testing lower could reflect the evaluation region rather than the training altitude. With a single run and no error bars, that confound cannot be dismissed. The authors should either recompute on a fixed central region for all altitudes, or explicitly present the experiment as measuring an interaction between altitude and evaluation protocol, not altitude alone.\n\nMinor issues: there is an internal contradiction about the water class—the text says it was omitted from Table 5 and the mIoU calculations, but Table 5 includes a water row and reports mIoU with and without water. Also, the paper provides a dataset URL but no checksums or downloadable annotation format details in the text, which makes independent verification harder than it should be for a dataset paper. These are not deal-breakers.\n\nBottom line: the central claim—that MESSI is a novel, well-annotated multi-elevation dataset that can support segmentation research—holds up. The altitude-transfer experiment in Fig. 7 does not, as currently presented. I would send this to peer review, but I would ask the authors to fix the mask confound and clean up the water contradiction before it gets archived.\n\nRecommendation: referee it, with the expectation of a substantial revision to the experimental section.","headline":"MESSI is a genuinely useful multi-elevation dataset for aerial segmentation, but the paper's headline altitude-effect experiment (Fig. 7) is confounded by the evaluation mask and needs to be redone or reframed.","tokens_in":13813,"tokens_out":1756,"would_cite":true,"duration_ms":18328,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 2,525-image drone dataset maps how altitude changes semantic segmentation accuracy, with matched views of the same urban areas from 30 to 100 meters and vertical descents down to 10 meters.","keywords":["semantic segmentation","drone imagery","UAV dataset","multi-elevation","urban scene understanding","altitude transfer","aerial images","pixel-wise annotation"],"falsifier":"Compare per-altitude mIoU on fixed ground-projected footprints of equal real-world area, selecting the same physical region in each image rather than fixed pixel crops; if the accuracy gap between training and testing altitudes disappears or reverses under this metric, the altitude-transfer result is an evaluation artifact.","tokens_in":12840,"feed_emoji":"🚁","tokens_out":5945,"duration_ms":57249,"temperature":0.7,"pith_summary":"This paper presents MESSI, a 2,525-image dataset in which the same urban areas are photographed from several altitudes and revisited during vertical descents, with pixel-level labels for 15 classes and per-image pose and camera intrinsics. The central claim is that this multi-elevation structure makes it possible, for the first time, to study systematically how altitude affects semantic-segmentation accuracy, rather than treating altitude as an incidental variation. The authors train several segmentation models on the dataset and report that SegFormer-B3 reaches 52.6 mean Intersection-over-Union on an out-of-distribution test set. They also present evidence that training at one altitude and testing at another degrades accuracy, with high-altitude training losing accuracy as the test images descend to low altitudes.","feed_headline":"Drone dataset maps how altitude changes segmentation accuracy","feed_subtitle":"MESSI revisits the same urban areas from 10 to 120 meters with 15 labeled classes to expose view-dependent errors.","key_machinery":"The dataset structure itself is the load-bearing mechanism: horizontal trajectories at four fixed altitudes combine with vertical descent sequences over the same urban locations to create matched image sets in which scale and view angle vary while scene content stays roughly fixed. This is paired with per-image pose and camera intrinsics and an evaluation protocol that scores accuracy over the overlapping field of view shared by altitudes; that protocol is what turns the raw images into measurements of altitude transfer.","core_discovery":"MESSI is a benchmark of downward-looking drone photographs over dense urban neighborhoods, annotated pixel-wise with 15 classes. Its distinguishing design is that flight paths revisit the same ground areas at 30, 50, 70, and 100 meters, and add 15 vertical descents from 120 to 10 meters, so every region appears at several scales and viewpoints. Each image carries the drone's position and orientation and the camera's intrinsic parameters. The authors claim that because the altitude variation is built into the capture design, MESSI can train and evaluate semantic segmentation across elevations and can quantify how training altitude transfers to test altitude; they report a best out-of-distribution mIoU of 52.6 for SegFormer-B3 and evidence that extrapolating across altitude degrades accuracy.","pith_inferences":["Beyond the paper, the matched multi-scale views make MESSI a natural probe for separating scale effects from viewpoint effects in aerial recognition; cropping fixed ground footprints would factor out resolution and isolate view angle.","Beyond the paper, the reported altitude-transfer asymmetry suggests that altitude-aware augmentation, such as random zooming during training, may be more effective than training at a single altitude; this is a testable hypothesis the paper does not itself test.","Beyond the paper, the availability of pose and intrinsics could enable cross-altitude ground-truth propagation, where labels from a low-altitude pass are projected onto higher-altitude images to reduce annotation cost; the geometry needed is present in the dataset, but the paper does not attempt this."],"forward_implications":["A segmentation network trained on MESSI's multiple altitudes should generalize better to unseen drone elevations than a network trained on a single altitude, because the training distribution covers the scale and viewpoint range of a 3D flight.","The altitude-transfer effect reported in the paper means deployers can expect accuracy to degrade when a model trained at high altitude is applied to low-altitude imagery, and the dataset provides a controlled way to measure the magnitude of that drop.","Because pose and intrinsics are included, MESSI also supports downstream tasks beyond segmentation, such as localization, navigation, and tracking.","The public test benchmark with undisclosed ground truth allows the community to compare segmentation architectures and class-weighting strategies on a common drone-imagery evaluation."],"supporting_citations":[{"why":"Motivates the altitude-related challenges of drone imagery that MESSI is designed to expose.","marker":"Han et al. (2022)"},{"why":"Closest prior urban drone segmentation dataset; supplies the tiling and pretraining recipe reused here.","marker":"Lyu et al. (2020)"},{"why":"Defines benchmark-property requirements for drone segmentation datasets that MESSI's design addresses.","marker":"Nigam et al. (2018)"},{"why":"Earlier two-altitude drone dataset; used as a comparison point for MESSI's systematic altitude variation.","marker":"Chen et al. (2018b)"},{"why":"Extended drone segmentation dataset captured at limited altitudes; baseline against which MESSI's multi-altitude advantage is argued.","marker":"Cai et al. (2023)"},{"why":"Large street-level urban dataset used for model initialization because its classes appear at many scales.","marker":"Cordts et al. (2016)"},{"why":"SegFormer is the backbone of the best-performing model, and its transformer design is central to the reported accuracy.","marker":"Xie et al. (2021)"},{"why":"The open-source toolbox used to train and test all models and to set the training hyper-parameters.","marker":"MMSegmentation Contributors (2020)"}],"fun_headline_variants":["Altitude shifts break drone-view segmentation models","Drone images at many heights stress-test segmentation nets","MESSI benchmark: multi-elevation drone images for segmentation","New drone dataset measures altitude impact on segmentation","Fly higher, segment worse: drone dataset quantifies drop"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that training altitude degrades performance at other altitudes rests on evaluating accuracy over the overlapping field of view, where higher-altitude scores are computed on smaller central crops; if that evaluation mask itself favors or penalizes certain altitudes, the observed transfer curve may be an artifact of the mask rather than a real altitude effect.","fun_headline_variants_meta":{"raw":{"variants":["Altitude shifts break drone-view segmentation models","Drone images at many heights stress-test segmentation nets","MESSI benchmark: multi-elevation drone images for segmentation","New drone dataset measures altitude impact on segmentation","Fly higher, segment worse: drone dataset quantifies drop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000496,"raw_usage":{"total_tokens":2408,"prompt_tokens":894,"completion_tokens":1514,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":1439}},"tokens_in":510,"tokens_out":1514,"duration_ms":12484,"temperature":1.0,"reasoning_tokens":1439,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:50:24.249658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare per-altitude mIoU on fixed ground-projected footprints of equal real-world area, selecting the same physical region in each image rather than fixed pixel crops; if the accuracy gap between training and testing altitudes disappears or reverses under this metric, the altitude-transfer result is an evaluation artifact.","supporting_citations":[{"cited_title":"Uavid: A semantic segmentation dataset for uav imagery","cited_arxiv_id":null,"evidence_quote":"Closest prior urban drone segmentation dataset; supplies the tiling and pretraining recipe reused here."},{"cited_title":"Ensemble knowledge transfer for semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Defines benchmark-property requirements for drone segmentation datasets that MESSI's design addresses."},{"cited_title":"The cityscapes dataset for semantic urban scene understanding","cited_arxiv_id":null,"evidence_quote":"Large street-level urban dataset used for model initialization because its classes appear at many scales."},{"cited_title":"MMSegmentation : Openmmlab semantic segmentation toolbox and benchmark","cited_arxiv_id":null,"evidence_quote":"The open-source toolbox used to train and test all models and to set the training hyper-parameters."}],"review_version":1}