{"id":"7b99d646-d0b7-47f3-a09a-a7e65e935c88","arxiv_id":"2507.22412","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"UAVScenes adds frame-wise image and LiDAR semantic labels, reconstructed 6-DoF poses, and 3D maps to 120k frames of the MARS-LVIG dataset, with six benchmark tasks.","lead":"UAVScenes is a new aerial-perception dataset built on top of the existing MARS-LVIG drone dataset. It adds manual semantic labels to 120,000 camera frames and matching LiDAR scans, together with 6-DoF poses, so researchers can benchmark segmentation, depth, localization, place recognition, and view synthesis on the same scenes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Terra pose validation is missing: the same DJI Terra 6-DoF poses are used as localization ground truth, as NVS supervision, and to render all semantic labels, yet no quantitative pose error is reported anywhere.","rationale":"The reader's weakest assumption is the right one. UAVScenes's novelty is not the raw data, which MARS-LVIG already provides as synchronized images, Livox LiDAR, and RTK positions; the novelty is the added 6-DoF poses and frame-wise annotations. Both additions are downstream of a single DJI Terra SfM reconstruction. Static semantic labels are rendered from Terra's map and poses (Sec. 3.2.1), LiDAR labels are projected through the same chain (Sec. 3.3), and the 6-DoF localization benchmark treats those poses as ground truth (Sec. 4.5). Any bias in Terra, such as roll/pitch error on ground-facing trajectories or drift in low-texture water and valley regions, would simultaneously corrupt labels and inflate or deflate benchmark numbers in ways that qualitative overlays cannot reveal. The paper has an independent reference for the translation component in the MARS-LVIG RTK/GNSS data but does not use it. The lack of inter-annotator agreement is a real secondary issue, but it is partly orthogonal: even perfect manual annotation cannot fix masks whose underlying geometry is wrong. Credit is due for the large labeling effort, the open repository, and the breadth of benchmarks, but these strengths do not remove the need for pose validation. The reader's CONDITIONAL verdict remains appropriate, so no verdict change is needed.","tokens_in":23899,"tokens_out":5733,"duration_ms":71369,"concrete_test":"Report the median and 90th-percentile absolute trajectory error (ATE) between the DJI Terra camera centers and the interpolated RTK/GNSS positions already available in MARS-LVIG, after performing a rigid Umeyama alignment of the translational components for each of the 8 splits. This is feasible because Terra uses GNSS only as initialization, so residual error is still meaningful. If median ATE exceeds 1 m or the 90th percentile exceeds 5 m on any split, the 'accurate 6-DoF poses' claim is unsupported and the localization, NVS, and rendered-label results must be re-examined.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The dataset's central added value is the set of 'accurate 6-DoF poses' and the frame-wise semantic masks. Both come from a single unvalidated reconstruction chain. Section 3.1 selects DJI Terra after trying FAST-LIVO, R3LIVE, COLMAP, RealityCapture, and Metashape, but the only evidence is qualitative alignment in Figures 2 and 3. Section 3.2.1 then states that static semantic classes are annotated on the Terra-reconstructed 3D map and 'rendered onto the corresponding camera views' to produce the 2D masks. Section 3.3 projects those image annotations onto LiDAR point clouds using the same calibration and pose chain. Section 4.5 uses the Terra poses directly as ground truth for the 6-DoF localization benchmark, and Section 4.4 uses them to supervise NVS. A systematic Terra bias would therefore propagate into every headline result: the semantic labels, the LiDAR labels, the localization numbers, and the NVS numbers. MARS-LVIG already provides RTK/GNSS positions (3-DoF location plus yaw), which could serve as an independent reference for the translational component, but the paper does not report any such comparison. The missing validation is more load-bearing than the lack of measured inter-annotator agreement, because even perfect manual labelers cannot correct masks whose underlying pose and geometry are wrong. The verdict should remain CONDITIONAL pending a quantitative pose check.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"UAVScenes is a multi-modal dataset paper built on the MARS-LVIG platform. It contributes 120k frame-wise semantic annotations for Hikvision images and Livox-Avia LiDAR point clouds (19 classes), 6-DoF poses and 3D maps reconstructed with DJI Terra, and benchmark evaluations of six perception tasks: image and LiDAR semantic segmentation, place recognition, novel view synthesis, 6-DoF visual localization, and depth estimation. The authors argue that no existing UAV dataset simultaneously provides 6-DoF poses and frame-wise annotations for both camera and LiDAR on real scenes, and they release code and train/test splits. The central quantitative claims are the accuracy of the Terra poses and the quality of the rendered/projected semantic labels.","tokens_in":24112,"tokens_out":7900,"duration_ms":90293,"significance":"The resource, if validated, would be a valuable addition to the UAV perception ecosystem. Its strengths are substantial: 120k annotated real frames, multiple environments with repeated traversals, open release of annotations and baseline code, and a broad set of benchmark tasks using recent state-of-the-art methods. The paper is careful in documenting reconstruction choices and recognizes limitations (e.g., the sparse pool class in the LiDAR segmentation benchmark). I do not see a derivational circularity in the benchmark methodology. The main risk is the unvalidated single-chain use of DJI Terra: the dataset's unique selling point is precisely the joint availability of pose, image labels, and LiDAR labels, and all three are generated from the same reconstruction. A quantitative pose check against MARS-LVIG's RTK data or an independent SfM/LiDAR-SLAM trajectory would materially raise confidence in every downstream number.","major_comments":[{"comment":"The paper's central 'accurate 6-DoF poses' claim is supported only by rendered-versus-real overlays. No pose-error statistics against an independent reference are reported, yet the Terra poses are used as ground truth for the localization benchmark in Section 4.5, as supervision for NVS in Section 4.4, as the geometry for rendering all static semantic masks in Section 3.2.1, and as the projection chain for obtaining LiDAR labels in Section 3.3. Any systematic bias in the Terra reconstruction therefore propagates into the semantic labels, LiDAR labels, localization numbers, and NVS numbers. This is more serious than the absence of inter-annotator agreement, because even perfect manual verification cannot fix masks whose underlying pose and 3D geometry are biased. Please add a quantitative validation: compare the Terra translational trajectory against the RTK/GNSS positions already available in MARS-LVIG, report per-split drift or loop-closure consistency on repeated traversals, or compare Terra poses with a LiDAR-inertial SLAM estimate on segments where it succeeds. Until such numbers are provided, the dataset's headline claim should be treated as conditional.","section":"Section 3.1, Figures 2 and 3"},{"comment":"Annotation quality assurance is described only as 'manual check' and 'thorough consistency checks,' with no inter-annotator agreement, no number of corrected frames, and no independent verification of the 3D map labels. Since the 2D masks are rendered from the Terra-reconstructed 3D map and the LiDAR masks are obtained by projecting those 2D masks through the same pose and calibration chain, the reported label quality is inseparable from the unvalidated reconstruction. Please report a measurable quality protocol, such as Intersection-over-Union between rendered masks and a manually annotated held-out subset, or inter-annotator agreement statistics, and state how dynamic objects are excluded from the static rendering step.","section":"Sections 3.2.1, 3.2.2, and 3.3"},{"comment":"The reported δ1 value for Depth Anything V2 (ViT-b) is 1.670, but Eq. (S7) defines δ1 as an average of indicator functions and hence it cannot exceed 1. This indicates either a unit error (values expressed as percentages while other rows are fractions) or a bug in the evaluation code, and it undermines the depth-estimation benchmark as a quantitative comparison. Please recompute the table, state the units for δ1 explicitly, and specify how invalid pixels (e.g., points beyond LiDAR range or on dynamic objects) are masked before computing AbsRel, SqRel, and δ1.","section":"Section 4.6, Table 7, and Eq. (S7)"}],"minor_comments":[{"comment":"Spacing artifacts such as 'UA Vs', 'UA VScenes', and 'MARS-LVIG'/'Mars-LVIG' appear in the title, abstract, and body; please standardize these names.","section":"Throughout"},{"comment":"The dataset is named MUM-FRL in the text but MUN-FRL in Table 1 and in reference [88]; please use one consistent name.","section":"Section 2.1 and Table 1"},{"comment":"The in-text citations for depth models are inconsistent: the text says 'Depth Anything [9]' but reference [9] is ZoeDepth, while [109] and [110] are the Depth Anything papers; please correct all citation keys.","section":"Section 4.6"},{"comment":"The GNSS interpolation step is described only briefly; please document the interpolation method, the coordinate-frame conventions, and whether the interpolated GNSS is used as hard constraints or only as initialization in Terra, since this affects the interpretation of any pose validation.","section":"Supplementary S6.2"},{"comment":"Please state explicitly whether mIoU and δ1 values are fractions or percentages; the δ1 column in Table 7 currently mixes conventions (e.g., 0, 0.010, and 1.670).","section":"Tables 3 and 7"},{"comment":"The paper should state the license and terms of use for the derived dataset relative to MARS-LVIG, especially because the base dataset is contributed by overlapping authors and the reconstruction uses the closed-source DJI Terra.","section":"Section 1 and dataset URL"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper builds on MARS-LVIG by overlapping authors (Shenghai Yuan and Lihua Xie appear in both author lists), and this derivative relationship is not explicitly framed as such in the main text. I do not view this as disqualifying, but the authors should state it plainly and confirm that the MARS-LVIG terms permit redistribution and reannotation. The 'only dataset' claim in Section 2.1 depends entirely on the comparison criteria in Table 1; the editor may want an independent check of that table, since a single misclassification would weaken the headline claim. My recommendation of major_revision is driven by the missing quantitative pose validation and the self-inconsistent δ1 value in Table 7, both of which I view as fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper delivers a genuinely useful resource: 120k frames of frame-wise semantic labels for both images and LiDAR, plus 6-DoF poses and a 3D map, built on MARS-LVIG. That combination does not exist in any cited UAV dataset, so the gap is real. The benchmarking across six tasks is extensive, and the pipeline write-up is transparent—they tried FAST-LIVO, R3LIVE, COLMAP, and a few commercial SfM pipelines before settling on DJI Terra, and they say why.\n\nThe weak point is exactly what the stress-test flagged. The Terra poses are used for three load-bearing purposes: as localization ground truth, as NVS supervision, and to render the semantic masks onto images and then project them onto LiDAR. A systematic pose bias would contaminate every headline number. Yet there is no quantitative pose error reported anywhere. The only evidence is qualitative overlay in Figs. 2 and 3. MARS-LVIG provides RTK/GNSS positions, so a simple comparison of the translational component against Terra would be cheap and would partially settle this. Not doing it leaves the core claim 'accurate 6-DoF poses' unsupported.\n\nThe annotation quality concern—manual verification without inter-annotator agreement—is real but softer; it's the pose validation that matters most. If that gets fixed, this is a strong dataset paper. The overlapping authorship with MARS-LVIG is not a problem given they clearly build on it and add new annotations.\n\nI'd send it to peer review, because the dataset fills a real gap and the evaluation is thorough, but the referee must insist on an independent pose check and a measurable error rate for labels. As is, the benchmark numbers should be treated as provisional. For my own work, I wouldn't cite it until that validation appears.","headline":"Useful multi-modal UAV dataset that currently lacks the one quantitative validation—pose accuracy—that its core claims depend on.","tokens_in":24785,"tokens_out":2661,"would_cite":false,"duration_ms":29148,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"UAVScenes provides the first UAV dataset with frame-wise semantic labels for both camera images and LiDAR point clouds together with 6-DoF poses, spanning over 120,000 frames across towns, valleys, airports, and islands.","keywords":["UAV dataset","multi-modal perception","semantic segmentation","LiDAR point cloud","6-DoF localization","place recognition","novel view synthesis","depth estimation"],"falsifier":"Take one of the eight splits and compute a reference trajectory with post-processed RTK-GNSS/INS; if the per-frame position error of the Terra poses is on the order of several meters, the 6-DoF localization and NVS benchmarks would no longer be measuring what they claim.","tokens_in":23653,"feed_emoji":"🚁","tokens_out":12695,"duration_ms":121614,"temperature":0.7,"pith_summary":"This paper introduces UAVScenes, a multi-modal benchmark dataset for aerial perception built on the raw SLAM-oriented MARS-LVIG data. The authors' claim is that UAVScenes is the first UAV dataset to offer frame-wise semantic annotations for both camera images and LiDAR point clouds on real scenes, together with 6-DoF poses, reconstructed 3D maps, and multiple traversals of towns, valleys, airports, and islands. The dataset contains over 120,000 annotated frames covering 19 semantic classes, and the authors demonstrate its value by benchmarking six perception tasks: image and LiDAR semantic segmentation, place recognition, novel view synthesis, 6-DoF visual localization, and depth estimation. If the claim holds, UAVScenes provides the research community a common ground for studying cross-modal and multi-task UAV perception on the same real-world scenes.","feed_headline":"120k labeled UAV frames open six perception benchmarks","feed_subtitle":"Per-frame labels for both sensors let a single scene set benchmark six perception tasks at once.","key_machinery":"The central mechanism is the annotation pipeline that turns raw SLAM data into multi-modal ground truth. Its key steps are: (1) DJI Terra SfM reconstructs one 3D map per split of one to three same-day sequences, yielding 6-DoF poses and geometry while accepting GNSS coordinates as initialization; (2) annotators label 16 static classes directly on the 3D maps, which are then rendered onto camera views to produce temporally consistent 2D semantic masks; (3) two dynamic classes (sedan, truck) are labeled instance-wise in 2D and fused with the rendered masks; (4) using the hardware-calibrated camera-LiDAR extrinsics, the image labels are projected into each Livox-Avia LiDAR frame to produce frame-wise 3D annotations. This pipeline is what makes the resulting 120k annotated frame pairs and eight reconstructed maps usable across six perception tasks.","core_discovery":"UAVScenes starts from the MARS-LVIG SLAM dataset, which provides hardware-synchronized camera and Livox-Avia LiDAR data but only 4-DoF RTK poses and no semantics. The paper's central discovery is the construction of a dataset that simultaneously provides 6-DoF poses, reconstructed 3D maps, and frame-wise semantic labels for both images and LiDAR point clouds—a combination no existing UAV dataset offers. The authors achieve this by splitting the sequences into eight consistent groups, reconstructing each group with DJI Terra structure-from-motion to get 6-DoF poses and geometry, manually labeling 16 static scene classes on the reconstructed 3D maps, rendering those labels onto camera frames to get image masks, adding manual instance labels for sedans and trucks, and projecting the image labels onto the LiDAR point clouds using the sensor calibration. They then benchmark six perception tasks with state-of-the-art methods, establishing baseline results on the new dataset.","pith_inferences":["Because the Terra-derived 6-DoF poses were validated only by qualitative image-overlay alignment, the localization and NVS numbers in the paper should be treated as method-comparison results rather than absolute pose-accuracy measurements.","The benchmark uses only the Livox-Avia LiDAR; excluding the encrypted DJI-L1 point clouds ties all 3D tasks to a single sensor's scan pattern, so the LiDAR segmentation and place-recognition results may not transfer to other LiDAR hardware.","The dataset's evening sequences and repeated traversals could support long-term place recognition and illumination-robustness studies, which the paper lists as data features but does not benchmark explicitly.","The pipeline of labeling 3D maps and rendering labels into frames could be adapted to add semantic annotations to other raw SLAM datasets at lower manual cost, a transfer the paper does not claim."],"forward_implications":["Image and LiDAR semantic segmentation can be benchmarked on the same real UAV scenes, with baseline results indicating Transformer-based image models (DeiT3) outperform ConvNets and the 'pool' class is a hard case for LiDAR segmentation.","Fusion-based place recognition methods outperformed camera-only and LiDAR-only baselines on this dataset, supporting the value of multi-modal fusion for aerial place retrieval.","Zero-shot monocular depth models perform poorly on UAV perspectives (best AbsRel 0.472), pointing to a clear gap for future depth estimation methods.","3D Gaussian Splatting methods (3DGS, Pixel-GS) clearly outperform the NeRF-based Instant-NGP on large-scale aerial scenes, giving a baseline for NVS research in this setting.","The dataset provides over 120k frames with multiple traversals across diverse environments, enabling robustness studies across lighting and speed variations."],"supporting_citations":[{"why":"The raw multi-modal sequences (camera, Livox-Avia LiDAR, DJI-L1, GNSS) and the calibrated camera-LiDAR suite come from this SLAM-oriented dataset; UAVScenes would not exist without it.","marker":"[57]"},{"why":"Provides the annotation methodology of segmenting the data into consistent splits and labeling static classes on a 3D map, which the authors follow for image annotations.","marker":"[7]"},{"why":"Shown in the comparison table as a UAV dataset with only map-level 3D annotations and no frame-wise labels, establishing the gap UAVScenes fills.","marker":"[50]"},{"why":"A real-world UAV dataset with map-level semantic labels only, used to argue that existing multi-modal datasets cannot support frame-wise image and LiDAR segmentation.","marker":"[62]"},{"why":"A camera-LiDAR UAV dataset whose DJI-L1 sensor produces encrypted point clouds, so frame-wise LiDAR access is impossible; this is load-bearing for the claim that UAVScenes is the first with frame-wise LiDAR annotations.","marker":"[104]"},{"why":"A UAV dataset equipped only with short-range LiDAR, cited to show why existing multi-modal UAV datasets cannot provide usable frame-wise point clouds from typical flight altitudes.","marker":"[88]"},{"why":"A camera+LiDAR SLAM dataset with no semantic annotations, representing the class of multi-modal datasets that cannot benchmark scene understanding tasks.","marker":"[75]"}],"fun_headline_variants":["UAV dataset with per-frame labels for both sensors tackles six tasks","New drone dataset offers 6-DoF poses and semantics for six benchmarks","Multi-modal aerial dataset benchmarks six perception tasks from one set","UAVScenes: labeled images and point clouds for six perception tasks","Dataset adds semantic labels to UAV SLAM frames for six benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The DJI Terra 6-DoF poses are treated as ground truth without being checked against an independent, quantitative pose reference.","fun_headline_variants_meta":{"raw":{"variants":["UAV dataset with per-frame labels for both sensors tackles six tasks","New drone dataset offers 6-DoF poses and semantics for six benchmarks","Multi-modal aerial dataset benchmarks six perception tasks from one set","UAVScenes: labeled images and point clouds for six perception tasks","Dataset adds semantic labels to UAV SLAM frames for six benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000363,"raw_usage":{"total_tokens":1963,"prompt_tokens":956,"completion_tokens":1007,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":916}},"tokens_in":572,"tokens_out":1007,"duration_ms":11616,"temperature":1.0,"reasoning_tokens":916,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:41:45.717017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the eight splits and compute a reference trajectory with post-processed RTK-GNSS/INS; if the per-frame position error of the Terra poses is on the order of several meters, the 6-DoF localization and NVS benchmarks would no longer be measuring what they claim.","supporting_citations":[{"cited_title":"Capturing, reconstructing, and simulat- ing: the urbanscene3d dataset","cited_arxiv_id":null,"evidence_quote":"A real-world UAV dataset with map-level semantic labels only, used to argue that existing multi-modal datasets cannot support frame-wise image and LiDAR segmentation."},{"cited_title":"Gauu-scene v2: Assessing the reliability of image-based metrics with expansive lidar image dataset using 3dgs and nerf","cited_arxiv_id":null,"evidence_quote":"A camera-LiDAR UAV dataset whose DJI-L1 sensor produces encrypted point clouds, so frame-wise LiDAR access is impossible; this is load-bearing for the claim that UAVScenes is the first with frame-wise LiDAR annotations."},{"cited_title":"Mun-frl: A visual-inertial-lidar dataset for aerial au- tonomous navigation and mapping","cited_arxiv_id":null,"evidence_quote":"A UAV dataset equipped only with short-range LiDAR, cited to show why existing multi-modal UAV datasets cannot provide usable frame-wise point clouds from typical flight altitudes."},{"cited_title":"Ntu viral: A visual- inertial-ranging-lidar dataset, from an aerial vehicle view- point","cited_arxiv_id":null,"evidence_quote":"A camera+LiDAR SLAM dataset with no semantic annotations, representing the class of multi-modal datasets that cannot benchmark scene understanding tasks."}],"review_version":1}