{"id":"ea28966a-56c3-4094-89e4-581c1e3b70e1","arxiv_id":"2505.02148","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"STU is the first public 3D LiDAR dataset to provide dense semantic and instance labels for road anomalies, and baseline tests show current methods detect very few of them.","lead":"The STU dataset is a new collection of 3D LiDAR scans from a car, with dense labels for unusual objects on the road such as chairs, buckets, and ladders. It aims to give autonomous driving researchers a public benchmark for detecting road debris that standard models often miss.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OOD premise unverified: staged anomaly objects may overlap with 'other-object' points in SemanticKITTI/Panoptic-CUDAL training data, so the benchmark may not measure true anomaly segmentation.","rationale":"The reader's weakest assumption focused on label trustworthiness (no inter-annotator agreement, pseudo-label initialization, no label-noise analysis). That concern is valid and remains. My review identifies a related but more specific load-bearing issue: the benchmark's claim to measure anomaly segmentation depends on the staged objects being truly absent from the training distribution, including the 'other-object' catch-all class. The paper asserts this design property but provides no evidence for it, and the supplementary material actually demonstrates that several staged object types (garbage bins, pots, bags) resemble 'other-object' instances in SemanticKITTI. If this overlap exists, the baselines' poor OOD scores are not evidence of the difficulty of 3D anomaly segmentation; they are evidence that the model correctly recognizes in-distribution miscellaneous objects. This concern is concrete and testable with the authors' own trained model and label files. It does not require external literature or assumptions about unreleased data. The paper could resolve it by releasing the per-object category list and a similarity analysis; because the paper is otherwise a useful dataset contribution and the authors promise release, the conditional verdict remains appropriate. I therefore keep the reader's CONDITIONAL verdict unchanged, while noting that the condition should explicitly include verification of the OOD/no-intersection claim and a fixed labeling protocol.","tokens_in":17350,"tokens_out":10525,"duration_ms":138232,"concrete_test":"Run the trained SemanticKITTI/Panoptic-CUDAL Mask4Former-3D model (without STU fine-tuning) on all STU anomaly sequences and record, for each anomaly instance, the majority predicted class and the mean softmax confidence on anomaly points. Report per object category the fraction of instances confidently classified (>0.5 confidence) into any inlier class, especially 'other-object' or the closest known class. Independently, compute feature-space nearest-neighbor distances between every staged anomaly instance and all 'other-object'/'movable object.debris' instances in the SemanticKITTI, Panoptic-CUDAL, and nuScenes training sets.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central premise is that the staged objects are truly out-of-distribution relative to the training data. Section 4 states: 'To strictly define anomalies, we analyze all other-objects present in our training set ... We specifically design our dataset such that we do not have an intersection between anomalous objects and the aforementioned classes.' No protocol, per-category object list, or quantitative similarity check is reported. The supplementary material (Section 11) shows that SemanticKITTI 'other-object' includes trash bins, garbage cans, pots, billboards, and small tables. The staged object list (Section 7) includes indoor garbage bins, buckets, pots, bags, and similar items. If any staged object would be assigned to the 'other-object' or 'movable object.debris' class by a model trained on SemanticKITTI/Panoptic-CUDAL, or is a close feature-space neighbor of such training instances, then it is present in the training distribution and is not an anomaly by the paper's own definition. In that case, the reported low OOD performance may reflect correct inlier recognition rather than the difficulty of 3D anomaly segmentation, which would undermine the central contribution. Additionally, Section 3.2 describes using baseline predictions to find missed objects and marking them 'unlabeled' for evaluation; without an independent, pre-registered labeling protocol, the evaluation mask is partially shaped by the baselines, making the benchmark scores difficult to interpret. Combined with the absence of inter-annotator agreement and label-noise analysis, neither the OOD status of the objects nor the fidelity of the labels is quantitatively established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces STU, a new dataset for road anomaly segmentation in 3D LiDAR point clouds. The dataset provides 70 fully annotated sequences with anomaly instances (19 validation, 51 test), dense semantic and instance labels, synchronized eight-camera images, and 128-beam LiDAR, plus two additional 'STU-inlier' sequences for training and validation. The authors adapt several 2D anomaly segmentation methods (MaxLogit, MC Dropout, Deep Ensembles, Void Classifier, RbA) to Mask4Former-3D and report that all baselines perform poorly on the OOD task, while closed-set performance on SemanticKITTI remains reasonable. They also provide dataset statistics and supplementary analyses, including distance/size breakdowns and a 2D control experiment on the front camera. The paper claims to be the first publicly available dataset for this task with dense 3D semantic labeling, LiDAR plus camera data, and temporal sequences.","tokens_in":17603,"tokens_out":9047,"duration_ms":98059,"significance":"If the dataset's ground truth is trustworthy and its anomalies are genuinely out-of-distribution relative to the training data, STU would be a valuable community benchmark: it is the first public LiDAR+camera anomaly segmentation dataset with instance-level dense labels and temporal sequences, and the baseline results convincingly show that 2D-derived methods do not transfer to 3D. The paper includes honest failure reporting, per-distance AP analysis, and a 2D control experiment on the front camera, which strengthen the empirical contribution. However, the value of the benchmark is conditional on two unverified assumptions: the non-overlap between staged anomalies and training-set 'other-object'/'debris' classes, and the quality and bias of the annotation process.","major_comments":[{"comment":"The OOD premise is not verified. The paper states: 'To strictly define anomalies, we analyze all other-objects present in our training set, as well as the \"Movable Object.Debris\" and \"Pushable.Pullable\" class objects in the NuScenes training set. We specifically design our dataset such that we do not have an intersection between anomalous objects and the aforementioned classes.' However, no protocol, per-category object list, or quantitative similarity check is provided. The supplementary (Section 11) shows that SemanticKITTI 'other-object' includes trash bins, garbage cans, pots, billboards, and small tables, while the staged object list (Section 7) includes indoor garbage bins, buckets, pots, bags, and similar items. Because Section 3.2 itself notes that 'current methods tend to be highly sensitive to objects that are present but ignored during training, such as those classified in the category \"other object\"', the reported low OOD performance could partly reflect correct inlier classification rather than genuine anomaly-detection difficulty. Please provide the per-category exclusion list, a feature-space or classifier-based check of non-overlap, and an analysis of what the trained baselines predict on the staged objects, for example confusion with the 'other-object' class.","section":"§4 (Training data) and Supplementary §7/§11"},{"comment":"The ground-truth labels are not quantitatively validated, and the evaluation mask is partially shaped by the baselines. The annotation starts from SemanticKITTI pseudo-labels and is refined by three annotators, but no inter-annotator agreement, no error analysis, and no check of pseudo-labeler mistakes are reported. More critically, the protocol states: 'we examined the predictions of the baseline methods to see if any known objects were missed... We annotate these objects as unlabeled and ignore them in the evaluations.' Combined with the metric description ('Ignore points are removed from the scene prior to evaluation and erroneous predictions in the ignore region are not penalized'), this means that if a baseline misses an anomaly and the annotators follow this step, that anomaly is excluded from the evaluation for all methods. This creates a feedback loop between the benchmarked models and the test mask, and the direction of the bias is uncontrolled. Please quantify how many points or instances were relabeled as unlabeled through this baseline-inspection step, report inter-annotator agreement statistics, and provide a labeling protocol that does not depend on baseline predictions.","section":"§3.2 (Annotation process) and §4 (Metrics)"}],"minor_comments":[{"comment":"Table 1 contains typos: '1 RBG' should be '1 RGB' and 'Augmentated' should be 'Augmented'.","section":"Table 1"},{"comment":"Reference [17] spells the first author's name as 'kuefeng Du'; this appears to be a typo for 'Xuefeng Du'.","section":"References"},{"comment":"The paper does not report training hyperparameters (learning rate, batch size, number of epochs, validation splits) beyond the note in Supplementary §12 about lowering the learning rate; including these details would improve reproducibility.","section":"§4.1 and Supplementary §12"},{"comment":"The OOD performance of all methods is very low (AP ≤ 5.17); the paper would benefit from per-sequence performance distributions, such as box plots over the 51 test sequences, so that readers can assess variance across object sizes and distances.","section":"Table 2"},{"comment":"The instruction to annotators that 'allowing for larger unlabeled regions where the annotator may be challenged' creates a potential bias toward conservative labeling; the paper should report how much of the point cloud is labeled 'unlabeled' and how this varies across sequences.","section":"§3.2"},{"comment":"The 2D control experiment in Supplementary Table 5 is an important sanity check and should be considered for inclusion in the main text.","section":"Supplementary §9.1"}],"recommendation":"major_revision","confidential_remarks":"The two baseline models used for the 3D adaptation (Mask4Former and Panoptic-CUDAL) originate from the authors' own research groups, which is not a correctness problem but should be kept in mind when interpreting the claim that 2D-derived methods do not transfer to 3D. The OOD-premise concern in Major Comment 1 is the main risk to the paper's central claim; if the authors can provide a convincing exclusion analysis, the dataset would be a solid contribution. The editor may also want to verify the 'first publicly available dataset' claim against SOD, which appears in Table 1 and seems to contain LiDAR and camera data for small obstacle segmentation, albeit with 16 beams and without instance-level dense semantic labels."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"STU is a genuinely useful resource: the first public 3D LiDAR benchmark for road anomaly segmentation with dense semantic and instance labels, 128-beam LiDAR, camera data, and temporal sequences. The baseline experiments are honestly reported and show that naively porting 2D anomaly methods to 3D is not enough; that alone is a useful result for the community. The dataset analysis (instance point counts, distances, sizes) is sensible, and the evaluation protocol is mostly reasonable, with the 50 m range and 5-point minimum justified by the sensor and training data. The paper deserves a serious referee.\n\nWhere I part ways with the authors' strongest claims: the OOD premise is asserted rather than demonstrated. Section 4 says they analyzed all other-objects in SemanticKITTI/Panoptic-CUDAL and designed the dataset so there is no intersection with the staged objects, but no protocol, per-category list, or similarity check is given. The supplementary shows SemanticKITTI other-object includes trash bins, pots, and small tables, and the staged list includes indoor garbage bins, buckets, pots, bags. If a staged object falls into the 'other-object' distribution, the benchmark measures inlier recognition rather than true anomaly segmentation. This is the most important soft spot, and it is addressable: release the exclusion analysis and a per-object category list.\n\nLabel quality is the second soft spot. Annotations start from pseudo-labels and are refined by three annotators, but there is no inter-annotator agreement or error analysis. For a benchmark that others will cite, that is a real gap. Also, the process of using baseline predictions to mark 'other-object' regions as unlabeled is a bit circular for evaluation, though it is understandable and not fatal. The data and code are not yet available from the preprint, so the conditional nature is justified.\n\nThe stress-test concern about OOD overlap is partly right, and the label-bias concern is right but secondary. These are not fatal flaws; they are missing evidence. If the authors provide the OOD analysis, label quality metrics, and release the data, this becomes a solid benchmark paper that should be published.","headline":"First public 3D LiDAR anomaly segmentation benchmark with real value; needs OOD verification and label-quality evidence before it becomes standard.","tokens_in":18213,"tokens_out":3126,"would_cite":true,"duration_ms":35562,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents STU, the first publicly available dataset for road anomaly segmentation with dense 3D semantic labels, LiDAR and camera data, and temporal sequences, and shows that 2D-derived baselines struggle on it.","keywords":["3D LiDAR anomaly segmentation","autonomous driving","out-of-distribution detection","point cloud dataset","benchmark","road debris","semantic segmentation","temporal sequences"],"falsifier":"Independently re-annotate a random subset of the 19 validation sequences with fresh annotators who do not see the published labels, then compute agreement on point-level anomaly labels. If agreement is low, or if the fresh labels contradict the published labels on a substantial fraction of anomaly points, the reported baseline scores cannot be treated as a reliable benchmark.","tokens_in":17147,"feed_emoji":"🚗","tokens_out":4406,"duration_ms":44982,"temperature":0.7,"pith_summary":"The paper presents STU, a new dataset for anomaly segmentation in autonomous driving, built from 128-beam LiDAR, eight cameras, and sequential recordings. Its central claim is that STU is the first publicly available benchmark with dense 3D semantic labels for road anomalies, combining LiDAR and camera data with temporal information. The authors argue that the field needs such a resource because existing anomaly segmentation research is dominated by 2D image benchmarks, while autonomous vehicles rely on LiDAR for range and robustness. Baseline experiments adapting 2D anomaly methods to 3D show that these methods perform far worse than in 2D, with models confidently labeling anomaly objects as inliers. If the dataset and its labels hold up, it supplies a missing evaluation ground for 3D and multimodal anomaly segmentation.","feed_headline":"First open 3D LiDAR benchmark for road anomaly segmentation","feed_subtitle":"The STU dataset adds dense 3D labels for unexpected road objects and shows 2D methods fall short.","key_machinery":"The load-bearing object is the dataset itself: 70 fully annotated sequences with a 128-beam LiDAR, dense point-level labels for inlier, anomaly, and unlabeled classes, and per-instance anomaly IDs, plus two anomaly-free sequences used to reduce the domain gap with SemanticKITTI. The annotation pipeline starts from pseudo-labels generated by a SemanticKITTI-trained model and refines them with three annotators, and the evaluation protocol follows SemanticKITTI's 50-meter range and requires at least five points per anomaly instance. This protocol turns the raw point clouds into a reproducible testbed where point-level metrics (AUROC, FPR@95, AP) and object-level metrics (PQ, UQ) can be computed for any 3D segmentation model.","core_discovery":"The core discovery is a public benchmark that makes 3D anomaly segmentation measurable for the first time. The dataset defines two label classes, inliers and outliers, with instance-level identity for each anomaly, and adds extra sequences without anomalies to train in-distribution models. The authors show that when standard 2D anomaly segmentation techniques—Max-Logit, Monte Carlo Dropout, Deep Ensembles, a void classifier, and RbA—are adapted to a Mask4Former-3D backbone, their point-level and object-level scores on STU are markedly lower than their 2D counterparts. Large anomalous objects are frequently predicted as familiar inlier classes such as \"other vehicle\" with high confidence, producing high false-positive rates at 95% recall and low average precision. The conclusion the paper draws is that directly transferring 2D anomaly methods to LiDAR does not work and the community needs 3D-specific approaches.","pith_inferences":["If the pseudo-labeler that seeds the annotations is biased toward SemanticKITTI classes, the benchmark may inherit that bias; a useful extension would be measuring agreement between human annotators and comparing labels produced by different seed models.","The 5-point threshold for evaluating an anomaly instance excludes many of the smallest and most distant objects shown in the dataset histograms, so the benchmark likely underestimates the true difficulty of far-range anomaly detection.","The fact that deep ensembles reduce false positives but still miss most anomalies suggests that uncertainty-based scoring alone is insufficient; a successful method may need to combine geometry cues, such as ground-plane inconsistency, with uncertainty.","A natural next step beyond the paper is a benchmark track that evaluates temporal anomaly detection, since the sequential structure is already present in STU but the reported baselines use single scans only."],"forward_implications":["3D anomaly segmentation can now be evaluated on real LiDAR data with dense labels, giving researchers a shared reference for comparing methods.","The poor baseline results imply that methods designed for 2D images, such as max-logit scoring or ensembling, do not transfer directly to LiDAR point clouds.","The dataset's temporal and multimodal setup makes it possible to design and test anomaly methods that exploit several frames or camera-LiDAR fusion.","Because anomalies appear as very few points among roughly 100,000 inlier points, the benchmark exposes class imbalance as a core difficulty for future methods.","The release of training, validation, and a closed test set with a submission procedure allows the community to track progress over time."],"supporting_citations":[{"why":"Supplies the in-distribution training data, the labeling tool, and the 50-meter evaluation convention used throughout the benchmark.","marker":"[1]"},{"why":"Provides the void-classifier baseline and the anomaly segmentation metric suite (AUROC, FPR@95, AP) inherited by this paper.","marker":"[4]"},{"why":"Defines the anomaly segmentation task with staged road objects and establishes the style of small-obstacle annotation that STU extends to 3D.","marker":"[45]"},{"why":"Offers the 2D benchmark protocol and comparison scores that frame the paper's claim that 2D methods underperform in 3D.","marker":"[10]"},{"why":"Supplies the state-of-the-art 2D anomaly method RbA, which the paper adapts to the Mask4Former-3D backbone.","marker":"[41]"},{"why":"Provides the Mask4Former architecture used as the backbone for all 3D baseline models in the evaluation.","marker":"[60]"},{"why":"Contributes additional rural training sequences and the vehicle sensor configuration matched by the STU acquisition platform.","marker":"[55]"},{"why":"Provides the LiDAR odometry pipeline, KISS-ICP, used to compute vehicle poses required by the labeling tool.","marker":"[57]"}],"fun_headline_variants":["3D LiDAR benchmark: 2D anomaly methods underperform","First 3D anomaly segmentation dataset shows 2D failure","New STU dataset: 3D anomaly segmentation challenge","Spotting unexpected: 3D LiDAR benchmark for AVs","2D anomaly methods fail on new 3D LiDAR dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth labels are trustworthy enough to benchmark methods, even though they originate from machine pseudo-labels and the paper reports no inter-annotator agreement or label-error analysis.","fun_headline_variants_meta":{"raw":{"variants":["3D LiDAR benchmark: 2D anomaly methods underperform","First 3D anomaly segmentation dataset shows 2D failure","New STU dataset: 3D anomaly segmentation challenge","Spotting unexpected: 3D LiDAR benchmark for AVs","2D anomaly methods fail on new 3D LiDAR dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000505,"raw_usage":{"total_tokens":2439,"prompt_tokens":892,"completion_tokens":1547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1459}},"tokens_in":508,"tokens_out":1547,"duration_ms":12032,"temperature":1.0,"reasoning_tokens":1459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T01:00:30.918294+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-annotate a random subset of the 19 validation sequences with fresh annotators who do not see the published labels, then compute agreement on point-level anomaly labels. If agreement is low, or if the fresh labels contradict the published labels on a substantial fraction of anomaly points, the reported baseline scores cannot be treated as a reliable benchmark.","supporting_citations":[{"cited_title":"Se- manticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences","cited_arxiv_id":null,"evidence_quote":"Supplies the in-distribution training data, the labeling tool, and the 50-meter evaluation convention used throughout the benchmark."},{"cited_title":"The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the void-classifier baseline and the anomaly segmentation metric suite (AUROC, FPR@95, AP) inherited by this paper."},{"cited_title":"Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles","cited_arxiv_id":null,"evidence_quote":"Defines the anomaly segmentation task with staged road objects and establishes the style of small-obstacle annotation that STU extends to 3D."},{"cited_title":"SegmentMeIfYou- Can: A Benchmark for Anomaly Segmentation","cited_arxiv_id":null,"evidence_quote":"Offers the 2D benchmark protocol and comparison scores that frame the paper's claim that 2D methods underperform in 3D."},{"cited_title":"Henriques, and Fatma G¨uney","cited_arxiv_id":null,"evidence_quote":"Supplies the state-of-the-art 2D anomaly method RbA, which the paper adapts to the Mask4Former-3D backbone."},{"cited_title":"Mask4Former: Mask Transformer for 4D Panoptic Segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the Mask4Former architecture used as the backbone for all 3D baseline models in the evaluation."},{"cited_title":"KISS-ICP: In Defense of Point-to-Point ICP – Simple, Accurate, and Ro- bust Registration If Done the Right Way","cited_arxiv_id":null,"evidence_quote":"Provides the LiDAR odometry pipeline, KISS-ICP, used to compute vehicle poses required by the labeling tool."}],"review_version":1}