REVIEW 4 major objections 5 minor 34 references
Model-Agnostic Open-Set Air-to-Air Visual Object Detection for Reliable UAV Perception
T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read By fusing softmax confidence with embedding-space Gaussian mixture entropy, this paper proposes a post-hoc open-set detection layer that lets real-time aerial detectors reject unseen objects without retraining, reporting that it raises OOD
desk verdict The paper builds a plausible UAV open-set detector from known components, but the main AUROC claim rests on an undefined score for Joint Thresholding and on selective reporting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is Joint Thresholding: each detection is kept only if its softmax confidence exceeds a threshold AND its GMM posterior entropy falls below a threshold. The GMMs are fit post-training on training embeddings matched to ground truth; spectral normalization on the backbone enforces a bi-Lipschitz constraint for well-behaved embeddings, and temperature scaling learned on validation recalibrates both logits and GMM log-likelihoods. The key unit is the per-detection embedding, which lets the same machinery work with any detector that outputs fixed-dimensional embeddings.
What would settle it
Re-run the Joint Thresholding pipeline with thresholds fixed using only the AOT-C training/validation splits, then evaluate on an independently collected aerial flight dataset that was never used to select the configuration; if the AUROC_bd gap over YOLOv5 (reported 0.883/0.887 vs 0.800) shrinks to near zero or reverses, the central claim would be falsified. A simpler check: repeat the real-flight evaluation with the softmax-entropy variant excluded from the best-configuration selection and verify the gap persists.
Extended reading notes
Core claim
The central discovery is that softmax-derived confidence and embedding-space GMM entropy carry complementary signals about out-of-distribution inputs, and fusing them via a simple threshold—Joint Thresholding—separates known aircraft from unknown drones better than any single score. The authors show that on real flight data, after training on AOT-C, Joint Thresholding reaches AUROC_bd of 0.883 (RT-DETR) and 0.887 (with spectral normalization), versus 0.800 for YOLOv5, while maintaining mAP around 40. The method also treats background detections as OOD and improves rejection there. This is presented as a post-hoc calibration layer that requires no retraining or detector modification, only acc
Load-bearing premise
The load-bearing premise is that the best-performing configuration found on the AOT validation OOD set (including the joint-threshold parameters) generalizes to new flight data without having been tuned on that data.
Editorial extensions
If this is right
- Any embedding-based detector (not just RT-DETR) can be retrofitted with open-set rejection by fitting GMMs on training embeddings and applying joint thresholding; no architecture change or retraining is required.
- Treating background detections as OOD rather than ignoring them improves AUROC and better reflects real UAV false-positive pressure, so background rejection becomes a trainable part of the open-set pipeline.
- Spectral normalization and temperature scaling are effective post-hoc calibrations: SN consistently raises AUROC across scoring methods, while temperature scaling is needed especially for GMM-based scores.
- Real-time operation is preserved; the method runs at about 24 FPS on an A10G, effectively identical to baseline RT-DETR.
- Softmax entropy alone, despite strong synthetic results, degrades on real flight data, so multi-signal fusion is necessary for deployment.
Reading between the lines
- The paper leaves implicit that its fixed-threshold fusion is a special case of score ensembling; a learned fusor (e.g., decision tree or shallow MLP over the same score vector) might close the remaining gap and would be directly testable with the released code.
- A natural extension the authors do not test is whether the same GMM-entropy layer transfers to other small-object aerial benchmarks (e.g., Det-Fly) or to different backbone architectures; if it does, the 'model-agnostic' claim would be stronger than the single RT-DETR demonstration.
- Because thresholds are tuned on a validation OOD split containing drones, operational deployment would need a principled way to set thresholds without such a split; an unsupervised entropy calibration or per-mission threshold adaptation might work, but the paper does not address this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a model-agnostic open-set detection framework for air-to-air UAV perception. It augments an embedding-based detector (RT-DETR-R50) with per-detection GMM density estimation, spectral normalization, temperature scaling, and a Joint Thresholding rule that combines softmax confidence and GMM entropy to reject OOD detections. Experiments on AOT-C and real flight data report AUROC gains over YOLOv5 and GMM-Det while preserving mAP and real-time speed. The central claims are that joint thresholding improves open-set rejection under corruption and domain shift while introducing negligible runtime overhead.
Significance. The paper addresses a safety-relevant problem in aerial robotics and gives a sensible combination of existing post-hoc uncertainty techniques (DDU, GMM-Det, temperature scaling) for small moving aerial targets. Contributions include a broad ablation over softmax/entropy/GMM scoring functions, investigation of spectral normalization, and an evaluation on real flight data rather than synthetic corruptions only. The authors also disclose their configuration-selection protocol, which is helpful. If the quantitative claims held, the method would be practically useful. However, as presented, the central quantitative claim is not verifiable because the primary metric (AUROC) is undefined for the proposed hard-decision rule, and the experimental reporting does not permit a fair comparison. The work is promising but requires substantial methodological clarification and additional experiments.
major comments (4)
- [Section III-D, Algorithm 1; Tables II, IV; Fig. 4] Joint Thresholding is defined as a hard decision rule: a detection is ID only if s_softmax >= tau_soft AND H_gmm <= tau_gmm, otherwise OOD. This yields a binary label, not a continuous ranking score. AUROC and ROC require a score over all thresholds, but the paper never defines such a score for this conjunction rule. The reported AUROC values (0.982 in Table II, 0.883-0.887 in Table IV) and Figure 4 are therefore not reproducible. Define a single continuous score (e.g., a monotone function of s_softmax and -H_gmm) that reduces to the stated rule, or replace AUROC with threshold-dependent metrics at specified operating points.
- [Section IV-C] The paper states: 'Each method is evaluated across multiple configurations ... and only the best-performing setup is shown. Results without softmax pruning are omitted.' This is post-hoc selection over a large configuration space (embedding layer, GMM components, temperatures, pruning threshold, spectral normalization). Without reporting all configurations or a pre-specified validation selection protocol, Tables II and IV do not support a fair comparison. No error bars or multiple seeds are reported, so the 0.982 vs. 0.929 gap may be within noise. Report full distributions, the split used for selection, and variability across seeds.
- [Section IV-C/D, Tables III-IV] The closed-set mAP claim is confounded: Table III reports mAP only for the best-AUROC configuration per method, with no comparison to the unfiltered detector or to non-best configurations. In Table IV, configurations are selected on AOT-C validation and the real-flight images come from the authors' prior benchmark [5], which weakens the transfer claim. Report mAP for all filtering modes and clarify that no test information was used in configuration selection; also consider an external test split.
- [Table IV] The headline gain versus YOLOv5 is not an apples-to-apples comparison of the proposed open-set method: YOLOv5 is evaluated with its standard inference, without the paper's embedding-GMM or pruning pipeline. Instead of comparing RT-DETR+Joint Thresholding to standard YOLOv5, the reader needs YOLOv5 baseline vs. YOLOv5 with the same open-set pipeline to support the model-agnostic claim. Apply the framework to YOLOv5 or another detector and report the relative gain with that detector's own baseline.
minor comments (5)
- [Section III-C] The pruning threshold S_max < 0.2 is introduced as a fixed choice, but no sensitivity analysis is given. A brief study over this threshold would clarify whether results depend strongly on it.
- [Table II] The header 'TPR@OSR 5%/10%/20%' uses 'OSR' without definition. Presumably it denotes a fixed open-set rate or false-positive rate; state this explicitly.
- [Algorithm 1] The notation in lines 9-11 is confusing: 'x_c' is used for both an image and a set of samples. Use clearer notation such as X_c for the set of matched embeddings.
- [Figure 3 caption] The caption says 'same image' but shows left/right panels from different detectors; clarify whether the panels show different views or crops of the same scene.
- [General] Typos and formatting issues: 'UA V' contains a spurious space; Table I headers use 'APclean' and 'APcor' without separation; reference [28] is an informal GitHub-style citation.
Circularity Check
No circular derivation: the reported AUROC is an empirical measurement, not a quantity forced by the paper's definitions; the main caveats (undefined ranking score for Joint Thresholding, best-config selection, self-created benchmark) are correctness/independence concerns rather than circularity.
full rationale
The claimed derivation chain is not circular in the sense defined by the review criteria. The GMM density model is fitted on in-distribution training embeddings only; the paper explicitly states 'No OOD data are used at this stage' (Section III-B2), so the OOD separation is not inserted by construction. Spectral normalization and temperature scaling are imported from external work ([3], [4]) and applied post hoc; the paper does not derive its headline result from these citations. Joint Thresholding is an empirical conjunction of a softmax score and a GMM-entropy threshold, and its reported AUROC is measured on data, not obtained by a definitional identity. The most serious concerns are: (i) Algorithm 1 returns a binary ID/OOD decision, yet Tables II and IV report AUROC/ROC curves, so a continuous ranking score for Joint Thresholding is never defined; (ii) Section IV-C states 'Each method is evaluated across multiple configurations ... and only the best-performing setup is shown', which implies configuration selection on the same OOD set whose AUROC is then reported — a statistical overfitting risk, but not a derivation equivalent to the inputs; (iii) the AOT-C and real-flight test data come from the authors' prior work [5], which is a dataset-independence caveat, not a circular step. These issues bear on reproducibility and validity more than on circularity. Accordingly, the circularity score is low.
Assumptions & free parameters
free parameters (8)
- tau_softmax
- tau_gmm
- T_model
- T_gmm
- GMM components K per class =
2, 3, 4
- embedding layer choice
- softmax pruning threshold =
0.2
- spectral normalization on/off =
bool
assumptions (5)
- domain assumption The detector's embedding space is regular enough for per-class Gaussian models to separate in-distribution from out-of-distribution objects.
- domain assumption The AOT-C corruption model is a representative proxy for real-world flight conditions.
- domain assumption The real-flight test set has reliable ID/OOD labels and a known object distribution.
- domain assumption Class-conditional GMMs fitted on training embeddings generalize to unseen OOD samples without using OOD data for fitting.
- domain assumption Temperature scaling learned on the validation set transfers to the test distribution.
Cite this review
Pith. "Pith review of Model-Agnostic Open-Set Air-to-Air Visual Object Detection for Reliable UAV Perception." pith.science (2026). https://pith.science/paper/MHJXRWHF
@misc{pith2026250909297,
author = {Pith},
title = {Pith review of: Model-Agnostic Open-Set Air-to-Air Visual Object Detection for Reliable UAV Perception},
year = {2026},
howpublished = {\url{https://pith.science/paper/MHJXRWHF}},
note = {Machine review of arXiv:2509.09297}
}
read the original abstract
Open-set detection is crucial for robust UAV autonomy in air-to-air object detection under real-world conditions. Traditional closed-set detectors degrade significantly under domain shifts and flight data corruption, posing risks to safety-critical applications. We propose a novel, model-agnostic open-set detection framework designed specifically for embedding-based detectors. The method explicitly handles unknown object rejection while maintaining robustness against corrupted flight data. It estimates semantic uncertainty via entropy modeling in the embedding space and incorporates spectral normalization and temperature scaling to enhance open-set discrimination. We validate our approach on the challenging AOT aerial benchmark and through extensive real-world flight tests. Comprehensive ablation studies demonstrate consistent improvements over baseline methods, achieving up to a 10\% relative AUROC gain compared to standard YOLO-based detectors. Additionally, we show that background rejection further strengthens robustness without compromising detection accuracy, making our solution particularly well-suited for reliable UAV perception in dynamic air-to-air environments.
Figures
Reference graph
Works this paper leans on
-
[5]
Common corruptions for evaluating and enhancing robustness in air-to-air visual object detection,
A. Arsenos, V . Karampinis, E. Petrongonas, C. Skliros, D. Kollias, S. Kollias, and A. V oulodimos, “Common corruptions for evaluating and enhancing robustness in air-to-air visual object detection,”IEEE Robotics and Automation Letters, vol. 9, no. 7, pp. 6688–6695, 2024
2024
-
[1]
Class semantics modulation for open-set instance segmentation,
Y . Yang, Z. Zhou, J. Wu, Y . Wang, and R. Xiong, “Class semantics modulation for open-set instance segmentation,”IEEE Robotics and Automation Letters, vol. 9, no. 3, pp. 2240–2247, 2024
2024
-
[2]
Open-set object detection using classification-free object proposal and instance-level contrastive learning,
Z. Zhou, Y . Yang, Y . Wang, and R. Xiong, “Open-set object detection using classification-free object proposal and instance-level contrastive learning,”IEEE Robotics and Automation Letters, vol. 8, no. 3, pp. 1691–1698, 2023
2023
-
[3]
Deep deterministic uncertainty: A new simple baseline,
J. Mukhoti, A. Kirsch, J. Van Amersfoort, P. H. Torr, and Y . Gal, “Deep deterministic uncertainty: A new simple baseline,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 24 384–24 394
2023
-
[4]
Uncertainty for identifying open-set errors in visual object detection,
D. Miller, N. S ¨underhauf, M. Milford, and F. Dayoub, “Uncertainty for identifying open-set errors in visual object detection,”IEEE Robotics and Automation Letters, vol. 7, no. 1, pp. 215–222, 2021
2021
-
[6]
Air-to-air visual detection of micro-uavs: An experimental evaluation of deep learning,
Y . Zheng, Z. Chen, D. Lv, Z. Li, Z. Lan, and S. Zhao, “Air-to-air visual detection of micro-uavs: An experimental evaluation of deep learning,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1020–1027, 2021
2021
-
[7]
Nefeli: A deep-learning detection and tracking pipeline for enhancing autonomy in advanced air mobility,
A. Arsenos, E. Petrongonas, O. Filippopoulos, C. Skliros, D. Kollias, and S. Kollias, “Nefeli: A deep-learning detection and tracking pipeline for enhancing autonomy in advanced air mobility,”Aerospace Science and Technology, vol. 155, p. 109613, 2024
2024
-
[8]
Airtrack: Onboard deep learning framework for long-range aircraft detection and tracking,
S. Ghosh, J. Patrikar, B. Moon, M. M. Hamidi, and S. Scherer, “Airtrack: Onboard deep learning framework for long-range aircraft detection and tracking,” in2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 1277–1283
2023
Show all 34 references
-
[9]
Visual-based obstacle detection and tracking, and conflict detection for small uas sense and avoid,
R. Opromolla and G. Fasano, “Visual-based obstacle detection and tracking, and conflict detection for small uas sense and avoid,”Aerospace Science and Technology, vol. 119, p. 107167, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1270963821006775
2021
-
[10]
Out-of-distribution identification: Let detector tell which i am not sure,
R. Li, C. Zhang, H. Zhou, C. Shi, and Y . Luo, “Out-of-distribution identification: Let detector tell which i am not sure,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 638–654
2022
-
[11]
Hyperdimensional feature fusion for out-of-distribution detection,
S. Wilson, T. Fischer, N. S ¨underhauf, and F. Dayoub, “Hyperdimensional feature fusion for out-of-distribution detection,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 2644–2654
2023
-
[12]
Open-set object detection: towards unified problem formulation and benchmarking,
H. Ammar, N. Kiselov, G. Lapouge, and R. Audigier, “Open-set object detection: towards unified problem formulation and benchmarking,” in European Conference on Computer Vision. Springer, 2024, pp. 46–61
2024
-
[13]
Grounding dino 1.5: Advance the
T. Ren, Q. Jiang, S. Liu, Z. Zeng, W. Liu, H. Gao, H. Huang, Z. Ma, X. Jiang, Y . Chenet al., “Grounding dino 1.5: Advance the” edge” of open-set object detection,”arXiv preprint arXiv:2405.10300, 2024
2024 arXiv
-
[14]
Uncertainty quantification for safe and reliable autonomous vehicles: A review of methods and applications,
K. Wang, C. Shen, X. Li, and J. Lu, “Uncertainty quantification for safe and reliable autonomous vehicles: A review of methods and applications,”IEEE Transactions on Intelligent Transportation Systems, 2025
2025
-
[15]
Gaussian mixture models,
D. Reynolds, “Gaussian mixture models,” inEncyclopedia of biometrics. Springer, 2015, pp. 827–832
2015
-
[16]
Certainnet: Sampling-free uncertainty estimation for object detection,
S. Gasperini, J. Haug, M.-A. N. Mahani, A. Marcos-Ramiro, N. Navab, B. Busam, and F. Tombari, “Certainnet: Sampling-free uncertainty estimation for object detection,”IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 698–705, 2021
2021
-
[17]
Spectral normal- ization for generative adversarial networks,
T. Miyato, T. Kataoka, M. Koyama, and Y . Yoshida, “Spectral normal- ization for generative adversarial networks,” inInternational Conference on Learning Representations (ICLR), 2018
2018
-
[18]
On calibration of modern neural networks,
C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” inProceedings of the 34th International Conference on Machine Learning (ICML), vol. 70, 2017, pp. 1321–1330
2017
-
[19]
Benchmarking neural network ro- bustness to common corruptions and perturbations,
D. Hendrycks and T. Dietterich, “Benchmarking neural network ro- bustness to common corruptions and perturbations,”Proceedings of the International Conference on Learning Representations, 2019
2019
-
[20]
The cityscapes dataset for semantic urban scene understanding,
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benen- son, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” inProc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016
2016
-
[21]
nuscenes: A multimodal dataset for autonomous driving,
H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11 621–11 631
2020
-
[22]
ultralytics/yolov5: v3.1 - bug fixes and performance improvements,
G. Jocher, A. Stoken, J. Borovec, NanoCode012, ChristopherSTAN, L. Changyu, Laughing, tkianai, A. Hogan, lorenzomammana, yxNONG, AlexWang1900, L. Diaconu, Marc, wanghaoyang0106, ml5ah, Doug, F. Ingham, Frederik, Guilhen, Hatovix, J. Poznanski, J. Fang, L. Yu, changyu98, M. Wan...
2020 doi
-
[23]
Ultralytics yolo,
G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics yolo,” https://github. com/ultralytics/ultralytics, Jan 2023, [Online; accessed August 2, 2025]
2023
-
[24]
Yolox: Exceeding yolo series in 2021,
Z. Ge, S. Liu, F. Wang, Z. Li, and J. Sun, “Yolox: Exceeding yolo series in 2021,”arXiv preprint arXiv:2107.08430, 2021
2021 arXiv
-
[25]
Focal loss for dense object detection,
T. Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” inProceedings of the IEEE International Conference on Computer Vision (ICCV), 2017
2017
-
[26]
Pytorch-retinanet: Pytorch implementation of retinanet,
Y . Henon, “Pytorch-retinanet: Pytorch implementation of retinanet,”
-
[27]
Faster r-cnn: Towards real-time object detection with region proposal networks,
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 28, 2015
2015
-
[28]
Faster r-cnn pytorch training pipeline,
sovit-123, “Faster r-cnn pytorch training pipeline,” 2025. [Online]. Avail- able: https://github.com/sovit-123/fasterrcnn-pytorch-training-pipeline
2025
-
[29]
Diffusiondet: Diffusion model for object detection,
S. Chen, P. Sun, Y . Song, and P. Luo, “Diffusiondet: Diffusion model for object detection,” in2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 19 773–19 786
2023
-
[30]
End-to-end object detection with transformers,
N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in European conference on computer vision. Springer, 2020, pp. 213– 229
2020
-
[31]
Probabilistic two-stage detec- tion,
X. Zhou, V . Koltun, and P. Kr ¨ahenb¨uhl, “Probabilistic two-stage detec- tion,” inarXiv preprint arXiv:2103.07461, 2021
2021 arXiv
-
[32]
DETRs Beat YOLOs on Real-time Object Detection,
Y . Zhao, W. Lv, S. Xu, J. Wei, G. Wang, Q. Dang, Y . Liu, and J. Chen, “DETRs Beat YOLOs on Real-time Object Detection,” Seattle, Washington, USA, pp. 16 965–16 974, June 2024
2024
-
[33]
Airborne object tracking dataset,
“Airborne object tracking dataset,” https://registry.opendata.aws/ airborne-object-tracking, accessed: 2023-07-23
2023
-
[2020]
Available: https://github.com/yhenon/pytorchretinanet
[Online]. Available: https://github.com/yhenon/pytorchretinanet
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.