REVIEW 4 major objections 5 minor 25 references
Inconsistency-based Active Learning for LiDAR Object Detection
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read An inconsistency score based on the number of detected boxes selects LiDAR training frames that match random sampling's mean average precision with half the labeled data.
desk verdict A practical, honest active-learning study for LiDAR detection whose headline 'half the labels' claim is not yet statistically established because of best-of-three curves and a post hoc ordering choice. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the normalized number-of-boxes inconsistency score, $S_{NoB} = \frac{|N_o - N_a|}{\max(N_o, N_a)}$, computed between predictions on a LiDAR point cloud and its horizontally mirrored version. It is a per-frame scalar that requires no matching of individual boxes, no class labels, and no extra models. The score is used to rank the working set, with ascending order (low inconsistency first) giving the best results, and with the training set initially containing only inconsistent samples until roughly 50% of the data is labeled. The normalization is what makes the score a relative measure, and the paper shows that this normalization, the ascending ordering, and the choice of count over IoU are each necessary for the reported gains.
What would settle it
Run the NoB selection strategy on a LiDAR dataset from a left-hand-traffic country or on scenes with strong directional asymmetry, and compare it against random sampling and against simply selecting frames with the most detected objects; if the advantage disappears or matches the object-count baseline exactly, the method is exploiting object density rather than detecting uncertain samples.
Extended reading notes
Core claim
The paper's central claim, stated in the abstract and in Section V-B, is that a naive inconsistency approach based on the number of detected boxes reaches the same mAP as the random sampling strategy with only 50% of the labeled point clouds on KITTI. Concretely, the method mirrors each LiDAR point cloud, runs the same PointPillars detector on both views, and computes $S_{NoB} = |N_o - N_a| / \max(N_o, N_a)$, where $N_o$ and $N_a$ are the numbers of detected boxes in the original and augmented clouds. Selecting the lowest-scoring samples first, and retraining from scratch while accumulating selected samples, improves mAP by about 1-2 percentage points over random sampling across the data range, which translates to the same accuracy at roughly half the labeling budget. The authors further report that this number-of-boxes score consistently outperforms a 3D-IoU-based inconsistency score, and they interpret the result as the score acting as a proxy for object count, effectively prioritizing frames with many objects during early cycles.
Load-bearing premise
The assumption that horizontal mirroring is a weak augmentation, so that a well-trained detector should produce nearly the same detections on the mirrored cloud; if that fails, the inconsistency signal is measuring domain shift rather than sample informativeness.
Editorial extensions
If this is right
- If the central claim holds, labeling budgets for LiDAR object detection can be cut roughly in half to reach a given mAP, by labeling frames selected with the NoB score instead of random frames.
- The success of ascending ordering implies that frames with many detected objects are the most cost-effective to label early, which is beneficial when labeling cost is charged per frame rather than per bounding box.
- IoU-based inconsistency measures, which are common in 3D active learning, appear to be noisier for LiDAR than a simple count-based measure, suggesting that count-based scores deserve more attention in future active learning pipelines.
- The class-wise results imply that gains concentrate in classes with enough samples and remaining headroom, so the method may be best applied after a class already has a moderate number of training instances.
- Retraining from scratch on the accumulated selected set works better than fine-tuning only on newly selected samples, pointing to how the active learning loop should be implemented in practice.
Reading between the lines
- Because the normalized score gives denser scenes lower inconsistency, ascending NoB selection is essentially ranking frames by object count; a direct ablation that selects frames with the most detections (without any mirroring) would reveal how much of the gain is simply object-density sampling rather than model uncertainty.
- Horizontal mirroring is not a distribution-preserving augmentation in a right-hand-traffic environment where car orientations and sensor placement are asymmetric; on such data the inconsistency signal may reflect domain shift, and the reported advantage may not transfer.
- The same number-of-boxes consistency could be inserted directly into the detector's training loss as a regularizer, which the paper mentions as future work; if that works, the metric is not just a selection signal but a learning signal.
- The method's value depends on the labeling cost model: it is well-matched to per-frame labeling budgets, but in settings where labeling is charged per box, favoring dense frames might not be the cheapest strategy.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an active learning framework for LiDAR-based 3D object detection. It selects point clouds for labeling based on inconsistency between the model's predictions on a frame and its horizontally mirrored version. Two scores are defined: a number-of-boxes score (NoB, Eq. 1) and an IoU-based score (Eq. 2). Using PointPillars on a custom split of the KITTI train set, the authors compare NoB Scratch, NoB Retrain, IoU Retrain, and a random baseline across active learning cycles. The headline claim is that NoB Scratch attains the same mAP as random sampling while using about half the labeled data, and that the method improves mAP by up to 2.5% over the random baseline.
Significance. If validated, the result would be practically valuable because the proposed score is simple, cheap, and uses only the detector's output rather than an ensemble or additional training. The paper also contains useful ablations in Table I and an honest discussion of the score's behavior as an object-count proxy. The experiments are described in enough detail that the pipeline could be reimplemented, and the comparison of training protocols (Scratch, Retrain, Fine-tuning) is informative. However, the statistical evidence for the headline gain is currently weak: best-of-three reporting, no error bars on the main curves, and a post hoc choice of ascending ordering compromise the claim.
major comments (4)
- [Section V-B (Figures 3-6)] The statement 'Each experiment was run three times with different seeds and the best value is plotted' means the displayed curves are upper order statistics rather than estimates of expected performance. The central claim that NoB Scratch matches random sampling with half the labels rests on 1-2% mAP differences and a horizontal gap of roughly a factor of two between curves; with only three seeds and no error bars or seed-level points on Figures 3-6, the reported gap could be produced by favorable seed draws. Please report mean +/- standard deviation (or all three runs) for every curve, and clarify whether the random baseline was also selected as best-of-three; if it was, the comparison may still be biased and needs a paired analysis.
- [Section V-A (Figure 2, Table I)] The choice of ascending vs descending ordering is described as made 'in accordance with these findings' from the pseudo-active experiment, and Table I shows the difference is large: +2.43% for Ascending NoB Retrain versus -0.05% for Descending NoB Retrain. Because the same dataset is used to inspect the ordering and to report the final result, the headline improvement may reflect model selection rather than a principled method. A held-out ordering-validation split, or a pre-registered hypothesis that ascending is best, is needed to support the claimed gain.
- [Section VI and Eq. (1)] The authors themselves observe that the normalized NoB score is a proxy for object count and that ascending ordering prioritizes frames with many detections. The paper therefore does not yet establish that the mirroring-inconsistency mechanism, rather than frame density, drives the improvement. A direct baseline that selects frames with the largest number of detected boxes, or a permutation control that randomizes which frames receive low scores, would isolate the contribution of the inconsistency computation and is necessary to justify the 'inconsistency-based' framing in the title and abstract.
- [Section IV] The evaluation uses a custom split of the KITTI train set into 3712 frames for active learning and 3769 frames for testing, rather than a standard KITTI train/val split. The x-axis 'percentage of labeled data' refers to the 3712-frame subset, which complicates comparison with prior work and may change the 'half the labels' statement. Please justify this split and report at least one set of results on the standard KITTI val split (or another commonly used split) to assess generalization.
minor comments (5)
- [Abstract and Section IV] The abstract says '50% of the labeled data' but the budget is defined over a custom 3712-frame subset; please make this explicit wherever the factor-of-two claim appears.
- [Section III-B and Section V-A] The sentence 'we discard samples for which both No and Na are 0' is clear, but the rule in Section V-A that 'the samples with inconsistency greater than 0 are used first' could be read as either 'score > 0' or 'nonzero score' including all inconsistent samples; please clarify whether zero-inconsistency samples (No = Na > 0) are used before or after the inconsistent subset, and how Eq. (1) assigns them in the ordering.
- [Figure 2] Figure 2 shows the standard deviation region for the random baseline but not for the pseudo-active strategies; adding repeated-run variability for both would help the reader assess whether the ascending/descending difference is meaningful.
- [Table I] Table I reports 'Normalised Improvement' without variance; given the best-of-three reporting, please state whether these numbers are means or maxima and provide a measure of dispersion.
- [Throughout] There are minor typos and formatting issues, e.g., 'has beeing developed' in the footnote and the missing space in 'I NTRODUCTION'; these do not affect the science but should be cleaned up.
Circularity Check
No circularity: the NoB score is externally defined and the central claim is evaluated against a random baseline; the self-citation [25] is motivational, not load-bearing.
full rationale
The paper's derivation chain is a standard active-learning experiment: an inconsistency score (Eq. 1) is defined externally from detector outputs on an original and a mirrored point cloud, used to rank unlabeled frames, and the selected frames are then labeled and used to train a PointPillars detector. The reported mAP is measured on a separate KITTI test split against a random-sampling baseline, so there is no equation-level identity between S_NoB and the target metric. The only self-citation is reference [25], used to motivate comparing box counts and to note agreement with prior results, but the central claim does not reduce to that citation; it rests on the new experiments reported in Figures 3-6. The choice of ascending ordering and the practice of plotting the best of three seeded runs are legitimate statistical-reliability concerns, but they are model-selection and reporting issues rather than circular reductions. Likewise, the Discussion's admission that the score acts as a proxy for object count affects the mechanistic interpretation of why the method works, but it does not make the evaluation circular, since the selection score is still computed from model predictions and compared with random sampling. The paper is therefore not circular, though its headline quantitative claim would benefit from error bars and a held-out ordering choice.
Assumptions & free parameters
free parameters (2)
- Selection ordering direction =
ascending
- IoU matching threshold =
not specified
assumptions (4)
- domain assumption Horizontal mirroring is a weak augmentation; a robust detector should produce similar outputs on a point cloud and its mirrored version.
- domain assumption KITTI ground-truth labels are accurate and can be used to simulate human labeling.
- domain assumption Labeling cost is incurred per frame, so selecting frames with more objects is a valid cost model.
- domain assumption PointPillars with fixed hyperparameters is a representative detector for comparing AL strategies.
Cite this review
Pith. "Pith review of Inconsistency-based Active Learning for LiDAR Object Detection." pith.science (2026). https://pith.science/paper/QBSW5XQC
@misc{pith2026250500511,
author = {Pith},
title = {Pith review of: Inconsistency-based Active Learning for LiDAR Object Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/QBSW5XQC}},
note = {Machine review of arXiv:2505.00511}
}
read the original abstract
Deep learning models for object detection in autonomous driving have recently achieved impressive performance gains and are already being deployed in vehicles worldwide. However, current models require increasingly large datasets for training. Acquiring and labeling such data is costly, necessitating the development of new strategies to optimize this process. Active learning is a promising approach that has been extensively researched in the image domain. In our work, we extend this concept to the LiDAR domain by developing several inconsistency-based sample selection strategies and evaluate their effectiveness in various settings. Our results show that using a naive inconsistency approach based on the number of detected boxes, we achieve the same mAP as the random sampling strategy with 50% of the labeled data.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Waymo. (2023) Waymo significantly outperforms comparable human benchmarks over 7+ million miles of rider-only driving. Accessed: 01, 11, 2024. [Online]. Available: https://waymo.com/blog/
work page 2023
-
[2]
A survey of deep active learning,
P. Ren, Y . Xiao, X. Chang, P.-Y . Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang, “A survey of deep active learning,” ACM Comput. Surv. , vol. 54, no. 9, oct 2021. [Online]. Available: https://doi.org/10.1145/3472291
doi:10.1145/3472291 2021
-
[3]
Are we ready for autonomous driving? the kitti vision benchmark suite,
A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2012
2012
-
[4]
Scalable active learning for object detection,
E. Haussmann, M. Fenzi, K. Chitta, J. Ivanecky, H. Xu, D. Roy, A. Mittel, N. Koumchatzky, C. Farabet, and J. M. Alvarez, “Scalable active learning for object detection,” in 2020 IEEE Intelligent Vehicles Symposium (IV) . IEEE Press, 2020, p. 1430–1435. [Online]. Available: https://doi.org/10.1109/IV47402.2020.9304793
arXiv 2020
-
[5]
Not all labels are equal: Rationalizing the labeling costs for training object detection,
I. Elezi, Z. Yu, A. Anandkumar, L. Leal-Taix ´e, and J. M. Alvarez, “Not all labels are equal: Rationalizing the labeling costs for training object detection,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022, pp. 14 472–14 481
work page 2022
-
[6]
Fixmatch: Simplifying semi- supervised learning with consistency and confidence,
K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel, “Fixmatch: Simplifying semi- supervised learning with consistency and confidence,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS’20. Red Hook, NY , USA: Curran Associates Inc., 2020
work page 2020
-
[7]
Consistency-based semi-supervised active learning: Towards mini- mizing labeling cost,
M. Gao, Z. Zhang, G. Yu, S. ¨O. Arık, L. S. Davis, and T. Pfister, “Consistency-based semi-supervised active learning: Towards mini- mizing labeling cost,” in Computer Vision – ECCV 2020 , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 510–526
work page 2020
-
[8]
Activeanno3d – an active learning framework for multi-modal 3d object detection,
A. Ghita, B. Antoniussen, W. Zimmer, R. Greer, C. Creß, A. Møgelmose, M. M. Trivedi, and A. C. Knoll, “Activeanno3d – an active learning framework for multi-modal 3d object detection,” 2024
work page 2024
Show all 25 references
-
[9]
Entropy-based active learning for object detection with progressive diversity constraint,
J. Wu, J. Chen, and D. Huang, “Entropy-based active learning for object detection with progressive diversity constraint,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Los Alamitos, CA, USA: IEEE Computer Society, jun 2022, pp. 9387–9396. [Onl...
2022
-
[10]
Exploring diversity-based active learning for 3d object detection in autonomous driving,
Z. Liang, X. Xu, S. Deng, L. Cai, T. Jiang, and K. Jia, “Exploring diversity-based active learning for 3d object detection in autonomous driving,” ArXiv, vol. abs/2205.07708, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:248811445
2022 arXiv
-
[11]
Query by committee,
H. S. Seung, M. Opper, and H. Sompolinsky, “Query by committee,” in Proceedings of the Fifth Annual Workshop on Computational Learning Theory, ser. COLT ’92. New York, NY , USA: Association for Computing Machinery, 1992, p. 287–294. [Online]. Available: https://doi.org/10.1145...
1992
-
[12]
Mobyv2al: Self-supervised active learning for image classification,
R. Caramalau, B. Bhattarai, D. Stoyanov, and T.-K. Kim, “Mobyv2al: Self-supervised active learning for image classification,” in 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022 . BMV A Press, 2022. [Online]. Available: https://bmvc2022....
2022
-
[13]
Deep active learning for image classification,
H. Ranganathan, H. Venkateswara, S. Chakraborty, and S. Pan- chanathan, “Deep active learning for image classification,” in 2017 IEEE International Conference on Image Processing (ICIP) , 2017, pp. 3934–3938
2017
-
[14]
Active learning for deep object detection via probabilistic modeling,
J. Choi, I. Elezi, H.-J. Lee, C. Farabet, and J. M. Alvarez, “Active learning for deep object detection via probabilistic modeling,” 2021
2021
-
[15]
Balanced sampling-based active learning for object detection,
S. J. Aithal, S. P. Adhikari, M. Ghorai, and H. Misra, “Balanced sampling-based active learning for object detection,” in Computer Vision and Machine Intelligence , M. Tistarelli, S. R. Dubey, S. K. Singh, and X. Jiang, Eds. Singapore: Springer Nature Singapore, 2023, pp. 323–333
2023
-
[16]
Active semi-supervised learning by exploring per-sample uncertainty and consistency,
J. Lim, J. Na, and N. Kwak, “Active semi-supervised learning by exploring per-sample uncertainty and consistency,” 2023
2023
-
[17]
Semi-supervised active learning based on semantic-aware crop consistency,
M. Cai, B. Wei, X. Li, and L. Li, “Semi-supervised active learning based on semantic-aware crop consistency,” 2022, pp. 655–660
2022
-
[18]
Semantic segmentation with active semi-supervised learning
M. H. Aneesh Rangnekar, Christopher Kanan, “Semantic segmentation with active semi-supervised learning.” 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (W ACV), 2023
2023
-
[19]
Semi-supervised active learning for object detection,
S. Chen, Y . Yang, and Y . Hua, “Semi-supervised active learning for object detection,” Electronics, vol. 12, no. 2, pp. 375–375, 2023
2023
-
[20]
Semi-supervised active learning with temporal output discrepancy,
S. Huang, T. Wang, H. Xiong, J. Huan, and D. Dou, “Semi-supervised active learning with temporal output discrepancy,” in ICCV, 10 2021, pp. 3427–3436
2021
-
[21]
Advanced active learning strategies for object detection,
S. Schmidt, Q. Rao, J. Tatsch, and A. Knoll, “Advanced active learning strategies for object detection,” in 2020 IEEE Intelligent Vehicles Symposium (IV), 2020, pp. 871–876
2020
-
[22]
Localization-aware active learning for object detection,
C.-C. Kao, T.-Y . Lee, P. Sen, and M.-Y . Liu, “Localization-aware active learning for object detection,” inComputer Vision – ACCV 2018, C. Jawahar, H. Li, G. Mori, and K. Schindler, Eds. Cham: Springer International Publishing, 2019, pp. 506–522
2018
-
[23]
Monocular 3d object detection with lidar guided semi supervised active learning,
A. Hekimoglu, M. Schmidt, and A. Marcos-Ramiro, “Monocular 3d object detection with lidar guided semi supervised active learning,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2024, pp. 2346–2355
2024
-
[24]
Camera-lidar active learning for ob- ject detecting deep neural networks,
A. Gunnarg ˚ard and D. Odin, “Camera-lidar active learning for ob- ject detecting deep neural networks,” Master’s thesis, University of Chalmers, Chalmers, Sweden, June 2020, master’s thesis
2020
-
[25]
Camera- lidar inconsistency analysis for active learning in object detection,
E. Rivera, A. Serra Do Nascimento, and M. Lienkamp, “Camera- lidar inconsistency analysis for active learning in object detection,” in 35th IEEE Intelligent Vehicles Symposium, IV 2024 , ser. IEEE Intelligent Vehicles Symposium, Proceedings. Institute of Electrical and Electro...
2024
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.