Pith. sign in

REVIEW 4 major objections 5 minor 25 references

Inconsistency-based Active Learning for LiDAR Object Detection

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read An inconsistency score based on the number of detected boxes selects LiDAR training frames that match random sampling's mean average precision with half the labeled data.

desk verdict A practical, honest active-learning study for LiDAR detection whose headline 'half the labels' claim is not yet statistically established because of best-of-three curves and a post hoc ordering choice. read the letter →

arxiv 2505.00511 v1 pith:QBSW5XQC submitted 2025-05-01 cs.CV

classification cs.CV
keywords activelearningLiDARobjectdetectioninconsistency-basedsamplingpointcloudaugmentation3DKITTIlabelingefficiencynumberofboxes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that active learning for 3D LiDAR object detection can be driven by a very cheap signal: how much a detector's output changes when a point cloud is horizontally mirrored. The central claim is that a normalized count of detected boxes, called the NoB inconsistency score, selects training frames that achieve the same mean average precision as random sampling using only half the labeled data on the KITTI dataset. A sympathetic reader would care because labeling LiDAR frames is expensive, and a selection method that halves that cost without needing extra models or dense annotations is directly useful for autonomous driving perception pipelines. The paper also argues that this simple count-based score beats a more elaborate IoU-based inconsistency score, and that ordering frames by low inconsistency first is the effective strategy.

What carries the argument

The load-bearing object is the normalized number-of-boxes inconsistency score, $S_{NoB} = \frac{|N_o - N_a|}{\max(N_o, N_a)}$, computed between predictions on a LiDAR point cloud and its horizontally mirrored version. It is a per-frame scalar that requires no matching of individual boxes, no class labels, and no extra models. The score is used to rank the working set, with ascending order (low inconsistency first) giving the best results, and with the training set initially containing only inconsistent samples until roughly 50% of the data is labeled. The normalization is what makes the score a relative measure, and the paper shows that this normalization, the ascending ordering, and the choice of count over IoU are each necessary for the reported gains.

What would settle it

Run the NoB selection strategy on a LiDAR dataset from a left-hand-traffic country or on scenes with strong directional asymmetry, and compare it against random sampling and against simply selecting frames with the most detected objects; if the advantage disappears or matches the object-count baseline exactly, the method is exploiting object density rather than detecting uncertain samples.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in the abstract and in Section V-B, is that a naive inconsistency approach based on the number of detected boxes reaches the same mAP as the random sampling strategy with only 50% of the labeled point clouds on KITTI. Concretely, the method mirrors each LiDAR point cloud, runs the same PointPillars detector on both views, and computes $S_{NoB} = |N_o - N_a| / \max(N_o, N_a)$, where $N_o$ and $N_a$ are the numbers of detected boxes in the original and augmented clouds. Selecting the lowest-scoring samples first, and retraining from scratch while accumulating selected samples, improves mAP by about 1-2 percentage points over random sampling across the data range, which translates to the same accuracy at roughly half the labeling budget. The authors further report that this number-of-boxes score consistently outperforms a 3D-IoU-based inconsistency score, and they interpret the result as the score acting as a proxy for object count, effectively prioritizing frames with many objects during early cycles.

Load-bearing premise

The assumption that horizontal mirroring is a weak augmentation, so that a well-trained detector should produce nearly the same detections on the mirrored cloud; if that fails, the inconsistency signal is measuring domain shift rather than sample informativeness.

Editorial extensions

If this is right

  • If the central claim holds, labeling budgets for LiDAR object detection can be cut roughly in half to reach a given mAP, by labeling frames selected with the NoB score instead of random frames.
  • The success of ascending ordering implies that frames with many detected objects are the most cost-effective to label early, which is beneficial when labeling cost is charged per frame rather than per bounding box.
  • IoU-based inconsistency measures, which are common in 3D active learning, appear to be noisier for LiDAR than a simple count-based measure, suggesting that count-based scores deserve more attention in future active learning pipelines.
  • The class-wise results imply that gains concentrate in classes with enough samples and remaining headroom, so the method may be best applied after a class already has a moderate number of training instances.
  • Retraining from scratch on the accumulated selected set works better than fine-tuning only on newly selected samples, pointing to how the active learning loop should be implemented in practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the normalized score gives denser scenes lower inconsistency, ascending NoB selection is essentially ranking frames by object count; a direct ablation that selects frames with the most detections (without any mirroring) would reveal how much of the gain is simply object-density sampling rather than model uncertainty.
  • Horizontal mirroring is not a distribution-preserving augmentation in a right-hand-traffic environment where car orientations and sensor placement are asymmetric; on such data the inconsistency signal may reflect domain shift, and the reported advantage may not transfer.
  • The same number-of-boxes consistency could be inserted directly into the detector's training loss as a regularizer, which the paper mentions as future work; if that works, the metric is not just a selection signal but a learning signal.
  • The method's value depends on the labeling cost model: it is well-matched to per-frame labeling budgets, but in settings where labeling is charged per box, favoring dense frames might not be the cheapest strategy.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an active learning framework for LiDAR-based 3D object detection. It selects point clouds for labeling based on inconsistency between the model's predictions on a frame and its horizontally mirrored version. Two scores are defined: a number-of-boxes score (NoB, Eq. 1) and an IoU-based score (Eq. 2). Using PointPillars on a custom split of the KITTI train set, the authors compare NoB Scratch, NoB Retrain, IoU Retrain, and a random baseline across active learning cycles. The headline claim is that NoB Scratch attains the same mAP as random sampling while using about half the labeled data, and that the method improves mAP by up to 2.5% over the random baseline.

Significance. If validated, the result would be practically valuable because the proposed score is simple, cheap, and uses only the detector's output rather than an ensemble or additional training. The paper also contains useful ablations in Table I and an honest discussion of the score's behavior as an object-count proxy. The experiments are described in enough detail that the pipeline could be reimplemented, and the comparison of training protocols (Scratch, Retrain, Fine-tuning) is informative. However, the statistical evidence for the headline gain is currently weak: best-of-three reporting, no error bars on the main curves, and a post hoc choice of ascending ordering compromise the claim.

major comments (4)
  1. [Section V-B (Figures 3-6)] The statement 'Each experiment was run three times with different seeds and the best value is plotted' means the displayed curves are upper order statistics rather than estimates of expected performance. The central claim that NoB Scratch matches random sampling with half the labels rests on 1-2% mAP differences and a horizontal gap of roughly a factor of two between curves; with only three seeds and no error bars or seed-level points on Figures 3-6, the reported gap could be produced by favorable seed draws. Please report mean +/- standard deviation (or all three runs) for every curve, and clarify whether the random baseline was also selected as best-of-three; if it was, the comparison may still be biased and needs a paired analysis.
  2. [Section V-A (Figure 2, Table I)] The choice of ascending vs descending ordering is described as made 'in accordance with these findings' from the pseudo-active experiment, and Table I shows the difference is large: +2.43% for Ascending NoB Retrain versus -0.05% for Descending NoB Retrain. Because the same dataset is used to inspect the ordering and to report the final result, the headline improvement may reflect model selection rather than a principled method. A held-out ordering-validation split, or a pre-registered hypothesis that ascending is best, is needed to support the claimed gain.
  3. [Section VI and Eq. (1)] The authors themselves observe that the normalized NoB score is a proxy for object count and that ascending ordering prioritizes frames with many detections. The paper therefore does not yet establish that the mirroring-inconsistency mechanism, rather than frame density, drives the improvement. A direct baseline that selects frames with the largest number of detected boxes, or a permutation control that randomizes which frames receive low scores, would isolate the contribution of the inconsistency computation and is necessary to justify the 'inconsistency-based' framing in the title and abstract.
  4. [Section IV] The evaluation uses a custom split of the KITTI train set into 3712 frames for active learning and 3769 frames for testing, rather than a standard KITTI train/val split. The x-axis 'percentage of labeled data' refers to the 3712-frame subset, which complicates comparison with prior work and may change the 'half the labels' statement. Please justify this split and report at least one set of results on the standard KITTI val split (or another commonly used split) to assess generalization.
minor comments (5)
  1. [Abstract and Section IV] The abstract says '50% of the labeled data' but the budget is defined over a custom 3712-frame subset; please make this explicit wherever the factor-of-two claim appears.
  2. [Section III-B and Section V-A] The sentence 'we discard samples for which both No and Na are 0' is clear, but the rule in Section V-A that 'the samples with inconsistency greater than 0 are used first' could be read as either 'score > 0' or 'nonzero score' including all inconsistent samples; please clarify whether zero-inconsistency samples (No = Na > 0) are used before or after the inconsistent subset, and how Eq. (1) assigns them in the ordering.
  3. [Figure 2] Figure 2 shows the standard deviation region for the random baseline but not for the pseudo-active strategies; adding repeated-run variability for both would help the reader assess whether the ascending/descending difference is meaningful.
  4. [Table I] Table I reports 'Normalised Improvement' without variance; given the best-of-three reporting, please state whether these numbers are means or maxima and provide a measure of dispersion.
  5. [Throughout] There are minor typos and formatting issues, e.g., 'has beeing developed' in the footnote and the missing space in 'I NTRODUCTION'; these do not affect the science but should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the NoB score is externally defined and the central claim is evaluated against a random baseline; the self-citation [25] is motivational, not load-bearing.

full rationale

The paper's derivation chain is a standard active-learning experiment: an inconsistency score (Eq. 1) is defined externally from detector outputs on an original and a mirrored point cloud, used to rank unlabeled frames, and the selected frames are then labeled and used to train a PointPillars detector. The reported mAP is measured on a separate KITTI test split against a random-sampling baseline, so there is no equation-level identity between S_NoB and the target metric. The only self-citation is reference [25], used to motivate comparing box counts and to note agreement with prior results, but the central claim does not reduce to that citation; it rests on the new experiments reported in Figures 3-6. The choice of ascending ordering and the practice of plotting the best of three seeded runs are legitimate statistical-reliability concerns, but they are model-selection and reporting issues rather than circular reductions. Likewise, the Discussion's admission that the score acts as a proxy for object count affects the mechanistic interpretation of why the method works, but it does not make the evaluation circular, since the selection score is still computed from model predictions and compared with random sampling. The paper is therefore not circular, though its headline quantitative claim would benefit from error bars and a held-out ordering choice.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim depends on domain assumptions (mirroring is a weak augmentation, KITTI labels are accurate, per-frame labeling cost) and on two post-hoc design choices (ascending ordering, best-of-three seed reporting). No new physical entities are introduced, and no numeric parameters are fit to data beyond the ordering direction and the unstated IoU threshold.

free parameters (2)
  • Selection ordering direction = ascending
    Section V-A tests both ascending and descending orderings; ascending is chosen for the main AL cycles after descending underperforms random baseline. This is a post-hoc model choice on the same data.
  • IoU matching threshold = not specified
    Section III-B defines SIoU with boxes matched only above a threshold, but the threshold value is never reported, preventing exact replication.
assumptions (4)
  • domain assumption Horizontal mirroring is a weak augmentation; a robust detector should produce similar outputs on a point cloud and its mirrored version.
    Section III-B states 'the model should, in general, be robust to weak augmentations, like mirroring'. This is the core premise of inconsistency-based AL and is not validated for LiDAR scenes with directional traffic asymmetries.
  • domain assumption KITTI ground-truth labels are accurate and can be used to simulate human labeling.
    Section III-A uses KITTI ground truth to simulate labeling in the AL cycle; if labels are noisy, selection results would differ.
  • domain assumption Labeling cost is incurred per frame, so selecting frames with more objects is a valid cost model.
    Discussion argues the method is beneficial 'in scenarios where labeling costs are incurred per frame rather than per bounding box'; this assumption drives the object-count proxy interpretation.
  • domain assumption PointPillars with fixed hyperparameters is a representative detector for comparing AL strategies.
    Section IV-A chooses PointPillars for speed and keeps settings constant; conclusions may not transfer to other detectors (e.g., CenterPoint).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Inconsistency-based Active Learning for LiDAR Object Detection." pith.science (2026). https://pith.science/paper/QBSW5XQC

@misc{pith2026250500511,
  author       = {Pith},
  title        = {Pith review of: Inconsistency-based Active Learning for LiDAR Object Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QBSW5XQC}},
  note         = {Machine review of arXiv:2505.00511}
}
read the original abstract

Deep learning models for object detection in autonomous driving have recently achieved impressive performance gains and are already being deployed in vehicles worldwide. However, current models require increasingly large datasets for training. Acquiring and labeling such data is costly, necessitating the development of new strategies to optimize this process. Active learning is a promising approach that has been extensively researched in the image domain. In our work, we extend this concept to the LiDAR domain by developing several inconsistency-based sample selection strategies and evaluate their effectiveness in various settings. Our results show that using a naive inconsistency approach based on the number of detected boxes, we achieve the same mAP as the random sampling strategy with 50% of the labeled data.

Figures

Figures reproduced from arXiv: 2505.00511 by the authors.

Figure 1
Figure 1. Point cloud and its augmented version side to side, with white [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pseudo active learning cycle results. Deviation for the baseline is [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 5
Figure 5. Results for Car instances of the desired class. It can be seen that for the 40% of the data or less, the three settings have lower or equal AP than the random baseline. For both the NoB Retrain settings, this trend continues for the whole range. On the other hand, the NoB Scratch setting is slightly better than the random baseline, with 1-2% on average. Contrary to the cyclists, the car is the most represented class… view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Results for mAP (all classes) 20 30 40 50 60 70 80 90 [%] of Labelled data 10 20 30 40 50 60 70 80 [%] per class Class proportion for different sampling methods Pedestrians Cyclists Cars [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Class distribution across cycles above the random baseline as well, with a more pronounced improvement on the low-data regime, where with only 30% of the data achieves the same performance as the random baseline with 60% or NoB Scratch with 40%. On the other hand, the …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 19 canonical work pages

  1. [1]

    (2023) Waymo significantly outperforms comparable human benchmarks over 7+ million miles of rider-only driving

    Waymo. (2023) Waymo significantly outperforms comparable human benchmarks over 7+ million miles of rider-only driving. Accessed: 01, 11, 2024. [Online]. Available: https://waymo.com/blog/

  2. [2]

    A survey of deep active learning,

    P. Ren, Y . Xiao, X. Chang, P.-Y . Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang, “A survey of deep active learning,” ACM Comput. Surv. , vol. 54, no. 9, oct 2021. [Online]. Available: https://doi.org/10.1145/3472291

  3. [3]

    Are we ready for autonomous driving? the kitti vision benchmark suite,

    A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2012

  4. [4]

    Scalable active learning for object detection,

    E. Haussmann, M. Fenzi, K. Chitta, J. Ivanecky, H. Xu, D. Roy, A. Mittel, N. Koumchatzky, C. Farabet, and J. M. Alvarez, “Scalable active learning for object detection,” in 2020 IEEE Intelligent Vehicles Symposium (IV) . IEEE Press, 2020, p. 1430–1435. [Online]. Available: https://doi.org/10.1109/IV47402.2020.9304793

  5. [5]

    Not all labels are equal: Rationalizing the labeling costs for training object detection,

    I. Elezi, Z. Yu, A. Anandkumar, L. Leal-Taix ´e, and J. M. Alvarez, “Not all labels are equal: Rationalizing the labeling costs for training object detection,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022, pp. 14 472–14 481

  6. [6]

    Fixmatch: Simplifying semi- supervised learning with consistency and confidence,

    K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel, “Fixmatch: Simplifying semi- supervised learning with consistency and confidence,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS’20. Red Hook, NY , USA: Curran Associates Inc., 2020

  7. [7]

    Consistency-based semi-supervised active learning: Towards mini- mizing labeling cost,

    M. Gao, Z. Zhang, G. Yu, S. ¨O. Arık, L. S. Davis, and T. Pfister, “Consistency-based semi-supervised active learning: Towards mini- mizing labeling cost,” in Computer Vision – ECCV 2020 , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 510–526

  8. [8]

    Activeanno3d – an active learning framework for multi-modal 3d object detection,

    A. Ghita, B. Antoniussen, W. Zimmer, R. Greer, C. Creß, A. Møgelmose, M. M. Trivedi, and A. C. Knoll, “Activeanno3d – an active learning framework for multi-modal 3d object detection,” 2024

Show all 25 references
  1. [9]

    Entropy-based active learning for object detection with progressive diversity constraint,

    J. Wu, J. Chen, and D. Huang, “Entropy-based active learning for object detection with progressive diversity constraint,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Los Alamitos, CA, USA: IEEE Computer Society, jun 2022, pp. 9387–9396. [Onl...

  2. [10]

    Exploring diversity-based active learning for 3d object detection in autonomous driving,

    Z. Liang, X. Xu, S. Deng, L. Cai, T. Jiang, and K. Jia, “Exploring diversity-based active learning for 3d object detection in autonomous driving,” ArXiv, vol. abs/2205.07708, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:248811445

  3. [11]

    Query by committee,

    H. S. Seung, M. Opper, and H. Sompolinsky, “Query by committee,” in Proceedings of the Fifth Annual Workshop on Computational Learning Theory, ser. COLT ’92. New York, NY , USA: Association for Computing Machinery, 1992, p. 287–294. [Online]. Available: https://doi.org/10.1145...

  4. [12]

    Mobyv2al: Self-supervised active learning for image classification,

    R. Caramalau, B. Bhattarai, D. Stoyanov, and T.-K. Kim, “Mobyv2al: Self-supervised active learning for image classification,” in 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022 . BMV A Press, 2022. [Online]. Available: https://bmvc2022....

  5. [13]

    Deep active learning for image classification,

    H. Ranganathan, H. Venkateswara, S. Chakraborty, and S. Pan- chanathan, “Deep active learning for image classification,” in 2017 IEEE International Conference on Image Processing (ICIP) , 2017, pp. 3934–3938

  6. [14]

    Active learning for deep object detection via probabilistic modeling,

    J. Choi, I. Elezi, H.-J. Lee, C. Farabet, and J. M. Alvarez, “Active learning for deep object detection via probabilistic modeling,” 2021

  7. [15]

    Balanced sampling-based active learning for object detection,

    S. J. Aithal, S. P. Adhikari, M. Ghorai, and H. Misra, “Balanced sampling-based active learning for object detection,” in Computer Vision and Machine Intelligence , M. Tistarelli, S. R. Dubey, S. K. Singh, and X. Jiang, Eds. Singapore: Springer Nature Singapore, 2023, pp. 323–333

  8. [16]

    Active semi-supervised learning by exploring per-sample uncertainty and consistency,

    J. Lim, J. Na, and N. Kwak, “Active semi-supervised learning by exploring per-sample uncertainty and consistency,” 2023

  9. [17]

    Semi-supervised active learning based on semantic-aware crop consistency,

    M. Cai, B. Wei, X. Li, and L. Li, “Semi-supervised active learning based on semantic-aware crop consistency,” 2022, pp. 655–660

  10. [18]

    Semantic segmentation with active semi-supervised learning

    M. H. Aneesh Rangnekar, Christopher Kanan, “Semantic segmentation with active semi-supervised learning.” 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (W ACV), 2023

  11. [19]

    Semi-supervised active learning for object detection,

    S. Chen, Y . Yang, and Y . Hua, “Semi-supervised active learning for object detection,” Electronics, vol. 12, no. 2, pp. 375–375, 2023

  12. [20]

    Semi-supervised active learning with temporal output discrepancy,

    S. Huang, T. Wang, H. Xiong, J. Huan, and D. Dou, “Semi-supervised active learning with temporal output discrepancy,” in ICCV, 10 2021, pp. 3427–3436

  13. [21]

    Advanced active learning strategies for object detection,

    S. Schmidt, Q. Rao, J. Tatsch, and A. Knoll, “Advanced active learning strategies for object detection,” in 2020 IEEE Intelligent Vehicles Symposium (IV), 2020, pp. 871–876

  14. [22]

    Localization-aware active learning for object detection,

    C.-C. Kao, T.-Y . Lee, P. Sen, and M.-Y . Liu, “Localization-aware active learning for object detection,” inComputer Vision – ACCV 2018, C. Jawahar, H. Li, G. Mori, and K. Schindler, Eds. Cham: Springer International Publishing, 2019, pp. 506–522

  15. [23]

    Monocular 3d object detection with lidar guided semi supervised active learning,

    A. Hekimoglu, M. Schmidt, and A. Marcos-Ramiro, “Monocular 3d object detection with lidar guided semi supervised active learning,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2024, pp. 2346–2355

  16. [24]

    Camera-lidar active learning for ob- ject detecting deep neural networks,

    A. Gunnarg ˚ard and D. Odin, “Camera-lidar active learning for ob- ject detecting deep neural networks,” Master’s thesis, University of Chalmers, Chalmers, Sweden, June 2020, master’s thesis

  17. [25]

    Camera- lidar inconsistency analysis for active learning in object detection,

    E. Rivera, A. Serra Do Nascimento, and M. Lienkamp, “Camera- lidar inconsistency analysis for active learning in object detection,” in 35th IEEE Intelligent Vehicles Symposium, IV 2024 , ser. IEEE Intelligent Vehicles Symposium, Proceedings. Institute of Electrical and Electro...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.