REVIEW 3 major objections 6 minor 38 references
Methodology for an Analysis of Influencing Factors on 3D Object Detection Performance
T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Camera-based 3D pedestrian detectors fail more at distance and occlusion, while LiDAR detectors fail more on still pedestrians, says a six-detector meta-information analysis.
desk verdict A useful, honest empirical comparison of 3D pedestrian detectors, but the central distance-modality result needs a matching-threshold sensitivity check before I'd trust it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a two-tier analysis. The univariate tier uses Kendall's tau to capture monotonic dependence and normalized mutual information (estimated with a k-nearest-neighbor estimator) to capture nonlinear, non-monotonic dependence between each meta-information feature and detection error. The meta-model tier trains a random forest to predict false-negative and false-positive errors from the same features, then extracts feature importance through SHAP values, computed with conditional sampling to avoid extrapolating correlated features. The random forest's role is to model interactions among factors, and SHAP converts that model into a ranked list of influencing factors. The two tiers are meant to be read together: agreement is treated as a reliable pattern, while disagreement signals that a factor's influence is conditional on other features.
What would settle it
Train the same six detectors on a validation split with balanced distance, occlusion, and weather distributions, and re-run the SHAP analysis: if camera detectors no longer show a distance-dependent drop in pedestrian recall or LiDAR detectors no longer show a pose/velocity-dependent drop, the claimed modality-specific error patterns would be falsified. More narrowly, if the random forest meta-model's top features for a detector change substantially under different random seeds or an alternative explainability method, then the meta-model-based rankings are not stable enough to represent the detector's behavior.
Extended reading notes
Core claim
The paper's central claim is that camera- and LiDAR-based 3D pedestrian detectors have distinct, measurable failure patterns that can be ranked by meta-information about the object and environment. On the NuScenes validation set, camera-based detectors show a stronger negative correlation between distance and correct detection than LiDAR detectors, and their false-positive rate also grows with distance; the authors attribute this to the LiDAR sensor's larger range. Occlusion, measured by NuScenes visibility tokens, degrades all detectors, with a stronger effect on cameras. LiDAR-based detectors in turn depend more on the pedestrian's attribute and velocity: moving pedestrians are detected more reliably, while standing, sitting, or lying pedestrians are more often missed, consistent with pose-dependent reflection properties in point clouds. Weather and location factors generally show small influence, which the authors tie to the limited weather diversity of the dataset. The paper also reports detector-specific deviations, most notably PointPillars, whose SHAP rankings diverge from the univariate analysis, indicating that the influence of factors can change when features are combined.
Load-bearing premise
The load-bearing premise is that the random forest meta-model trained on NuScenes validation data faithfully represents each detector's true error behavior, so its SHAP rankings reflect real influencing factors; the paper itself notes the Rashomon effect means different equally good meta-models can give different explanations.
Editorial extensions
If this is right
- Camera-based pedestrian detection should be stress-tested at long range: cameras both miss more pedestrians and produce more ghost detections as distance grows.
- LiDAR-based systems should be specifically validated for stationary, sitting, and lying pedestrians, since attribute and velocity are strong drivers of their failures.
- Occlusion should be a standard safety-validation scenario for all detectors, and especially for camera systems, because it degrades every detector examined and cameras most.
- Weather-related conclusions from NuScenes should not be generalized to adverse conditions; the dataset's warm, low-rain, high-visibility range limits the generalizability of the small weather effects.
- Detector-specific differences, such as PointPillars' divergent rankings, imply that per-modality conclusions should be re-checked for each new architecture rather than assumed to transfer.
Reading between the lines
- A practical consequence not developed in the paper: because camera false positives increase with distance, a safety system could gate distant camera-only detections behind a LiDAR confirmation before acting on them.
- The same methodology could be applied to other object classes and to camera-LiDAR fusion detectors; distance and occlusion rankings would likely shift because fusion inherits the stronger long-range modality, but that is a testable extension, not a result of this paper.
- The paper's limited weather diversity suggests that replicating the analysis on data with fog, heavy rain, and low visibility would likely raise weather factors in the importance rankings—an inference, since the paper does not test those conditions.
- Disagreements between univariate and SHAP rankings, as seen for PointPillars, could serve as a diagnostic signal that an architecture's failure modes are interaction-driven and therefore hard to capture with single-factor tests.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a methodology for analyzing how object- and environment-related 'meta-information' influences pedestrian detection errors in 3D object detection, and applies it to six detectors (FCOS3D, MonoDIS, SpatialDETR for cameras; Megvii, PointPillars, TransFusion-L for LiDAR) on the NuScenes validation set. Three analyses are performed: Kendall's tau rank correlation, mutual information, and a random forest meta-model with SHAP values, separately for false negatives and false positives. The central findings are that camera-based detectors are more strongly affected by distance and occlusion, while LiDAR-based detectors depend more on pedestrian attributes and velocity; weather and location factors show small effects. The authors propose the methodology as a support tool for safety assurance of DNN-based object detectors.
Significance. The paper's comparative design is a genuine contribution: it considers both error types, a broad set of meta-information, and three complementary analytical lenses, with multiple detectors per modality. The authors are also transparent about limitations (weather data diversity, Rashomon effect for the meta-model). If the findings are robust, they provide an actionable map of modality-specific failure modes. However, the quantitative conclusions currently rest on point estimates without uncertainty quantification and on a fixed 2 m matching threshold that may penalize camera depth error; the central modality comparison therefore needs additional robustness evidence before the method can serve as a reliable safety-analysis tool.
major comments (3)
- [§3.2] The central claim that camera-based detectors misdetect distant pedestrians more often and produce more distant FPs than LiDAR detectors is directly tied to the BEV center-distance matching threshold of 2 m. Camera-based detectors, especially monocular ones, have larger depth-localization error at range, so a detection with correct classification but 2–3 m center offset is counted as a false negative, and an offset prediction becomes a false positive. Under the fixed 2 m rule, the distance-dependent FN/FP patterns in Figs. 1–6 (and the corresponding statements in §5) may partly reflect localization accuracy rather than detection or ghost-detection behavior. Please run a threshold-robustness analysis (e.g., matching at 1.0, 1.5, 2.5, 3.0 m) and report whether the modality differences persist. If they do not, the paper's headline should be reframed.
- [§4.1] Figures 1–6 report Kendall's tau, MI, and mean absolute SHAP values without confidence intervals or significance tests. Point estimates alone are insufficient to support claims such as 'camera detectors have a relatively strong negative correlation with distance compared to LiDAR detectors' (Fig. 1) or the detector-level differences in §4.2. Please add bootstrap confidence intervals or permutation tests for the correlations and SHAP values, and clearly indicate when observed differences are not statistically distinguishable.
- [§3.3] The random forest meta-model is trained on the same NuScenes validation set used for the univariate analyses and for confidence-threshold optimization, but the manuscript does not describe a data split or cross-validation for the RF itself. Grid-search tuning on the same set risks overfitting, so the SHAP importances in Figs. 3 and 6 may reflect quirks of that split. Additionally, the authors themselves acknowledge in §5 that due to the Rashomon effect the meta-model does not necessarily represent the true behavior of the detectors. To make the meta-model-based results interpretable, report holdout performance (e.g., AUROC/F1 on a separate subset) and discuss the consistency of SHAP rankings across multiple fitted models.
minor comments (6)
- [§4.2] In the text near Fig. 6, 'the mean absolute SHAP values in Fig. 5 for temperature and humidity' should reference Fig. 6, since it describes the SHAP analysis rather than the MI analysis.
- [§3.1] The phrase 'the width, height, and length of the 3D bounding box around the pedestrian m3' mixes units incorrectly; should be 'in meters' for each dimension or 'volume in m^3' if intended.
- [Introduction] The phrase 'A Vs' should be 'AVs' or 'automated vehicles'.
- [References] Reference [18] appears to have a placeholder or incorrect title ('MS Windows NT kernel description'); the Visual Crossing weather API reference needs correct metadata.
- [§5] The sentence about visibility contains a typo: 'visibilites' should be 'visibilities'.
- [§5] The statement 'The examined dataset consists of only 150 scenes, while it contains about 6000 different samples' should clarify that this is the validation subset and, for context, state the total number of scenes in NuScenes.
Circularity Check
No circularity: the analysis is an empirical measurement; caveats are acknowledged limitations, not self-referential reductions.
full rationale
The paper does not derive detector behavior from first principles; it proposes a measurement methodology applied to NuScenes. The Section 3.2 matching rule (BEV center distance < 2 m) and the F1-optimized confidence thresholds are fixed evaluation choices, not parameters fitted and then relabeled as predictions. The Kendall tau, mutual information, and RF/SHAP analyses are descriptive statistics of those labeled errors; the central finding that camera detectors are more distance-sensitive is not defined in terms of the fitted thresholds. The RF surrogate is explicitly qualified in Section 5: 'the interpretations derived from the meta-model-based approach do not necessarily represent causal relationships' and the Rashomon effect is named, so SHAP rankings are not presented as a forced uniqueness result. The only self-citation [36] appears in the outlook as a pointer to explainable-AI literature and is not load-bearing. The skeptic's concern about the 2 m matching rule is a threshold-robustness or measurement-validity question, not circularity: the paper never asserts that TP/FP labels are independent of localization error, and no equation reduces a claimed prediction to an input by construction.
Assumptions & free parameters
free parameters (4)
- Per-detector confidence threshold =
FCOS3D: 0.2, MonoDIS: 0.2, SpatialDETR: 0.36, Megvii: 0.27, PointPillars: 0.25, TransFusion-L: 0.14
- Random forest hyperparameters =
not reported
- k for k-NN mutual information estimator =
3
- Matching threshold for true positive =
2m center distance in BEV
assumptions (4)
- domain assumption Ground-truth annotations in NuScenes are accurate and complete
- domain assumption Weather API data from the nearest weather station represents the conditions at the ego vehicle
- ad hoc to paper The random forest meta-model is a valid surrogate for detector error behavior
- domain assumption The NuScenes validation set covers sufficient environmental diversity for conclusions about weather and location
Cite this review
Pith. "Pith review of Methodology for an Analysis of Influencing Factors on 3D Object Detection Performance." pith.science (2026). https://pith.science/paper/JUCBJMCJ
@misc{pith2026241108482,
author = {Pith},
title = {Pith review of: Methodology for an Analysis of Influencing Factors on 3D Object Detection Performance},
year = {2026},
howpublished = {\url{https://pith.science/paper/JUCBJMCJ}},
note = {Machine review of arXiv:2411.08482}
}
read the original abstract
In automated driving, object detection is crucial for perceiving the environment. Although deep learning-based detectors offer high performance, their black-box nature complicates safety assurance. We propose a novel methodology to analyze how object- and environment-related factors affect LiDAR- and camera-based 3D object detectors. A statistical univariate analysis relates each factor to pedestrian detection errors. Additionally, a Random Forest (RF) model predicts errors from meta-information, with Shapley Values interpreting feature importance. By capturing feature dependencies, the RF enables a nuanced analysis of detection errors. Understanding these factors reveals detector performance gaps and supports safer object detection system development.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
An overview of sensors in autonomous vehicles,
H. A. Ignatious, M. Khan et al., “An overview of sensors in autonomous vehicles,” Procedia Computer Science, vol. 198, pp. 736–741, 2022
work page 2022
-
[2]
C. Sun, R. Zhang, Y . Lu, Y . Cui, Z. Deng, D. Cao, and A. Khajepour, “Toward ensuring safety for autonomous driv- ing perception: Standardization progress, research advances, and perspectives,” IEEE Transactions on Intelligent Trans- portation Systems, 2023
work page 2023
-
[3]
3d object detection for autonomous driving: A comprehensive survey,
J. Mao, S. Shi, X. Wang, and H. Li, “3d object detection for autonomous driving: A comprehensive survey,” International Journal of Computer Vision, vol. 131, no. 8, pp. 1909–1963, 2023
1909
-
[4]
O. Willers, S. Sudholt, S. Raafatnia, and S. Abrecht, Safety Concerns and Mitigation Approaches Regarding the Use of Deep Learning in Safety-Critical Perception Tasks. Springer International Publishing, 2020, p. 336–350. [Online]. Available: http://dx.doi.org/10.1007/978-3-030-55583-2 25
-
[5]
Safety assurance of machine learning for perception func- tions,
S. Burton, C. Hellert, F. H ¨uger, M. Mock, and A. Rohatschek, “Safety assurance of machine learning for perception func- tions,” in Deep Neural Networks and Data for Automated Driving: Robustness, Uncertainty Quantification, and In- sights Towards Safety . Springer International Publishing Cham, 2022, pp. 335–358
work page 2022
-
[6]
Addressing uncertainty in the safety assurance of machine-learning,
S. Burton and B. Herd, “Addressing uncertainty in the safety assurance of machine-learning,” Frontiers in Computer Science , vol. 5, Apr. 2023. [Online]. Available: http://dx.doi.org/10.3389/fcomp.2023.1132580
arXiv 2023
-
[7]
Cv-hazop: Introducing test data validation for computer vision,
O. Zendel, M. Murschitz, M. Humenberger, and W. Herzner, “Cv-hazop: Introducing test data validation for computer vision,” in 2015 IEEE International Conference on Computer Vision (ICCV). IEEE, 2015, pp. 2066–2074
work page 2015
-
[8]
Joint camera and lidar risk analysis,
O. Zendel, J. Huemer, M. Murschitz, G. F. Dominguez, and A. Lobe, “Joint camera and lidar risk analysis,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, 2023, pp. 88–97
work page 2023
Show all 38 references
-
[9]
A2d2: Audi autonomous driving dataset,
J. Geyer, Y . Kassahun, M. Mahmudi, X. Ricou, R. Durgesh, A. S. Chung, L. Hauswald, V . H. Pham, M. M ¨uhlegg, S. Dorn, T. Fernandez, M. J ¨anicke, S. Mirashi, C. Savani, M. Sturm, O. V orobiov, M. Oelker, S. Garreis, and P. Schuberth, “A2d2: Audi autonomous driving dataset,”
-
[10]
Towards serious perception sensor simu- lation for safety validation of automated driving - a col- laborative method to specify sensor models,
C. Linnhoff, P. Rosenberger, S. Schmidt, L. Elster, R. Stark, and H. Winner, “Towards serious perception sensor simu- lation for safety validation of automated driving - a col- laborative method to specify sensor models,” in 2021 IEEE International Intelligent Transportation S...
2021
-
[11]
Dark-skin individuals are at more risk on the street: Unmasking fairness issues of autonomous driving systems
X. Li, Z. Chen, J. M. Zhang, F. Sarro, Y . Zhang, and X. Liu, “Dark-skin individuals are at more risk on the street: Unmasking fairness issues of autonomous driving systems.” [Online]. Available: http://arxiv.org/pdf/2308.02935v1
-
[12]
Performance analysis of camera-based object detection for automated vehicles,
T. Ponn, T. Kr ¨oger, and F. Diermeyer, “Performance analysis of camera-based object detection for automated vehicles,” Sensors (Basel, Switzerland) , vol. 20, no. 13, 2020
2020
-
[13]
Random decision forests,
T. K. Ho, “Random decision forests,” in Proceedings of 3rd international conference on document analysis and recogni- tion, vol. 1. IEEE, 1995, pp. 278–282
1995
-
[14]
A unified approach to in- terpreting model predictions,
S. M. Lundberg and S.-I. Lee, “A unified approach to in- terpreting model predictions,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY , USA: Curran Asso- ciates Inc., 2017, p. 4768–4777
2017
-
[15]
Freiesleben and C
T. Freiesleben and C. Molnar, Supervised Machine Learning for Science , 2024. [Online]. Available: https://ml-science- book.com/
2024
-
[16]
Development of a methodology for deriving in- terpretable causes for perception failures,
X. Liang, “Development of a methodology for deriving in- terpretable causes for perception failures,” Master’s thesis, Technical University of Darmstadt, Institute of Automotive Engineering, 2022
2022
-
[17]
nuscenes: A multimodal dataset for autonomous driving,
H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020, pp. 11 618–11 628
2020
-
[18]
Crossing
V . Crossing. MS Windows NT kernel description. [Online]. Available: https://www.visualcrossing.com/weather-api
-
[19]
2d and 3d object detection algorithms from images: A survey,
W. Chen, Y . Li, Z. Tian, and F. Zhang, “2d and 3d object detection algorithms from images: A survey,” Array, vol. 19, p. 100305, 2023
2023
-
[20]
Fcos3d: Fully convolutional one-stage monocular 3d object detection,
T. Wang, X. Zhu, J. Pang, and D. Lin, “Fcos3d: Fully convolutional one-stage monocular 3d object detection,” in 2021 IEEE/CVF International Conference on Computer Vi- sion Workshops (ICCVW). IEEE, 2021, pp. 913–922
2021
-
[21]
Disentangling monocular 3d object detec- tion: From single to multi-class recognition,
A. Simonelli, S. R. Bulo, L. Porzi, M. L. Antequera, and P. Kontschieder, “Disentangling monocular 3d object detec- tion: From single to multi-class recognition,” IEEE transac- tions on pattern analysis and machine intelligence , vol. 44, no. 3, pp. 1219–1231, 2022
2022
-
[22]
Spatialdetr: Robust scal- able transformer-based 3d object detection from multi-view camera images with global cross-sensor attention,
S. Doll, R. Schulz, L. Schneider, V . Benzin, M. En- zweiler, and H. P. A. Lensch, “Spatialdetr: Robust scal- able transformer-based 3d object detection from multi-view camera images with global cross-sensor attention,” in Com- puter Vision – ECCV 2022: 17th European Conferenc...
2022
-
[23]
Class-balanced grouping and sampling for point cloud 3d object detection
B. Zhu, Z. Jiang, X. Zhou, Z. Li, and G. Yu, “Class-balanced grouping and sampling for point cloud 3d object detection.” [Online]. Available: http://arxiv.org/pdf/1908.09492v1
1908 arXiv
-
[24]
Pointpillars: Fast encoders for object detection from point clouds,
A. H. Lang, S. V ora, H. Caesar, L. Zhou, J. Yang, and O. Bei- jbom, “Pointpillars: Fast encoders for object detection from point clouds,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2019, pp. 12 689–12 697
2019
-
[25]
Transfusion: Robust lidar-camera fusion for 3d object detection with transformers,
X. Bai, Z. Hu, X. Zhu, Q. Huang, Y . Chen, H. Fu, and C.-L. Tai, “Transfusion: Robust lidar-camera fusion for 3d object detection with transformers,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022, pp. 1080–1089
2022
-
[26]
A new measure of rank correlation,
M. G. KENDALL, “A new measure of rank correlation,” Biometrika, vol. 30, no. 1–2, p. 81–93, Jun. 1938. [Online]. Available: http://dx.doi.org/10.1093/biomet/30.1-2.81
1938 doi
-
[27]
A mathematical theory of communication,
C. E. Shannon, “A mathematical theory of communication,” The Bell System Technical Journal , vol. 27, no. 3, pp. 379– 423, 1948
1948
-
[28]
Sample estimate of the entropy of a random vector,
L. Kozachenko, “Sample estimate of the entropy of a random vector,” Probl. Peredachi Inf., vol. 23, pp. 9–16, 1987
1987
-
[29]
Mutual information between discrete and con- tinuous data sets,
B. C. Ross, “Mutual information between discrete and con- tinuous data sets,” PloS one, vol. 9, no. 2, p. e87357, 2014
2014
-
[30]
Estimating mutual information,
A. Kraskov, H. St ¨ogbauer, and P. Grassberger, “Estimating mutual information,” Physical review. E, Statistical, nonlin- ear, and soft matter physics , vol. 69, no. 6 Pt 2, p. 066138, 2004
2004
-
[31]
Entropy and correlation: Some comments,
T. O. Kvalseth, “Entropy and correlation: Some comments,” IEEE Transactions on Systems, Man, and Cybernetics , vol. 17, no. 3, pp. 517–519, 1987
1987
-
[32]
Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features,
F. Pargent, F. Pfisterer, J. Thomas, and B. Bischl, “Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features,” Computational Statistics , vol. 37, no. 5, p. 2671–2692, Mar. 2022. [Online]. Available: http://dx.d...
2022 doi
-
[33]
Pedestrian detection with lidar technology in smart-city deployments–challenges and recommendations,
P. Torres, H. Marques, and P. Marques, “Pedestrian detection with lidar technology in smart-city deployments–challenges and recommendations,” Computers, vol. 12, no. 3,
-
[34]
Optimation free space optic (fso) design with kim model using space diversity,
T. Subekti, A. F. Isnawati, and D. Zulherman, “Optimation free space optic (fso) design with kim model using space diversity,” JURNAL INFOTEL , vol. 11, no. 3, Sep. 2019. [Online]. Available: http://dx.doi.org/10.20895/infotel.v11i3.444
2019 doi
-
[35]
An innovative acoustic rain gauge based on convolutional neural networks,
R. Avanzato and F. Beritelli, “An innovative acoustic rain gauge based on convolutional neural networks,” Information, vol. 11, no. 4, p. 183, Mar. 2020. [Online]. Available: http://dx.doi.org/10.3390/info11040183
2020 doi
-
[36]
Explainable ai for safe and trustworthy autonomous driving: A systematic review,
A. Kuznietsov, B. Gyevnar, C. Wang, S. Peters, and S. V . Al- brecht, “Explainable ai for safe and trustworthy autonomous driving: A systematic review,” IEEE Transactions on Intel- ligent Transportation Systems , vol. 25, no. 12, pp. 19 342– 19 364, 2024
2024
-
[2020]
Available: https://arxiv.org/abs/2004.06320
[Online]. Available: https://arxiv.org/abs/2004.06320
2004 arXiv
-
[2023]
Available: https://www.mdpi.com/2073- 431X/12/3/65
[Online]. Available: https://www.mdpi.com/2073- 431X/12/3/65
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.