{"id":"a52fb2b1-c75c-40c0-96d7-f69c138301de","arxiv_id":"2411.08482","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On NuScenes, camera-based 3D pedestrian detectors fail more on distant and occluded pedestrians, while LiDAR-based detectors fail more on stationary or non-moving pedestrians, with PointPillars showing the most distinct sensitivity pattern.","lead":"The paper tests how factors like distance, occlusion, weather, and pedestrian pose affect 3D pedestrian detection errors across camera- and LiDAR-based detectors on the NuScenes dataset. It combines correlation, mutual information, and a random forest with Shapley values to rank these factors and spot detector-specific weaknesses.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The §3.2 2 m BEV center-distance matching rule likely turns camera depth-localization error into apparent FN/FP at range, so the central distance-modality finding needs a threshold-robustness check.","rationale":"The reader identified the random-forest surrogate and missing uncertainty quantification as the weakest assumption. Those are real limitations, but the most load-bearing assumption for the paper's central modality-difference claim is the 2 m matching threshold used to define TP/FN. That threshold is applied uniformly to all detectors and is particularly punishing for camera-based detectors at distance because their depth estimates are less accurate; thus the distance-dependent error pattern could be an evaluation artifact rather than a true behavioral difference. This concern is concrete, testable, and directly affects the strongest claim, while the SHAP/Rashomon issue mostly affects the meta-model-based analysis of PointPillars, which is peripheral to the headline comparison. The CONDITIONAL verdict remains appropriate: the paper's methodology is reasonable, but the central finding should not be relied upon until the matching-threshold robustness check is performed and the results are reported with scene-level or bootstrap uncertainty.","tokens_in":11039,"tokens_out":4889,"duration_ms":47672,"concrete_test":"Recompute FN and FP labels for all detectors using relaxed matching thresholds (e.g., BEV center distance < 4 m and < 5 m, or an IoU-based criterion), then rerun the Kendall's Tau and MI analyses for distance, angular size, and occlusion. If the camera-vs-LiDAR gap in distance dependence shrinks or reverses, the 2 m matching criterion is driving the central finding. Additionally, decompose FN errors into 'no prediction near the object' versus 'prediction present but outside 2 m' to quantify how much of the camera distance effect is localization error rather than missed detection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that camera-based detectors misdetect distant pedestrians more often and produce more distant FPs than LiDAR-based detectors rests on the matching criterion defined in §3.2: a prediction is a true positive only if the BEV center distance to the ground-truth box is less than 2 m, and a prediction with no ground-truth box within 2 m is a false positive. Camera-based detectors, especially monocular ones, have markedly larger depth estimation error at range than LiDAR detectors. Under this fixed metric, a camera detection that correctly classifies a pedestrian but localizes the center 2–3 m off is scored as a false negative, and a true pedestrian whose predicted center is offset by more than 2 m is scored as a false positive. Consequently, the paper's headline observation — that cameras 'misdetect distant pedestrians more frequently' and 'generate more FP detections at greater distances' — may be an artifact of the 2 m threshold penalizing camera depth uncertainty rather than a genuine difference in missed or ghost detections. This affects the univariate Kendall's Tau and MI analyses directly, and it would persist even if the random-forest surrogate were perfectly faithful. The reader's concern about the Rashomon effect is secondary: PointPillars' SHAP ranking is only a small part of the paper's argument, whereas the distance-dependent FN/FP pattern is the core conclusion.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a methodology for analyzing how object- and environment-related 'meta-information' influences pedestrian detection errors in 3D object detection, and applies it to six detectors (FCOS3D, MonoDIS, SpatialDETR for cameras; Megvii, PointPillars, TransFusion-L for LiDAR) on the NuScenes validation set. Three analyses are performed: Kendall's tau rank correlation, mutual information, and a random forest meta-model with SHAP values, separately for false negatives and false positives. The central findings are that camera-based detectors are more strongly affected by distance and occlusion, while LiDAR-based detectors depend more on pedestrian attributes and velocity; weather and location factors show small effects. The authors propose the methodology as a support tool for safety assurance of DNN-based object detectors.","tokens_in":11321,"tokens_out":5892,"duration_ms":50573,"significance":"The paper's comparative design is a genuine contribution: it considers both error types, a broad set of meta-information, and three complementary analytical lenses, with multiple detectors per modality. The authors are also transparent about limitations (weather data diversity, Rashomon effect for the meta-model). If the findings are robust, they provide an actionable map of modality-specific failure modes. However, the quantitative conclusions currently rest on point estimates without uncertainty quantification and on a fixed 2 m matching threshold that may penalize camera depth error; the central modality comparison therefore needs additional robustness evidence before the method can serve as a reliable safety-analysis tool.","major_comments":[{"comment":"The central claim that camera-based detectors misdetect distant pedestrians more often and produce more distant FPs than LiDAR detectors is directly tied to the BEV center-distance matching threshold of 2 m. Camera-based detectors, especially monocular ones, have larger depth-localization error at range, so a detection with correct classification but 2–3 m center offset is counted as a false negative, and an offset prediction becomes a false positive. Under the fixed 2 m rule, the distance-dependent FN/FP patterns in Figs. 1–6 (and the corresponding statements in §5) may partly reflect localization accuracy rather than detection or ghost-detection behavior. Please run a threshold-robustness analysis (e.g., matching at 1.0, 1.5, 2.5, 3.0 m) and report whether the modality differences persist. If they do not, the paper's headline should be reframed.","section":"§3.2"},{"comment":"Figures 1–6 report Kendall's tau, MI, and mean absolute SHAP values without confidence intervals or significance tests. Point estimates alone are insufficient to support claims such as 'camera detectors have a relatively strong negative correlation with distance compared to LiDAR detectors' (Fig. 1) or the detector-level differences in §4.2. Please add bootstrap confidence intervals or permutation tests for the correlations and SHAP values, and clearly indicate when observed differences are not statistically distinguishable.","section":"§4.1"},{"comment":"The random forest meta-model is trained on the same NuScenes validation set used for the univariate analyses and for confidence-threshold optimization, but the manuscript does not describe a data split or cross-validation for the RF itself. Grid-search tuning on the same set risks overfitting, so the SHAP importances in Figs. 3 and 6 may reflect quirks of that split. Additionally, the authors themselves acknowledge in §5 that due to the Rashomon effect the meta-model does not necessarily represent the true behavior of the detectors. To make the meta-model-based results interpretable, report holdout performance (e.g., AUROC/F1 on a separate subset) and discuss the consistency of SHAP rankings across multiple fitted models.","section":"§3.3"}],"minor_comments":[{"comment":"In the text near Fig. 6, 'the mean absolute SHAP values in Fig. 5 for temperature and humidity' should reference Fig. 6, since it describes the SHAP analysis rather than the MI analysis.","section":"§4.2"},{"comment":"The phrase 'the width, height, and length of the 3D bounding box around the pedestrian m3' mixes units incorrectly; should be 'in meters' for each dimension or 'volume in m^3' if intended.","section":"§3.1"},{"comment":"The phrase 'A Vs' should be 'AVs' or 'automated vehicles'.","section":"Introduction"},{"comment":"Reference [18] appears to have a placeholder or incorrect title ('MS Windows NT kernel description'); the Visual Crossing weather API reference needs correct metadata.","section":"References"},{"comment":"The sentence about visibility contains a typo: 'visibilites' should be 'visibilities'.","section":"§5"},{"comment":"The statement 'The examined dataset consists of only 150 scenes, while it contains about 6000 different samples' should clarify that this is the validation subset and, for context, state the total number of scenes in NuScenes.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper fits the journal's scope in intelligent vehicles and perception safety. The central modality-comparison claim is plausible but currently too sensitive to the matching threshold and lacks statistical uncertainty; these issues are addressable within a revision and should gate acceptance. The authors might also be encouraged to release their analysis code to strengthen reproducibility, given the methodological character of the contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent empirical study with a real but modest contribution, and the authors are refreshingly honest about its limits. The main caveat is that the headline result — cameras misdetect distant pedestrians more and produce more distant false positives — could be inflated by the fixed 2 m BEV matching rule, which penalizes camera depth error at range.\n\nWhat's actually new: prior work (Ponn et al.) did RF+SHAP on 2D camera detection and only false negatives. This paper extends the recipe to six 3D detectors (three camera, three LiDAR) on NuScenes, includes false positives, and triangulates with Kendall's tau, mutual information, and SHAP. That triangulation is the methodological core, and it's reasonable. The qualitative agreement across the three measures for most detectors gives me some confidence the modality differences are real and not just noise.\n\nWhat it does well: the weather-data limitations are stated plainly (nearest-station data, low diversity, lowest visibility 6.3 km, no heavy rain). The Rashomon effect caveat on the random forest surrogate is acknowledged in Section 5. That is more honesty than most papers in this area.\n\nSoft spots, in order of importance:\n\n1. The 2 m center-distance matching rule. Camera detectors, especially monocular ones, have larger depth error at range. Under a hard 2 m threshold, a correct classification with a 2–3 m localization error is scored as a false negative, and a true pedestrian whose predicted center is off by >2 m is scored as a false positive. So the distance-dependent FN/FP pattern may partly reflect localization error, not missed or ghost detections. The paper needs a sensitivity analysis at, say, 1 m, 2 m, 4 m, or a distance-dependent threshold, to show the modality comparison holds. This is the load-bearing concern.\n\n2. No confidence intervals or significance tests. The bar plots look clean, but without error bars we can't tell which detector differences are meaningful. Minor-to-moderate, because the consistency across Kendall/MI/SHAP partially compensates.\n\n3. No code or data release. For a methodology paper that claims to be usable for safety assurance, artifacts matter. Minor as a scientific flaw, but it limits adoption.\n\nThe Rashomon-effect concern the reader raised is secondary: it's acknowledged, it mainly affects the PointPillars SHAP ranking, which is not the core argument.\n\nWho this is for: safety engineers and perception researchers who want a low-cost way to profile detector failure modes before deployment. It deserves a serious referee. My recommendation: send it to review, with a required robustness check on the matching threshold and ideally confidence intervals.","headline":"A useful, honest empirical comparison of 3D pedestrian detectors, but the central distance-modality result needs a matching-threshold sensitivity check before I'd trust it.","tokens_in":11825,"tokens_out":1773,"would_cite":true,"duration_ms":15961,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Camera-based 3D pedestrian detectors fail more at distance and occlusion, while LiDAR detectors fail more on still pedestrians, says a six-detector meta-information analysis.","keywords":["3D object detection","pedestrian detection","LiDAR","camera","random forest","SHAP","mutual information","NuScenes"],"falsifier":"Train the same six detectors on a validation split with balanced distance, occlusion, and weather distributions, and re-run the SHAP analysis: if camera detectors no longer show a distance-dependent drop in pedestrian recall or LiDAR detectors no longer show a pose/velocity-dependent drop, the claimed modality-specific error patterns would be falsified. More narrowly, if the random forest meta-model's top features for a detector change substantially under different random seeds or an alternative explainability method, then the meta-model-based rankings are not stable enough to represent the detector's behavior.","tokens_in":10842,"feed_emoji":"🚶","tokens_out":9690,"duration_ms":80966,"temperature":0.7,"pith_summary":"To make black-box 3D object detectors safer for automated driving, this paper develops a methodology for finding which scene and object factors drive detection errors. It applies the methodology to six pedestrian detectors—three camera-based and three LiDAR-based—on the NuScenes validation set, analyzing both missed pedestrians (false negatives) and ghost detections (false positives). The analysis pairs univariate statistics (Kendall's tau and mutual information) with a random forest meta-model explained by SHAP values, so feature interactions are considered. The paper's central findings are that camera detectors miss more pedestrians and create more ghost detections at larger distances, occlusion hurts all detectors but hits cameras harder, and LiDAR detectors depend more on pedestrian attributes and velocity. The authors argue that this comparative, factor-level evaluation identifies modality-specific and detector-specific weaknesses that safety assurance should target.","feed_headline":"Cameras miss distant pedestrians; LiDAR misses still ones","feed_subtitle":"A six-detector study maps which failure scenarios each sensing modality must be safety-checked for.","key_machinery":"The central machinery is a two-tier analysis. The univariate tier uses Kendall's tau to capture monotonic dependence and normalized mutual information (estimated with a k-nearest-neighbor estimator) to capture nonlinear, non-monotonic dependence between each meta-information feature and detection error. The meta-model tier trains a random forest to predict false-negative and false-positive errors from the same features, then extracts feature importance through SHAP values, computed with conditional sampling to avoid extrapolating correlated features. The random forest's role is to model interactions among factors, and SHAP converts that model into a ranked list of influencing factors. The two tiers are meant to be read together: agreement is treated as a reliable pattern, while disagreement signals that a factor's influence is conditional on other features.","core_discovery":"The paper's central claim is that camera- and LiDAR-based 3D pedestrian detectors have distinct, measurable failure patterns that can be ranked by meta-information about the object and environment. On the NuScenes validation set, camera-based detectors show a stronger negative correlation between distance and correct detection than LiDAR detectors, and their false-positive rate also grows with distance; the authors attribute this to the LiDAR sensor's larger range. Occlusion, measured by NuScenes visibility tokens, degrades all detectors, with a stronger effect on cameras. LiDAR-based detectors in turn depend more on the pedestrian's attribute and velocity: moving pedestrians are detected more reliably, while standing, sitting, or lying pedestrians are more often missed, consistent with pose-dependent reflection properties in point clouds. Weather and location factors generally show small influence, which the authors tie to the limited weather diversity of the dataset. The paper also reports detector-specific deviations, most notably PointPillars, whose SHAP rankings diverge from the univariate analysis, indicating that the influence of factors can change when features are combined.","pith_inferences":["A practical consequence not developed in the paper: because camera false positives increase with distance, a safety system could gate distant camera-only detections behind a LiDAR confirmation before acting on them.","The same methodology could be applied to other object classes and to camera-LiDAR fusion detectors; distance and occlusion rankings would likely shift because fusion inherits the stronger long-range modality, but that is a testable extension, not a result of this paper.","The paper's limited weather diversity suggests that replicating the analysis on data with fog, heavy rain, and low visibility would likely raise weather factors in the importance rankings—an inference, since the paper does not test those conditions.","Disagreements between univariate and SHAP rankings, as seen for PointPillars, could serve as a diagnostic signal that an architecture's failure modes are interaction-driven and therefore hard to capture with single-factor tests."],"forward_implications":["Camera-based pedestrian detection should be stress-tested at long range: cameras both miss more pedestrians and produce more ghost detections as distance grows.","LiDAR-based systems should be specifically validated for stationary, sitting, and lying pedestrians, since attribute and velocity are strong drivers of their failures.","Occlusion should be a standard safety-validation scenario for all detectors, and especially for camera systems, because it degrades every detector examined and cameras most.","Weather-related conclusions from NuScenes should not be generalized to adverse conditions; the dataset's warm, low-rain, high-visibility range limits the generalizability of the small weather effects.","Detector-specific differences, such as PointPillars' divergent rankings, imply that per-modality conclusions should be re-checked for each new architecture rather than assumed to transfer."],"supporting_citations":[{"why":"Supplies the random-forest-with-SHAP meta-model template that this paper extends from 2D camera-only analysis to 3D multi-detector, false-negative and false-positive analysis.","marker":"[12]"},{"why":"Provides the NuScenes multimodal dataset and annotations from which all meta-information, ground truth, visibility tokens, and weather labels are drawn.","marker":"[17]"},{"why":"Introduces SHAP values, the method used to rank the random forest's feature importances for each detector.","marker":"[14]"},{"why":"Supplies the Rashomon-effect limitation and the conditional-sampling procedure used to compute Shapley values without extrapolating correlated features.","marker":"[15]"},{"why":"Defines random decision forests, the meta-model trained to predict detection errors from meta-information.","marker":"[13]"},{"why":"Provides Kendall's tau, the rank correlation coefficient used in the univariate monotonic analysis.","marker":"[26]"},{"why":"Provides the k-nearest-neighbor mutual information estimator used to capture nonlinear dependencies in the univariate analysis.","marker":"[30]"},{"why":"Supplies evidence that human pose changes the number of reflections and intensities in point clouds, which the paper uses to explain why LiDAR detectors depend on pedestrian attributes and velocity.","marker":"[33]"}],"fun_headline_variants":["Distance trips cameras; stillness trips LiDAR in 3D detection","Cameras + range, LiDAR + pose: 3D detector failure factors","Range fails cameras, pose fails LiDAR: pedestrian detector gaps","Far pedestrians stump cameras; still ones stump LiDAR","Pedestrian detection gaps: distance for cameras, stillness for LiDAR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the random forest meta-model trained on NuScenes validation data faithfully represents each detector's true error behavior, so its SHAP rankings reflect real influencing factors; the paper itself notes the Rashomon effect means different equally good meta-models can give different explanations.","fun_headline_variants_meta":{"raw":{"variants":["Distance trips cameras; stillness trips LiDAR in 3D detection","Cameras + range, LiDAR + pose: 3D detector failure factors","Range fails cameras, pose fails LiDAR: pedestrian detector gaps","Far pedestrians stump cameras; still ones stump LiDAR","Pedestrian detection gaps: distance for cameras, stillness for LiDAR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001179,"raw_usage":{"total_tokens":4826,"prompt_tokens":855,"completion_tokens":3971,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":3881}},"tokens_in":471,"tokens_out":3971,"duration_ms":27812,"temperature":1.0,"reasoning_tokens":3881,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:31:04.664840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same six detectors on a validation split with balanced distance, occlusion, and weather distributions, and re-run the SHAP analysis: if camera detectors no longer show a distance-dependent drop in pedestrian recall or LiDAR detectors no longer show a pose/velocity-dependent drop, the claimed modality-specific error patterns would be falsified. More narrowly, if the random forest meta-model's top features for a detector change substantially under different random seeds or an alternative explainability method, then the meta-model-based rankings are not stable enough to represent the detector's behavior.","supporting_citations":[{"cited_title":"Performance analysis of camera-based object detection for automated vehicles,","cited_arxiv_id":null,"evidence_quote":"Supplies the random-forest-with-SHAP meta-model template that this paper extends from 2D camera-only analysis to 3D multi-detector, false-negative and false-positive analysis."},{"cited_title":"nuscenes: A multimodal dataset for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Provides the NuScenes multimodal dataset and annotations from which all meta-information, ground truth, visibility tokens, and weather labels are drawn."},{"cited_title":"A unified approach to in- terpreting model predictions,","cited_arxiv_id":null,"evidence_quote":"Introduces SHAP values, the method used to rank the random forest's feature importances for each detector."},{"cited_title":"Freiesleben and C","cited_arxiv_id":null,"evidence_quote":"Supplies the Rashomon-effect limitation and the conditional-sampling procedure used to compute Shapley values without extrapolating correlated features."},{"cited_title":"Random decision forests,","cited_arxiv_id":null,"evidence_quote":"Defines random decision forests, the meta-model trained to predict detection errors from meta-information."},{"cited_title":"Estimating mutual information,","cited_arxiv_id":null,"evidence_quote":"Provides the k-nearest-neighbor mutual information estimator used to capture nonlinear dependencies in the univariate analysis."},{"cited_title":"Pedestrian detection with lidar technology in smart-city deployments–challenges and recommendations,","cited_arxiv_id":null,"evidence_quote":"Supplies evidence that human pose changes the number of reflections and intensities in point clouds, which the paper uses to explain why LiDAR detectors depend on pedestrian attributes and velocity."}],"review_version":1}