REVIEW 3 major objections 6 minor 19 references
DecoratingFusion: A LiDAR-Camera Fusion Network with the Combination of Point-level and Feature-level Fusion
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Calibration-guided image features lift 3D detection on KITTI and Waymo.
desk verdict Competent fusion-of-known-ideas paper whose central heatmap query initialization lacks a specified training signal; promising but needs clarification before the numbers can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the decorated point cloud, written as $(x, y, z, r, f)$, where $(x, y, z, r)$ is a LiDAR point and $f$ is the image feature at the pixel obtained by projecting it with the calibration matrix $T$. The mechanism that carries the argument is the query-generation pipeline built on it: voxelization, two independent sparse convolutions that keep LiDAR and image features separate until BEV concatenation, a center-heatmap selection of query positions, a class one-hot embedding appended to each query, and cross-attention in which LiDAR-derived queries attend to image-derived keys and values. Together these turn hard geometric alignment into a soft, learnable fusion at the feature level.
What would settle it
Re-run the listed baselines under DecoratingFusion's exact settings (same SECOND backbone, DLA34 at 448x800, GT-Paste with five-epoch fading, AdamW one-cycle schedule) on KITTI and Waymo; if the re-trained baselines close the margins to less than one AP point, the claimed superiority is not established. Separately, test which of Waymo's five cameras supplies the decorating image features and what happens to points that project outside the 448x800 image — if that choice changes the decorated point set, the ranking may shift.
Extended reading notes
Core claim
DecoratingFusion's central claim is that point-level fusion and feature-level fusion are not alternatives: hard calibration-based decoration and learned cross-attention can be stacked in one end-to-end network. The network decorates each LiDAR point with the 2D CNN feature of its calibrated image pixel, passes the decorated cloud through two independent sparse-convolution streams (one for LiDAR features, one for image features), and then uses the concatenated BEV features to generate object queries. Those queries start from a center heatmap rather than random positions, carry class one-hot information, and are fused with image keys and values by cross-attention. With SECOND as the 3D backbone and a DLA34 image backbone, this configuration is reported to reach 77.30 mAP on KITTI validation and 74.80 L2 mAPH on Waymo validation, placing first among the compared methods and strongest on pedestrians and cyclists.
Load-bearing premise
The load-bearing premise is that Tables 1 and 2 compare like with like: the baseline numbers come from their original papers rather than being re-run under DecoratingFusion's training recipe, so if the baselines were re-trained or the Waymo camera arrangement were specified differently, the reported ranking could change.
Editorial extensions
If this is right
- Hard geometric correspondence from calibration matrices can be carried into a feature-level fusion network, not just early-fusion point painting, without sacrificing the benefits of learned cross-attention.
- End-to-end training of the 2D image backbone removes the need for separately pre-trained segmentation or detection models, cutting annotation cost and cross-domain mismatch.
- Decorative image features and LiDAR features should be extracted by separate sparse convolutions and only concatenated in BEV space, because the domain gap makes joint feature extraction ineffective.
- Center-heatmap query initialization plus class embedding brings query starting positions closer to true object centers and helps cross-attention focus on intra-class variation.
- On KITTI and Waymo, the largest gains are on small objects (pedestrians and cyclists), where LiDAR-only methods are weakest.
Reading between the lines
- Because decoration gives every point a pixel feature only where the image covers the scene, applying the same recipe across all of Waymo's five cameras or a full surround view would change which points are decorated; the paper does not specify its multi-camera handling, so the reported Waymo ranking could shift under that choice.
- The method's success on pedestrians and cyclists suggests that image texture supplies what sparse LiDAR points lack; a direct test would be to degrade or corrupt the image branch (for example, simulate camera failure or fog) and measure whether small-object gains disappear first.
- The ablations attribute most of the gain to decoration itself, not to the heatmap query or class embedding, so a simpler design that only decorates points may already capture most of the benefit; the paper's mid-level additions buy about 0.3 AP on Waymo.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DecoratingFusion, a LiDAR-camera 3D detection network that combines point-level and feature-level fusion. Point-level fusion uses calibration matrices to decorate each LiDAR point with the corresponding image feature from a CenterNet DLA34 backbone; the decorated point cloud is processed by two independent sparse convolutional branches, one for lidar features and one for image features, and then combined in BEV. Feature-level fusion initializes cross-attention queries from a predicted center heatmap and augments each query with a one-hot category embedding derived from that heatmap. The loss is a weighted sum of focal classification and smooth-L1 regression losses. Experiments on KITTI and Waymo report mAP of 77.30 and L2 mAPH of 74.80, respectively, with ablations on Waymo L2 attributing gains to the proposed components.
Significance. If the reported results are reproducible, the paper makes a useful empirical contribution: it demonstrates a concrete calibration-based decoration mechanism, trained end-to-end, combined with heatmap-initialized queries, and it reports consistent improvements on small objects (pedestrians and cyclists) over a wide set of baselines on two standard datasets. The architecture is coherent, the experimental protocol uses official metrics, and the end-to-end training of the 2D backbone is a practical advantage over PointPainting/PointAugmenting-style approaches. However, the current manuscript does not establish that the heatmap-branch mechanism is learned as described, and the baseline comparisons are not controlled enough to support the claimed margins either. The paper would be significantly strengthened by clarifying (or fixing) the supervision of the heatmap branch, re-running at least the closest baselines in a common codebase, and reporting variance estimates.
major comments (3)
- [§3.3 and §3.4, Eq. (1)] The center-heatmap branch that supplies query positions and class embeddings in §3.3 receives no explicitly defined training signal. Equation (1) defines only L_cls and L_reg, and the paper does not state that L_cls includes a heatmap focal loss; if L_cls is the final detection-head classification loss, the top-n selection and one-hot class encoding are non-differentiable in the heatmap scores and no gradient reaches the heatmap predictor. Consequently, the claimed benefits of Heatmap Init. and Category Embedding in Table 4 (+0.13/+0.06 and +0.18/+0.08 on Waymo L2) are uninterpretable as described. The authors must specify a heatmap supervision loss or otherwise explain how this branch is trained; without that, the central mechanism of the paper cannot be validated.
- [§4.3, Tables 1 and 2] The performance comparison is not a controlled experiment: the baseline numbers are quoted from their original publications, while DecoratingFusion uses a specific training recipe described in §4.2 (SECOND backbone, CenterNet DLA34 at 448x800, GT-Paste with 5-epoch fading, AdamW one-cycle). Because several winning margins are below 1 point (e.g., Table 1, mAP 77.30 vs CAT-Det 75.42; Table 2, L2 mAPH 74.80 vs DeepFusion 74.20), the claimed superiority could be reversed under a common codebase, and no error bars are reported. To support the central claim, the authors should either re-run at least the closest competing methods under their own pipeline or provide variance estimates and explicitly state the comparison protocol.
- [§4.2 and Table 2] The Waymo experimental setup is under-specified: the paper does not state which of the five Waymo cameras is used to decorate the point cloud, nor what happens to points whose projections fall outside the 448x800 image. Since the detection range is [-75.2, 75.2] m on X and Y, a single front camera covers only a subset of the points, and the choice of camera(s) changes the decorated point set and therefore the reported Waymo results. This detail is necessary for reproduction and for interpreting the Waymo L2 mAPH of 74.80.
minor comments (6)
- [Table 3] The checkmark rows do not unambiguously indicate which components are active in each combination; please label each row (e.g., 'Decoration only', '+E2E', '+2SparseConv') so the reader can follow the incremental additions.
- [Table 4] The columns 'Vehicle' and 'Pedestrian' should state whether these are AP, APH, or mAPH and at which difficulty; the text says L2 but the table header does not.
- [Fig. 1] Figure 1 contains untranslated Chinese text in the diagram labels (e.g., '点云', '图像'); please replace these with English labels.
- [§2.3] The related-work statement that the current state-of-the-art fusion method is mid-level feature fusion cites CenterFormer [19], which is evaluated as a lidar-only method in Table 2; please clarify the citation or rephrase the claim.
- [§4.1] The KITTI evaluation description says AP with an IoU threshold of 0.7, but KITTI benchmarks commonly use 0.7 for cars and 0.5 for pedestrians and cyclists; please specify the exact evaluation setting used.
- [General] The paper does not mention code or model release; adding a reproducibility statement would strengthen the empirical claims.
Circularity Check
No circularity: the paper's claims are empirical benchmark results; flagged supervision and baseline-comparison issues are correctness/reproducibility risks, not definitional reductions.
full rationale
The paper's central claim is an empirical benchmark comparison: DecoratingFusion reports mAP values on held-out KITTI and Waymo validation sets and compares them with existing detectors. The path from inputs to outputs is a trained network, not a derivation whose conclusion is identical to its assumptions. The hard-association decoration step uses fixed calibration matrices, which are external sensor parameters, not fitted outputs; the heatmap-based query initialization and category embedding are architectural choices whose contributions are assessed by ablations, not by defining the evaluation metric in terms of those components. No equation in the paper defines the reported results as a function of the design choices in a way that forces the outcome, and no fitted parameter is renamed as a prediction. There is no load-bearing self-citation: the cited prior works are external baselines and inspirations, and no uniqueness or existence theorem from the authors' own prior work is invoked to rule out alternatives. The reviewer-flagged concerns, such as the absence of an explicit heatmap loss in Eq. (1) and the use of quoted rather than re-run baseline numbers, are internal-consistency and experimental-control issues that could affect interpretability or fairness of comparison, but they are not circularity. Therefore, no circular step is exhibited, and the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- number of object queries per class =
not reported
- training epochs =
not reported
- dropout rate =
0.3
- loss weight w =
2
- image size =
448x800
assumptions (3)
- domain assumption Calibration matrices between LiDAR and cameras are accurate and available for every frame
- domain assumption A single 448x800 image or a known camera selection is sufficient to decorate all relevant points in Waymo
- domain assumption The SECOND backbone can consume decorated voxel features after two separate sparse convolutions
Cite this review
Pith. "Pith review of DecoratingFusion: A LiDAR-Camera Fusion Network with the Combination of Point-level and Feature-level Fusion." pith.science (2026). https://pith.science/paper/YM2VHGEB
@misc{pith2026250100220,
author = {Pith},
title = {Pith review of: DecoratingFusion: A LiDAR-Camera Fusion Network with the Combination of Point-level and Feature-level Fusion},
year = {2026},
howpublished = {\url{https://pith.science/paper/YM2VHGEB}},
note = {Machine review of arXiv:2501.00220}
}
read the original abstract
Lidars and cameras play essential roles in autonomous driving, offering complementary information for 3D detection. The state-of-the-art fusion methods integrate them at the feature level, but they mostly rely on the learned soft association between point clouds and images, which lacks interpretability and neglects the hard association between them. In this paper, we combine feature-level fusion with point-level fusion, using hard association established by the calibration matrices to guide the generation of object queries. Specifically, in the early fusion stage, we use the 2D CNN features of images to decorate the point cloud data, and employ two independent sparse convolutions to extract the decorated point cloud features. In the mid-level fusion stage, we initialize the queries with a center heatmap and embed the predicted class labels as auxiliary information into the queries, making the initial positions closer to the actual centers of the targets. Extensive experiments conducted on two popular datasets, i.e. KITTI, Waymo, demonstrate the superiority of DecoratingFusion.
Figures
Reference graph
Works this paper leans on
-
[1]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Bai, X., Hu, Z., Zhu, X., Huang, Q., Chen, Y.: Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1090–1099 (2022)
work page 2022
-
[2]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Chen, Y., Li, Y., Zhang, X., Sun, J., Jia, J.: Focal sparse convolutional networks for 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5428–5437 (2022)
work page 2022
-
[3]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Hu, J.S., Kuai, T., Waslander, S.L.: Point density-aware voxels for lidar 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8469–8478 (2022)
work page 2022
-
[4]
In: Proceedings of the European conference on computer vision (ECCV)
Huang, T., Liu, Z., Chen, X., Bai, X.: Epnet: Enhancing point features with image semantics for 3d object detection. In: Proceedings of the European conference on computer vision (ECCV). pp. 35–52. Springer (2020)
work page 2020
-
[5]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Li, Y., Yu, A.W., Meng, T., Caine, B., Ngiam, J.: Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17182–17191 (2022)
work page 2022
-
[6]
In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Pang, S., Morris, D., Radha, H.: Clocs: Camera-lidar object candidates fusion for 3d object detection. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 10386–10393. IEEE (2020)
work page 2020
-
[7]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Qi, C.R., Liu, W., Wu, C., Su, H., Guibas, L.J.: Frustum pointnets for 3d object detection from rgb-d data. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 918–927 (2018)
work page 2018
-
[8]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., Li, H.: Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10529–10538 (2020)
work page 2020
Show all 19 references
-
[9]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Shi, S., Wang, X., Li, H.: Pointrcnn: 3d object proposal generation and detection from point cloud. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 770–779 (2019)
2019
-
[10]
IEEE transactions on pattern analysis and machine intelligence 43(8), 2647–2664 (2020)
Shi, S., Wang, Z., Shi, J., Wang, X., Li, H.: From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE transactions on pattern analysis and machine intelligence 43(8), 2647–2664 (2020)
2020
-
[11]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Vora, S., Lang, A.H., Helou, B., Beijbom, O.: Pointpainting: Sequential fusion for 3d object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4604–4612 (2020)
2020
-
[12]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Wang, C., Ma, C., Zhu, M., Yang, X.: Pointaugmenting: Cross-modal augmentation for 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11794–11803 (2021)
2021
-
[13]
Sensors 18(10), 3337 (2018)
Yan, Y., Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. Sensors 18(10), 3337 (2018)
2018
-
[14]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Yang, Z., Zhou, Y., Chen, Z., Ngiam, J.: 3d-man: 3d multi-frame attention network for object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1863–1872 (2021)
2021
-
[15]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Yin, T., Zhou, X., Krahenbuhl, P.: Center-based 3d object detection and track- ing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11784–11793 (2021) 12 Z. Yin et al
2021
-
[16]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Zhang, Y., Chen, J., Huang, D.: Cat-det: Contrastively augmented transformer for multi-modal 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 908–917 (2022)
2022
-
[17]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Zheng, W., Tang, W., Jiang, L., Fu, C.W.: Se-ssd: Self-ensembling single-stage object detector from point cloud. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14494–14503 (2021)
2021
-
[18]
arXiv preprint arXiv:1904.07850 (2019)
Zhou, X., Wang, D., Kr¨ ahenb¨ uhl, P.: Objects as points. arXiv preprint arXiv:1904.07850 (2019)
2019 arXiv
-
[19]
In: European Conference on Computer Vision
Zhou, Z., Zhao, X., Wang, Y., Wang, P., Foroosh, H.: Centerformer: Center-based transformer for 3d object detection. In: European Conference on Computer Vision. pp. 496–513. Springer (2022)
2022
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.