Pith. sign in

REVIEW 3 major objections 6 minor 19 references

DecoratingFusion: A LiDAR-Camera Fusion Network with the Combination of Point-level and Feature-level Fusion

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Calibration-guided image features lift 3D detection on KITTI and Waymo.

desk verdict Competent fusion-of-known-ideas paper whose central heatmap query initialization lacks a specified training signal; promising but needs clarification before the numbers can be trusted. read the letter →

arxiv 2501.00220 v1 pith:YM2VHGEB submitted 2024-12-31 cs.CV cs.LG

classification cs.CVcs.LG
keywords LiDAR-camerafusion3Dobjectdetectionpoint-levelfeature-levelcross-attentioncenterheatmapqueryinitializationautonomousdrivingKITTIandWaymobenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that LiDAR-camera fusion for 3D object detection is best done in two stages: first decorate every LiDAR point with the image feature of the pixel it projects onto, then fuse the decorated point cloud with image features through cross-attention. The decoration step is a point-level (early) fusion held together by the calibration matrix, giving interpretable hard correspondence; the cross-attention step is feature-level (mid-level) fusion, letting the network learn soft associations on top of that hard alignment. The authors report that this combination, called DecoratingFusion, reaches 77.30 mAP on KITTI validation and 74.80 L2 mAPH on Waymo validation, beating the listed baselines on both benchmarks, with the clearest margins on pedestrians and cyclists. If true, it would make hard geometric priors a practical ingredient in modern feature-fusion detectors rather than a relic of early-fusion methods.

What carries the argument

The central object is the decorated point cloud, written as $(x, y, z, r, f)$, where $(x, y, z, r)$ is a LiDAR point and $f$ is the image feature at the pixel obtained by projecting it with the calibration matrix $T$. The mechanism that carries the argument is the query-generation pipeline built on it: voxelization, two independent sparse convolutions that keep LiDAR and image features separate until BEV concatenation, a center-heatmap selection of query positions, a class one-hot embedding appended to each query, and cross-attention in which LiDAR-derived queries attend to image-derived keys and values. Together these turn hard geometric alignment into a soft, learnable fusion at the feature level.

What would settle it

Re-run the listed baselines under DecoratingFusion's exact settings (same SECOND backbone, DLA34 at 448x800, GT-Paste with five-epoch fading, AdamW one-cycle schedule) on KITTI and Waymo; if the re-trained baselines close the margins to less than one AP point, the claimed superiority is not established. Separately, test which of Waymo's five cameras supplies the decorating image features and what happens to points that project outside the 448x800 image — if that choice changes the decorated point set, the ranking may shift.

Watch

Extended reading notes

Core claim

DecoratingFusion's central claim is that point-level fusion and feature-level fusion are not alternatives: hard calibration-based decoration and learned cross-attention can be stacked in one end-to-end network. The network decorates each LiDAR point with the 2D CNN feature of its calibrated image pixel, passes the decorated cloud through two independent sparse-convolution streams (one for LiDAR features, one for image features), and then uses the concatenated BEV features to generate object queries. Those queries start from a center heatmap rather than random positions, carry class one-hot information, and are fused with image keys and values by cross-attention. With SECOND as the 3D backbone and a DLA34 image backbone, this configuration is reported to reach 77.30 mAP on KITTI validation and 74.80 L2 mAPH on Waymo validation, placing first among the compared methods and strongest on pedestrians and cyclists.

Load-bearing premise

The load-bearing premise is that Tables 1 and 2 compare like with like: the baseline numbers come from their original papers rather than being re-run under DecoratingFusion's training recipe, so if the baselines were re-trained or the Waymo camera arrangement were specified differently, the reported ranking could change.

Editorial extensions

If this is right

  • Hard geometric correspondence from calibration matrices can be carried into a feature-level fusion network, not just early-fusion point painting, without sacrificing the benefits of learned cross-attention.
  • End-to-end training of the 2D image backbone removes the need for separately pre-trained segmentation or detection models, cutting annotation cost and cross-domain mismatch.
  • Decorative image features and LiDAR features should be extracted by separate sparse convolutions and only concatenated in BEV space, because the domain gap makes joint feature extraction ineffective.
  • Center-heatmap query initialization plus class embedding brings query starting positions closer to true object centers and helps cross-attention focus on intra-class variation.
  • On KITTI and Waymo, the largest gains are on small objects (pedestrians and cyclists), where LiDAR-only methods are weakest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because decoration gives every point a pixel feature only where the image covers the scene, applying the same recipe across all of Waymo's five cameras or a full surround view would change which points are decorated; the paper does not specify its multi-camera handling, so the reported Waymo ranking could shift under that choice.
  • The method's success on pedestrians and cyclists suggests that image texture supplies what sparse LiDAR points lack; a direct test would be to degrade or corrupt the image branch (for example, simulate camera failure or fog) and measure whether small-object gains disappear first.
  • The ablations attribute most of the gain to decoration itself, not to the heatmap query or class embedding, so a simpler design that only decorates points may already capture most of the benefit; the paper's mid-level additions buy about 0.3 AP on Waymo.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes DecoratingFusion, a LiDAR-camera 3D detection network that combines point-level and feature-level fusion. Point-level fusion uses calibration matrices to decorate each LiDAR point with the corresponding image feature from a CenterNet DLA34 backbone; the decorated point cloud is processed by two independent sparse convolutional branches, one for lidar features and one for image features, and then combined in BEV. Feature-level fusion initializes cross-attention queries from a predicted center heatmap and augments each query with a one-hot category embedding derived from that heatmap. The loss is a weighted sum of focal classification and smooth-L1 regression losses. Experiments on KITTI and Waymo report mAP of 77.30 and L2 mAPH of 74.80, respectively, with ablations on Waymo L2 attributing gains to the proposed components.

Significance. If the reported results are reproducible, the paper makes a useful empirical contribution: it demonstrates a concrete calibration-based decoration mechanism, trained end-to-end, combined with heatmap-initialized queries, and it reports consistent improvements on small objects (pedestrians and cyclists) over a wide set of baselines on two standard datasets. The architecture is coherent, the experimental protocol uses official metrics, and the end-to-end training of the 2D backbone is a practical advantage over PointPainting/PointAugmenting-style approaches. However, the current manuscript does not establish that the heatmap-branch mechanism is learned as described, and the baseline comparisons are not controlled enough to support the claimed margins either. The paper would be significantly strengthened by clarifying (or fixing) the supervision of the heatmap branch, re-running at least the closest baselines in a common codebase, and reporting variance estimates.

major comments (3)
  1. [§3.3 and §3.4, Eq. (1)] The center-heatmap branch that supplies query positions and class embeddings in §3.3 receives no explicitly defined training signal. Equation (1) defines only L_cls and L_reg, and the paper does not state that L_cls includes a heatmap focal loss; if L_cls is the final detection-head classification loss, the top-n selection and one-hot class encoding are non-differentiable in the heatmap scores and no gradient reaches the heatmap predictor. Consequently, the claimed benefits of Heatmap Init. and Category Embedding in Table 4 (+0.13/+0.06 and +0.18/+0.08 on Waymo L2) are uninterpretable as described. The authors must specify a heatmap supervision loss or otherwise explain how this branch is trained; without that, the central mechanism of the paper cannot be validated.
  2. [§4.3, Tables 1 and 2] The performance comparison is not a controlled experiment: the baseline numbers are quoted from their original publications, while DecoratingFusion uses a specific training recipe described in §4.2 (SECOND backbone, CenterNet DLA34 at 448x800, GT-Paste with 5-epoch fading, AdamW one-cycle). Because several winning margins are below 1 point (e.g., Table 1, mAP 77.30 vs CAT-Det 75.42; Table 2, L2 mAPH 74.80 vs DeepFusion 74.20), the claimed superiority could be reversed under a common codebase, and no error bars are reported. To support the central claim, the authors should either re-run at least the closest competing methods under their own pipeline or provide variance estimates and explicitly state the comparison protocol.
  3. [§4.2 and Table 2] The Waymo experimental setup is under-specified: the paper does not state which of the five Waymo cameras is used to decorate the point cloud, nor what happens to points whose projections fall outside the 448x800 image. Since the detection range is [-75.2, 75.2] m on X and Y, a single front camera covers only a subset of the points, and the choice of camera(s) changes the decorated point set and therefore the reported Waymo results. This detail is necessary for reproduction and for interpreting the Waymo L2 mAPH of 74.80.
minor comments (6)
  1. [Table 3] The checkmark rows do not unambiguously indicate which components are active in each combination; please label each row (e.g., 'Decoration only', '+E2E', '+2SparseConv') so the reader can follow the incremental additions.
  2. [Table 4] The columns 'Vehicle' and 'Pedestrian' should state whether these are AP, APH, or mAPH and at which difficulty; the text says L2 but the table header does not.
  3. [Fig. 1] Figure 1 contains untranslated Chinese text in the diagram labels (e.g., '点云', '图像'); please replace these with English labels.
  4. [§2.3] The related-work statement that the current state-of-the-art fusion method is mid-level feature fusion cites CenterFormer [19], which is evaluated as a lidar-only method in Table 2; please clarify the citation or rephrase the claim.
  5. [§4.1] The KITTI evaluation description says AP with an IoU threshold of 0.7, but KITTI benchmarks commonly use 0.7 for cars and 0.5 for pedestrians and cyclists; please specify the exact evaluation setting used.
  6. [General] The paper does not mention code or model release; adding a reproducibility statement would strengthen the empirical claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are empirical benchmark results; flagged supervision and baseline-comparison issues are correctness/reproducibility risks, not definitional reductions.

full rationale

The paper's central claim is an empirical benchmark comparison: DecoratingFusion reports mAP values on held-out KITTI and Waymo validation sets and compares them with existing detectors. The path from inputs to outputs is a trained network, not a derivation whose conclusion is identical to its assumptions. The hard-association decoration step uses fixed calibration matrices, which are external sensor parameters, not fitted outputs; the heatmap-based query initialization and category embedding are architectural choices whose contributions are assessed by ablations, not by defining the evaluation metric in terms of those components. No equation in the paper defines the reported results as a function of the design choices in a way that forces the outcome, and no fitted parameter is renamed as a prediction. There is no load-bearing self-citation: the cited prior works are external baselines and inspirations, and no uniqueness or existence theorem from the authors' own prior work is invoked to rule out alternatives. The reviewer-flagged concerns, such as the absence of an explicit heatmap loss in Eq. (1) and the use of quoted rather than re-run baseline numbers, are internal-consistency and experimental-control issues that could affect interpretability or fairness of comparison, but they are not circularity. Therefore, no circular step is exhibited, and the circularity score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim depends on standard dataset calibrations, an unspecified Waymo camera selection, and the existence of a compatible 3D decoder for the concatenated features. All learned network weights are trained on the benchmark training splits; they are not listed individually here, but the hyperparameters above are the ones the paper fixes by hand.

free parameters (5)
  • number of object queries per class = not reported
    Section 3.3 selects the top n local-maximum heatmap peaks per class as queries, but n is never given; it controls model capacity and affects all downstream results.
  • training epochs = not reported
    Section 4.2 gives optimizer and learning rates but not epoch count or batch size; these are free choices that can change final AP by more than the reported margins.
  • dropout rate = 0.3
    Applied to the cross-attention correlation matrix (Section 4.2) with no sensitivity analysis; a hand-chosen regularization hyperparameter.
  • loss weight w = 2
    Weights regression loss in Eq. (1), borrowed from SECOND [13] without re-tuning; a free scalar in the objective.
  • image size = 448x800
    Input resolution for CenterNet DLA34 (Section 4.2); affects the image features attached to points and is a manually chosen constant.
assumptions (3)
  • domain assumption Calibration matrices between LiDAR and cameras are accurate and available for every frame
    Section 3.2 projects each point to the image plane using the calibration matrix; if calibration drifts or sensors are unsynchronized, attached image features are wrong.
  • domain assumption A single 448x800 image or a known camera selection is sufficient to decorate all relevant points in Waymo
    Section 4.2 specifies one image size but does not say which of Waymo's five cameras is used or how projections outside that image are handled; the point-level fusion depends on this choice.
  • domain assumption The SECOND backbone can consume decorated voxel features after two separate sparse convolutions
    The architecture in Section 3.3 concatenates BEV features from separate sparse convolutions and feeds them to the detection head; this assumes no harmful distribution shift from adding image features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DecoratingFusion: A LiDAR-Camera Fusion Network with the Combination of Point-level and Feature-level Fusion." pith.science (2026). https://pith.science/paper/YM2VHGEB

@misc{pith2026250100220,
  author       = {Pith},
  title        = {Pith review of: DecoratingFusion: A LiDAR-Camera Fusion Network with the Combination of Point-level and Feature-level Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YM2VHGEB}},
  note         = {Machine review of arXiv:2501.00220}
}
read the original abstract

Lidars and cameras play essential roles in autonomous driving, offering complementary information for 3D detection. The state-of-the-art fusion methods integrate them at the feature level, but they mostly rely on the learned soft association between point clouds and images, which lacks interpretability and neglects the hard association between them. In this paper, we combine feature-level fusion with point-level fusion, using hard association established by the calibration matrices to guide the generation of object queries. Specifically, in the early fusion stage, we use the 2D CNN features of images to decorate the point cloud data, and employ two independent sparse convolutions to extract the decorated point cloud features. In the mid-level fusion stage, we initialize the queries with a center heatmap and embed the predicted class labels as auxiliary information into the queries, making the initial positions closer to the actual centers of the targets. Extensive experiments conducted on two popular datasets, i.e. KITTI, Waymo, demonstrate the superiority of DecoratingFusion.

Figures

Figures reproduced from arXiv: 2501.00220 by the authors.

Figure 1
Figure 1. An overview of DecoratingFusion framework. DecoratingFusion consists of two parts: the point-level fusion stage and the feature-level fusion stage, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The two independent sparse convolutions used to extract lidar and image fea￾ture [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 18 canonical work pages

  1. [1]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Bai, X., Hu, Z., Zhu, X., Huang, Q., Chen, Y.: Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1090–1099 (2022)

  2. [2]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, Y., Li, Y., Zhang, X., Sun, J., Jia, J.: Focal sparse convolutional networks for 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5428–5437 (2022)

  3. [3]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Hu, J.S., Kuai, T., Waslander, S.L.: Point density-aware voxels for lidar 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8469–8478 (2022)

  4. [4]

    In: Proceedings of the European conference on computer vision (ECCV)

    Huang, T., Liu, Z., Chen, X., Bai, X.: Epnet: Enhancing point features with image semantics for 3d object detection. In: Proceedings of the European conference on computer vision (ECCV). pp. 35–52. Springer (2020)

  5. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, Y., Yu, A.W., Meng, T., Caine, B., Ngiam, J.: Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17182–17191 (2022)

  6. [6]

    In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

    Pang, S., Morris, D., Radha, H.: Clocs: Camera-lidar object candidates fusion for 3d object detection. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 10386–10393. IEEE (2020)

  7. [7]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Qi, C.R., Liu, W., Wu, C., Su, H., Guibas, L.J.: Frustum pointnets for 3d object detection from rgb-d data. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 918–927 (2018)

  8. [8]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., Li, H.: Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10529–10538 (2020)

Show all 19 references
  1. [9]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Shi, S., Wang, X., Li, H.: Pointrcnn: 3d object proposal generation and detection from point cloud. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 770–779 (2019)

  2. [10]

    IEEE transactions on pattern analysis and machine intelligence 43(8), 2647–2664 (2020)

    Shi, S., Wang, Z., Shi, J., Wang, X., Li, H.: From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE transactions on pattern analysis and machine intelligence 43(8), 2647–2664 (2020)

  3. [11]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Vora, S., Lang, A.H., Helou, B., Beijbom, O.: Pointpainting: Sequential fusion for 3d object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4604–4612 (2020)

  4. [12]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, C., Ma, C., Zhu, M., Yang, X.: Pointaugmenting: Cross-modal augmentation for 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11794–11803 (2021)

  5. [13]

    Sensors 18(10), 3337 (2018)

    Yan, Y., Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. Sensors 18(10), 3337 (2018)

  6. [14]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Yang, Z., Zhou, Y., Chen, Z., Ngiam, J.: 3d-man: 3d multi-frame attention network for object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1863–1872 (2021)

  7. [15]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Yin, T., Zhou, X., Krahenbuhl, P.: Center-based 3d object detection and track- ing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11784–11793 (2021) 12 Z. Yin et al

  8. [16]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhang, Y., Chen, J., Huang, D.: Cat-det: Contrastively augmented transformer for multi-modal 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 908–917 (2022)

  9. [17]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zheng, W., Tang, W., Jiang, L., Fu, C.W.: Se-ssd: Self-ensembling single-stage object detector from point cloud. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14494–14503 (2021)

  10. [18]

    arXiv preprint arXiv:1904.07850 (2019)

    Zhou, X., Wang, D., Kr¨ ahenb¨ uhl, P.: Objects as points. arXiv preprint arXiv:1904.07850 (2019)

  11. [19]

    In: European Conference on Computer Vision

    Zhou, Z., Zhao, X., Wang, Y., Wang, P., Foroosh, H.: Centerformer: Center-based transformer for 3d object detection. In: European Conference on Computer Vision. pp. 496–513. Springer (2022)

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.