{"id":"dd57392a-dd78-43ff-abbc-77d0aeaf9c36","arxiv_id":"2501.00220","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DecoratingFusion attaches image features to LiDAR points using calibration and then fuses both through cross-attention queries, improving 3D detection on KITTI and Waymo.","lead":"DecoratingFusion is a 3D object detection network that attaches camera image features to LiDAR points using the calibration between the two sensors, then fuses both modalities with cross-attention. It reports higher detection scores than several published baselines on KITTI and Waymo, especially for small objects such as pedestrians.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The center-heatmap branch that drives query initialization and class embedding appears to receive no training signal: Eq. 1 omits a heatmap loss, and the top-n/one-hot selection is non-differentiable. Table 4's gains are uninterpretable unless this is clarified.","rationale":"The reader's weakest assumption concerns apples-to-apples baseline comparisons. That is a valid secondary concern, but the more load-bearing issue is internal: the paper's central contribution is the heatmap-based query initialization and class embedding, yet Equation 1 does not specify any loss for the heatmap branch. Because the top-n selection and one-hot class embedding are non-differentiable in the heatmap scores, the heatmap branch cannot be trained by the downstream detection loss alone. If the heatmap loss is in fact L_cls, the paper should say so explicitly and also account for the detection head's classification loss. This affects the interpretability of Table 4 and the first claimed contribution, and it is easy to resolve by inspecting the training configuration or running the ablation with and without a heatmap loss. I therefore keep the reader's CONDITIONAL verdict but for a different, more specific reason. The paper does provide a plausible architecture, clear writing, and consistent directional improvements on two datasets, so no rejection is warranted; the condition should be: clarify (or release) the loss/configuration for the center-heatmap branch and the Waymo camera selection.","tokens_in":8399,"tokens_out":11179,"duration_ms":106067,"concrete_test":"Ask the authors for the training config/code and identify every loss term attached to the center-heatmap branch. Then run a minimal ablation on Waymo: remove the heatmap loss while keeping HeatmapInit and CategoryEmbedding; if vehicle/pedestrian L2 APH do not return to the 72.84/76.35 baseline of Table 4, the heatmap branch is being trained through an unspecified path. Also require a sentence stating whether L_cls in Eq. 1 is applied to the heatmap branch, the detection head, or both.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is the supervision of the center-heatmap branch that drives both query initialization and category embedding. Section 3.3 predicts a center heatmap \\hat{Y} in R^{X x Y x K}, selects top-n local maxima as queries, and one-hot encodes the predicted class into each query. Section 3.4 defines the only loss as L = L_cls + 2*L_reg (Eq. 1), with no separate heatmap loss and no statement of which outputs L_cls and L_reg apply to. Query selection (top-n) and the one-hot class embedding are non-differentiable in the heatmap scores, so if L_cls is the final detection-head classification, no gradient reaches the heatmap branch. The claimed 'initial positions closer to actual centers' and the class-embedding benefit in Table 4 then cannot be learned; the +0.13/+0.06 and +0.05/+0.02 gains are uninterpretable. If L_cls is instead the heatmap focal loss, the paper must still specify the final-head classification loss, since the architecture has two classification outputs. This is more fundamental than the baseline-comparison issue: it concerns whether the mechanism central to the title and contributions can work as described.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DecoratingFusion, a LiDAR-camera 3D detection network that combines point-level and feature-level fusion. Point-level fusion uses calibration matrices to decorate each LiDAR point with the corresponding image feature from a CenterNet DLA34 backbone; the decorated point cloud is processed by two independent sparse convolutional branches, one for lidar features and one for image features, and then combined in BEV. Feature-level fusion initializes cross-attention queries from a predicted center heatmap and augments each query with a one-hot category embedding derived from that heatmap. The loss is a weighted sum of focal classification and smooth-L1 regression losses. Experiments on KITTI and Waymo report mAP of 77.30 and L2 mAPH of 74.80, respectively, with ablations on Waymo L2 attributing gains to the proposed components.","tokens_in":8600,"tokens_out":6691,"duration_ms":55815,"significance":"If the reported results are reproducible, the paper makes a useful empirical contribution: it demonstrates a concrete calibration-based decoration mechanism, trained end-to-end, combined with heatmap-initialized queries, and it reports consistent improvements on small objects (pedestrians and cyclists) over a wide set of baselines on two standard datasets. The architecture is coherent, the experimental protocol uses official metrics, and the end-to-end training of the 2D backbone is a practical advantage over PointPainting/PointAugmenting-style approaches. However, the current manuscript does not establish that the heatmap-branch mechanism is learned as described, and the baseline comparisons are not controlled enough to support the claimed margins either. The paper would be significantly strengthened by clarifying (or fixing) the supervision of the heatmap branch, re-running at least the closest baselines in a common codebase, and reporting variance estimates.","major_comments":[{"comment":"The center-heatmap branch that supplies query positions and class embeddings in §3.3 receives no explicitly defined training signal. Equation (1) defines only L_cls and L_reg, and the paper does not state that L_cls includes a heatmap focal loss; if L_cls is the final detection-head classification loss, the top-n selection and one-hot class encoding are non-differentiable in the heatmap scores and no gradient reaches the heatmap predictor. Consequently, the claimed benefits of Heatmap Init. and Category Embedding in Table 4 (+0.13/+0.06 and +0.18/+0.08 on Waymo L2) are uninterpretable as described. The authors must specify a heatmap supervision loss or otherwise explain how this branch is trained; without that, the central mechanism of the paper cannot be validated.","section":"§3.3 and §3.4, Eq. (1)"},{"comment":"The performance comparison is not a controlled experiment: the baseline numbers are quoted from their original publications, while DecoratingFusion uses a specific training recipe described in §4.2 (SECOND backbone, CenterNet DLA34 at 448x800, GT-Paste with 5-epoch fading, AdamW one-cycle). Because several winning margins are below 1 point (e.g., Table 1, mAP 77.30 vs CAT-Det 75.42; Table 2, L2 mAPH 74.80 vs DeepFusion 74.20), the claimed superiority could be reversed under a common codebase, and no error bars are reported. To support the central claim, the authors should either re-run at least the closest competing methods under their own pipeline or provide variance estimates and explicitly state the comparison protocol.","section":"§4.3, Tables 1 and 2"},{"comment":"The Waymo experimental setup is under-specified: the paper does not state which of the five Waymo cameras is used to decorate the point cloud, nor what happens to points whose projections fall outside the 448x800 image. Since the detection range is [-75.2, 75.2] m on X and Y, a single front camera covers only a subset of the points, and the choice of camera(s) changes the decorated point set and therefore the reported Waymo results. This detail is necessary for reproduction and for interpreting the Waymo L2 mAPH of 74.80.","section":"§4.2 and Table 2"}],"minor_comments":[{"comment":"The checkmark rows do not unambiguously indicate which components are active in each combination; please label each row (e.g., 'Decoration only', '+E2E', '+2SparseConv') so the reader can follow the incremental additions.","section":"Table 3"},{"comment":"The columns 'Vehicle' and 'Pedestrian' should state whether these are AP, APH, or mAPH and at which difficulty; the text says L2 but the table header does not.","section":"Table 4"},{"comment":"Figure 1 contains untranslated Chinese text in the diagram labels (e.g., '点云', '图像'); please replace these with English labels.","section":"Fig. 1"},{"comment":"The related-work statement that the current state-of-the-art fusion method is mid-level feature fusion cites CenterFormer [19], which is evaluated as a lidar-only method in Table 2; please clarify the citation or rephrase the claim.","section":"§2.3"},{"comment":"The KITTI evaluation description says AP with an IoU threshold of 0.7, but KITTI benchmarks commonly use 0.7 for cars and 0.5 for pedestrians and cyclists; please specify the exact evaluation setting used.","section":"§4.1"},{"comment":"The paper does not mention code or model release; adding a reproducibility statement would strengthen the empirical claims.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The main technical gap is the supervision of the heatmap branch; if the authors can clarify or fix this, the paper might become publishable. The baseline-comparison issue is common in this field, but the small margins make it important here, so I would encourage the editor to require at least one controlled re-run or variance estimate. The novelty over DeepFusion is incremental, but the combination of calibration-based decoration, separate sparse convolutions, and heatmap-initialized queries is reasonably clear and the small-object results are encouraging."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nI've read DecoratingFusion. The one thing to know: this is a reasonably put-together engineering paper that combines ideas from PointPainting, DeepFusion, CenterPoint and TransFusion into one detector and reports consistent gains on KITTI and Waymo, especially for pedestrians and cyclists. But the center-heatmap branch that supposedly drives query initialization and class embedding is never given an explicit training signal, and that leaves the central claim less solid than the tables suggest.\n\nWhat's genuinely new is the specific combination: calibration-based point decoration with image features, two separate sparse convolution streams, and heatmap-initialized queries with one-hot class embeddings. Previous papers had each piece; I don't know of one that combined them exactly this way. The writing is clear, the architecture is plausible, and the ablations are structured to isolate each component. The small-object improvements are directionally consistent across both datasets.\n\nThe soft spots are real but mostly fixable. The most important is the heatmap supervision gap. Section 3.3 describes a predicted heatmap used to select top-n local maxima and to produce class embeddings. Section 3.4 defines only a single classification loss and a regression loss; no heatmap loss is mentioned, and the top-n selection is non-differentiable. If L_cls is the final detection head classification, the heatmap branch receives no gradient and the claimed improvements in Table 4 are uninterpretable. If L_cls is actually the heatmap focal loss, the paper needs to say so and also specify the final-head supervision. This is an ambiguity, not necessarily a fatal flaw—probably an oversight in the write-up—but it must be resolved before the numbers can be trusted.\n\nThe other concerns are more standard: baselines are quoted from their original papers rather than re-run under a shared training recipe, margins are sometimes below a point, and there's no code or error bars. I also can't tell from Section 4.2 which Waymo cameras are used for decoration or what happens to points that project outside the single 448x800 image. These are addressable but they do affect the read on the ranking.\n\nWho gets value from this: anyone working on LiDAR-camera fusion for 3D detection will find the combination instructive, and the paper deserves a serious referee rather than a desk reject. The right outcome would be revision: clarify the loss, unify or at least disclose the baseline reproduction, release code, and specify the Waymo camera handling. If that happens, the claim of a modest but useful fusion improvement would be credible.\n\nMy recommendation: send it to review, with the heatmap supervision issue as the first question to the authors.\n\nBest,\n[Your name]","headline":"Competent fusion-of-known-ideas paper whose central heatmap query initialization lacks a specified training signal; promising but needs clarification before the numbers can be trusted.","tokens_in":9201,"tokens_out":3417,"would_cite":false,"duration_ms":34496,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Calibration-guided image features lift 3D detection on KITTI and Waymo.","keywords":["LiDAR-camera fusion","3D object detection","point-level fusion","feature-level fusion","cross-attention","center heatmap query initialization","autonomous driving","KITTI and Waymo benchmarks"],"falsifier":"Re-run the listed baselines under DecoratingFusion's exact settings (same SECOND backbone, DLA34 at 448x800, GT-Paste with five-epoch fading, AdamW one-cycle schedule) on KITTI and Waymo; if the re-trained baselines close the margins to less than one AP point, the claimed superiority is not established. Separately, test which of Waymo's five cameras supplies the decorating image features and what happens to points that project outside the 448x800 image — if that choice changes the decorated point set, the ranking may shift.","tokens_in":8142,"feed_emoji":"🚗","tokens_out":7892,"duration_ms":68006,"temperature":0.7,"pith_summary":"The paper tries to establish that LiDAR-camera fusion for 3D object detection is best done in two stages: first decorate every LiDAR point with the image feature of the pixel it projects onto, then fuse the decorated point cloud with image features through cross-attention. The decoration step is a point-level (early) fusion held together by the calibration matrix, giving interpretable hard correspondence; the cross-attention step is feature-level (mid-level) fusion, letting the network learn soft associations on top of that hard alignment. The authors report that this combination, called DecoratingFusion, reaches 77.30 mAP on KITTI validation and 74.80 L2 mAPH on Waymo validation, beating the listed baselines on both benchmarks, with the clearest margins on pedestrians and cyclists. If true, it would make hard geometric priors a practical ingredient in modern feature-fusion detectors rather than a relic of early-fusion methods.","feed_headline":"Calibration-guided image features lift 3D detection on KITTI and Waymo","feed_subtitle":"LiDAR points carry corresponding image pixels; cross-attention fuses both — biggest gains on pedestrians and cyclists.","key_machinery":"The central object is the decorated point cloud, written as $(x, y, z, r, f)$, where $(x, y, z, r)$ is a LiDAR point and $f$ is the image feature at the pixel obtained by projecting it with the calibration matrix $T$. The mechanism that carries the argument is the query-generation pipeline built on it: voxelization, two independent sparse convolutions that keep LiDAR and image features separate until BEV concatenation, a center-heatmap selection of query positions, a class one-hot embedding appended to each query, and cross-attention in which LiDAR-derived queries attend to image-derived keys and values. Together these turn hard geometric alignment into a soft, learnable fusion at the feature level.","core_discovery":"DecoratingFusion's central claim is that point-level fusion and feature-level fusion are not alternatives: hard calibration-based decoration and learned cross-attention can be stacked in one end-to-end network. The network decorates each LiDAR point with the 2D CNN feature of its calibrated image pixel, passes the decorated cloud through two independent sparse-convolution streams (one for LiDAR features, one for image features), and then uses the concatenated BEV features to generate object queries. Those queries start from a center heatmap rather than random positions, carry class one-hot information, and are fused with image keys and values by cross-attention. With SECOND as the 3D backbone and a DLA34 image backbone, this configuration is reported to reach 77.30 mAP on KITTI validation and 74.80 L2 mAPH on Waymo validation, placing first among the compared methods and strongest on pedestrians and cyclists.","pith_inferences":["Because decoration gives every point a pixel feature only where the image covers the scene, applying the same recipe across all of Waymo's five cameras or a full surround view would change which points are decorated; the paper does not specify its multi-camera handling, so the reported Waymo ranking could shift under that choice.","The method's success on pedestrians and cyclists suggests that image texture supplies what sparse LiDAR points lack; a direct test would be to degrade or corrupt the image branch (for example, simulate camera failure or fog) and measure whether small-object gains disappear first.","The ablations attribute most of the gain to decoration itself, not to the heatmap query or class embedding, so a simpler design that only decorates points may already capture most of the benefit; the paper's mid-level additions buy about 0.3 AP on Waymo."],"forward_implications":["Hard geometric correspondence from calibration matrices can be carried into a feature-level fusion network, not just early-fusion point painting, without sacrificing the benefits of learned cross-attention.","End-to-end training of the 2D image backbone removes the need for separately pre-trained segmentation or detection models, cutting annotation cost and cross-domain mismatch.","Decorative image features and LiDAR features should be extracted by separate sparse convolutions and only concatenated in BEV space, because the domain gap makes joint feature extraction ineffective.","Center-heatmap query initialization plus class embedding brings query starting positions closer to true object centers and helps cross-attention focus on intra-class variation.","On KITTI and Waymo, the largest gains are on small objects (pedestrians and cyclists), where LiDAR-only methods are weakest."],"supporting_citations":[{"why":"Supplies the mid-level fusion baseline whose query-from-BEV design and cross-attention DecoratingFusion extends, and whose reported numbers anchor the Waymo comparison.","marker":"[5]"},{"why":"Introduces the point-painting idea of decorating point clouds with camera outputs, which DecoratingFusion adapts from segmentation scores to learned image features.","marker":"[11]"},{"why":"Provides the decoration-with-image-features baseline that motivates end-to-end training and the GT-Paste fading strategy.","marker":"[12]"},{"why":"Supplies the SECOND 3D sparse-convolution backbone, voxel settings, and loss weight used by DecoratingFusion.","marker":"[13]"},{"why":"Source of the center heatmap initialization for object queries.","marker":"[15]"},{"why":"Provides the CenterNet DLA34 2D backbone that produces the image features used to decorate the point cloud.","marker":"[18]"},{"why":"The lidar-only CenterFormer is the top competitor on Waymo vehicle detection that DecoratingFusion trails by 0.69% but beats overall.","marker":"[19]"},{"why":"Source of the class-embedding idea for queries in TransFusion, which DecoratingFusion adapts.","marker":"[1]"}],"fun_headline_variants":["Hard association fusion beats soft-only for 3D detection","Decorating LiDAR points with image pixels boosts 3D detection","Point-level + feature-level fusion: best of both for 3D","Calibration-linked image and LiDAR fusion wins on pedestrians"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Tables 1 and 2 compare like with like: the baseline numbers come from their original papers rather than being re-run under DecoratingFusion's training recipe, so if the baselines were re-trained or the Waymo camera arrangement were specified differently, the reported ranking could change.","fun_headline_variants_meta":{"raw":{"variants":["Hard association fusion beats soft-only for 3D detection","Decorating LiDAR points with image pixels boosts 3D detection","Point-level + feature-level fusion: best of both for 3D","Calibration-linked image and LiDAR fusion wins on pedestrians"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1234,"prompt_tokens":916,"completion_tokens":318,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":246}},"tokens_in":532,"tokens_out":318,"duration_ms":3890,"temperature":1.0,"reasoning_tokens":246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:56:24.975045+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the listed baselines under DecoratingFusion's exact settings (same SECOND backbone, DLA34 at 448x800, GT-Paste with five-epoch fading, AdamW one-cycle schedule) on KITTI and Waymo; if the re-trained baselines close the margins to less than one AP point, the claimed superiority is not established. Separately, test which of Waymo's five cameras supplies the decorating image features and what happens to points that project outside the 448x800 image — if that choice changes the decorated point set, the ranking may shift.","supporting_citations":[{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the mid-level fusion baseline whose query-from-BEV design and cross-attention DecoratingFusion extends, and whose reported numbers anchor the Waymo comparison."},{"cited_title":"In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","cited_arxiv_id":null,"evidence_quote":"Introduces the point-painting idea of decorating point clouds with camera outputs, which DecoratingFusion adapts from segmentation scores to learned image features."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Provides the decoration-with-image-features baseline that motivates end-to-end training and the GT-Paste fading strategy."},{"cited_title":"Sensors 18(10), 3337 (2018)","cited_arxiv_id":null,"evidence_quote":"Supplies the SECOND 3D sparse-convolution backbone, voxel settings, and loss weight used by DecoratingFusion."},{"cited_title":"In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","cited_arxiv_id":null,"evidence_quote":"Source of the center heatmap initialization for object queries."},{"cited_title":"In: European Conference on Computer Vision","cited_arxiv_id":null,"evidence_quote":"The lidar-only CenterFormer is the top competitor on Waymo vehicle detection that DecoratingFusion trails by 0.69% but beats overall."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Source of the class-embedding idea for queries in TransFusion, which DecoratingFusion adapts."}],"review_version":1}