{"id":"ddd56f75-d9a6-463f-81f8-6d729e15f10f","arxiv_id":"2412.12725","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RaCFormer achieves 64.9% mAP and 70.2% NDS on nuScenes test for radar-camera 3D detection via query-based dual-view feature sampling, circular query initialization, radar-aided depth estimation, and a ConvGRU temporal module.","lead":"RaCFormer is a radar-camera fusion system for 3D object detection that samples object features from both top-down and perspective views, reporting state-of-the-art results on the nuScenes and View-of-Delft benchmarks. A generalist reader should care because the results suggest camera-plus-radar sensors can approach LiDAR-level accuracy in autonomous driving perception at lower cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on unmatched evaluation protocol: RaCFormer uses 6+6 future frames, DD3D-pretrained V2-99, and 36 epochs while key baselines use ≤20 epochs and no future frames; a controlled comparison is needed.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: favorable evaluation setup. The paper's own tables show epoch counts of 36 for RaCFormer versus 20 for HyDRa and 12 for RCBEVDet, and the test-set headline uses future frames and DD3D-pretrained V2-99 while several baselines do not. This unmatched protocol directly threatens the SOTA claim because the margin is large relative to typical training-schedule and temporal-input gains. My stress-test pass did not find a more fundamental internal inconsistency: the architecture is coherent, the ablations are plausible, and the Doppler/IDC criticisms are secondary to the evaluation confound. Since the reader already recommends CONDITIONAL acceptance based on this same issue, no verdict change is needed. The concrete test of retraining under matched epochs and frames would settle whether the concern lands by quantifying how much of the gap is due to the proposed method versus the evaluation setup.","tokens_in":15538,"tokens_out":3908,"duration_ms":39304,"concrete_test":"On the nuScenes val set, train RaCFormer and HyDRa under identical conditions: same backbone (ResNet-50), same image size (256x704), same 8 historical frames (no future frames), and the same number of epochs (e.g., 20 or 24). If RaCFormer's mAP/NDS advantage over HyDRa shrinks to less than ~1–2 points, the reported SOTA gap is mostly an artifact of training schedule and future-frame access. For the test-set claim, additionally run RaCFormer without future frames using V2-99 and compare against HyDRa and CRN at matched epochs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of SOTA radar-camera 3D detection is not yet supported by a controlled comparison. In Table 1, RaCFormer with ResNet-50 is trained for 36 epochs, while HyDRa is trained for 20, RCBEVDet for 12, and CRN for 24. Section 4.2 states that 'Unless specifically indicated, training is conducted for a standard 24 epochs for all models,' but the table shows RaCFormer at 36 against shorter-schedule baselines. On the test set (Table 2), the headline 64.9 mAP / 70.2 NDS uses 6 past and 6 future frames, while most radar-camera baselines use no future frames; RaCFormer also uses a V2-99 backbone pretrained on DD3D with extra data, which is not matched across all rows. The claimed gains of 4.7 mAP / 2.8 NDS over HyDRa on val and 4.0 / 2.8 over HVDetFusion on test could therefore reflect longer training, oracle future frames, and stronger pretraining rather than the proposed modules. The ablations in Tables 4–7 are internally informative, but they do not isolate this confound. A matched-protocol comparison is required before the SOTA claim is accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents RaCFormer, a query-based radar-camera fusion framework for 3D object detection. Three main components are introduced: (i) a linearly increasing circular query initialization that places object queries on concentric circles with increasing density toward the outer range, (ii) a radar-aware depth head that projects radar points (with z set to 1) onto the image plane, extends their depth to the full image height, and fuses radar depth and RCS values with image features before the LSS view transformation, and (iii) an implicit dynamic catcher built on ConvGRU that uses multi-frame radar BEV features to exploit Doppler velocity information. The method is evaluated on nuScenes and View-of-Delft (VoD). The authors report state-of-the-art results on the nuScenes test set (64.9% mAP and 70.2% NDS) and on VoD (78.57% mAP in the region of interest), along with extensive ablations and robustness analyses. The central claim is that the query-based dual-view sampling paradigm, together with radar-guided depth and temporal radar encoding, substantially improves radar-camera 3D detection.","tokens_in":15842,"tokens_out":9495,"duration_ms":76849,"significance":"If the reported improvements hold under controlled comparisons, this paper offers a valuable contribution to radar-camera 3D detection. The motivation is clear—addressing the misalignment between sparse radar and depth-estimated camera BEV features—and the ablations in Tables 4–7 provide evidence that each proposed module contributes positively. The availability of code is a strength for reproducibility, and the robustness evaluations under weather, lighting, and sensor dropout conditions are commendable. However, the state-of-the-art claim is currently not supported by an evaluation protocol that matches training epochs, input frames, backbone pretraining, and schedule across methods. The significance of the paper cannot be fully assessed until these confounds are removed.","major_comments":[{"comment":"The comparison with HyDRa on the nuScenes validation set is not epoch-matched: RaCFormer with ResNet-50 is trained for 36 epochs, while HyDRa is reported at 20 epochs and RCBEVDet at 12 epochs, despite Section 4.2 stating a 'standard 24 epochs' unless otherwise indicated. Since longer training is expected to improve mAP and NDS, the 4.7 mAP gain over HyDRa cannot be attributed to the proposed modules without a controlled experiment. Please provide RaCFormer results at 20 epochs and HyDRa results at 36 epochs, or otherwise demonstrate that the gap persists under matched training schedules.","section":"Section 4.3, Table 1"},{"comment":"The headline test-set results (64.9 mAP and 70.2 NDS) use 6 historical and 6 future frames, while most radar-camera baselines in the same table do not use future frames. The comparison with HVDetFusion, which does use future frames, is not matched in backbone (V2-99 with DD3D pretraining vs InternImage-B), training epochs (24 vs 20), or input resolution. A controlled comparison is needed in which only the fusion and architecture components vary while the number of input frames, backbone, pretraining, and epochs are held constant. Without this, the claim of outperforming HVDetFusion 'with more input frames' is not conclusive.","section":"Section 4.3, Table 2"},{"comment":"The statement that RaCFormer 'partially bridges the modality gap' to LiDAR-based methods compares against CenterPoint and VoxelNeXt, which are evaluated without temporal context. Since RaCFormer's best performance uses future frames, this is not a like-for-like comparison between sensor modalities. Please either provide a comparison where the LiDAR baselines also receive equivalent multi-frame input (including future frames) or explicitly qualify the claim as applying only to settings with temporal look-ahead.","section":"Section 4.3"},{"comment":"The paper's statement 'Unless specifically indicated, training is conducted for a standard 24 epochs for all models' is inconsistent with Table 1, where RaCFormer-ResNet50 and RayFormer-ResNet50 are trained for 36 epochs. Please clarify the actual training protocols used for each row in Tables 1 and 2, and ensure that models compared under the same heading are trained under the same schedule, or clearly mark which rows use non-standard schedules.","section":"Section 4.2"}],"minor_comments":[{"comment":"The closed-form expression for the geometric sum when α ≠ 1 appears ambiguous as typeset; it should read (α^k − 1)/(α − 1) × n, which is consistent with the reported total of about 900 queries for α=1.25, k=6, n=80.","section":"Equation (4)"},{"comment":"The text uses 'V oVNet-99 (V2-99)'; the correct backbone name is VoVNet-99. Please correct the typo.","section":"Section 4.2"},{"comment":"Please specify the training schedule and hyperparameters used on the VoD dataset, and confirm that the baseline numbers in Table 3 are obtained under the same evaluation protocol (including IoU thresholds and region definitions).","section":"Table 3"},{"comment":"The layout of the robustness table is confusing: the 'C' and 'R' entries under 'CRN' and 'RaCFormer' are not clearly labeled as indicating which sensor modality is dropped (or which views are dropped). Please clarify the row and column semantics.","section":"Table 9"},{"comment":"The paper does not report variance over multiple runs or seeds. Adding a statement about whether reported numbers are single-run or averaged would help calibrate the significance of the observed performance differences.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The core technical idea is sound and the ablations are informative, but the evaluation protocol needs substantial revision before the SOTA claim can be considered supported. The authors should be asked to provide controlled comparisons with matched epochs, input frames, backbone, and pretraining. The heavy self-citation of RayFormer is transparent given that RaCFormer is an extension, and the paper is within the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RaCFormer is a well-assembled query-based radar-camera detector with three concrete, ablatable ideas, but the headline SOTA numbers are not yet supported by a controlled comparison.\n\nThe paper extends the authors' RayFormer into a radar-camera fusion framework. The contributions are incremental: a circular query initialization with linearly increasing density, a radar-aware depth head that projects radar points with a fixed height and extends them to the full image height, and a ConvGRU-based implicit dynamic catcher on radar BEV features. Each is ablated cleanly in Tables 4-7, and the ablations show consistent gains. That kind of incremental, well-ablated engineering is useful. The View-of-Delft results are also strong.\n\nThe main issue is the evaluation protocol. On the test set, the 64.9 mAP / 70.2 NDS number uses 6 past and 6 future frames, which gives the model oracle access to future motion. Most radar-camera baselines in the same table do not use future frames. Future frames are known to be a strong advantage on nuScenes. On the validation set, RaCFormer with ResNet-50 is trained for 36 epochs while HyDRa is trained for 20, RCBEVDet for 12, and CRN for 24, even though Section 4.2 says 'Unless specifically indicated, training is conducted for a standard 24 epochs for all models.' The validation gains of 4.7 mAP over HyDRa could be in large part training schedule. The authors need to retrain baselines under a matched protocol, or at least clearly separate matched and unmatched comparisons. The Doppler-effect language overstates the mechanism: the ConvGRU does not use radial velocity directly, but that is a framing issue, not a load-bearing flaw.\n\nThis paper will get value from readers working on radar-camera fusion and query-based detection. It deserves a serious referee: the architecture is coherent, the ablations are honest, and the code is released. I would send it to review, with the instruction that the referee verify whether the SOTA claim survives a matched-protocol comparison. If it doesn't, the SOTA claim should be pulled back.","headline":"Solid query-based radar-camera fusion with strong ablations, but the SOTA claims rest on future frames and unequal training schedules that need a controlled comparison.","tokens_in":16378,"tokens_out":4145,"would_cite":true,"duration_ms":35858,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RaCFormer claims query-based radar-camera fusion is the key to high-quality 3D detection, reporting 64.9% mAP and 70.2% NDS on nuScenes and 78.6% mAP on View-of-Delft.","keywords":["3D object detection","radar-camera fusion","query-based transformer","bird's-eye view","depth estimation","temporal modeling","autonomous driving","sensor fusion"],"falsifier":"A controlled comparison that trains RaCFormer and the published radar-camera baselines under identical settings—same number of future and historical frames, same backbone pretraining, same epochs and data augmentations—on the nuScenes validation split. If the 4.7 mAP margin over HyDRa at ResNet-50 resolution shrinks or reverses, the architecture is not the source of the claimed advantage.","tokens_in":15359,"feed_emoji":"🚗","tokens_out":8328,"duration_ms":65796,"temperature":0.7,"pith_summary":"RaCFormer sets out to show that radar and camera data are best fused not by combining bird's-eye-view (BEV) feature maps, but by letting a set of object queries sample features from both the perspective image view and the BEV. The paper argues that BEV-only fusion inherits two weaknesses: image-to-BEV transformation distorts features when pixel depth is inaccurate, while radar BEV features are too sparse to anchor alignment. Its query-based framework, supported by a radar-guided depth head and a Doppler-aware temporal module, is reported to reach 64.9% mAP and 70.2% NDS on the nuScenes test set and 78.57% mAP in the region of interest on View-of-Delft, outperforming prior radar-camera detectors and partially closing the gap to LiDAR-based systems. A sympathetic reader would care because radar-camera perception is a far cheaper alternative to LiDAR for autonomous driving, and the reported gains appear in exactly the conditions where cameras fail—night, rain, and fast-moving objects.","feed_headline":"Radar-camera 3D detection hits 64.9% mAP on nuScenes","feed_subtitle":"Sampling from both image and BEV views with radar-guided depth and Doppler-aware timing beats prior camera-radar detectors.","key_machinery":"The carrying mechanism is query-based dual-view sampling: 3D object queries serve as the fusion medium, with a ray-sampling module taking sample points along each query's ray segment and gathering deformable-attention features from both the image plane and the BEV, then aggregating them with an adaptive mixer. Three components strengthen it. First, linearly increasing circular query initialization places queries on concentric circles with $n$ queries in the innermost circle and $\\alpha$ times more per outer circle, so the total is $N = (1+\\alpha+\\dots+\\alpha^{k-1})n$, giving denser coverage at distance than the radial layout it replaces. Second, the radar-aware depth head projects radar points with $z_r=1$ onto the full image height, discretizes their depths with increasing spacing, embeds depth and radar cross section, and feeds this into the depth head to refine the depth distribution used by the lift-splat-shoot view transformation. Third, the implicit dynamic catcher feeds multi-frame radar BEV features through a ConvGRU whose hidden state accumulates motion evidence across frames, using the Doppler information implicit in radar velocity measurements to sharpen detection of moving objects.","core_discovery":"The paper's central claim is that cross-modal fusion for 3D detection should be query-based and cross-perspective: each object query defines sampling locations that are projected into the original images and onto the BEV, so the detector can pull instance-relevant evidence from both views instead of requiring the two BEV maps to be spatially aligned first. On top of this, RaCFormer adds a circular query initialization whose density grows linearly with distance from the ego vehicle, a radar-aware depth head that projects radar returns to the full image height and embeds their range and radar cross section to sharpen the lift-splat-shoot transformation, and an implicit dynamic catcher that runs multi-frame radar BEV features through a convolutional gated recurrent unit to exploit the Doppler-derived motion signal. With these components the method reports state-of-the-art numbers on nuScenes and View-of-Delft, including a test-set result of 64.9% mAP and 70.2% NDS with six past and six future frames, which the paper reads as evidence that radar-camera fusion can approach—and on some metrics match—LiDAR-based detectors.","pith_inferences":["The circular query prior encodes a physical assumption—object density in driving scenes grows roughly linearly with distance from the ego sensor within range. A natural extension is to estimate $\\alpha$ and $k$ per dataset from annotated object distributions instead of hand-setting them, which could transfer the method to intersection or highway geometries.","The same query-based dual-view sampling idea should transfer to LiDAR-camera fusion, where BEV features are denser but still distorted near occlusion boundaries; if the mechanism is truly density-agnostic, it would reduce the alignment burden there too.","Because the implicit dynamic catcher already processes a temporal sequence of BEV features, an explicit radar-Doppler velocity supervision head could be added to predict radial velocity directly from hidden states, potentially cutting the average velocity error further.","The graceful-degradation table suggests training with random per-frame sensor dropout could make the system even more robust; this is a testable variant of the paper's claim that query-based fusion tolerates missing modalities."],"forward_implications":["On the nuScenes validation set, RaCFormer with a ResNet-50 at 256x704 reaches 54.1% mAP and 61.3% NDS, beating the HyDRa baseline by 4.7 mAP and 2.8 NDS, and with ResNet-101 at 512x1408 reaches 57.3% mAP and 63.0% NDS.","On the nuScenes test set, using six past and six future frames, RaCFormer reports 64.9% mAP and 70.2% NDS, 4.0 mAP and 2.8 NDS above HVDetFusion even though HVDetFusion uses more frames.","On View-of-Delft, RaCFormer reports 54.44% mAP over the whole annotated area and 78.57% mAP in the region of interest, exceeding RCBEVDet by 4.45 and 8.77 points respectively.","Under sensor dropout, RaCFormer keeps a car AP of 27.2% with cameras entirely off and 52.8% with radar off, well above the CRN baseline, indicating the fusion remains usable when one sensor fails.","Ablations attribute the gains to the proposed modules: query-based dual-view decoding adds 3.1 mAP over BEV-only queries, radar depth and RCS embeddings add 1.1 mAP, and the implicit dynamic catcher improves moving-object mAP while lowering velocity error."],"supporting_citations":[{"why":"Supplies the ray-sampling mechanism and the radial query baseline that circular initialization improves on.","marker":"[6]"},{"why":"Defines the standard BEV concatenation/cross-attention fusion paradigm that the paper argues against.","marker":"[34]"},{"why":"Provides the lift-splat-shoot view transformation whose depth distribution the radar-aware depth head refines.","marker":"[43]"},{"why":"Gives the pillar encoding used to turn raw radar points into BEV features.","marker":"[19]"},{"why":"Supplies the adaptive mixing operation used to aggregate sampled image and BEV features.","marker":"[9]"},{"why":"Serves as the primary radar-camera baseline on nuScenes validation and test sets.","marker":"[48]"},{"why":"Provides a radar-camera fusion baseline for comparison and for the sensor-dropout robustness study.","marker":"[18]"},{"why":"Is the radar-camera BEV fusion baseline compared against on nuScenes and View-of-Delft.","marker":"[29]"},{"why":"Supplies the extra-data pretraining for the V2-99 backbone used in the top test-set result.","marker":"[42]"}],"fun_headline_variants":["Query-based radar-camera fusion sets new 3D detection record","Radar-camera fusion with circular queries: 64.9% mAP on nuScenes","Doppler-aware BEV fusion pushes radar-camera 3D detection to SOTA","Cross-view query sampling beats aligned BEV for radar-camera 3D detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported state-of-the-art numbers assume the gains come from the proposed modules rather than from a favorable evaluation setup: the top test-set result uses six past and six future frames plus a V2-99 backbone pretrained on extra data, while several baselines run with fewer frames, shorter schedules, or weaker backbones.","fun_headline_variants_meta":{"raw":{"variants":["Query-based radar-camera fusion sets new 3D detection record","Radar-camera fusion with circular queries: 64.9% mAP on nuScenes","Doppler-aware BEV fusion pushes radar-camera 3D detection to SOTA","Cross-view query sampling beats aligned BEV for radar-camera 3D detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000789,"raw_usage":{"total_tokens":3538,"prompt_tokens":1062,"completion_tokens":2476,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":2369}},"tokens_in":678,"tokens_out":2476,"duration_ms":15802,"temperature":1.0,"reasoning_tokens":2369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:46:56.031634+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled comparison that trains RaCFormer and the published radar-camera baselines under identical settings—same number of future and historical frames, same backbone pretraining, same epochs and data augmentations—on the nuScenes validation split. If the 4.7 mAP margin over HyDRa at ResNet-50 resolution shrinks or reverses, the architecture is not the source of the claimed advantage.","supporting_citations":[{"cited_title":"Rayformer: Improving query- based multi-camera 3d object detection via ray-centric strate- gies","cited_arxiv_id":null,"evidence_quote":"Supplies the ray-sampling mechanism and the radial query baseline that circular initialization improves on."},{"cited_title":"Bevfusion: Multi- task multi-sensor fusion with unified bird’s-eye view repre- sentation","cited_arxiv_id":null,"evidence_quote":"Defines the standard BEV concatenation/cross-attention fusion paradigm that the paper argues against."},{"cited_title":"Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d","cited_arxiv_id":null,"evidence_quote":"Provides the lift-splat-shoot view transformation whose depth distribution the radar-aware depth head refines."},{"cited_title":"Lang, Sourabh V ora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom","cited_arxiv_id":null,"evidence_quote":"Gives the pillar encoding used to turn raw radar points into BEV features."},{"cited_title":"Adamixer: A fast-converging query-based object detector","cited_arxiv_id":null,"evidence_quote":"Supplies the adaptive mixing operation used to aggregate sampled image and BEV features."},{"cited_title":"CRN: camera radar net for accurate, robust, efficient 3d perception","cited_arxiv_id":null,"evidence_quote":"Provides a radar-camera fusion baseline for comparison and for the sensor-dropout robustness study."},{"cited_title":"Rcbevdet: Radar-camera fusion in bird’s eye view for 3d object detection","cited_arxiv_id":null,"evidence_quote":"Is the radar-camera BEV fusion baseline compared against on nuScenes and View-of-Delft."},{"cited_title":"Is pseudo-lidar needed for monocular 3d object detection? In IEEE/CVF International Conference on Computer Vision (ICCV), 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the extra-data pretraining for the V2-99 backbone used in the top test-set result."}],"review_version":1}