{"id":"5203d192-6395-436e-9d5c-700d18027531","arxiv_id":"2412.18884","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HV-BEV decouples bird's-eye-view feature sampling into adaptive height-aware vertical sampling and graph-based horizontal aggregation, improving 3D detection on nuScenes.","lead":"HV-BEV is a camera-only 3D object detection method that samples image features at object-relevant heights and aggregates features from neighboring horizontal positions. It beats the BEVFormer baseline on the nuScenes dataset, reaching 50.5% mAP and 59.8% NDS on the test set.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed gains may reflect more sampled image features per BEV query, not the decoupled horizontal/vertical design; Table IX does not equalize sampling count or compute.","rationale":"The reader's weakest assumption targets the VHA module's reliance on flattened BEV features containing height information. That is a valid concern, but VHA alone contributes only 0.5 mAP in the small-config ablation (Table VI), so it is not the main support for the central claim. The dominant component is DHCA, whose gain could plausibly come from the large increase in sampled image features per BEV query rather than from the proposed horizontal-plane graph structure. Table IX attempts to control for added reference points, but it changes only Nref in the vanilla uniform-sampling scheme and does not match the number of deformable sampling points or neighbor samples. This makes the attribution of the overall improvement to the decoupled design the most load-bearing open issue. A compute-matched baseline that equals HV-BEV's sampling count would settle whether the specific geometry matters; until then, the conditional verdict is appropriate. The paper otherwise provides consistent ablations, competitive public-benchmark numbers, and code availability, which I weigh as supporting evidence for the method's practical value even while the mechanism attribution remains unproven.","tokens_in":19579,"tokens_out":9212,"duration_ms":96660,"concrete_test":"On the nuScenes val set with the small config, train a BEVFormer baseline with Nref=4 but 2 deformable sampling points per view per reference point (matching HV-BEV's center-point sample count), and a second variant with Nref=4 plus four learnable image-plane offsets per reference point (matching DHCA's ~20 sampled locations per query). If either variant reaches roughly 39.9 mAP / 50.2 NDS, the HV-BEV gain is explained by sample count or compute rather than by the height-aware or horizontal-plane decoupling mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim attributes the mAP/NDS improvements to the decoupled sampling design (VHA plus DHCA). Table VI shows the gain is dominated by DHCA (+2.2 mAP in the small config), so the load-bearing question is whether DHCA's improvement comes from the proposed horizontal-plane graph or simply from sampling more image features. In vanilla BEVFormer each query uses Nref=4 reference points with one deformable-attention sample per hit view. In HV-BEV, each of the 4 reference points first computes x_p via deformable attention with 2 sampling points per view (Sec. IV-B) and then aggregates 4 neighboring points via bilinear sampling in every hit view (Eq. 14). This multiplies the number of feature locations per query by roughly 4-5x. The control in Table IX only increases Nref for uniform height sampling in the vanilla model; it does not increase the number of deformable sampling points per reference point, nor does it add learnable offsets in the image plane or BEV plane. Therefore it cannot rule out that the gain is a sampling-capacity/compute effect. If a compute-matched BEVFormer variant with the same number of sampled locations reaches HV-BEV's 39.9 mAP / 50.2 NDS, the paper's attribution to decoupled horizontal and vertical sampling is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HV-BEV, a multi-view 3D object detector built on BEVFormer that decouples feature sampling into a horizontal and a vertical component. The vertical adaptive height-aware (VHA) module predicts a discrete height distribution from current and historical BEV features, fuses the two predictions with a dual-branch cross-attention, and uses a top-k selection to choose reference-point heights, supervised by Gaussian-smoothed ground-truth center heights. The dynamic horizontal cross-view aggregation (DHCA) module learns 2D offsets in the BEV plane for each 3D reference point, projects the resulting neighboring points into the camera views, and aggregates their bilinearly sampled features with learned edge weights. Experiments on nuScenes validation and test sets and on Lyft report consistent improvements over the BEVFormer baseline, with the base configuration improving mAP from 41.6% to 43.9% and NDS from 51.7% to 53.3% on nuScenes validation, and the largest configuration reaching 50.5% mAP and 59.8% NDS on the nuScenes test set. The paper also reports a transfer of the modules to 3D occupancy prediction.","tokens_in":19840,"tokens_out":4953,"duration_ms":49991,"significance":"If the reported gains are attributable to the proposed decoupled horizontal and vertical sampling, the work is a useful contribution to dense BEV perception: it preserves the multi-task extendability of BEV grid queries while adding object-aware height sampling and cross-view feature aggregation. The paper ships code, evaluates on two datasets, and provides ablations for both modules, which is a strength. The main significance caveat is that the ablation evidence does not yet establish that the gains come from the decoupled design rather than from a larger number of sampled image features or from increased model capacity. The VHA module in isolation adds only 0.5 mAP and 0.3 NDS, and the DHCA control in Table IX does not equalize sampling counts or compute. Consequently, the central attribution claim is plausible but not fully supported.","major_comments":[{"comment":"The control experiment in Table IX varies only the number of uniformly sampled reference points Nref in the vanilla BEVFormer (4, 8, 16). It does not equalize the number of sampled feature locations per query with HV-BEV, nor does it equalize compute. In HV-BEV, each of the Nref reference points first computes x_p with deformable attention and two sampling points per view (Sec. IV-B), and then samples M=4 neighboring points with bilinear interpolation in every hit view (Eq. 14). This multiplies the number of image-feature locations per query by roughly a factor of 1+M/n_deform relative to a baseline that uses n_deform deformable points per reference point. The results in Table IX therefore cannot rule out the explanation that the DHCA gain comes from sampling more image features or from the extra learnable offsets, rather than from the horizontal graph structure itself. I ask for an additional baseline that matches the total number of sampled locations and approximately the same FLOPs/FPS, for example a BEVFormer variant with more deformable sampling points per reference point or with an equivalent number of additional bilinear samples arranged in a non-graph (e.g., random or uniform) configuration. Without such a control, the central claim that the decoupled horizontal design is responsible for the improvement is not established.","section":"Table IX, Sec. IV-E"},{"comment":"The VHA module alone yields only +0.5 mAP and +0.3 NDS over the small-configuration baseline, and the paper reports a single run for each configuration. Given typical run-to-run variance in end-to-end trained detection models with this training setup, this difference is not demonstrably significant. The paper should provide at least three seeds with mean and standard deviation for the main ablation rows, or otherwise justify that the VHA-only gain is outside noise. This matters because the paper's central narrative attributes part of the improvement to adaptive height sampling.","section":"Table VI, Sec. IV-E"},{"comment":"The height-aware module selects the height coordinates of the Nref reference points by applying a top-k operation to the fused height distribution H_fus (Sec. III-B: 'we apply a top-k function to sample Nref discrete height values'). Top-k selection is not differentiable, and the manuscript does not specify a straight-through estimator, Gumbel-top-k, or a continuous relaxation. As written, gradients from the detection loss cannot flow through the selected height indices into H_fus; only the auxiliary height loss L_hgt (Eq. 8) can supervise H_fus. This creates an inconsistency with the claim that the model is trained end-to-end and leaves unclear how the height-sampling pathway is optimized for the detection objective. Please state the exact sampling/differentiation procedure and, if the top-k is used as a non-differentiable selector, discuss the implications and provide evidence that the height distribution is nevertheless learned effectively.","section":"Sec. III-B, Sec. III-C"},{"comment":"The temporal warp of the historical height distribution uses a 2D rotation matrix and translation vector around the vertical z axis only. This ignores vehicle pitch and roll, which can be non-negligible on slopes and could misalign the historical height distribution with the current grid. The paper does not discuss this assumption or provide evidence that the 2D warp is sufficient. Since the baseline BEVFormer also operates in a flattened BEV plane, this may be an inherited limitation rather than a new error, but it should be stated explicitly and, if possible, quantified on nuScenes (e.g., by comparing against a warp that uses the full 6-DoF ego pose).","section":"Eq. (3)-(4), Sec. III-C"}],"minor_comments":[{"comment":"The conditional definition of H_gt_{p,m} for the two cases I_p=1 and I_p=0 is not typeset clearly in Eq. (6); please separate the cases explicitly so the uniform distribution for background cells is unambiguous.","section":"Sec. III-C, Eq. (6)"},{"comment":"The loss weights lambda_1, lambda_2, and lambda_3 in Eq. (16) are introduced but their numerical values are never reported. Please provide the values used in all experiments.","section":"Sec. IV-B, Eq. (16)"},{"comment":"After the top-k operation selects height bins, the manuscript does not state how the discrete bins are converted to continuous height values for the 3D reference points (e.g., bin centers or interpolation). Please specify this conversion.","section":"Sec. III-C"},{"comment":"The number of deformable attention sampling points used by the baseline BEVFormer spatial cross-attention is not stated, which is needed to compute the sample-count ratio discussed in the major comments. Please report this value for both baseline and HV-BEV.","section":"Sec. IV-B"},{"comment":"The conclusion states that the introduced loss provides 'depth supervision,' but the loss in Eq. (8) supervises height distributions, not depth. This appears to be a typo and should be corrected.","section":"Sec. V"},{"comment":"In the comparison with state-of-the-art, the phrase 'surpassing one of the latest methods' is supported by Table II, but the image sizes differ slightly between HV-BEV small (1280x720) and BEVFormer-small-QAF2D (1280x736). Please note whether this difference affects the comparison.","section":"Sec. IV-C"},{"comment":"There are several typographical and formatting issues, including 'odject' in Sec. IV-C, the rendered 'V oVnet' in Tables and text, and the citation for the Lyft dataset missing a full bibliographic entry. These should be cleaned up.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical architecture study with a useful baseline comparison and code release. The main risk for the journal is the attribution problem: Table IX is the only control against the 'more samples / more compute' explanation, and it does not control for either. If the authors can add a compute- and sample-matched baseline and clarify the top-k differentiability issue, the central claim would be substantially stronger. Given the small VHA-only gain and the lack of error bars, I would not recommend acceptance before these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent BEVFormer variant, better than most of the recent height-sampling tweaks, and the main numbers likely hold up. The genuinely new piece is the two-module decoupling: VHA predicts a height distribution from BEV queries and fuses it with a history-derived distribution before sampling reference heights, and DHCA builds a learnable horizontal graph around each 3D reference point and aggregates neighbor features from all hit views. That combination is new relative to BEVFormer, OC-BEV, HeightFormer, and Graph-DETR3D. The occupancy transfer experiment and the Lyft result are good evidence that DHCA generalizes beyond the primary task.\n\nWhat the paper does well: ablations are honestly laid out, the baseline is standard, and the authors include a control (Tab. IX) meant to argue that the gain is not just from sampling more reference points. The test-set result (50.5 mAP / 59.8 NDS with VoVNet) is competitive with LiDAR-trained methods, and code is promised.\n\nSoft spots, in order of severity:\n- Single runs, no error bars. The VHA-only gain is 0.5 mAP; that could easily be run-to-run noise. They should report variance or at least a couple of seeds for the small config.\n- The Tab. IX control is not a proper compute-matched or sampling-count-matched ablation. HV-BEV's DHCA samples M neighbor locations per reference point in every hit view in addition to the deformable-attention samples, so it extracts substantially more image features per query than the Nref=8 or 16 uniform-sampling baseline. The control doesn't add those extra samples to the baseline, nor does it match learnable offsets. The Nref=16 result (mAP drops to 35.1) partially shows that naive scaling doesn't help, but a skeptic can still claim the DHCA gain comes from more features, not from the horizontal graph, and a baseline that increases deformable sampling points per reference point would be more convincing. The stress-test note's exact arithmetic about sampling counts may be off, but the underlying concern is fair.\n- The top-k height sampling is not discussed in terms of gradient flow. Since the height distribution is supervised directly with cross-entropy, the reference heights don't strictly need to be differentiable, but that should be stated; otherwise readers will wonder how the sampled heights affect training.\n- Minor: the conclusion says 'depth supervision' where the method actually supervises height, and the 2D ego-motion warp ignores pitch and roll; both are worth a sentence.\n\nI don't see a load-bearing flaw. The math is consistent, the data is public benchmarks, and the citations to the height-aware and graph-based literature are in order. This is an incremental but solid contribution for researchers building on BEVFormer.\n\nRecommendation: send it to review. A serious referee will want the variance and the sampling-count control addressed, but the paper deserves referee time. I'd bring it to a reading group if your group works on camera-based perception; otherwise a skim of the ablation section is enough.","headline":"A solid BEVFormer upgrade with a genuinely new height-aware sampling mechanism; the main gain is real but needs a compute-matched control to prove it comes from the decoupled design.","tokens_in":20391,"tokens_out":7803,"would_cite":true,"duration_ms":65383,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that decoupling bird's-eye-view feature sampling into adaptive height-aware vertical sampling and dynamic horizontal cross-view neighbor aggregation improves multi-view 3D object detection, raising base-validation mAP…","keywords":["multi-view 3D object detection","bird's-eye view","adaptive height-aware sampling","cross-view feature aggregation","deformable attention","temporal feature fusion","nuScenes benchmark","BEV grid queries"],"falsifier":"Train the vertical height-aware module while corrupting the predicted height distributions at inference, for example replacing the fused distribution with a uniform one or with random heights while keeping the horizontal module intact; if mAP and NDS do not drop below the intact model, then the adaptive heights themselves carry no object-height information and the module's reported gain must come from something else.","tokens_in":19344,"feed_emoji":"🚗","tokens_out":8664,"duration_ms":73630,"temperature":0.7,"pith_summary":"This paper proposes a dense bird's-eye-view (BEV) multi-camera 3D object detector that separates feature sampling into two directions. On the horizontal plane, each 3D reference point learns a small set of neighboring points across grid cells and camera views, so an object spanning many BEV cells or several images is aggregated as one structure. In the vertical direction, instead of sampling reference points uniformly over a fixed height range, the model predicts a height distribution per grid cell from current and temporally aligned historical BEV features and samples heights near where objects of that scene actually sit. The paper reports that these two changes raise validation mAP from 41.6% to 43.9% and NDS from 51.7% to 53.3% in the base configuration over the grid-query baseline, and 50.5% mAP and 59.8% NDS on the nuScenes test set.","feed_headline":"Sampling by height and view lifts camera 3D detection to 50.5% mAP","feed_subtitle":"Horizontal and vertical sampling beats the BEV baseline on nuScenes and lifts occupancy too","key_machinery":"The load-bearing mechanism is the decoupled sampling scheme applied inside each BEV encoder layer. A vertical adaptive height-aware sampling module takes the current BEV queries and the aligned historical BEV features, feeds both through a shared MLP into $D=8$ discrete height bins, combines the two distributions with a dual-branch cross-attention fusion, and selects the top $N_{ref}=4$ heights as the z-coordinates of each query's 3D reference points. A dynamic horizontal cross-view feature aggregation module then learns 2D offsets on the ground-aligned plane for each reference point, forming a four-neighbor graph whose nodes are projected into the images and aggregated by learned softmax edge weights. The two modules replace uniform global-height sampling and isolated single-point cross-attention, so the model samples where height information is concentrated and shares features across the spatial extent of an object.","core_discovery":"On its own terms, the paper's central claim is that the two quantities BEV-grid query sampling usually collapses—where along the height axis an object sits, and which neighboring grid cells belong to the same object—can be recovered explicitly and separately, and that doing so improves multi-view 3D detection. The vertical module fuses a predicted current height distribution with a historical distribution warped by ego-motion, supervises the fused distribution with annotated center heights of 3D boxes, and uses top-k sampling to place reference points at object-relevant heights. The horizontal module then treats each reference point as the center of a dynamically learned neighborhood on its height plane, projects all neighboring points into the camera views, and combines their bilinearly sampled features with learned edge weights. The paper reports that both modules together beat the baseline consistently across three configurations on nuScenes, that the horizontal module contributes the larger share of the gain in ablations, and that the same BEV features improve 3D semantic occupancy prediction when the task head is replaced.","pith_inferences":["Inference: the ablation pattern—VHA alone gains 0.5 mAP while DHCA alone gains 2.2 mAP—suggests the headline improvement is mostly driven by horizontal neighborhood aggregation, and the vertical module's contribution may be to make those neighbors more informative rather than to add large independent accuracy; an experiment running DHCA with fixed uniform heights would separate these two readings.","Inference: the paper supervises heights only from annotated 3D box centers; if the same module were trained with dense or voxel-level height labels, the predicted height distribution might also serve as a lightweight 3D geometric prior for downstream planning, not just a sampling guide.","Inference: the historical-height fusion assumes the ego vehicle only rotates around the vertical axis and translates horizontally; on roads with significant pitch or roll, the 2D warping could misalign the historical height distribution, so a 6-DoF alignment or learning-based correction is a natural extension the paper does not test."],"forward_implications":["If the gains reproduce, dense BEV-grid perception can keep a single shared representation for detection and other tasks while still exploiting object height, because the added modules live inside the feature encoder rather than in a task-specific sparse query head.","The base configuration's 2.3-point mAP and 1.6-point NDS improvements imply that structured horizontal aggregation, not just more reference points, is what recovers objects spanning multiple BEV cells or views, since increasing uniform samples from 4 to 8 gives only 0.4 mAP in the paper's control experiment.","The occupancy experiments imply the same height-aware BEV features are not detection-specific: replacing the detection head with a voxel segmentation head raised scene-completion IoU by 1.28 points and semantic mIoU by 3.22 points over the baseline.","The Lyft results imply the approach is not overfit to nuScenes camera geometry, at least under the evaluation protocol used there, beating the camera and LiDAR baselines listed in the paper on that validation split."],"supporting_citations":[{"why":"Supplies the dense BEV grid-query baseline whose temporal self-attention, spatial cross-attention, and uniform height sampling are replaced by the proposed modules.","marker":"[15]"},{"why":"Provides the deformable attention primitive used in temporal, spatial, height-fusion, and neighboring-point aggregation.","marker":"[19]"},{"why":"Introduces height-specific BEV slice features and the height-range prior that inspires the dual-branch cross-attention fusion of height distributions.","marker":"[23]"},{"why":"Shows how standard spatial cross-attention implicitly encodes height, motivating explicit adaptive height sampling as an alternative.","marker":"[37]"},{"why":"Offers the object-focused reference-point sampling comparison on the same benchmark that the vertical module seeks to improve upon.","marker":"[40]"},{"why":"Defines the nuScenes benchmark and metrics (mAP, NDS) on which all main comparisons and ablations are run.","marker":"[44]"},{"why":"Defines the Lyft dataset and evaluation protocol used for the generalization experiment.","marker":"[54]"},{"why":"Provides the occupancy predictor into which the horizontal module is inserted for the transfer experiment.","marker":"[57]"}],"fun_headline_variants":["HV-BEV: Decoupled sampling for camera 3D detection hits 50.5% mAP","Height-aware sampling boosts multi-view 3D detection to 50.5% mAP","Decoupling horizontal and vertical features lifts nuScenes detection to 50.5%","Camera-only 3D detection gets height-adaptive sampling edge: 50.5% mAP","HV-BEV: Separate horizontal and vertical sampling to lift 3D object detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole vertical module rests on the assumption that the flattened 2D bird's-eye-view features already contain enough hidden information about object heights for the shared MLP to predict where to sample; if that height signal is not actually present in the BEV features, the adaptive sampling just moves reference points around without adding information.","fun_headline_variants_meta":{"raw":{"variants":["HV-BEV: Decoupled sampling for camera 3D detection hits 50.5% mAP","Height-aware sampling boosts multi-view 3D detection to 50.5% mAP","Decoupling horizontal and vertical features lifts nuScenes detection to 50.5%","Camera-only 3D detection gets height-adaptive sampling edge: 50.5% mAP","HV-BEV: Separate horizontal and vertical sampling to lift 3D object detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001091,"raw_usage":{"total_tokens":4623,"prompt_tokens":1077,"completion_tokens":3546,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":3428}},"tokens_in":693,"tokens_out":3546,"duration_ms":22350,"temperature":1.0,"reasoning_tokens":3428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:22:06.478491+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the vertical height-aware module while corrupting the predicted height distributions at inference, for example replacing the fused distribution with a uniform one or with random heights while keeping the horizontal module intact; if mAP and NDS do not drop below the intact model, then the adaptive heights themselves carry no object-height information and the module's reported gain must come from something else.","supporting_citations":[{"cited_title":"Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers,","cited_arxiv_id":null,"evidence_quote":"Supplies the dense BEV grid-query baseline whose temporal self-attention, spatial cross-attention, and uniform height sampling are replaced by the proposed modules."},{"cited_title":"Deformable detr: De- formable transformers for end-to-end object detection,","cited_arxiv_id":null,"evidence_quote":"Provides the deformable attention primitive used in temporal, spatial, height-fusion, and neighboring-point aggregation."},{"cited_title":"Bev-san: Accurate bev 3d object detection via slice attention networks,","cited_arxiv_id":null,"evidence_quote":"Introduces height-specific BEV slice features and the height-range prior that inspires the dual-branch cross-attention fusion of height distributions."},{"cited_title":"Heightformer: Explicit height modeling without extra data for camera-only 3d object detection in bird’s eye view,","cited_arxiv_id":null,"evidence_quote":"Shows how standard spatial cross-attention implicitly encodes height, motivating explicit adaptive height sampling as an alternative."},{"cited_title":"Ocbev: Object-centric bev transformer for multi-view 3d object detection,","cited_arxiv_id":null,"evidence_quote":"Offers the object-focused reference-point sampling comparison on the same benchmark that the vertical module seeks to improve upon."},{"cited_title":"nuscenes: A multimodal dataset for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Defines the nuScenes benchmark and metrics (mAP, NDS) on which all main comparisons and ablations are run."},{"cited_title":"Lyft level 5 av dataset 2019,","cited_arxiv_id":null,"evidence_quote":"Defines the Lyft dataset and evaluation protocol used for the generalization experiment."},{"cited_title":"Surroundocc: Multi-camera 3d occupancy prediction for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Provides the occupancy predictor into which the horizontal module is inserted for the transfer experiment."}],"review_version":1}