{"id":"7f9c1a15-af3b-49d5-b8f6-2d677619d441","arxiv_id":"2506.10242","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"DySS combines state-space feature learning with dynamic query merging and pruning to improve both accuracy and speed for camera-based 3D detection on nuScenes.","lead":"A camera-only 3D object detector for self-driving cars uses a state-space model to summarize multi-camera video and prunes, merges, and splits its detection queries to stay fast. It reports state-of-the-art nuScenes scores and faster inference than its baseline, SparseBEV.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dynamic query decisions in Section 3.1 have no specified training signal or gradient path, so the claimed accuracy-preserving query reduction is not yet reproducible.","rationale":"The reader identified the missing supervision for dynamic query operations as the weakest assumption, and I agree: it is the single most load-bearing concern because the paper's main efficiency contribution is precisely the learned merge/remove/split schedule, and the reported 33 FPS with only a small accuracy change is attributed to this mechanism. The paper does specify auxiliary losses for the state-space model (Eq. 2) and provides ablations that are directionally consistent, which is real support for the SSM component. The test-split section contains a numeric inconsistency (65.1/56.8 in the text versus 65.3/57.4 in Table 2 and the abstract), but this appears to be a typo rather than a challenge to the central mechanism. The dynamic-query supervision gap, by contrast, affects reproducibility and the interpretation of the main result. Since the authors could plausibly resolve this by specifying a training objective or releasing code, and since there is no demonstrated contradiction in the reported numbers, the reader's CONDITIONAL verdict remains appropriate; my read does not change it.","tokens_in":12546,"tokens_out":3749,"duration_ms":46561,"concrete_test":"Run a control ablation on nuScenes val using the same DySS pipeline, replacing the learned merge/remove/split decisions of Section 3.1 with random labels drawn from the same quotas (merge index/ratio, remove 20–30% of Nq, split at most 5% of Nq), keeping all other components unchanged. If the random-label variant reproduces 56.2 NDS and 46.2 mAP at roughly 33 FPS, the observed gains are not attributable to learned dynamic query selection; if it drops measurably, the learned decisions are functional. Additionally, report the agreement of predicted merge/remove/split labels with oracle heuristics (e.g., IoU-based duplicate/occlusion criteria) to confirm the labels carry meaningful information.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central efficiency claim of DySS is that merge, remove, and split operations cut the query count from 900 to 269 while preserving detection accuracy. Section 3.1 describes these operations as linear layers producing merge labels/indexes, remove labels in [0,1] with a removal ratio in [0.2,0.3], and split labels in [0,1] with a split count in [0,5], operating on query features and a covariance matrix Cq. However, the paper never defines a loss or supervision target for any of these labels. Section 4.1 states that training uses the standard detection losses plus the state-space auxiliary losses Lr and Lf from Eq. (2); neither loss involves merge/remove/split decisions. Because these decisions are discrete, detection loss alone cannot provide gradients through them unless a relaxation or estimator (e.g., Gumbel-Softmax, straight-through, or REINFORCE) is specified, and none is described. The Table 3 ablation cannot distinguish learned query selection from arbitrary quota-based pruning: the comparison is between fixed 900 queries and 269 queries after the dynamic operations, with no control that random labels under the same quotas would behave differently. This makes the central mechanism of the paper underspecified and its reported speed/accuracy trade-off unattributable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"DySS proposes a sparse-query, multi-camera 3D object detector that combines a state-space model (Mamba) with dynamic query operations (merge, remove, split). The state-space model is trained with auxiliary masked-reconstruction and future-prediction losses, and its features are mixed into the query features. The claimed contributions are improved accuracy and efficiency: on nuScenes test the paper reports 65.3 NDS / 57.4 mAP, on val 56.2 NDS / 46.2 mAP at 33 FPS, and an ablation on val shows that reducing queries from 900 to 269 via the dynamic operations roughly preserves accuracy while increasing speed.","tokens_in":12826,"tokens_out":4641,"duration_ms":48644,"significance":"If the claims hold, the paper makes a practical contribution: it shows that a state-space representation can summarize multi-view video efficiently, and that pruning detection queries dynamically can cut compute substantially without losing accuracy. The external nuScenes test-server numbers and the clear ablation table are strengths, and the comparison against SparseBEV and StreamPETR is appropriate. The main reason for caution is that the training signal for the dynamic query module is not specified, so the central efficiency-accuracy trade-off is not yet reproducible from the manuscript.","major_comments":[{"comment":"The dynamic query operations are not supervised or otherwise given a training signal. Section 3.1.1-3.1.3 says linear layers produce merge labels, removal labels in [0,1], a removal ratio in [0.2,0.3], split labels in [0,1], and a split count in [0,5], but no loss term or reward is defined for any of these outputs. Section 4.1 lists only the detection losses and the state-space losses Lr and Lf from Eq. (2); none involve merge/remove/split decisions. Because the operations are discrete, detection losses cannot backpropagate through them unless a relaxation or estimator (e.g., Gumbel-Softmax, straight-through, REINFORCE) is described, and none is. This is load-bearing: Table 3 attributes the preserved accuracy and the FPS gain to the dynamic query module, yet the mechanism by which the module learns to merge, remove, and split is absent. The authors should specify the supervision or training procedure for these decisions, or add a control experiment showing that random labels under the same quotas do not produce the same result.","section":"Section 3.1, Eq. (2), Section 4.1"},{"comment":"The nuScenes test numbers are internally inconsistent. The abstract states 65.31 NDS and 57.4 mAP, Table 2 reports 65.3 NDS and 57.4 mAP, but Section 4.3.2 first says DySS achieves 65.1 NDS and 56.8 mAP and later, in the same paragraph, says it achieves 65.3 NDS and 57.4 mAP. Since the test split result is the paper's headline claim, the correct numbers need to be stated consistently in all three places.","section":"Section 4.3.2 vs. Table 2 vs. Abstract"},{"comment":"The comparison against SparseBEV in Section 4.3.1 says DySS is 'more than 20% faster than SparseBEV (27 FPS)', but the SparseBEV baseline in Table 3 is measured at 14 FPS with 900 queries (or 10-11 FPS when the SSM is added). The source of the 27 FPS figure is not given, and the two numbers are not compatible. The speed comparison should state which configuration of SparseBEV is used and under which hardware/software conditions, or the ablation-table baseline should be used as the reference.","section":"Section 4.3.1 and Table 3"}],"minor_comments":[{"comment":"Typo: 'It outputs an enhanced version of the sampled features TF̂_t=1' should presumably be 'TF̂_t=i'.","section":"Section 3.2"},{"comment":"References [6] and [7] are the same paper (Mamba by Gu and Dao); the duplicate citation should be merged.","section":"References"},{"comment":"The merge operation is described only as producing a label and an index; the manuscript does not state how the selected queries are actually merged (e.g., weighted averaging of features, which query is kept). This is needed for reproducibility even if the supervision question is resolved.","section":"Section 3.1.1"},{"comment":"The state-space feature is written as S^l_{t=T}, but the decoder layer index l is not defined at that point; it should be introduced or the superscript omitted.","section":"Eq. (3)-(4)"},{"comment":"The FPS column shows that adding SSM with identity/FFT drops speed from 14 to 10 FPS, and then dynamic queries raise it to 33-35 FPS; a sentence explaining that the SSM overhead is amortized by the query reduction would help readers interpret the trade-off.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's main result is attractive and the test-server numbers, if correct, would be competitive. The blocking issue is the missing training signal for the dynamic query module; this is not a matter of taste but of the central mechanism being underspecified. If the authors can supply the exact loss or estimator for merge/remove/split and reconcile the three conflicting test numbers, I would be inclined to accept after a revision. I would also ask the editor to verify the duplicate Mamba reference, which suggests the manuscript was assembled in haste."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: DySS reports a new SOTA on nuScenes test (65.3 NDS, 57.4 mAP) using a Mamba-style SSM plus dynamic query pruning. The test-server results are genuinely external evidence, and the direction of the ablations is consistent. The problem is that the merge/remove/split decisions in Section 3.1 have no specified training signal or gradient path, so the central efficiency claim is not yet reproducible.\n\nWhat's actually new: the combination of state-space learning with future prediction and masked reconstruction losses for sparse BEV detection, and the merge/remove/split query schedule that cuts 900 queries down to 269 with no accuracy loss. Each ingredient has been seen before, but the combination is fresh. The paper does what a solid empirical paper should: val ablations, test-server comparison, a tracking side-experiment, and clear FPS reporting. The SSM with auxiliary losses adds about 0.7 NDS and 1.25 mAP over the SparseBEV baseline; the dynamic queries then hold accuracy while tripling throughput. That's a real result, if the mechanism works as claimed.\n\nThe soft spots. First and most importantly, the training of the dynamic query module is underspecified. The text describes linear layers that emit merge labels, remove scores, and split counts, but never defines a loss for them, and no straight-through estimator, Gumbel-Softmax, or REINFORCE is mentioned. The detection losses (focal + L1) plus the SSM auxiliary losses cannot backprop through hard discrete decisions unless such an estimator exists and is described. Without that, the Table 3 ablation cannot separate learned pruning from arbitrary quota-based pruning. A random-label control under the same quotas would settle this. Second, the test split numbers conflict: Section 4.3.2 says 65.1 NDS and 56.8 mAP, then two sentences later 65.3 and 57.4, matching Table 2 and the abstract. Minor, but sloppy. Third, no code or error bars; field-standard, but combined with the supervision gap it makes the speed claim hard to verify.\n\nWho should read it: anyone working on camera-only 3D detection or efficient video perception. The test-server numbers make it a useful data point. I'd accept it for peer review—the issues are addressable—but not as-is. The authors need to specify how the dynamic operations are trained, add a random-pruning baseline, and reconcile the numbers.","headline":"Solid nuScenes SOTA claim with a reproducible gap: the dynamic query training signal is never specified, so the speed-up story is unattributable as written.","tokens_in":13341,"tokens_out":4423,"would_cite":true,"duration_ms":49557,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that DySS makes camera-only 3D detection from multi-camera video both more accurate and faster by summarizing scene history in a state-space model and by dynamically pruning, merging, and splitting the detection queries…","keywords":["dynamic queries","state-space model","3D object detection","multi-camera video","BEV perception","nuScenes","temporal modeling","query pruning"],"falsifier":"Run DySS with the learned merge, remove, and split heads replaced by fixed random decisions that produce the same final query count (e.g., always dropping 30% and splitting 5% at random, and merging at random); if NDS and mAP stay within noise of the learned-head model, the reported gains come from the reduced query count or the SSM, not from learned query editing. A complementary check is to inspect the remove labels after training: if queries marked for removal sit as close to ground-truth objects as queries kept, the head is not learning redundancy.","tokens_in":12344,"feed_emoji":"🚗","tokens_out":5475,"duration_ms":62799,"temperature":0.7,"pith_summary":"DySS sets out to show that camera-only 3D detection from multi-camera video does not need a fixed, large set of queries or a dense BEV grid to be state of the art. Instead, a state-space model reads the image features sampled by the queries over consecutive frames and compresses them into a hidden state that summarizes motion and scene correspondence; auxiliary losses on masked reconstruction and future-frame prediction train this state to be informative. The decoder then edits its queries between layers, merging similar ones, removing redundant ones, and splitting others, shrinking the query set from 900 to 269 on the validation split and from 1600 to 729 on the test split. If the claim holds, the practical consequence is that autonomous-driving perception can get higher accuracy at real-time speed: 56.2 NDS and 46.2 mAP at 33 FPS on the validation split, and 65.31 NDS and 57.4 mAP on the test benchmark.","feed_headline":"Camera-only 3D detection hits 65.3 NDS with dynamic queries","feed_subtitle":"A state-space memory plus merge-remove-split query edits beat prior sparse detectors at 33 FPS.","key_machinery":"The load-bearing machinery is the recurrent state update $h(t) = \\boldsymbol A h(t-1) + \\boldsymbol B x(t)$, $y(t) = \\boldsymbol C h(t)$, realized with Mamba-style selective state-space blocks; one block operates in the time domain and one in the frequency domain via FFT. The SSM consumes sampled features $F_{t=i}$ plus predicted features $\\tilde{F}_{t=i+1}$ and is trained with reconstruction and prediction losses $L_r$ and $L_f$ in Eq. (2), which is what makes the hidden state encode motion rather than simply pass features through. The second mechanism is the dynamic query editor: after cross-attention with state-space features, linear layers read the query features and a covariance matrix $\\boldsymbol C_q$ to emit merge, remove, and split labels, removing 20-30% of queries and splitting at most 5%, so the query count falls adaptively through the six decoder layers.","core_discovery":"The central claim is that temporal context for sparse query detection should be represented as the evolving state of a state-space model rather than as extra attention over many sampled points from many frames. DySS feeds the SSM the features sampled at each time step, together with a predicted version of those features from the previous step, and uses the final hidden state as a compact spatial-temporal summary of the scene. Supervised by masked reconstruction and future-feature prediction during training, the SSM is claimed to capture motion and cross-view correspondence that improves both detection and tracking. On top of this, merge, remove, and split operations driven by query-feature covariances let the decoder maintain a lean set of queries; the paper reports that this dynamic editing keeps accuracy essentially equal to the 900-query baseline while nearly tripling the frame rate, and that the full system outperforms prior state of the art on the nuScenes benchmark.","pith_inferences":["If dynamic query editing is what carries the speed gain, then the merge/remove/split ratios could be made a continuous function of a latency budget, letting one network trade accuracy for speed at inference time.","The auxiliary masked-reconstruction and future-prediction losses suggest a self-supervised pretraining recipe: train the SSM on unlabeled driving video with those two losses, then fine-tune with detection losses, which the paper does not explore.","A direct test of the query editor would be to supervise merge/remove/split labels with a sparse-set matching loss; lack of such supervision is the main unstated assumption, and adding it would either strengthen or refute the claimed mechanism.","Using FFT as one of the two SSM branches hints that spectral features help motion modeling; a natural extension is replacing the fixed transform with a learned frequency filter bank and measuring the change on high-velocity classes."],"forward_implications":["Temporal history becomes cheap: a compact SSM state replaces the expensive per-query multi-frame sampling and attention of earlier sparse detectors, so longer video context does not multiply compute.","Query editing gives an accuracy-preserving speed dial: cutting queries from 900 to 269 raises FPS from 11 to 35 while NDS stays near the full-query baseline.","The learned temporal representation transfers to downstream tasks: DySS improves AMOTA over SparseBEV and StreamPETR, not just box accuracy.","Camera-only detection reaches a real-time operating point on a single GPU, 33 FPS on an RTX-3080, while setting a new nuScenes test score of 65.31 NDS."],"supporting_citations":[{"why":"Supplies the sparse query-sampling baseline that DySS builds on and compares against for both accuracy and speed.","marker":"[25]"},{"why":"Provides the Mamba state-space architecture that carries the temporal sequence modeling in DySS.","marker":"[7]"},{"why":"Defines the nuScenes dataset and the NDS metric used for the paper's central accuracy and tracking claims.","marker":"[2]"},{"why":"Is the object-centric temporal memory baseline whose query design and tracking results DySS is compared with.","marker":"[37]"},{"why":"Inspires the auxiliary future-prediction training that the paper credits for making the SSM learn motion.","marker":"[40]"},{"why":"Provides the learnable proposal-query formulation that DySS adopts for encoding instance information in each query.","marker":"[36]"},{"why":"Contributes the channel-and-point adaptive mixing idea that DySS adapts to fuse query features with state-space features.","marker":"[5]"}],"fun_headline_variants":["State-space memory sharpens camera 3D object detection","Dynamic queries + state-space = lean 3D detection","DySS: 65.3 NDS at 33 FPS with sparse queries","Efficient BEV detection via state-space and query editing","Camera 3D detection gets faster with state-space learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ordinary detection losses alone teach the merge, remove, and split heads to make useful query-editing decisions, because the paper specifies no loss or ground-truth label for those operations.","fun_headline_variants_meta":{"raw":{"variants":["State-space memory sharpens camera 3D object detection","Dynamic queries + state-space = lean 3D detection","DySS: 65.3 NDS at 33 FPS with sparse queries","Efficient BEV detection via state-space and query editing","Camera 3D detection gets faster with state-space learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3559,"prompt_tokens":1004,"completion_tokens":2555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":2468}},"tokens_in":620,"tokens_out":2555,"duration_ms":17525,"temperature":1.0,"reasoning_tokens":2468,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:31:17.297873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DySS with the learned merge, remove, and split heads replaced by fixed random decisions that produce the same final query count (e.g., always dropping 30% and splitting 5% at random, and merging at random); if NDS and mAP stay within noise of the learned-head model, the reported gains come from the reduced query count or the SSM, not from learned query editing. A complementary check is to inspect the remove labels after training: if queries marked for removal sit as close to ground-truth objects as queries kept, the head is not learning redundancy.","supporting_citations":[{"cited_title":"Sparsebev: High-performance sparse 3d object de- tection from multi-camera videos","cited_arxiv_id":null,"evidence_quote":"Supplies the sparse query-sampling baseline that DySS builds on and compares against for both accuracy and speed."},{"cited_title":"nuscenes: A multi- modal dataset for autonomous driving","cited_arxiv_id":null,"evidence_quote":"Defines the nuScenes dataset and the NDS metric used for the paper's central accuracy and tracking claims."},{"cited_title":"Exploring object-centric temporal modeling for efficient multi-view 3d object detection","cited_arxiv_id":null,"evidence_quote":"Is the object-centric temporal memory baseline whose query design and tracking results DySS is compared with."},{"cited_title":"Futuredepth: Learning to predict the future improves video depth estimation","cited_arxiv_id":null,"evidence_quote":"Inspires the auxiliary future-prediction training that the paper credits for making the SSM learn motion."},{"cited_title":"Adamixer: A fast-converging query-based object detector","cited_arxiv_id":null,"evidence_quote":"Contributes the channel-and-point adaptive mixing idea that DySS adapts to fuse query features with state-space features."}],"review_version":1}