REVIEW 4 major objections 8 minor 35 references
NOVA: Next-step Open-Vocabulary Autoregression for 3D Multi-Object Tracking in Autonomous Driving
T0 review · 4 major / 8 minor · reviewed 2026-07-15 · grok-4.5
Pith's one-line read NOVA recasts open-vocabulary 3D multi-object tracking as next-step sequence completion so a 0.5B language model can keep identities for unseen classes, raising novel AMOTA on nuScenes from 2.2% to 22.4%.
desk verdict Solid OV-3D association recipe with real novel-category gains on a 0.5B model; the big AMOTA jump is only partly isolated from the detector. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Next-step Open-Vocabulary Autoregression (NOVA): each track–detection pair is serialized into a prompt of recent boxes and a candidate; a lightweight LLM emits a Yes/No decision token whose probability becomes the association score. Geometry Encoder embeddings, Hybrid Prompting (base names kept, novel names replaced by Unknown), and hard negative mining supply the inductive bias that makes the decision reliable under open-vocabulary noise.
What would settle it
Replace the open-vocabulary detector with one that produces systematically mislocalized or empty novel boxes on the same nuScenes Novel split and re-run the identical NOVA association pipeline; if Novel AMOTA collapses back near the prior baseline, the claim that trajectory-conditioned next-token scoring is the primary source of the gain is falsified.
Extended reading notes
Core claim
The central claim is that online open-vocabulary 3D multi-object association is better solved as autoregressive next-token prediction over serialized trajectory context than as hand-crafted geometric or semantic cost matching. With a Geometry Encoder that maps 3D boxes into the model embedding space plus IoU-quality supervision, Hybrid Prompting that masks novel labels, and hard negative mining, a compact 0.5B model produces association scores that substantially improve novel-category tracking—most notably lifting nuScenes Novel AMOTA from 2.20% to 22.41%—while remaining detector-agnostic across the evaluated benchmarks.
Load-bearing premise
The method assumes upstream open-vocabulary 3D detectors already supply usable novel-class boxes, and that training with those labels masked as Unknown plus a short motion history is enough for the language model to recover correct identities under the same detection noise at test time.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. NOVA reformulates open-vocabulary 3D multi-object tracking (OV-3D-MOT) association as autoregressive next-token prediction over serialized trajectory context. A lightweight LLM (default Qwen2.5-0.5B) scores track–detection pairs by the probability of generating “Yes,” using a Geometry Encoder that injects continuous 3D box features via a <box> token (with IoU-quality auxiliary loss), Hybrid Prompting that keeps base class names but masks novel labels as “Unknown,” and hard-negative mining of spatially proximate identity-inconsistent candidates. Online inference builds a cost matrix from these scores and applies Hungarian matching with standard birth/death logic. Experiments on nuScenes, V2X-Seq-SPD, and KITTI report large novel-category gains versus Open3DTrack—most notably nuScenes Novel AMOTA 22.41% vs 2.20%—with ablations on LLM size, geometry encoding, prompting, sampling, history length, and epochs.
Significance. OV-3D-MOT is still nascent; moving association from hand-crafted geometric costs toward a learned, trajectory-conditioned policy is a timely and useful direction for open-world driving perception. If the reported novel-category gains hold under controlled isolation of the association module, the work would be a solid empirical contribution: multi-benchmark evaluation, systematic ablations, a compact 0.5B backbone with reported FPS, and promised code. The Geometry Encoder + IoU quality head and Hybrid Prompting are concrete, transferable design choices even if the “LLM commonsense” framing is secondary to the binary association scorer. The paper does not claim machine-checked proofs or parameter-free theory; its value is systems evidence for generative association under open-vocabulary detections.
major comments (4)
- [Table I, §IV.B, Fig. 2] Table I / §IV.B: The central attribution of the +20.21 Novel AMOTA gain (2.20→22.41 on nuScenes) to autoregressive association is not fully isolated. Although both methods are listed under Find n’ Propagate on nuScenes, Open3DTrack is defined (§II, Fig. 2) as 2D-OV semantics on closed-set 3D proposals with conventional cost matching. The manuscript does not report (i) a standard non-LLM association baseline (e.g., 3D IoU/distance/Mahalanobis + Hungarian, or AB3DMOT-style) on exactly the same OV boxes NOVA receives, nor (ii) NOVA’s scorer on the proposal stream Open3DTrack originally used. Without that swap, detector/proposal quality and association formulation remain partially confounded, and Hybrid Prompting / L=3 ablations (Tables IV, VI) only validate design choices inside NOVA.
- [Table I, §IV.B, Abstract] Table I (V2X-Seq-SPD rows): Novel AMOTA is largely negative for NOVA (−141.78 and −129.85) while the text emphasizes sAMOTA gains and “superior performance.” Negative AMOTA typically indicates performance below the untracked baseline under the official averaging; this is load-bearing for the open-vocabulary claim and is not explained in §IV.B or the conclusion. Please (a) state when AMOTA vs sAMOTA is the primary metric and why, (b) discuss what drives Novel AMOTA collapse (recall, confidence calibration, score thresholds, ID switches), and (c) temper abstract/intro claims of broad superiority where Novel AMOTA fails.
- [Abstract, §I, §III.B Hybrid Prompting, Eq. (2), Table II] Abstract / §I / §III.B: The narrative that the model “utilizes the hierarchical structure of language space” and “high-level commonsense reasoning” overstates the implemented mechanism. Hybrid Prompting masks novel names as “Unknown,” the decision is a binary Yes/No token (Eq. 2), and Table II shows a 0.5B model outperforming larger LLMs—consistent with learning geometric/motion discrimination more than deep linguistic category reasoning. Please align claims with the actual task (trajectory-conditioned match scoring) and clarify what, if any, language priors remain after novel-label masking; otherwise the contribution is better framed as learned geometric association with optional base-class text, not open-vocabulary linguistic reasoning.
- [§IV.B, Table I KITTI] §IV.A–B / Table I (KITTI): Open3DTrack is omitted on KITTI as “degenerate,” so the only multi-method comparison is incomplete on a third benchmark. Given that the paper’s strongest claim is cross-dataset OV-3D-MOT superiority, either provide at least one conventional association baseline on the same Find n’ Propagate (+ GroundingDINO/YOLO-World) detections for KITTI, or clearly restrict the SOTA claim to settings where the baseline is reported and non-degenerate.
minor comments (8)
- [Title, headers, Fig. 1] Throughout: the title/name is inconsistently rendered as “NOV A” (space) vs “NOVA”; unify for searchability and typesetting.
- [§III.B Geometry Encoder, Eq. (3)] Eq. (3): f_raw includes vol=lwh which is deterministically redundant with (l,w,h); a one-line justification or ablation note would help.
- [§III.B, Eq. (4)] Eq. (4): λ_reg is never specified numerically; please report the value used and sensitivity if any.
- [Table I nuScenes] Table I: AMOTP for NOVA Base on nuScenes rises to 100.09 from 63.70 (worse if lower-is-better); briefly discuss the precision trade-off when claiming overall superiority.
- [Fig. 4] Fig. 4: failure callouts are informative but dense; consider marking which objects are Novel vs Base and ensuring IDs are legible in print.
- [§IV.A Datasets and Metrics] §IV.A: Base/Novel splits differ across datasets (e.g., Pedestrian is Base on nuScenes/V2X but Novel on KITTI); a short rationale table would reduce confusion when comparing Novel metrics across rows.
- [§II Autoregressive Modeling] Related work: AR-MOT and recent OV tracking citations are present; a clearer sentence on how binary Yes/No association differs from full sequence-generation MOT would help position the method.
- [§IV.A Implementation Details] Implementation: LoRA rank, learning rate, batch construction (pairs per step), and track age K are underspecified relative to reproducibility claims; add to §IV.A or appendix.
Circularity Check
No circularity: empirical tracking system with learned association scores evaluated on held-out standard MOT metrics.
full rationale
NOVA is a systems/ML paper: it trains an LLM association scorer and reports AMOTA/sAMOTA/MOTA on nuScenes, V2X-Seq-SPD, and KITTI under fixed Base/Novel splits. The association score pi,j = Pr(Yes|P(Ti,bj_t)) is a learned model output, not a quantity defined from the evaluation metrics or from parameters fitted to those same metrics. Geometry Encoder, Hybrid Prompting, hard negatives, history length L, and epoch count are design choices and hyperparameters ablated on the same benchmarks; none of the reported gains reduce by construction to the training objective or to a self-cited uniqueness theorem. Citations (Open3DTrack, Find n' Propagate, Qwen, etc.) are external baselines or tools, not load-bearing self-proofs. Concerns about detector confounds or attribution of gains to association vs. proposals are experimental-isolation issues, not circular derivation. Derivation chain is self-contained against external benchmarks; steps empty by design.
Assumptions & free parameters
free parameters (6)
- inference history length L
- training epochs
- IoU-quality loss weight λ_reg
- track age threshold K
- association 3D IoU evaluation threshold 0.25
- hard-negative spatial selection / positive jitter magnitude
assumptions (5)
- domain assumption Online tracking-by-detection with one-to-one bipartite matching is an adequate formulation for OV-3D-MOT association.
- domain assumption Open-vocabulary 3D detectors used (Find n’ Propagate ± GroundingDINO/YOLO-World) produce novel-class boxes usable enough that association, not proposal generation, is the main bottleneck.
- ad hoc to paper Masking novel class names as “Unknown” during training teaches class-agnostic association without discarding necessary identity cues.
- ad hoc to paper Probability of generating the token “Yes” is a calibrated enough match affinity for cost-based assignment across base and novel classes.
- domain assumption Standard MOT metrics at fixed IoU, with author-defined Base/Novel splits, measure open-vocabulary tracking success.
invented entities (3)
-
Geometry Encoder with <box> embedding injection
-
Hybrid Prompting (base names kept, novel names → Unknown)
-
NOVA autoregressive association policy
Cite this review
Pith. "Pith review of NOVA: Next-step Open-Vocabulary Autoregression for 3D Multi-Object Tracking in Autonomous Driving." pith.science (2026). https://pith.science/paper/GBKSPFC5
@misc{pith2026260306254,
author = {Pith},
title = {Pith review of: NOVA: Next-step Open-Vocabulary Autoregression for 3D Multi-Object Tracking in Autonomous Driving},
year = {2026},
howpublished = {\url{https://pith.science/paper/GBKSPFC5}},
note = {Machine review of arXiv:2603.06254}
}
read the original abstract
Generalizing across unknown targets is critical for open-world perception, yet existing 3D Multi-Object Tracking (3D MOT) pipelines remain limited by closed-set assumptions and ``semantic-blind'' heuristics. To address this, we propose Next-step Open-Vocabulary Autoregression (NOVA), an autoregressive association formulation that shifts the data association stage from fragmented distance-based matching toward trajectory-conditioned spatio-semantic modeling. NOVA reformulates 3D trajectories as structured spatio-temporal semantic sequences, enabling the simultaneous encoding of physical motion continuity and deep linguistic priors. By leveraging the autoregressive capabilities of Large Language Models (LLMs), we transform the tracking task into a principled process of next-step sequence completion. This mechanism allows the model to explicitly utilize the hierarchical structure of language space to resolve fine-grained semantic ambiguities and maintain identity consistency across complex long-range sequences through high-level commonsense reasoning. Extensive experiments on nuScenes, V2X-Seq-SPD, and KITTI demonstrate the superior performance of NOVA. Notably, on the nuScenes dataset, NOVA achieves an AMOTA of 22.41% for Novel categories, yielding a significant 20.21% absolute improvement over the baseline. These gains are realized through a compact 0.5B autoregressive model. Code will be available at https://github.com/xifen523/NOVA.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Open3DTrack: Towards open-vocabulary 3D multi- object tracking,
A. Ishaqet al., “Open3DTrack: Towards open-vocabulary 3D multi- object tracking,” inProc. ICRA, 2025, pp. 787–793
2025
-
[2]
nuScenes: A multimodal dataset for autonomous driving,
H. Caesaret al., “nuScenes: A multimodal dataset for autonomous driving,” inProc. CVPR, 2020, pp. 11 618–11 628
2020
-
[3]
V2X-Seq: A large-scale sequential dataset for vehicle- infrastructure cooperative perception and forecasting,
H. Yuet al., “V2X-Seq: A large-scale sequential dataset for vehicle- infrastructure cooperative perception and forecasting,” inProc. CVPR, 2023, pp. 5486–5495
2023
-
[4]
A survey of autonomous driving: Common practices and emerging technologies,
E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of autonomous driving: Common practices and emerging technologies,” IEEE Access, vol. 8, pp. 58 443–58 469, 2020
2020
-
[5]
AB3DMOT: A baseline for 3D multi-object tracking and new evaluation metrics,
X. Weng, J. Wang, D. Held, and K. Kitani, “AB3DMOT: A baseline for 3D multi-object tracking and new evaluation metrics,”arXiv preprint arXiv:2008.08063, 2020
arXiv 2008
-
[6]
Towards long-tailed 3D detection,
N. Peri, A. Dave, D. Ramanan, and S. Kong, “Towards long-tailed 3D detection,” inProc. CoRL, vol. 205, 2023, pp. 1904–1915
2023
-
[7]
Towards open world recognition,
A. Bendale and T. Boult, “Towards open world recognition,” inProc. CVPR, 2015, pp. 1893–1902
2015
-
[8]
Are we ready for autonomous driving? The KITTI vision benchmark suite,
A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? The KITTI vision benchmark suite,” inProc. CVPR, 2012, pp. 3354–3361
2012
Show all 35 references
-
[9]
Learning transferable visual models from natural language supervision,
A. Radfordet al., “Learning transferable visual models from natural language supervision,” inProc. ICML, vol. 139, 2021, pp. 8748–8763
2021
-
[10]
Grounded language-image pre-training,
L. H. Liet al., “Grounded language-image pre-training,” inProc. CVPR, 2022, pp. 10 955–10 965
2022
-
[11]
OVTrack: Open-vocabulary multiple object tracking,
S. Li, T. Fischer, L. Ke, H. Ding, M. Danelljan, and F. Yu, “OVTrack: Open-vocabulary multiple object tracking,” inProc. CVPR, 2023, pp. 5567–5577
2023
-
[12]
VOVTrack: Exploring the potentiality in videos for open-vocabulary object tracking,
Z. Qian, R. Han, J. Hou, L. Song, and W. Feng, “VOVTrack: Exploring the potentiality in videos for open-vocabulary object tracking,”arXiv preprint arXiv:2410.08529, 2024
2024 arXiv
-
[13]
Center-based 3D object detection and tracking,
T. Yin, X. Zhou, and P. Krähenbühl, “Center-based 3D object detection and tracking,” inProc. CVPR, 2021, pp. 11 784–11 793
2021
-
[14]
EagerMOT: 3D multi-object tracking via sensor fusion,
A. Kim, A. Osep, and L. Leal-Taixé, “EagerMOT: 3D multi-object tracking via sensor fusion,” inProc. ICRA, 2021, pp. 11 315–11 321
2021
-
[15]
MCTrack: A unified 3D multi-object tracking framework for autonomous driving,
X. Wanget al., “MCTrack: A unified 3D multi-object tracking framework for autonomous driving,” inProc. IROS, 2025, pp. 4551– 4558
2025
-
[16]
PointPillars: Fast encoders for object detection from point clouds,
A. H. Lang, S. V ora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “PointPillars: Fast encoders for object detection from point clouds,” in Proc. CVPR, 2019, pp. 12 697–12 705
2019
-
[17]
PV-RCNN: Point-voxel feature set abstraction for 3D object detection,
S. Shiet al., “PV-RCNN: Point-voxel feature set abstraction for 3D object detection,” inProc. CVPR, 2020, pp. 10 526–10 535
2020
-
[18]
Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection,
S. Liuet al., “Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection,” inProc. ECCV, 2024, pp. 38–55
2024
-
[19]
YOLO- World: Real-time open-vocabulary object detection,
T. Cheng, L. Song, Y . Ge, W. Liu, X. Wang, and Y . Shan, “YOLO- World: Real-time open-vocabulary object detection,” inProc. CVPR, 2024, pp. 16 901–16 911
2024
-
[20]
PointCLIP: Point cloud understanding by CLIP,
R. Zhanget al., “PointCLIP: Point cloud understanding by CLIP,” in Proc. CVPR, 2022, pp. 8542–8552
2022
-
[21]
OpenScene: 3D scene understanding with open vocabularies,
S. Peng, K. Genova, C. M. Jiang, A. Tagliasacchi, M. Pollefeys, and T. A. Funkhouser, “OpenScene: 3D scene understanding with open vocabularies,” inProc. CVPR, 2023, pp. 815–824
2023
-
[22]
ImOV3D: Learning open vocabulary point clouds 3D object detection from only 2D images,
T. Yang, Y . Ju, and L. Yi, “ImOV3D: Learning open vocabulary point clouds 3D object detection from only 2D images,” inProc. NeurIPS, 2024
2024
-
[23]
FM-OV3D: Foundation model-based cross-modal knowledge blending for open-vocabulary 3D detection,
D. Zhanget al., “FM-OV3D: Foundation model-based cross-modal knowledge blending for open-vocabulary 3D detection,” inProc. AAAI, 2024, pp. 16 723–16 731
2024
-
[24]
Find n’ propagate: Open-vocabulary 3D object detection in urban environments,
D. Etchegaray, Z. Huang, T. Harada, and Y . Luo, “Find n’ propagate: Open-vocabulary 3D object detection in urban environments,” inProc. ECCV, 2024, pp. 133–151
2024
-
[25]
CoDA: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3D object detection,
Y . Cao, Z. Yihan, H. Xu, and D. Xu, “CoDA: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3D object detection,” inProc. NeurIPS, 2023, pp. 71 862–71 873
2023
-
[26]
OVTR: End-to-end open-vocabulary multiple object tracking with transformer,
J. Li, E. Yu, S. Chen, and W. Tao, “OVTR: End-to-end open-vocabulary multiple object tracking with transformer,” inProc. ICLR, 2025
2025
-
[27]
Monocular multi-object 3D visual language tracking,
H. Weiet al., “Monocular multi-object 3D visual language tracking,” IEEE Transactions on Image Processing, 2026
2026
-
[28]
Class-balanced grouping and sampling for point cloud 3D object detection,
B. Zhu, Z. Jiang, X. Zhou, Z. Li, and G. Yu, “Class-balanced grouping and sampling for point cloud 3D object detection,”arXiv preprint arXiv:1908.09492, 2019
1908 arXiv
-
[29]
BEVFusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation,
Z. Liuet al., “BEVFusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation,” inProc. ICRA, 2022, pp. 2774–2781
2022
-
[30]
Recurrent autoregressive networks for online multi-object tracking,
K. Fang, Y . Xiang, X. Li, and S. Savarese, “Recurrent autoregressive networks for online multi-object tracking,” inProc. WACV, 2018, pp. 466–475
2018
-
[31]
TrackFormer: Multi-object tracking with transformers,
T. Meinhardt, A. Kirillov, L. Leal-Taixé, and C. Feichtenhofer, “TrackFormer: Multi-object tracking with transformers,” inProc. CVPR, 2022, pp. 8834–8844
2022
-
[32]
3DMOT- Former: Graph transformer for online 3D multi-object tracking,
S. Ding, E. Rehder, L. Schneider, M. Cordts, and J. Gall, “3DMOT- Former: Graph transformer for online 3D multi-object tracking,” in Proc. ICCV, 2023, pp. 9750–9760
2023
-
[33]
TrackFusion: Enhancing multi-object tracking with temporal trajectory modeling and frame-integrated detection,
X. Zhanget al., “TrackFusion: Enhancing multi-object tracking with temporal trajectory modeling and frame-integrated detection,” inProc. ICASSP, 2025, pp. 1–5
2025
-
[34]
AR-MOT: Autoregressive multi-object tracking,
L. Jia, Y . Wu, B. Ran, Y . Wang, L. Wang, and H. Lu, “AR-MOT: Autoregressive multi-object tracking,”arXiv preprint arXiv:2601.01925, 2026
2026
-
[35]
Tracking meets large multimodal models for driving scenario understanding,
A. Ishaq, J. Lahoud, F. S. Khan, S. Khan, H. Cholakkal, and R. M. Anwer, “Tracking meets large multimodal models for driving scenario understanding,”arXiv preprint arXiv:2503.14498, 2025
2025 arXiv
Reviewed July 15, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.