{"id":"e09f410c-ae85-4c9a-b5f6-9fac3e33782f","arxiv_id":"2603.09420","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"OMEN is the first end-to-end class-incremental motion forecaster that retains old-class accuracy via VLM-filtered future-detection pseudo-labels and variance-based sequence replay.","lead":"This paper defines class-incremental (open-world) motion forecasting: predict agent trajectories from cameras while new object classes arrive over time without re-annotating old data. It offers OMEN, an end-to-end system that uses VLM-filtered pseudo-labels and motion-query variance replay so fleets can add classes under storage and re-labeling constraints.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Preservation of old-class forecasting rests on unquantified fidelity of VLM-filtered future-detection pseudo-labels that replace GT.","rationale":"The reader correctly isolates the pseudo-label + VLM fidelity assumption as the weakest link supporting the strongest claim. Ablations (Table V) and relative gains over forgetting/CL-DETR are real, yet absolute PL quality remains unmeasured and the experimental protocol never removes old-class instances, so the numbers do not fully stress a pure class-incremental regime. No deeper internal inconsistency or over-claim on the forecasting core was found; planning/zero-shot results are secondary and open-loop/qualitative. The contribution stays a solid first systems paper, so the CONDITIONAL verdict (pending code and multi-seed stats) needs no adjustment.","tokens_in":20828,"tokens_out":584,"duration_ms":31292,"concrete_test":"After each per-class step i on nuScenes, compute precision/recall of the VLM-filtered future-detection pseudo-labels for every previous class against the original GT annotations of those classes that appear in D_i (same bipartite matching used at training). Separately retrain the final model with all PL loss weights set to zero (replay only, K=30). If PL precision for the first class falls below ~0.6 after step ≥3, or if the zero-PL ablation drops All mAP_f by >3 points relative to full OMEN, the substitution assumption is load-bearing and currently under-supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (OMEN preserves known-class forecasting while adapting; Tables I–III, abstract, §I) requires that motion pseudo-labels composed from Φ_{i-1} future-frame 3D detections (Eq. 1) and retained only after Grounded-SAM-2 keypoint majority vote (Eq. 2, §III-B.2) are accurate/calibrated substitutes for true labels of C_0…C_{i-1} on D_i. The paper notes rising confidence/FPs, keeps unmatched proposals at zero loss weight, and does not enforce absence of old classes so that PLs remain available (§H, §III-B.2). No precision, recall, or calibration of these filtered PLs vs. original GT is reported. Consequently it is unclear how much of the observed retention (e.g., car mAP_f 29.95 vs. joint 32.75) is produced by high-quality PLs versus the small variance-based replay buffer alone. If PL quality is only moderate, the “mitigates catastrophic forgetting” claim is weaker than the tables suggest.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper formalizes open-world / class-incremental end-to-end motion forecasting: future trajectories are predicted directly from multi-view camera images while new semantic classes are introduced sequentially, with ground-truth labels available only for the current subset Ci and no access to the original full training set. It proposes OMEN, built on SparseDrive, that (i) generates motion pseudo-labels for previously seen classes by composing 3D detections of the prior model Φi−1 at future frames (Eq. 1), (ii) filters false positives / over-confident proposals via 3D-to-2D keypoint majority voting against Grounded-SAM-2 instance masks (Eq. 2), and (iii) maintains a fixed-size sequence replay buffer selected by per-class variance of latent motion queries (Eqs. 4–5). Evaluations on nuScenes (per-class 7-step and group 3-step splits) and Argoverse 2 (20-2 overlapping) show that OMEN retains mAPf / EPA on old classes better than forgetting, plain pseudo-labeling and adapted CL-DETR baselines while approaching a joint-training upper bound; ablations isolate the contributions of future-detection PLs, VLM filtering and variance replay; the framework is extended to open-loop ego planning (Table IV) and demonstrated qualitatively zero-shot on real-vehicle data. Code is promised.","tokens_in":21140,"tokens_out":1191,"duration_ms":20325,"significance":"The work identifies a practically relevant gap—closed-world forecasting models cannot cheaply absorb novel classes (e-scooters, etc.) under storage and re-annotation constraints—and supplies the first end-to-end camera-based solution together with a reproducible experimental protocol (multiple incremental splits, joint upper bound, strong continual-detection baseline). The combination of future-detection pseudo-labels, open-vocabulary VLM filtering and motion-query variance replay is novel for this task; public code and multi-dataset results (including planning transfer) make the contribution usable by the community. If the retention numbers hold under stricter PL-quality scrutiny, the method offers a concrete path toward continual adaptation of the full perception–prediction–planning stack.","major_comments":[{"comment":"§III-B (Eqs. 1–2) and the central retention claims of Tables I–III: preservation of old-class forecasting is attributed to VLM-filtered future-detection pseudo-labels that replace ground truth for C0…Ci−1 on Di. The manuscript acknowledges rising confidence / residual false positives and keeps unmatched proposals at zero loss weight, yet reports no precision, recall, calibration or AP of the filtered PLs against the original GT of those classes. Without this quantification it remains unclear how much of the observed gap closure (e.g., car mAPf 29.95 vs joint 32.75) is produced by high-fidelity PLs versus the small variance replay buffer alone; an explicit PL-quality table (or oracle-PL ablation) is needed to make the “mitigates catastrophic forgetting” claim fully load-bearing.","section":"§III-B, Tables I–III"},{"comment":"Abstract and §I claim that the framework “extends naturally to open- and closed-loop end-to-end class-incremental planning on nuScenes and NeuroNCAP.” The body (§III-D, Table IV) only presents open-loop L2 / collision numbers on the nuScenes per-class split; no closed-loop results and no NeuroNCAP experiments appear. Either the missing experiments must be added or the abstract / contribution list must be revised to match the delivered content.","section":"Abstract, §III-D, Table IV"}],"minor_comments":[{"comment":"Title uses “Class-Incremental Motion Forecasting” while the body consistently speaks of “open-world motion forecasting”; align terminology for clarity.","section":"Title / Abstract"},{"comment":"§H Limitations correctly notes that old classes need not be fully absent and that GT is still required for novel classes; these caveats should also be reflected briefly in the abstract so readers do not over-interpret the “no original training set” claim.","section":"§H, Abstract"},{"comment":"Table V ablation rows are dense; adding a short textual summary of the incremental gains (future-det PL \to +VLM \to +var buffer) would improve readability.","section":"Table V"},{"comment":"Real-world zero-shot results (Fig. 4, §G) are purely qualitative; a short note on domain-gap statistics (camera FOV, mounting height, country) would help readers gauge transfer strength.","section":"§G"},{"comment":"Minor notation: the homogeneous transform in Eq. (1) is written T\tau\to t but later text uses T\tau\to t inconsistently; fix subscript style.","section":"Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The novelty claim as “first end-to-end class-incremental forecasting” appears solid relative to the cited tracklet-based continual works; the main risk is the unquantified PL fidelity rather than prior art. The abstract–body mismatch on closed-loop/NeuroNCAP looks like residual text from a longer draft and should be cleaned before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is that this is the first coherent end-to-end class-incremental motion forecasting setup from cameras, with a concrete pipeline that actually works on nuScenes and Argoverse 2. They formalize the open-world setting (labels only for the current class subset, raw multi-view images, no full re-annotation), then ship OMEN: future-frame 3D detections as motion pseudo-labels, Grounded-SAM-2 keypoint majority filter, and motion-query variance sequence replay. That combination is new at the task+pipeline level even though the pieces are known.\n\nWhat they do well: multi-split experiments (per-class and group on nuScenes, 20-2 on AV2), joint-training upper bound, forgetting and two pseudo-labeling baselines, adapted CL-DETR, replay-selection ablations (variance vs random/DINOv3/CL-DETR), buffer-size study, open-loop planning extension that improves L2/collision as classes arrive, and a zero-shot real-vehicle demo. Tables I–III show clear retention on early classes (especially linear/non-linear) and adaptation; the variance buffer preferentially keeps moving scenes, which is the right inductive bias for forecasting. Code is promised. Math is standard DETR-style matching plus the two simple equations; citations cover the right prior work and correctly claim the gap.\n\nSoft spots, in proportion: the stress-test concern lands. They never report precision/recall/calibration of the VLM-filtered future-detection PLs against original GT, so we cannot cleanly separate how much retention comes from high-quality PLs versus the small replay buffer. Confidence inflation is acknowledged, unmatched proposals get zero loss weight, and they do not force old classes to disappear. That weakens the strongest “mitigates forgetting” claim a bit, but the ablations still show each piece helps and the gap to joint training is not catastrophic. No main-table error bars / multi-seed numbers is a minor reporting gap. Dependence on Grounded SAM 2 quality is real but disclosed.\n\nThis is for people working on continual robot perception or end-to-end AV stacks who need a practical way to add classes without full re-labeling. It deserves a serious referee; the contribution is accept-shaped once code lands and they quantify PL fidelity. I would engage.","headline":"Solid first systems paper on class-incremental end-to-end camera forecasting; the task framing and pipeline are real, the PL-fidelity gap is real but not fatal.","tokens_in":21802,"tokens_out":576,"would_cite":true,"duration_ms":5736,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A motion forecaster can learn new road-agent classes over time from cameras without forgetting old ones.","keywords":["class-incremental learning","motion forecasting","end-to-end autonomous driving","catastrophic forgetting","pseudo-labeling","experience replay","open-world perception","camera-based trajectory prediction"],"falsifier":"Run the full per-class incremental protocol on nuScenes with the VLM filter disabled or replaced by a weaker detector; if mAP_f and non-linear AP_f on the first classes (car, pedestrian) collapse toward the forgetting baseline while joint-training remains high, the pseudo-label quality claim fails.","tokens_in":21715,"feed_emoji":"🚗","tokens_out":682,"duration_ms":5215,"temperature":0.7,"pith_summary":"Autonomous vehicles need to predict where other agents will go, but real roads keep introducing new object types—scooters, new vehicles—while perception is imperfect and full re-annotation of past data is impractical. This paper defines class-incremental (open-world) motion forecasting: the model must predict future trajectories directly from multi-view camera images while new semantic classes arrive in successive steps, with labels only for the current subset. The authors introduce OMEN, which generates motion pseudo-labels for previously learned classes from the old model’s future detections, filters false positives by matching 3D keypoints to open-vocabulary 2D masks, and replays past sequences chosen by high variance in motion-query features. On nuScenes and Argoverse 2 the method retains accuracy on early classes, adapts to new ones, transfers zero-shot to real driving, and extends to class-incremental planning. A sympathetic reader cares because this removes the closed-world barrier that forces full re-training whenever the object taxonomy grows.","feed_headline":"Cars learn new road agents without forgetting old ones","feed_subtitle":"Camera-only forecaster adds classes over time via filtered pseudo-labels and smart replay","key_machinery":"VLM-guided pseudo-label generation plus variance-based sequence replay: the previous model’s detections at future frames become motion pseudo-labels; they are kept only if a majority of projected 3D keypoints fall inside open-vocabulary 2D instance masks of the same class; past sequences with highest sum-of-squared deviations of motion queries from the class mean are stored in a fixed-size replay buffer.","core_discovery":"OMEN is the first end-to-end class-incremental motion forecasting framework: when new classes are introduced sequentially with labels only for the current subset, it preserves forecasting performance on previously learned classes while adapting to novel ones by combining future-detection pseudo-labels, VLM-based 3D-to-2D keypoint filtering, and variance-based sequence replay.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["OMEN learns new road classes without forgetting old ones","First end-to-end forecaster adds agents over time via replay","Camera-only motion model adapts to novel classes sequentially","Class-incremental forecasting preserves prior agents with filtering","Forecaster expands taxonomy while retaining known trajectories"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That pseudo-labels built from the previous model’s future detections and filtered by open-vocabulary masks are accurate and calibrated enough to stand in for ground truth on all old classes without ever seeing the original labeled data again.","fun_headline_variants_meta":{"raw":{"variants":["OMEN learns new road classes without forgetting old ones","First end-to-end forecaster adds agents over time via replay","Camera-only motion model adapts to novel classes sequentially","Class-incremental forecasting preserves prior agents with filtering","Forecaster expands taxonomy while retaining known trajectories"]},"model":"grok-4.5","effort":"low","cost_usd":0.005304,"raw_usage":{"total_tokens":1480,"prompt_tokens":798,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":53040000,"prompt_tokens_details":{"text_tokens":798,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":617,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":798,"tokens_out":65,"duration_ms":4887,"temperature":1.0,"reasoning_tokens":617,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-15T00:15:17.850082+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the full per-class incremental protocol on nuScenes with the VLM filter disabled or replaced by a weaker detector; if mAP_f and non-linear AP_f on the first classes (car, pedestrian) collapse toward the forgetting baseline while joint-training remains high, the pseudo-label quality claim fails.","supporting_citations":[],"review_version":1}