{"id":"8bebe831-ec29-4969-abad-c331884fef69","arxiv_id":"2411.18850","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A two-stage camera-LiDAR 3D tracker that generates coarse dual-stream trajectories and then refines them via cross-modal correction, achieving state-of-the-art KITTI results.","lead":"CrossTracker is a two-stage 3D multi-object tracking system for self-driving cars that first builds rough camera and LiDAR tracks, then refines them by correcting each sensor's mistakes using the other. It reports top results on the KITTI benchmark, beating 18 existing trackers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ablation in Table IV conflates adding the camera stream with the cross-correction steps, so the claimed source of the HOTA gains is not isolated.","rationale":"The reader's weakest assumption is that 2D-IoU overlap reliably indicates the same physical object during cross correction. That is a plausible robustness concern, but it is secondary: even if 2D-IoU pairing sometimes fails, the overall ablation and comparison with EagerMOT suggest the pipeline works on average. The more load-bearing issue is that the paper's own ablation does not control for the presence of the camera stream. The first row of Table IV uses only LiDAR, while all dual-stream rows also include the camera stream in C-TG and in the final output-matching step. Since the final output selects LiDAR trajectories that match any camera set, a dual-stream configuration with TR completely disabled is a materially different baseline from the LiDAR-only row. Without that baseline, the +5.70 HOTA improvement cannot be attributed specifically to the cross-correction cases. This directly bears on the strongest claim that the two-stage TR module, not just the multi-modal feature module, produces the reported gains. The fair comparison with EagerMOT in Table I provides some support for the two-stage advantage, so the paper should not be rejected outright; but the missing control should be supplied before the causal attribution is accepted. This is why the verdict remains CONDITIONAL and the reader's verdict does not change, though the justification is sharper: an additional ablation is required, not merely disclosure of thresholds or error bars.","tokens_in":17664,"tokens_out":10878,"duration_ms":101510,"concrete_test":"Run the Table IV validation-split experiment under a Dual Stream / No TR condition: both camera and LiDAR C-TG active, all five cross-correction cases disabled, final output generated by matching updated T_l_t against T_c_t, UD_c_t, and UT_c_t as described in Section III-D. Compare HOTA and IDSW to the LiDAR-only row (78.08 HOTA on Car) and the full-TR row (83.78 HOTA). If the no-TR dual-stream HOTA is close to 83.78, the TR steps are not the primary source of improvement; if it is close to 78.08, the two-stage attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the trajectory refinement (TR) cross-correction, not merely multi-modal feature fusion, drives the reported improvements. Table IV is offered as evidence: the LiDAR-stream row gives 78.08 HOTA on Car, and enabling cases (a)-(e) in dual-stream mode reaches 83.78. But the control is missing. The first row is single-stream (LiDAR only); every subsequent row is dual-stream (camera + LiDAR) with at least one cross-correction case enabled. There is no dual-stream condition with TR disabled. This matters because the final output already selects LiDAR trajectories by matching them against camera trajectories (Section III-D, final paragraph), so simply adding the camera stream to C-TG and to output gating can change HOTA independently of the five cross-correction cases. The reported +5.70 HOTA therefore conflates 'use camera information' with 'apply cross-correction'. The paper's attribution of the improvement to the two-stage refinement is not established by the provided ablation.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CrossTracker proposes a two-stage online multi-modal 3D multi-object tracking pipeline for the KITTI benchmark. The first stage (C-TG) generates coarse camera and LiDAR trajectories using a multi-modal modeling (M3) module that fuses image, point-cloud, and plane-geometry features to estimate inter-object consistency probabilities. The second stage (TR) refines these coarse trajectories through five cross-correction cases that address false detections, missed detections in one modality, and missed detections in both modalities. On the KITTI test set, the method outperforms EagerMOT under identical detections and also surpasses CasTrack when sharing the same LiDAR detections, with reported HOTA gains of 2-4 points on the Car category and 5+ points on the Pedestrian category. Ablations on the validation set show monotonic HOTA improvements as cross-correction cases are enabled sequentially.","tokens_in":17899,"tokens_out":4943,"duration_ms":42239,"significance":"If the claimed results hold, the paper makes a useful contribution by identifying a concrete architectural limitation of existing single-stage sensor-fusion trackers and by proposing a modular two-stage design with cross correction. The comparison against EagerMOT using identical detections is a well-controlled benchmark, and the M3 classifier is evaluated on a held-out validation split. The method is compatible with several off-the-shelf detectors, which increases its practical relevance. However, the central attribution of the reported gains to the trajectory-refinement stage is not currently established by the ablation design, because the key control condition is missing.","major_comments":[{"comment":"The ablation does not include the critical control: a dual-stream run with all cross-correction cases disabled. The first row is single-stream (LiDAR only), while every dual-stream row enables at least case (a). The +5.70 HOTA improvement on Car between the first and last rows therefore conflates the addition of the camera stream (which affects the C-TG association and the final output selection by matching T^l_t against T^c_t, UD^c_t, and UT^c_t in Section III-D) with the effect of the TR cross-correction. To support the paper's central claim that trajectory refinement drives the gains, please add a dual-stream no-TR baseline and, if possible, enable each cross-correction case independently from that baseline.","section":"Section IV-F2, Table IV"},{"comment":"The cross-correction logic relies on the assumptions that 2D-IoU overlap between a LiDAR detection and a camera trajectory, or between two predicted trajectories, indicates the same physical object, and that objects or trajectories unmatched for N consecutive frames are false detections or terminated tracks. These assumptions are not validated quantitatively; no appearance check is used in the greedy pairing, and the failure case in Fig. 7 is only a qualitative example. The paper should report how often cross-correction introduces false trajectories or revives already-terminated ones, for instance by comparing the current 2D-IoU-only pairing with a variant that also uses the M3 classifier in the cross-correction steps.","section":"Section III-D, Steps 1-3"},{"comment":"All reported numbers are single runs on one validation split, and the thresholds θS, θG, θiou, θhits, and N are selected on the same split. No sensitivity analysis or variance over runs is provided. Given that some incremental gains are small (e.g., 0.17% HOTA for case (d) in Table IV, and a 0.01% HOTA decrease for case (e) on Pedestrian), the reader cannot judge whether these differences are meaningful or artifacts of threshold choices. At minimum, please report the chosen hyperparameter values and a sensitivity study for the most sensitive thresholds, especially θiou and N.","section":"Section IV-D and IV-F"}],"minor_comments":[{"comment":"The heading 'LIMILATION' should be 'LIMITATION', and in the second paragraph 'HOA' should be 'HOTA'.","section":"Section V"},{"comment":"The column header 'IDWS' should be 'IDSW' to match the standard metric name and the text in Section IV-C.","section":"Table I"},{"comment":"References [33] and [34] appear to be the same paper (same title, authors, and page range) and should be merged or one should be removed.","section":"References"},{"comment":"The text contains a LaTeX artifact 'textitCar' in the discussion of Table IV; it should simply read 'Car'.","section":"Section IV-F2"},{"comment":"The phrase 'using Cross Entropy Loss' should be 'using a cross-entropy loss' for grammatical correctness.","section":"Section III-B4"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the missing dual-stream no-TR control in Table IV, which is load-bearing for the claimed contribution. This is fixable within a revision by adding the missing baseline and, ideally, non-cumulative ablations. The paper is otherwise competently written; the EagerMOT comparison with identical detections and the held-out validation of the M3 classifier are notable strengths."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this if you work on 3D MOT. The two-stage idea — generate coarse dual-stream trajectories, then refine via cross correction between camera and LiDAR — is a real organizational novelty, not just a new fusion module. Existing single-stage fusion trackers correct LiDAR with camera, but don't reciprocally fix the camera stream. CrossTracker does, and the M3 module (image + plane geometry + point cloud features for consistency) looks sound.\n\nThe paper also does some things right. The comparison against EagerMOT uses identical detections, which is fair and useful. Table V shows the method works across several detector combinations, so the gains aren't detector-specific. The five cross-correction cases are well-motivated and the incremental ablations in Table IV show each case adds something, with case (e) even acknowledged to slightly hurt pedestrian HOTA.\n\nThe soft spot is real, though. Table IV jumps from a LiDAR-only stream straight to dual-stream with cross-correction cases enabled. There's no dual-stream condition with TR disabled. Since the final output already selects LiDAR trajectories by matching against camera trajectories, simply adding the camera stream — without any of the five correction cases — could plausibly account for most of the +5.7 HOTA jump. The incremental rows after that do isolate the cases somewhat, but the central claim that the TR refinement, not merely camera information, drives the result isn't pinned down by the provided ablation. I'd want to see the missing baseline before accepting that attribution.\n\nAlso, the hyperparameters θS, θG, θiou, θhits, and N are never reported, no code is available, and there are no error bars. None of these sink the paper, but they'd need to be fixed in review. The 2D-IoU pairing without appearance verification is a reasonable worry in occlusion-heavy scenes, but the paper doesn't show it failing in practice beyond the acknowledged boundary case.\n\nVerdict: worth a serious referee. The idea is clear, the experiments are extensive, and the cost of the missing baseline is a revision, not a rejection. I'd cite this for the two-stage architecture if I were working on multi-modal tracking.","headline":"A genuinely new two-stage design for multi-modal 3D MOT, but the headline ablation misses a control that blurs the source of the gains.","tokens_in":18397,"tokens_out":3062,"would_cite":true,"duration_ms":26487,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CrossTracker claims the first online two-stage 3D tracker where camera and LiDAR streams correct each other's failures.","keywords":["3D multi-object tracking","camera-LiDAR fusion","two-stage tracking","cross correction","trajectory refinement","KITTI benchmark","online tracking"],"falsifier":"Re-run CrossTracker on the KITTI validation split with the camera–LiDAR extrinsic calibration perturbed by a few degrees: if HOTA drops sharply, the 2D-IoU cross-modal pairing is the sensitive link. A second check is to count wrongly initialized trajectories in scenes where a car and a pedestrian project to strongly overlapping 2D boxes.","tokens_in":17466,"feed_emoji":"🚗","tokens_out":5550,"duration_ms":45218,"temperature":0.7,"pith_summary":"This paper claims that reliable camera–LiDAR 3D multi-object tracking needs a dedicated refinement stage in which the two sensor streams correct each other, not just a fused feature metric. Existing fusion trackers use a single-stage architecture that only lets camera detections patch gaps in LiDAR tracks, leaving camera-only misses and simultaneous double misses uncorrected. CrossTracker instead generates coarse camera and LiDAR trajectories in a first stage, then a trajectory-refinement module pairs unmatched detections and unmatched tracks across the two streams using 2D IoU, a hit-count requirement, and an image-boundary check. On the KITTI tracking benchmark the method reports higher HOTA, MOTA, and lower ID switch counts than eighteen comparison methods, including gains of roughly four HOTA points over the same-detector EagerMOT baseline.","feed_headline":"Camera and LiDAR tracks correct each other in two stages","feed_subtitle":"Two-stage refinement beats all 18 comparison methods on KITTI 3D multi-object tracking","key_machinery":"The load-bearing mechanism is the trajectory refinement (TR) module's five-case cross-correction procedure. The five cases are: new objects identified from unmatched LiDAR detections using camera tracks as reference; new objects from unmatched camera detections; recovery of unmatched LiDAR trajectories using camera tracks; recovery of unmatched camera trajectories using LiDAR tracks; and recovery when both modalities simultaneously miss an object, gated by an image-boundary check. Each pairing is done with a greedy 2D-IoU association at threshold $\\theta_{iou}$, a minimum-hit requirement $\\theta_{hits}$, and in the last case a boundary test, with unmatched items discarded after $N$ consecutive unobserved frames. The M3 network supplies the learned consistency probabilities used in stage one, but the cross-correction steps in stage two are geometric, which is the paper's main architectural departure.","core_discovery":"The paper's central claim is that the two-stage coarse-to-fine scheme is what unlocks multi-modal robustness: stage one (C-TG) builds separate coarse trajectory sets for camera and LiDAR, and stage two (TR) performs cross correction, transferring newly appearing objects from unmatched detections into live tracks and reviving unmatched trajectories that the other modality still sees. The authors argue this bidirectional correction is exactly what distinguishes their method from prior single-stage fusion trackers, which only correct LiDAR failures using camera data. To support the scheme, the M3 module learns object-pair consistency probabilities from image patches, point-cloud patches, and pseudo-point patches sampled inside 2D boxes (plane geometry), which then feed the stage-one association cost along with spatial geometric constraints such as 2D IoU and 3D centroid distance. The reported results on KITTI show the full pipeline, with all five cross-correction cases active, outperforming all eighteen competitors, and the ablation attributes the largest single gains to cases (a), (c), and (e) — new-object discovery from unmatched LiDAR detections, camera-referenced recovery of unmatched LiDAR trajectories, and recovery when both modalities miss.","pith_inferences":["This paper's own ablation shows case (e) slightly reduces pedestrian HOTA, which suggests the boundary-check rule could be made class-dependent rather than global; that is an extension the authors did not explore.","If the central claim is right, the 2D-IoU pairing rule creates a direct sensitivity to extrinsic calibration and to small, distant objects whose projected boxes barely overlap; a natural test is to measure HOTA versus calibration error.","The same cross-correction logic could be extended to multi-camera setups by treating each camera as one stream and LiDAR as another, which would generalize the five cases to N-way correction.","The separation of M3 and TR means the learned metric could be swapped for a cheaper appearance model in resource-constrained settings, isolating the contribution of the two-stage architecture."],"forward_implications":["The reported improvements over EagerMOT with identical detectors imply that adding a second refinement stage to an existing single-stage fusion tracker can recover most of the gains without changing detectors.","The gap in ID switch counts (82 vs 239 on cars with the same detectors) suggests that cross correction chiefly stabilizes identity maintenance, which would matter for downstream prediction and planning.","Because the TR module is geometric and detector-agnostic, the two-stage scheme should transfer to other detection pairs, such as radar and camera, in the same tracking-by-detection framework.","The ablation numbers indicate that the largest contributions come from the new-object and single-modality-recovery cases, so future work can prioritize those steps when computation is limited."],"supporting_citations":[{"why":"Baseline single-stage fusion tracker whose camera-to-LiDAR correction CrossTracker extends; same detectors used in the main comparison.","marker":"[8]"},{"why":"DeepFusionMOT, an example of single-stage camera-LiDAR fusion that only corrects LiDAR failures.","marker":"[9]"},{"why":"StrongFusionMOT, another single-stage fusion baseline representing the unidirectional correction approach.","marker":"[10]"},{"why":"AB3DMOT supplies the 3D Kalman-filter tracking baseline and the 3D centroid-distance spatial constraint used in stage one.","marker":"[4]"},{"why":"Faster R-CNN provides the two-stage detector analogy that motivates the coarse-to-fine tracking architecture.","marker":"[12]"},{"why":"CasTrack, the strongest LiDAR-only competitor, which CrossTracker improves on when using the same 3D detections.","marker":"[41]"}],"fun_headline_variants":["Two-stage cross correction outperforms 18 trackers","Cross correction revives lost tracks in 3D tracking","Both sensors fix each other's tracking gaps","Coarse-to-fine cross correction for robust 3D tracking","Reviving lost tracks with cross-modal correction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole cross-correction hinges on 2D-IoU overlap between a LiDAR detection's projected box and a camera trajectory (or between two predicted trajectories) reliably indicating the same physical object, and on the rule that an object unmatched for N consecutive frames is a false detection or a terminated track.","fun_headline_variants_meta":{"raw":{"variants":["Two-stage cross correction outperforms 18 trackers","Cross correction revives lost tracks in 3D tracking","Both sensors fix each other's tracking gaps","Coarse-to-fine cross correction for robust 3D tracking","Reviving lost tracks with cross-modal correction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001295,"raw_usage":{"total_tokens":5330,"prompt_tokens":1035,"completion_tokens":4295,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":4219}},"tokens_in":651,"tokens_out":4295,"duration_ms":25629,"temperature":1.0,"reasoning_tokens":4219,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:48:34.829769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run CrossTracker on the KITTI validation split with the camera–LiDAR extrinsic calibration perturbed by a few degrees: if HOTA drops sharply, the 2D-IoU cross-modal pairing is the sensitive link. A second check is to count wrongly initialized trajectories in scenes where a car and a pedestrian project to strongly overlapping 2D boxes.","supporting_citations":[{"cited_title":"Eagermot: 3d multi-object track- ing via sensor fusion,","cited_arxiv_id":null,"evidence_quote":"Baseline single-stage fusion tracker whose camera-to-LiDAR correction CrossTracker extends; same detectors used in the main comparison."},{"cited_title":"Deepfusionmot: A 3d multi-object tracking framework based on camera-lidar fusion with deep association,","cited_arxiv_id":null,"evidence_quote":"DeepFusionMOT, an example of single-stage camera-LiDAR fusion that only corrects LiDAR failures."},{"cited_title":"Strongfusionmot: A multi-object tracking method based on lidar-camera fusion,","cited_arxiv_id":null,"evidence_quote":"StrongFusionMOT, another single-stage fusion baseline representing the unidirectional correction approach."},{"cited_title":"Faster R-CNN: towards real-time object detection with region proposal networks,","cited_arxiv_id":null,"evidence_quote":"Faster R-CNN provides the two-stage detector analogy that motivates the coarse-to-fine tracking architecture."},{"cited_title":"3d multi-object tracking in point clouds based on prediction confidence-guided data association,","cited_arxiv_id":null,"evidence_quote":"CasTrack, the strongest LiDAR-only competitor, which CrossTracker improves on when using the same 3D detections."}],"review_version":1}