{"id":"f699e35a-6016-4252-a404-c02ffdef00c1","arxiv_id":"1908.04349","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A tracking-by-detection system that runs a detector ensemble only every f frames and reports the fastest speed among compared online MOT16 entries, at reduced MOTA.","lead":"This paper combines two object detectors that run only every few frames, plus a Kalman-filter tracker, to track people faster than most MOT16 benchmark entries. The practical appeal is that skipping frames keeps the system real-time while still tracking with acceptable accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I's MOTA values are internally inconsistent with their FP/FN/IDSw counts and with the SORT row; f=5/f=10 appear evaluated only on sampled frames, so the accuracy claim is not established.","rationale":"I read the paper as a tracking-by-detection system whose headline is a speed/accuracy trade-off. For that headline, both the speed and accuracy metrics must be comparable to the comparison set. The speed metric is explicitly tracker-only, and the authors concede detector time separately; that is recoverable with a full end-to-end measurement. The accuracy inconsistency is not recoverable because the published table is self-contradictory. The GT ratios are not near-equal; they are proportional to the sampling interval f, which is strong evidence of a sampled-frame evaluation. A reader could not reproduce the claimed MOTA from the table alone. A metric-configuration error in the evaluation script is plausible, but as published the numbers cannot support the claim of surpassing other online entries in speed while maintaining acceptable accuracy. The correct response is to reject or require a corrected evaluation before any such claim is made. I am not accusing the authors of dishonesty; an evaluation-script bug is a common failure mode, but it is fatal to the current claim as stated.","tokens_in":7278,"tokens_out":7333,"duration_ms":80166,"concrete_test":"Using the official MOT16 ground-truth count (inferable from the SORT row, ≈61,500 boxes), recompute MOTA for each row from its own FP/FN/IDSw totals. If OURS(f=1) does not equal 43.7 (it cannot: errors ≈62,200 > GT) and if f=5/f=10 require GT denominators of ≈22,100/11,200, the table's accuracy numbers are inconsistent. Then rerun the tracker on MOT16 evaluating all frames, including Kalman-propagated detections in skipped frames; if the resulting MOTA at f=5/f=10 is materially lower than Table I, the headline accuracy claim fails.","verdict_should_be":"REJECT","load_bearing_attack":"Central claim requires Table I's accuracy numbers to be comparable to official MOT16 scores, but they are not. Using the standard MOTA formula, MOTA = 1 − (FP+FN+IDSw)/GT, the rows imply mutually inconsistent GT counts: f=1 implies GT ≈ 110,400; f=5 implies GT ≈ 22,100; f=10 implies GT ≈ 11,200; the SORT row implies GT ≈ 61,500. The f=1 row is therefore arithmetically impossible on the official MOT16 GT: its errors of 62,169 exceed the SORT-implied GT, giving a negative MOTA rather than 43.7. The f=5 and f=10 rows are consistent with evaluating only every 5th/10th frame, exactly the frames where the detector ensemble runs, rather than every output frame. Thus the accuracy side of the central claim is not established: 'acceptable accuracy' at high speeds appears to be measured under a different, undisclosed evaluation protocol, and the comparison to other entries is invalid.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-object tracking system based on tracking-by-detection, in which an ensemble of two detectors (YOLOv3 and Lighthead R-CNN) runs only every f frames, and the resulting detections are combined via a soft-NMS variant before being fed to a Kalman-filter-based tracker with Mahalanobis-distance assignment. The authors report results on the MOT16 benchmark for f = 1, 5, and 10, claiming the system achieves higher speed than other online entries while maintaining acceptable accuracy. The paper also includes an ablation of the ensemble versus individual detectors and a comparison with several published MOT16 results.","tokens_in":7484,"tokens_out":6354,"duration_ms":67040,"significance":"If the empirical claims were correct, the contribution would be practically relevant: running the detector ensemble only every f frames is a simple and potentially effective way to reduce the computational bottleneck of tracking-by-detection, and the reported tracker-only speeds are very high. The paper is, however, purely empirical, with no theoretical derivation and no released code, so its value rests entirely on the soundness and reproducibility of the evaluation. The reported accuracy numbers are internally inconsistent and appear to have been computed under a different evaluation protocol from the comparison methods, which currently undermines the main claim.","major_comments":[{"comment":"The rows for OURS in Table I are internally inconsistent when checked against the standard MOTA formula MOTA = 1 - (FP + FN + IDSw) / GT. For f = 1, the error sum is 62,169; since the official MOT16 ground-truth count implied by the SORT row included in the same table is about 61,000, this sum already exceeds the total number of ground-truth boxes, making the reported MOTA of 43.7% arithmetically impossible. For f = 5 and f = 10, the error sums imply ground-truth counts of roughly 22,000 and 11,000, respectively, which is consistent with evaluating only on every 5th or 10th frame rather than on the full test sequences. If that is the case, the f = 5 and f = 10 MOTA values are not comparable to the official MOT16 scores of the other methods, and the claim of 'acceptable accuracy' at high speed is not established. The authors must report the exact evaluation protocol for each row, including the number of ground-truth boxes used, and recompute all OURS rows with the official evaluation code on the full ground-truth sequences.","section":"Section III, Table I"},{"comment":"The headline speeds of 249.6, 1,431.5, and 3,000.1 Hz are explicitly defined as 'processing speed in frames per second excluding the detector.' The paper states in the final paragraph of Section III that the application 'was able to run with real-time performance using f = 5,' but no end-to-end wall-clock time for the complete pipeline (detector ensemble plus tracker) is reported. Without this measurement, the real-time claim is unverifiable, and the comparison with other entries is ambiguous because it is unclear whether the published speeds of the other methods include detection time. Please report measured end-to-end runtimes for at least f = 1, f = 5, and f = 10, and clearly label which values in Table I and Fig. 3 include detection.","section":"Section III, Speed definition"},{"comment":"All reported results appear to come from a single run, with no error bars, no multiple seeds, and no sensitivity analysis for the free parameters (beta, Max.Age, detector confidence thresholds). The ablation in Fig. 4 is also based on one run per configuration. Since the central claim is empirical, the robustness of the claimed accuracy and speed values cannot be assessed. The authors should provide variance over several runs or, failing that, a sensitivity analysis over the main parameters, and state how the parameters were selected (e.g., on a validation split).","section":"Section III, Evaluation methodology"}],"minor_comments":[{"comment":"The caption states that all compared methods 'use their own detections' and are online, but the SORT row appears to be the published result from the original SORT paper, which uses public detections. Please clarify which rows are from the authors' own runs and which are taken from the literature, and specify the detection source for every method.","section":"Section III, Table I caption"},{"comment":"References [21] and [22] appear to be mismatched with the methods CNNKCF and FMOT BL; for example, [21] cites a 1999 paper on tracking with multiple cameras, which does not correspond to the cited CNNKCF method. Please verify and correct these citations.","section":"References"},{"comment":"The notation p(˜Z1:t|X) uses a tilde over Z that is not defined; please define it or replace it with a consistent notation such as p(Z_1:t|X).","section":"Section II, Equation (2)"},{"comment":"The figure legend shows the ensemble and two individual detectors, but the underlying MOTA values are not given in a table, and the y-axis title 'Accuracy (MOTA)' omits units and error bars; please include the numerical values and, if possible, variability across runs.","section":"Section III, Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The errors in Table I are severe enough that the paper's central claim is currently unsupported. The arithmetic inconsistency should be caught by any careful reader, and the authors need to re-run their evaluation on the full ground-truth sequences for all f values and report end-to-end runtime. If the re-evaluation shows that the accuracy numbers were computed only on detection frames, the comparison to official MOT16 scores would be invalid, and the paper would need substantial rewriting. The paper is within scope for the venue, but the current evaluation is not acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this is a well-intentioned engineering paper that builds a fast MOT system by running an ensemble of YOLOv3 and Lighthead R-CNN every f frames, combining detections with a soft-NMS variant, and feeding them into a SORT-like Kalman/assignment tracker. The idea is simple, the write-up is mostly clear, and the package is plausible for retail video analytics. But the evaluation as presented does not support the headline claim of acceptable accuracy at speeds of 1,400–3,000 Hz.\n\nThe new bit is modest: the frame-skipping detector ensemble is a natural extension of known components, not a new formulation. The gain is speed at the cost of accuracy, which is exactly what you’d expect. The paper does a reasonable job describing the algorithm, and the authors correctly identify detection as the bottleneck.\n\nThe soft spots are in the numbers, and they’re serious. The FP/FN/IDSw counts and MOTA values in Table I are arithmetically inconsistent across rows under the standard MOTA formula. The SORT row implies a GT count around 61k, while the OURS f=1 row implies about 110k. Those can’t both be from the same evaluation set, so the comparison is not apples-to-apples. More importantly, the f=5 and f=10 rows have error sums that scale roughly proportionally to 1/f — exactly what you’d see if the evaluation was run only on the frames where the detector executes, not on every output frame. That means the reported MOTA values for the fast settings are not comparable to official MOT16 evaluations, and the accuracy half of the central claim is not established. The speed numbers also exclude detector time; the authors mention end-to-end real-time at f=5 but never give that end-to-end number.\n\nAlso missing: no code, no error bars, no sensitivity to the beta/free parameters. On a positive note, the mathematical formulation is standard and not the issue; the paper is readable and the method is reproducible in principle.\n\nWho gets value from this? Practitioners looking for a recipe for a fast tracker might try the frame-skipping ensemble and see if it works for them. But as a scientific claim about MOT accuracy, the current evidence doesn’t hold.\n\nI’d send it to a serious referee — the flaws are fixable with a re-evaluation on all frames, an end-to-end speed measurement, and a clean comparison — but I would not accept it as is.","headline":"A simple frame-skipping ensemble gives impressive tracker-only speeds, but the accuracy evaluation in Table I is internally inconsistent and likely not comparable to official MOT16 numbers; the central 'acceptable accuracy' claim is not established.","tokens_in":7961,"tokens_out":7184,"would_cite":false,"duration_ms":76282,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-detector ensemble that runs only every f frames lets a Kalman-filter tracker reach 249.6–3,000.1 Hz on MOT16 while keeping acceptable accuracy.","keywords":["multi-object tracking","detector ensemble","tracking-by-detection","Kalman filter","Soft-NMS","MOT16","real-time tracking","Mahalanobis distance"],"falsifier":"Measure the full pipeline end-to-end—detector ensemble plus tracker, including I/O—on MOT16 with the authors' hardware and settings; if the f=5 configuration does not sustain real-time throughput, or if the reported Hz change materially when detector time is included, the paper's central speed claim fails.","tokens_in":7085,"feed_emoji":"⚡","tokens_out":5506,"duration_ms":53600,"temperature":0.7,"pith_summary":"The paper proposes a tracking-by-detection system whose central trick is to run an ensemble of object detectors only once every f frames rather than every frame, then fuse their outputs with a Soft-NMS variant and feed the fused detections to a Kalman-filter tracker. The authors claim that this sparse-detection schedule turns the detector from a bottleneck into a tunable cost: speed grows roughly linearly with f while accuracy degrades gracefully, so the system can beat every other online MOT16 entry on speed while keeping accuracy acceptable for applications such as people counting. They report tracker-only speeds of 249.6 Hz at f=1, 1,431.5 Hz at f=5, and 3,000.1 Hz at f=10, with MOTA falling from 43.7 to 34.3 to 28.3. A sympathetic reader should care because it reframes real-time MOT as a sampling-rate problem: choose f for the latency your hardware and application allow, and the same pipeline serves online and offline use.","feed_headline":"Skipping detector frames yields 1,431 Hz tracking on MOT16","feed_subtitle":"A two-detector ensemble plus Kalman gating keeps MOTA near 34 while beating every online entry on speed.","key_machinery":"The load-bearing mechanism is the f-frame detector schedule combined with an ensemble fusion rule: detections from m detectors are merged by repeatedly taking the highest-confidence box and down-weighting overlapping boxes with g = exp(-β IoU), a Soft-NMS variant; fused detections feed a Kalman-filter tracker whose assignment cost is the Mahalanobis distance between predicted track state and detection, with Σ taken from the filter. This is what lets the system skip detector computation for f-1 frames while trusting motion continuity to keep identities.","core_discovery":"The central discovery is that a detector ensemble scheduled every f frames—two independently trained detectors, YOLOv3 and Lighthead R-CNN, fused by an exponential-decay Soft-NMS variant—permits a deliberately simple online tracker (Kalman filters plus a {0,1}-assignment problem with Mahalanobis distance) to run orders of magnitude faster than competing online entries on MOT16 with only moderate accuracy loss. The paper argues that the ensemble reduces false negatives enough to relax the tracking subproblem, replacing the need for complex appearance models with cheap motion-based association. The contribution is an extension of SORT: the same fast assignment machinery, but with the Euclidean/overlap distance replaced by a Mahalanobis distance that carries Kalman uncertainty, and with detections arriving every f frames. This is presented as a tunable speed-accuracy tradeoff rather than a fixed architecture.","pith_inferences":["The reported Hz exclude detector time; a fair cross-system comparison would need an end-to-end runtime, and the real-time claim at f=5 is conditional on the detector ensemble cost being amortized over five frames.","The same sparse-detection schedule could be tested with a learned motion model or a re-identification module: if the Kalman model is the bottleneck at large f, replacing it should push the accuracy-speed frontier further than the paper's measured curve.","Because the ensemble fusion is detector-agnostic, one testable extension is to swap in newer one-stage detectors and measure whether the MOTA-versus-f curve shifts upward without changing the tracking code.","Applying the same system to a dense-crowd sequence with fast motion would stress whether Mahalanobis gating alone can bridge f-1 skipped frames."],"forward_implications":["Running the detector ensemble every f frames makes tracker speed grow roughly linearly with f, letting users trade accuracy for throughput on the same hardware.","At f=5 the full pipeline (detector plus tracker) runs in real time on the authors' hardware, and at f=10 it becomes suitable for offline batch processing at thousands of frames per second.","The ensemble's false-negative reduction improves MOTA by roughly 7.8 percentage points at f=1 and 5.5 at f=5 over the best single detector, so detector diversity is doing real work.","Because the tracker uses only motion and no appearance re-identification, its speed comes with elevated identity switches; applications tolerant of ID switches, such as counting, are the natural fit."],"supporting_citations":[{"why":"Supplies the SORT tracking algorithm—Kalman-filter state prediction and {0,1} assignment—that this system extends with sparse detections and Mahalanobis costs.","marker":"[1]"},{"why":"YOLOv3 is one of the two ensemble detectors, providing one-stage detections.","marker":"[10]"},{"why":"Lighthead R-CNN is the other ensemble detector, providing two-stage detections.","marker":"[25]"},{"why":"Soft-NMS is the fusion rule the authors vary with exponential IoU decay to combine detector outputs.","marker":"[14]"},{"why":"MOT16 is the benchmark whose sequences and metrics ground all reported MOTA, IDF1, and speed comparisons.","marker":"[11]"},{"why":"Provides the ensemble-accuracy justification that diversity among models improves combined prediction.","marker":"[13]"},{"why":"DeepSORT is the main online accuracy-oriented baseline the speed comparison must be read against.","marker":"[18]"},{"why":"Supplies the tracking-by-detection posterior formulation that the paper's objective follows.","marker":"[12]"}],"fun_headline_variants":["Detector ensemble every f frames: 1,431 Hz MOT16 speed","Two-detector ensemble, Kalman gating: 1,431 Hz on MOT16","Skipping detector frames: 1,431 Hz MOT16 with acceptable accuracy","Cheap motion tracking with detector ensemble hits 1,431 Hz","Frame-skipped detector ensemble beats online MOT16 speed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The real-time and speed claims rest on running the detector ensemble only every f frames and on the Kalman motion model plus Mahalanobis assignment carrying tracks across skipped frames, with reported speeds excluding detector time.","fun_headline_variants_meta":{"raw":{"variants":["Detector ensemble every f frames: 1,431 Hz MOT16 speed","Two-detector ensemble, Kalman gating: 1,431 Hz on MOT16","Skipping detector frames: 1,431 Hz MOT16 with acceptable accuracy","Cheap motion tracking with detector ensemble hits 1,431 Hz","Frame-skipped detector ensemble beats online MOT16 speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001092,"raw_usage":{"total_tokens":4483,"prompt_tokens":793,"completion_tokens":3690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":409,"completion_tokens_details":{"reasoning_tokens":3591}},"tokens_in":409,"tokens_out":3690,"duration_ms":28128,"temperature":1.0,"reasoning_tokens":3591,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:45:20.944068+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the full pipeline end-to-end—detector ensemble plus tracker, including I/O—on MOT16 with the authors' hardware and settings; if the f=5 configuration does not sustain real-time throughput, or if the reported Hz change materially when detector time is included, the paper's central speed claim fails.","supporting_citations":[{"cited_title":"Simple online and realtime tracking,","cited_arxiv_id":null,"evidence_quote":"Supplies the SORT tracking algorithm—Kalman-filter state prediction and {0,1} assignment—that this system extends with sparse detections and Mahalanobis costs."},{"cited_title":"You only look once: Unified, real-time object detection,","cited_arxiv_id":null,"evidence_quote":"YOLOv3 is one of the two ensemble detectors, providing one-stage detections."},{"cited_title":"Soft-nmsimproving object detection with one line of code,","cited_arxiv_id":null,"evidence_quote":"Soft-NMS is the fusion rule the authors vary with exponential IoU decay to combine detector outputs."},{"cited_title":"Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy,","cited_arxiv_id":null,"evidence_quote":"Provides the ensemble-accuracy justification that diversity among models improves combined prediction."},{"cited_title":"Simple online and realtime tracking with a deep association metric,","cited_arxiv_id":null,"evidence_quote":"DeepSORT is the main online accuracy-oriented baseline the speed comparison must be read against."},{"cited_title":"Global data association for multi- object tracking using network flows,","cited_arxiv_id":null,"evidence_quote":"Supplies the tracking-by-detection posterior formulation that the paper's objective follows."}],"review_version":1}