REVIEW 3 major objections 4 minor 29 references
A fast multi-object tracking system using an object detector ensemble
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A two-detector ensemble that runs only every f frames lets a Kalman-filter tracker reach 249.6–3,000.1 Hz on MOT16 while keeping acceptable accuracy.
desk verdict A simple frame-skipping ensemble gives impressive tracker-only speeds, but the accuracy evaluation in Table I is internally inconsistent and likely not comparable to official MOT16 numbers; the central 'acceptable accuracy' claim is not established. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the f-frame detector schedule combined with an ensemble fusion rule: detections from m detectors are merged by repeatedly taking the highest-confidence box and down-weighting overlapping boxes with g = exp(-β IoU), a Soft-NMS variant; fused detections feed a Kalman-filter tracker whose assignment cost is the Mahalanobis distance between predicted track state and detection, with Σ taken from the filter. This is what lets the system skip detector computation for f-1 frames while trusting motion continuity to keep identities.
What would settle it
Measure the full pipeline end-to-end—detector ensemble plus tracker, including I/O—on MOT16 with the authors' hardware and settings; if the f=5 configuration does not sustain real-time throughput, or if the reported Hz change materially when detector time is included, the paper's central speed claim fails.
Extended reading notes
Core claim
The central discovery is that a detector ensemble scheduled every f frames—two independently trained detectors, YOLOv3 and Lighthead R-CNN, fused by an exponential-decay Soft-NMS variant—permits a deliberately simple online tracker (Kalman filters plus a {0,1}-assignment problem with Mahalanobis distance) to run orders of magnitude faster than competing online entries on MOT16 with only moderate accuracy loss. The paper argues that the ensemble reduces false negatives enough to relax the tracking subproblem, replacing the need for complex appearance models with cheap motion-based association. The contribution is an extension of SORT: the same fast assignment machinery, but with the Euclidean/overlap distance replaced by a Mahalanobis distance that carries Kalman uncertainty, and with detections arriving every f frames. This is presented as a tunable speed-accuracy tradeoff rather than a fixed architecture.
Load-bearing premise
The real-time and speed claims rest on running the detector ensemble only every f frames and on the Kalman motion model plus Mahalanobis assignment carrying tracks across skipped frames, with reported speeds excluding detector time.
Editorial extensions
If this is right
- Running the detector ensemble every f frames makes tracker speed grow roughly linearly with f, letting users trade accuracy for throughput on the same hardware.
- At f=5 the full pipeline (detector plus tracker) runs in real time on the authors' hardware, and at f=10 it becomes suitable for offline batch processing at thousands of frames per second.
- The ensemble's false-negative reduction improves MOTA by roughly 7.8 percentage points at f=1 and 5.5 at f=5 over the best single detector, so detector diversity is doing real work.
- Because the tracker uses only motion and no appearance re-identification, its speed comes with elevated identity switches; applications tolerant of ID switches, such as counting, are the natural fit.
Reading between the lines
- The reported Hz exclude detector time; a fair cross-system comparison would need an end-to-end runtime, and the real-time claim at f=5 is conditional on the detector ensemble cost being amortized over five frames.
- The same sparse-detection schedule could be tested with a learned motion model or a re-identification module: if the Kalman model is the bottleneck at large f, replacing it should push the accuracy-speed frontier further than the paper's measured curve.
- Because the ensemble fusion is detector-agnostic, one testable extension is to swap in newer one-stage detectors and measure whether the MOTA-versus-f curve shifts upward without changing the tracking code.
- Applying the same system to a dense-crowd sequence with fast motion would stress whether Mahalanobis gating alone can bridge f-1 skipped frames.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a multi-object tracking system based on tracking-by-detection, in which an ensemble of two detectors (YOLOv3 and Lighthead R-CNN) runs only every f frames, and the resulting detections are combined via a soft-NMS variant before being fed to a Kalman-filter-based tracker with Mahalanobis-distance assignment. The authors report results on the MOT16 benchmark for f = 1, 5, and 10, claiming the system achieves higher speed than other online entries while maintaining acceptable accuracy. The paper also includes an ablation of the ensemble versus individual detectors and a comparison with several published MOT16 results.
Significance. If the empirical claims were correct, the contribution would be practically relevant: running the detector ensemble only every f frames is a simple and potentially effective way to reduce the computational bottleneck of tracking-by-detection, and the reported tracker-only speeds are very high. The paper is, however, purely empirical, with no theoretical derivation and no released code, so its value rests entirely on the soundness and reproducibility of the evaluation. The reported accuracy numbers are internally inconsistent and appear to have been computed under a different evaluation protocol from the comparison methods, which currently undermines the main claim.
major comments (3)
- [Section III, Table I] The rows for OURS in Table I are internally inconsistent when checked against the standard MOTA formula MOTA = 1 - (FP + FN + IDSw) / GT. For f = 1, the error sum is 62,169; since the official MOT16 ground-truth count implied by the SORT row included in the same table is about 61,000, this sum already exceeds the total number of ground-truth boxes, making the reported MOTA of 43.7% arithmetically impossible. For f = 5 and f = 10, the error sums imply ground-truth counts of roughly 22,000 and 11,000, respectively, which is consistent with evaluating only on every 5th or 10th frame rather than on the full test sequences. If that is the case, the f = 5 and f = 10 MOTA values are not comparable to the official MOT16 scores of the other methods, and the claim of 'acceptable accuracy' at high speed is not established. The authors must report the exact evaluation protocol for each row, including the number of ground-truth boxes used, and recompute all OURS rows with the official evaluation code on the full ground-truth sequences.
- [Section III, Speed definition] The headline speeds of 249.6, 1,431.5, and 3,000.1 Hz are explicitly defined as 'processing speed in frames per second excluding the detector.' The paper states in the final paragraph of Section III that the application 'was able to run with real-time performance using f = 5,' but no end-to-end wall-clock time for the complete pipeline (detector ensemble plus tracker) is reported. Without this measurement, the real-time claim is unverifiable, and the comparison with other entries is ambiguous because it is unclear whether the published speeds of the other methods include detection time. Please report measured end-to-end runtimes for at least f = 1, f = 5, and f = 10, and clearly label which values in Table I and Fig. 3 include detection.
- [Section III, Evaluation methodology] All reported results appear to come from a single run, with no error bars, no multiple seeds, and no sensitivity analysis for the free parameters (beta, Max.Age, detector confidence thresholds). The ablation in Fig. 4 is also based on one run per configuration. Since the central claim is empirical, the robustness of the claimed accuracy and speed values cannot be assessed. The authors should provide variance over several runs or, failing that, a sensitivity analysis over the main parameters, and state how the parameters were selected (e.g., on a validation split).
minor comments (4)
- [Section III, Table I caption] The caption states that all compared methods 'use their own detections' and are online, but the SORT row appears to be the published result from the original SORT paper, which uses public detections. Please clarify which rows are from the authors' own runs and which are taken from the literature, and specify the detection source for every method.
- [References] References [21] and [22] appear to be mismatched with the methods CNNKCF and FMOT BL; for example, [21] cites a 1999 paper on tracking with multiple cameras, which does not correspond to the cited CNNKCF method. Please verify and correct these citations.
- [Section II, Equation (2)] The notation p(˜Z1:t|X) uses a tilde over Z that is not defined; please define it or replace it with a consistent notation such as p(Z_1:t|X).
- [Section III, Fig. 4] The figure legend shows the ensemble and two individual detectors, but the underlying MOTA values are not given in a table, and the y-axis title 'Accuracy (MOTA)' omits units and error bars; please include the numerical values and, if possible, variability across runs.
Circularity Check
No circularity: the contribution is an empirical system evaluation, not a derived prediction.
full rationale
The paper's claims are measurements: MOTA, IDF1, MT/ML, FP/FN/IDSw, and Hz values reported in Table I are presented as results of running the proposed tracker on MOT16, not as predictions derived from fitted parameters. The free parameter beta in the soft-NMS ensemble is a scaling knob of a standard post-processing rule, and the paper does not fit beta to the reported MOTA numbers nor rename a fitted quantity as a prediction. The tracking formulation (Kalman filters plus Mahalanobis-distance assignment) follows SORT and standard tracking-by-detection, with citations used as background rather than as load-bearing self-citations; no uniqueness theorem or prior-work assertion is invoked to force the result. The speeds exclude detector time, and the f=5/f=10 accuracy rows raise legitimate evaluation-protocol questions (e.g., apparent inconsistency with the official ground-truth count), but those are correctness and comparability concerns, not circularity: measured numbers are still measured numbers. No equation in the paper reduces by construction to an input, and no term is defined in terms of the quantity it is supposed to derive. The paper is therefore self-contained as an empirical benchmark report, and the appropriate circularity finding is none.
Assumptions & free parameters
free parameters (4)
- beta =
not specified
- f =
1, 5, 10
- Max.Age =
not specified
- detector confidence thresholds c(i) =
not specified
assumptions (3)
- domain assumption Detections are conditionally independent given tracks and tracks are independent (Eq. 1).
- domain assumption Track transitions follow a first-order Markov model.
- domain assumption Kalman filter linear-Gaussian dynamics adequately model human motion across skipped frames.
Cite this review
Pith. "Pith review of A fast multi-object tracking system using an object detector ensemble." pith.science (2026). https://pith.science/paper/DJGLEDZ7
@misc{pith2026190804349,
author = {Pith},
title = {Pith review of: A fast multi-object tracking system using an object detector ensemble},
year = {2026},
howpublished = {\url{https://pith.science/paper/DJGLEDZ7}},
note = {Machine review of arXiv:1908.04349}
}
read the original abstract
Multiple-Object Tracking (MOT) is of crucial importance for applications such as retail video analytics and video surveillance. Object detectors are often the computational bottleneck of modern MOT systems, limiting their use for real-time applications. In this paper, we address this issue by leveraging on an ensemble of detectors, each running every f frames. We measured the performance of our system in the MOT16 benchmark. The proposed model surpassed other online entries of the MOT16 challenge in speed, while maintaining an acceptable accuracy.
Figures
Reference graph
Works this paper leans on
-
[1]
Simple online and realtime tracking,
A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “Simple online and realtime tracking,” in Image Processing (ICIP), 2016 IEEE International Conference on. IEEE, 2016, pp. 3464–3468
work page 2016
-
[2]
Multiple object tracking using k-shortest paths optimization,
J. Berclaz, F. Fleuret, E. Turetken, and P. Fua, “Multiple object tracking using k-shortest paths optimization,” IEEE transactions on pattern analysis and machine intelligence , vol. 33, no. 9, pp. 1806–1819, 2011
work page 2011
-
[3]
Development of a n-type gm-phd filter for multiple target, multiple type visual tracking,
N. L. Baisa and A. Wallace, “Development of a n-type gm-phd filter for multiple target, multiple type visual tracking,” Journal of Visual Communication and Image Representation , 2019
work page 2019
-
[4]
Poi: Multiple object tracking with high performance detection and appearance feature,
F. Yu, W. Li, Q. Li, Y . Liu, X. Shi, and J. Yan, “Poi: Multiple object tracking with high performance detection and appearance feature,” in European Conference on Computer Vision. Springer, 2016, pp. 36–42
work page 2016
-
[5]
Estimating pedestrian counts in groups,
P. Kilambi, E. Ribnick, A. J. Joshi, O. Masoud, and N. Papanikolopoulos, “Estimating pedestrian counts in groups,” Computer Vision and Image Understanding, vol. 110, no. 1, pp. 43–59, 2008
work page 2008
-
[6]
M. Li, Z. Zhang, K. Huang, and T. Tan, “Estimating the number of people in crowded scenes by mid based foreground segmentation and head-shoulder detection,” inPattern Recognition, 2008. ICPR 2008. 19th International Conference on . IEEE, 2008, pp. 1–4
work page 2008
-
[7]
A camera-based system for tracking people in real time,
J. Segen, “A camera-based system for tracking people in real time,” in Proceedings of 13th International Conference on Pattern Recognition , vol. 3. IEEE, 1996, pp. 63–67
work page 1996
-
[8]
Robust head-shoulder detection by pca-based multilevel hog-lbp detector for people counting,
C. Zeng and H. Ma, “Robust head-shoulder detection by pca-based multilevel hog-lbp detector for people counting,” in Pattern Recognition (ICPR), 2010 20th International Conference on. IEEE, 2010, pp. 2069– 2072
work page 2010
Show all 29 references
-
[9]
Faster r-cnn: Towards real-time object detection with region proposal networks,
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in neural information processing systems , 2015, pp. 91–99
2015
-
[10]
You only look once: Unified, real-time object detection,
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 779– 788
2016
-
[11]
Mot16: A benchmark for multi-object tracking,
A. Milan, L. Leal-Taix ´e, I. Reid, S. Roth, and K. Schindler, “Mot16: A benchmark for multi-object tracking,” arXiv preprint arXiv:1603.00831, 2016
2016 arXiv
-
[12]
Global data association for multi- object tracking using network flows,
L. Zhang, Y . Li, and R. Nevatia, “Global data association for multi- object tracking using network flows,” in Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on . IEEE, 2008, pp. 1–8
2008
-
[13]
Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy,
L. I. Kuncheva and C. J. Whitaker, “Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy,” Machine learning, vol. 51, no. 2, pp. 181–207, 2003
2003
-
[14]
Soft-nmsimproving object detection with one line of code,
N. Bodla, B. Singh, R. Chellappa, and L. S. Davis, “Soft-nmsimproving object detection with one line of code,” in Computer Vision (ICCV), 2017 IEEE International Conference on . IEEE, 2017, pp. 5562–5570
2017
-
[15]
Multi-person tracking by multicut and deep matching,
S. Tang, B. Andres, M. Andriluka, and B. Schiele, “Multi-person tracking by multicut and deep matching,” in European Conference on Computer Vision. Springer, 2016, pp. 100–111
2016
-
[16]
Globally-optimal greedy algorithms for tracking a variable number of objects,
H. Pirsiavash, D. Ramanan, and C. C. Fowlkes, “Globally-optimal greedy algorithms for tracking a variable number of objects,” in Com- puter Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2011, pp. 1201–1208
2011
-
[17]
Subgraph decompo- sition for multi-target tracking,
S. Tang, B. Andres, M. Andriluka, and B. Schiele, “Subgraph decompo- sition for multi-target tracking,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 5033–5041
2015
-
[18]
Simple online and realtime tracking with a deep association metric,
N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in Image Processing (ICIP), 2017 IEEE International Conference on . IEEE, 2017, pp. 3645–3649
2017
-
[19]
Online multi-target tracking with tensor-based high-order graph matching,
Z. Zhou, J. Xing, M. Zhang, and W. Hu, “Online multi-target tracking with tensor-based high-order graph matching,” in 2018 24th Interna- tional Conference on Pattern Recognition (ICPR) . IEEE, 2018, pp. 1809–1814
2018
-
[20]
Online multi-target tracking with strong and weak detections,
R. Sanchez-Matilla, F. Poiesi, and A. Cavallaro, “Online multi-target tracking with strong and weak detections,” in European Conference on Computer Vision. Springer, 2016, pp. 84–99
2016
-
[21]
Tracking multiple people with multiple cameras,
S. Stillman, R. Tanawongsuwan, and I. Essa, “Tracking multiple people with multiple cameras,” 01 1999
1999
-
[22]
Kurkova, Y
V . Kurkova, Y . Manolopoulos, B. Hammer, L. Iliadis, and I. Maglogian- nis, Eds., Artificial Neural Networks and Machine Learning ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part I, ser. Theoretical Co...
2018
-
[23]
Multi-target tracking using cnn-based features: Cnnmtt,
N. Mahmoudi, S. M. Ahadi, and M. Rahmati, “Multi-target tracking using cnn-based features: Cnnmtt,” Multimedia Tools and Applications , pp. 1–20, 2018
2018
-
[24]
Recurrent autoregressive networks for online multi-object tracking,
K. Fang, Y . Xiang, X. Li, and S. Savarese, “Recurrent autoregressive networks for online multi-object tracking,” in 2018 IEEE Winter Con- ference on Applications of Computer Vision (WACV) . IEEE, 2018, pp. 466–475
2018
-
[25]
Light- head r-cnn: In defense of two-stage object detector,
Z. Li, C. Peng, G. Yu, X. Zhang, Y . Deng, and J. Sun, “Light- head r-cnn: In defense of two-stage object detector,” arXiv preprint arXiv:1711.07264, 2017
2017 arXiv
-
[26]
Microsoft coco: Common objects in context,
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision . Springer, 2014, pp. 740–755
2014
-
[27]
The pascal visual object classes challenge 2007 (voc2007) results,
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zis- serman, “The pascal visual object classes challenge 2007 (voc2007) results,” 2007
2007
-
[28]
Automatic differentiation in pytorch,
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” in NIPS-W, 2017
2017
-
[29]
Performance measures and a data set for multi-target, multi-camera tracking,
E. Ristani, F. Solera, R. Zou, R. Cucchiara, and C. Tomasi, “Performance measures and a data set for multi-target, multi-camera tracking,” in European Conference on Computer Vision . Springer, 2016, pp. 17– 35
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.