Pith. sign in

REVIEW 3 major objections 4 minor 29 references

A fast multi-object tracking system using an object detector ensemble

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A two-detector ensemble that runs only every f frames lets a Kalman-filter tracker reach 249.6–3,000.1 Hz on MOT16 while keeping acceptable accuracy.

desk verdict A simple frame-skipping ensemble gives impressive tracker-only speeds, but the accuracy evaluation in Table I is internally inconsistent and likely not comparable to official MOT16 numbers; the central 'acceptable accuracy' claim is not established. read the letter →

arxiv 1908.04349 v1 pith:DJGLEDZ7 submitted 2019-08-06 cs.CV cs.LG

classification cs.CVcs.LG
keywords multi-objecttrackingdetectorensembletracking-by-detectionKalmanfilterSoft-NMSMOT16real-timeMahalanobisdistance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a tracking-by-detection system whose central trick is to run an ensemble of object detectors only once every f frames rather than every frame, then fuse their outputs with a Soft-NMS variant and feed the fused detections to a Kalman-filter tracker. The authors claim that this sparse-detection schedule turns the detector from a bottleneck into a tunable cost: speed grows roughly linearly with f while accuracy degrades gracefully, so the system can beat every other online MOT16 entry on speed while keeping accuracy acceptable for applications such as people counting. They report tracker-only speeds of 249.6 Hz at f=1, 1,431.5 Hz at f=5, and 3,000.1 Hz at f=10, with MOTA falling from 43.7 to 34.3 to 28.3. A sympathetic reader should care because it reframes real-time MOT as a sampling-rate problem: choose f for the latency your hardware and application allow, and the same pipeline serves online and offline use.

What carries the argument

The load-bearing mechanism is the f-frame detector schedule combined with an ensemble fusion rule: detections from m detectors are merged by repeatedly taking the highest-confidence box and down-weighting overlapping boxes with g = exp(-β IoU), a Soft-NMS variant; fused detections feed a Kalman-filter tracker whose assignment cost is the Mahalanobis distance between predicted track state and detection, with Σ taken from the filter. This is what lets the system skip detector computation for f-1 frames while trusting motion continuity to keep identities.

What would settle it

Measure the full pipeline end-to-end—detector ensemble plus tracker, including I/O—on MOT16 with the authors' hardware and settings; if the f=5 configuration does not sustain real-time throughput, or if the reported Hz change materially when detector time is included, the paper's central speed claim fails.

Watch

Extended reading notes

Core claim

The central discovery is that a detector ensemble scheduled every f frames—two independently trained detectors, YOLOv3 and Lighthead R-CNN, fused by an exponential-decay Soft-NMS variant—permits a deliberately simple online tracker (Kalman filters plus a {0,1}-assignment problem with Mahalanobis distance) to run orders of magnitude faster than competing online entries on MOT16 with only moderate accuracy loss. The paper argues that the ensemble reduces false negatives enough to relax the tracking subproblem, replacing the need for complex appearance models with cheap motion-based association. The contribution is an extension of SORT: the same fast assignment machinery, but with the Euclidean/overlap distance replaced by a Mahalanobis distance that carries Kalman uncertainty, and with detections arriving every f frames. This is presented as a tunable speed-accuracy tradeoff rather than a fixed architecture.

Load-bearing premise

The real-time and speed claims rest on running the detector ensemble only every f frames and on the Kalman motion model plus Mahalanobis assignment carrying tracks across skipped frames, with reported speeds excluding detector time.

Editorial extensions

If this is right

  • Running the detector ensemble every f frames makes tracker speed grow roughly linearly with f, letting users trade accuracy for throughput on the same hardware.
  • At f=5 the full pipeline (detector plus tracker) runs in real time on the authors' hardware, and at f=10 it becomes suitable for offline batch processing at thousands of frames per second.
  • The ensemble's false-negative reduction improves MOTA by roughly 7.8 percentage points at f=1 and 5.5 at f=5 over the best single detector, so detector diversity is doing real work.
  • Because the tracker uses only motion and no appearance re-identification, its speed comes with elevated identity switches; applications tolerant of ID switches, such as counting, are the natural fit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported Hz exclude detector time; a fair cross-system comparison would need an end-to-end runtime, and the real-time claim at f=5 is conditional on the detector ensemble cost being amortized over five frames.
  • The same sparse-detection schedule could be tested with a learned motion model or a re-identification module: if the Kalman model is the bottleneck at large f, replacing it should push the accuracy-speed frontier further than the paper's measured curve.
  • Because the ensemble fusion is detector-agnostic, one testable extension is to swap in newer one-stage detectors and measure whether the MOTA-versus-f curve shifts upward without changing the tracking code.
  • Applying the same system to a dense-crowd sequence with fast motion would stress whether Mahalanobis gating alone can bridge f-1 skipped frames.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a multi-object tracking system based on tracking-by-detection, in which an ensemble of two detectors (YOLOv3 and Lighthead R-CNN) runs only every f frames, and the resulting detections are combined via a soft-NMS variant before being fed to a Kalman-filter-based tracker with Mahalanobis-distance assignment. The authors report results on the MOT16 benchmark for f = 1, 5, and 10, claiming the system achieves higher speed than other online entries while maintaining acceptable accuracy. The paper also includes an ablation of the ensemble versus individual detectors and a comparison with several published MOT16 results.

Significance. If the empirical claims were correct, the contribution would be practically relevant: running the detector ensemble only every f frames is a simple and potentially effective way to reduce the computational bottleneck of tracking-by-detection, and the reported tracker-only speeds are very high. The paper is, however, purely empirical, with no theoretical derivation and no released code, so its value rests entirely on the soundness and reproducibility of the evaluation. The reported accuracy numbers are internally inconsistent and appear to have been computed under a different evaluation protocol from the comparison methods, which currently undermines the main claim.

major comments (3)
  1. [Section III, Table I] The rows for OURS in Table I are internally inconsistent when checked against the standard MOTA formula MOTA = 1 - (FP + FN + IDSw) / GT. For f = 1, the error sum is 62,169; since the official MOT16 ground-truth count implied by the SORT row included in the same table is about 61,000, this sum already exceeds the total number of ground-truth boxes, making the reported MOTA of 43.7% arithmetically impossible. For f = 5 and f = 10, the error sums imply ground-truth counts of roughly 22,000 and 11,000, respectively, which is consistent with evaluating only on every 5th or 10th frame rather than on the full test sequences. If that is the case, the f = 5 and f = 10 MOTA values are not comparable to the official MOT16 scores of the other methods, and the claim of 'acceptable accuracy' at high speed is not established. The authors must report the exact evaluation protocol for each row, including the number of ground-truth boxes used, and recompute all OURS rows with the official evaluation code on the full ground-truth sequences.
  2. [Section III, Speed definition] The headline speeds of 249.6, 1,431.5, and 3,000.1 Hz are explicitly defined as 'processing speed in frames per second excluding the detector.' The paper states in the final paragraph of Section III that the application 'was able to run with real-time performance using f = 5,' but no end-to-end wall-clock time for the complete pipeline (detector ensemble plus tracker) is reported. Without this measurement, the real-time claim is unverifiable, and the comparison with other entries is ambiguous because it is unclear whether the published speeds of the other methods include detection time. Please report measured end-to-end runtimes for at least f = 1, f = 5, and f = 10, and clearly label which values in Table I and Fig. 3 include detection.
  3. [Section III, Evaluation methodology] All reported results appear to come from a single run, with no error bars, no multiple seeds, and no sensitivity analysis for the free parameters (beta, Max.Age, detector confidence thresholds). The ablation in Fig. 4 is also based on one run per configuration. Since the central claim is empirical, the robustness of the claimed accuracy and speed values cannot be assessed. The authors should provide variance over several runs or, failing that, a sensitivity analysis over the main parameters, and state how the parameters were selected (e.g., on a validation split).
minor comments (4)
  1. [Section III, Table I caption] The caption states that all compared methods 'use their own detections' and are online, but the SORT row appears to be the published result from the original SORT paper, which uses public detections. Please clarify which rows are from the authors' own runs and which are taken from the literature, and specify the detection source for every method.
  2. [References] References [21] and [22] appear to be mismatched with the methods CNNKCF and FMOT BL; for example, [21] cites a 1999 paper on tracking with multiple cameras, which does not correspond to the cited CNNKCF method. Please verify and correct these citations.
  3. [Section II, Equation (2)] The notation p(˜Z1:t|X) uses a tilde over Z that is not defined; please define it or replace it with a consistent notation such as p(Z_1:t|X).
  4. [Section III, Fig. 4] The figure legend shows the ensemble and two individual detectors, but the underlying MOTA values are not given in a table, and the y-axis title 'Accuracy (MOTA)' omits units and error bars; please include the numerical values and, if possible, variability across runs.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the contribution is an empirical system evaluation, not a derived prediction.

full rationale

The paper's claims are measurements: MOTA, IDF1, MT/ML, FP/FN/IDSw, and Hz values reported in Table I are presented as results of running the proposed tracker on MOT16, not as predictions derived from fitted parameters. The free parameter beta in the soft-NMS ensemble is a scaling knob of a standard post-processing rule, and the paper does not fit beta to the reported MOTA numbers nor rename a fitted quantity as a prediction. The tracking formulation (Kalman filters plus Mahalanobis-distance assignment) follows SORT and standard tracking-by-detection, with citations used as background rather than as load-bearing self-citations; no uniqueness theorem or prior-work assertion is invoked to force the result. The speeds exclude detector time, and the f=5/f=10 accuracy rows raise legitimate evaluation-protocol questions (e.g., apparent inconsistency with the official ground-truth count), but those are correctness and comparability concerns, not circularity: measured numbers are still measured numbers. No equation in the paper reduces by construction to an input, and no term is defined in terms of the quantity it is supposed to derive. The paper is therefore self-contained as an empirical benchmark report, and the appropriate circularity finding is none.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The contribution is an engineering combination of known components; new free parameters are limited to beta, f, Max.Age, and detector thresholds, none of which are reported with values.

free parameters (4)
  • beta = not specified
    Soft-NMS exponential decay scaling parameter, described as a free parameter in Algorithm 1.
  • f = 1, 5, 10
    Frame skip rate, evaluated at three values.
  • Max.Age = not specified
    Tracker deletion threshold listed as an input to Algorithm 2, not specified.
  • detector confidence thresholds c(i) = not specified
    Used to drop detections in Algorithm 1, not specified.
assumptions (3)
  • domain assumption Detections are conditionally independent given tracks and tracks are independent (Eq. 1).
    Standard tracking-by-detection factorization, stated in Section II.
  • domain assumption Track transitions follow a first-order Markov model.
    Standard assumption used to derive the recursive update in Eq. 3.
  • domain assumption Kalman filter linear-Gaussian dynamics adequately model human motion across skipped frames.
    The tracker relies on Kalman filtering and a Mahalanobis distance, which is calibrated to the filter's uncertainty.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A fast multi-object tracking system using an object detector ensemble." pith.science (2026). https://pith.science/paper/DJGLEDZ7

@misc{pith2026190804349,
  author       = {Pith},
  title        = {Pith review of: A fast multi-object tracking system using an object detector ensemble},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DJGLEDZ7}},
  note         = {Machine review of arXiv:1908.04349}
}
read the original abstract

Multiple-Object Tracking (MOT) is of crucial importance for applications such as retail video analytics and video surveillance. Object detectors are often the computational bottleneck of modern MOT systems, limiting their use for real-time applications. In this paper, we address this issue by leveraging on an ensemble of detectors, each running every f frames. We measured the performance of our system in the MOT16 benchmark. The proposed model surpassed other online entries of the MOT16 challenge in speed, while maintaining an acceptable accuracy.

Figures

Figures reproduced from arXiv: 1908.04349 by the authors.

Figure 1
Figure 1. Proposed multi-object tracking system with an object detector [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. State transition diagram for a tracker. The described MOT system is depicted in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Accuracy vs Sample rate of individual detectors and the ensemble. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 24 canonical work pages

  1. [1]

    Simple online and realtime tracking,

    A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “Simple online and realtime tracking,” in Image Processing (ICIP), 2016 IEEE International Conference on. IEEE, 2016, pp. 3464–3468

  2. [2]

    Multiple object tracking using k-shortest paths optimization,

    J. Berclaz, F. Fleuret, E. Turetken, and P. Fua, “Multiple object tracking using k-shortest paths optimization,” IEEE transactions on pattern analysis and machine intelligence , vol. 33, no. 9, pp. 1806–1819, 2011

  3. [3]

    Development of a n-type gm-phd filter for multiple target, multiple type visual tracking,

    N. L. Baisa and A. Wallace, “Development of a n-type gm-phd filter for multiple target, multiple type visual tracking,” Journal of Visual Communication and Image Representation , 2019

  4. [4]

    Poi: Multiple object tracking with high performance detection and appearance feature,

    F. Yu, W. Li, Q. Li, Y . Liu, X. Shi, and J. Yan, “Poi: Multiple object tracking with high performance detection and appearance feature,” in European Conference on Computer Vision. Springer, 2016, pp. 36–42

  5. [5]

    Estimating pedestrian counts in groups,

    P. Kilambi, E. Ribnick, A. J. Joshi, O. Masoud, and N. Papanikolopoulos, “Estimating pedestrian counts in groups,” Computer Vision and Image Understanding, vol. 110, no. 1, pp. 43–59, 2008

  6. [6]

    Estimating the number of people in crowded scenes by mid based foreground segmentation and head-shoulder detection,

    M. Li, Z. Zhang, K. Huang, and T. Tan, “Estimating the number of people in crowded scenes by mid based foreground segmentation and head-shoulder detection,” inPattern Recognition, 2008. ICPR 2008. 19th International Conference on . IEEE, 2008, pp. 1–4

  7. [7]

    A camera-based system for tracking people in real time,

    J. Segen, “A camera-based system for tracking people in real time,” in Proceedings of 13th International Conference on Pattern Recognition , vol. 3. IEEE, 1996, pp. 63–67

  8. [8]

    Robust head-shoulder detection by pca-based multilevel hog-lbp detector for people counting,

    C. Zeng and H. Ma, “Robust head-shoulder detection by pca-based multilevel hog-lbp detector for people counting,” in Pattern Recognition (ICPR), 2010 20th International Conference on. IEEE, 2010, pp. 2069– 2072

Show all 29 references
  1. [9]

    Faster r-cnn: Towards real-time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in neural information processing systems , 2015, pp. 91–99

  2. [10]

    You only look once: Unified, real-time object detection,

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 779– 788

  3. [11]

    Mot16: A benchmark for multi-object tracking,

    A. Milan, L. Leal-Taix ´e, I. Reid, S. Roth, and K. Schindler, “Mot16: A benchmark for multi-object tracking,” arXiv preprint arXiv:1603.00831, 2016

  4. [12]

    Global data association for multi- object tracking using network flows,

    L. Zhang, Y . Li, and R. Nevatia, “Global data association for multi- object tracking using network flows,” in Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on . IEEE, 2008, pp. 1–8

  5. [13]

    Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy,

    L. I. Kuncheva and C. J. Whitaker, “Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy,” Machine learning, vol. 51, no. 2, pp. 181–207, 2003

  6. [14]

    Soft-nmsimproving object detection with one line of code,

    N. Bodla, B. Singh, R. Chellappa, and L. S. Davis, “Soft-nmsimproving object detection with one line of code,” in Computer Vision (ICCV), 2017 IEEE International Conference on . IEEE, 2017, pp. 5562–5570

  7. [15]

    Multi-person tracking by multicut and deep matching,

    S. Tang, B. Andres, M. Andriluka, and B. Schiele, “Multi-person tracking by multicut and deep matching,” in European Conference on Computer Vision. Springer, 2016, pp. 100–111

  8. [16]

    Globally-optimal greedy algorithms for tracking a variable number of objects,

    H. Pirsiavash, D. Ramanan, and C. C. Fowlkes, “Globally-optimal greedy algorithms for tracking a variable number of objects,” in Com- puter Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2011, pp. 1201–1208

  9. [17]

    Subgraph decompo- sition for multi-target tracking,

    S. Tang, B. Andres, M. Andriluka, and B. Schiele, “Subgraph decompo- sition for multi-target tracking,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 5033–5041

  10. [18]

    Simple online and realtime tracking with a deep association metric,

    N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in Image Processing (ICIP), 2017 IEEE International Conference on . IEEE, 2017, pp. 3645–3649

  11. [19]

    Online multi-target tracking with tensor-based high-order graph matching,

    Z. Zhou, J. Xing, M. Zhang, and W. Hu, “Online multi-target tracking with tensor-based high-order graph matching,” in 2018 24th Interna- tional Conference on Pattern Recognition (ICPR) . IEEE, 2018, pp. 1809–1814

  12. [20]

    Online multi-target tracking with strong and weak detections,

    R. Sanchez-Matilla, F. Poiesi, and A. Cavallaro, “Online multi-target tracking with strong and weak detections,” in European Conference on Computer Vision. Springer, 2016, pp. 84–99

  13. [21]

    Tracking multiple people with multiple cameras,

    S. Stillman, R. Tanawongsuwan, and I. Essa, “Tracking multiple people with multiple cameras,” 01 1999

  14. [22]

    Kurkova, Y

    V . Kurkova, Y . Manolopoulos, B. Hammer, L. Iliadis, and I. Maglogian- nis, Eds., Artificial Neural Networks and Machine Learning ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part I, ser. Theoretical Co...

  15. [23]

    Multi-target tracking using cnn-based features: Cnnmtt,

    N. Mahmoudi, S. M. Ahadi, and M. Rahmati, “Multi-target tracking using cnn-based features: Cnnmtt,” Multimedia Tools and Applications , pp. 1–20, 2018

  16. [24]

    Recurrent autoregressive networks for online multi-object tracking,

    K. Fang, Y . Xiang, X. Li, and S. Savarese, “Recurrent autoregressive networks for online multi-object tracking,” in 2018 IEEE Winter Con- ference on Applications of Computer Vision (WACV) . IEEE, 2018, pp. 466–475

  17. [25]

    Light- head r-cnn: In defense of two-stage object detector,

    Z. Li, C. Peng, G. Yu, X. Zhang, Y . Deng, and J. Sun, “Light- head r-cnn: In defense of two-stage object detector,” arXiv preprint arXiv:1711.07264, 2017

  18. [26]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision . Springer, 2014, pp. 740–755

  19. [27]

    The pascal visual object classes challenge 2007 (voc2007) results,

    M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zis- serman, “The pascal visual object classes challenge 2007 (voc2007) results,” 2007

  20. [28]

    Automatic differentiation in pytorch,

    A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” in NIPS-W, 2017

  21. [29]

    Performance measures and a data set for multi-target, multi-camera tracking,

    E. Ristani, F. Solera, R. Zou, R. Cucchiara, and C. Tomasi, “Performance measures and a data set for multi-target, multi-camera tracking,” in European Conference on Computer Vision . Springer, 2016, pp. 17– 35

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.