REVIEW 3 major objections 4 minor 48 references
Multiple Object Tracking with Motion and Appearance Cues
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Optical-flow compensation, an auxiliary tracker, and cascade matching lift multiple-object tracking accuracy on drone video.
desk verdict A plausible integration of known ideas for drone MOT with an internally consistent ablation, but the headline gain is a grid-search maximum on the validation set and the test set doesn't confirm it—worth reviewing with the evaluation protocol as the main demand. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the optical-flow shift of Equations (2)-(3): for each tracked box in the previous frame, the top-left and bottom-right corners are moved by the flow values at those corners to produce a predicted box in the current frame. This single predicted box replaces the raw previous box in association, simultaneously undoing global camera motion and acting as a position predictor. The second mechanism is the cascade: matches above a strict IoU threshold are accepted first; remaining tracks are matched by combined appearance cosine distance and IoU. The third is the auxiliary tracker, which keeps an unmatched track alive with flow predictions for a maximum of $t_{\max}$ frames. The gating rule—run optical flow only when unmatched detections exceed half of matched ones—yields the 100 FPS variant.
What would settle it
Take VisDrone validation clips with obvious drone rotation or zoom and compute the mean IoU between boxes predicted by Equations (2)-(3) and the ground-truth boxes before any matching; if this predicted IoU is no better than simply reusing the previous box, the claimed reduction in ID switches from motion compensation would not transfer to those maneuvers.
Extended reading notes
Core claim
The central claim is that camera motion and missing detections in drone videos can be handled by reusing a single pretrained optical-flow network for two jobs: shifting each track's bounding box into the current frame before matching, and serving as an auxiliary tracker that predicts positions for up to $t_{\max}$ frames when a detection is absent. The paper further claims that cascade matching—first a high IoU threshold, then appearance cosine distance plus IoU via Hungarian assignment—reduces false matches between crowded objects. Evidence is the ablation on VisDrone2019-MOT validation: replacing the Faster R-CNN detector with an improved Cascade R-CNN raises MOTA from 12.6 to 26.7; adding motion estimation, auxiliary tracker, and cascade matching raises it to 32.1 and lowers ID switches from 576 to 112. On the test set it reports AP 30.87, above all compared baselines. The paper concludes that the tracker achieves a practical trade-off between accuracy and speed.
Load-bearing premise
The whole improvement rests on the assumption that shifting a bounding box by the optical-flow vectors at its top-left and bottom-right corners gives the object's true position in the next frame; if drone motion involves rotation, scale change, or heavy occlusion, those two corner vectors are unreliable.
Editorial extensions
If this is right
- A tracker can inherit camera-motion robustness from a pretrained optical-flow network without specialized motion models or training on tracking data.
- The largest single improvement comes from the detector, so tracking quality on drone video remains bottlenecked by detection quality, not association alone.
- Bridging missing detections with a short-lived position predictor reduces trajectory fragmentation and identity switches, which directly improves downstream trajectory analytics.
- The gated-flow variant shows that most frames in drone video do not need explicit motion compensation, so accuracy and real-time speed can be traded by a simple per-frame test.
Reading between the lines
- The two-corner flow model assumes near-rigid translation; drone yaw, roll, or zoom will stress it. A denser flow aggregation over the box, or an affine flow model, would test whether the remaining ID switches are motion-model errors.
- Since the detector replacement accounts for most of the MOTA gain, the association modules may transfer to other detection-based trackers with stronger detectors; comparing on other aerial or surveillance sequences would reveal how much of the gain is detector-specific.
- The frame-gating heuristic could be replaced by a learned or inertial predictor of camera motion, potentially recovering most of the accuracy gap between the 5 FPS and 100 FPS variants.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Flow-Tracker, an online tracking-by-detection method built on IoU Tracker, with three additions: optical-flow-based global motion compensation via PWC-Net (Section 3.2), an auxiliary tracker that propagates unmatched tracks for up to t_max frames (Section 3.3), and a cascade association policy combining IoU and a deep appearance metric (Section 3.4). On VisDrone2019-MOT, the authors report that MOTA on the validation set rises from 12.6 (IoU Tracker) to 32.1 (Flow-Tracker), and that a speed-optimized variant runs at 100 FPS (Tables 4 and 5). They also report AP on the test set (Table 6). The central claim is that the combination of these modules significantly improves tracking accuracy while remaining efficient.
Significance. The modular design targets genuine problems in drone video—camera motion, missing detections, and association ambiguity—and the ablation tables (Tables 3 and 4) are internally consistent: each added module improves MOTA or reduces ID switches in the expected direction. If the reported gains were unbiased, the work would be a useful practical contribution. The strengths include a clear decomposition of the tracking pipeline and explicit reporting of per-module effects. However, the paper does not release code, and no machine-checked derivations are involved; more importantly, the evaluation methodology leaves the headline effect size unverified. The main significance of the paper therefore depends on whether the reported validation-set numbers survive an independent evaluation.
major comments (3)
- [Section 4.1, Tables 2 and 4] Section 4.1 states that all hyper-parameters were obtained by grid search on the VisDrone2019-MOT validation set, and Tables 2 and 4 report results on exactly that same validation set. With roughly eight tunable thresholds (sigma_IoU, sigma_IoU1, t_min, sigma_h, t_max, sigma_nms, the appearance threshold, and the camera-motion trigger) and only seven validation videos, the headline MOTA improvement from 12.6 to 32.1 is a selected maximum rather than an unbiased estimate of expected performance. The paper should provide an independent held-out evaluation, cross-validation over the 63 training/validation videos, or at minimum report the corresponding MOTA/IDS numbers on the test set, and the abstract's claim of 'significant improvement' should be tempered until such evidence is available.
- [Section 4.2, Table 4 and Section 4.3, Table 6] The test-set evidence does not confirm the identity-stability claim. Table 6 reports only AP on the test set; MOTA, IDF1, IDS, and FM are absent. Furthermore, Section 4.2 shows that changing the detector from Faster R-CNN to Cascade R-CNN alone raises MOTA from 12.6 to 26.7, and the final row of Table 4 adds detector improvements (Soft-NMS, deformable convolution) together with the last tracker component. The test-set comparison in Table 6 therefore does not hold the detector fixed across methods, and the tracking modules' contribution to the reported gain is not isolated. The authors should report test-set MOTA/IDS for all compared trackers using the same detector, and should separate the detector improvement from the tracker module improvements in the ablation.
- [Section 3.2, Equations (2)-(3)] The motion-compensation step uses optical flow at only the top-left and bottom-right corners of each previous-frame bounding box to shift the entire box into the current frame. This assumes that the object's apparent motion is a rigid translation well represented by those two corner points. For articulated objects, partial occlusion, or non-rigid motion in drone footage, this assumption may fail, and the paper provides no quantitative evidence for its accuracy. Since the reported reduction in ID switches is attributed in part to this module, the authors should either justify the two-corner approximation (for example, by comparing against a dense-flow average or an affine fit) or provide an ablation showing that the results are insensitive to this choice.
minor comments (4)
- [Tables 1 and 2, references] The baseline is labeled 'Deep Sort [42]' in Tables 1 and 2, but reference [42] is Milan et al.'s CEM; the Deep Sort citation should be [4] (Wojke et al.). The reference list should be checked for other mismatches after renumbering.
- [Equations (2)-(3)] The notation in Equations (2)-(3) is confusing: if (u,v) is the optical flow from frame t-1 to frame t, the current frame's corner should be (x+u, y+v), but the equations as written place the primed bounding box at the old coordinates. Please rewrite these equations with clearer notation, e.g., (x1', y1') and (x2', y2') for the shifted corners.
- [Section 4.2, Table 5] The speed comparison lacks measurement details: the reported 100 FPS for Flow-Tracker-fast and 5 FPS for Flow-Tracker do not state image resolution, GPU model (the GTX1080Ti is mentioned only for detector training), whether optical-flow timing includes PWC-Net GPU inference, or whether the numbers are averaged over the same video subset. Please specify the measurement protocol so the speed claim is reproducible.
- [Section 3.2, Section 4.2] The camera-motion trigger (using optical flow when the number of unmatched objects exceeds half the number of matched objects) is introduced in Section 3.2 and used in Table 5, but no sensitivity analysis is given for the trigger fraction. A short experiment varying this threshold would help establish that the 100 FPS variant is not operating at a fragile operating point.
Circularity Check
No derivation-level circularity, but the headline accuracy gain is evaluated on the same validation set used to grid-search all hyperparameters.
-
fitted input called prediction
[Section 4.1 (Experiment setup) and Tables 2-5 (validation-set results)]
"We use a GTX1080Ti GPU to train it on all the images in the training set. And all the hyper-parameters in the experiments are obtained by grid search on the validation set."
The central quantitative claim (MOTA 12.6 -> 32.1, IDS 576 -> 112 in Table 4) is reported on the VisDrone2019-MOT validation set, and the same validation set was used to choose all hyper-parameters by grid search, including the IoU/appearance thresholds and track-length limits. The reported numbers are therefore a post-selection maximum over the tuned parameters, not an independent estimate of the modules' effect. This is a fitted-input/evaluation overlap: the evidence for 'improve the performance significantly' is the same data that determined the parameters. It does not make the tracker equations equivalent to their inputs, but it partially fits the central claim to the benchmark.
full rationale
Flow-Tracker is an integration of externally validated components (PWC-Net for optical flow, ResNet for appearance, Faster/Cascade R-CNN for detection, Hungarian assignment), and no author self-citation chain carries the argument. The motion-compensation equations (2)-(3) simply translate box corners by optical-flow offsets; the cascade matching is a decision procedure; neither defines its target result in terms of that result. The ablation tables are empirical measurements, not derivations. The main circularity-like weakness is that all hyper-parameters were grid-searched on the validation set and the headline Tables 1-5 report exactly that set, so the improved MOTA/IDS figures are a tuned maximum rather than an unbiased expected-performance estimate. The test-set Table 6 reports only AP, not MOTA or IDS, so it does not independently confirm the identity-stability claim. These are evidence-loop concerns rather than equation-level self-definition, so the score is moderate.
Assumptions & free parameters
free parameters (8)
- IoU association threshold sigma_IoU
- First-stage IoU threshold sigma_IoU1
- Minimum track length t_min
- High-confidence detection score sigma_h
- Maximum auxiliary tracking frames t_max
- NMS threshold sigma_nms
- Appearance distance threshold
- Camera motion trigger fraction =
0.5
assumptions (5)
- domain assumption PWC-Net pretrained on FlyingChairs and FlyingThings3D transfers to drone videos.
- domain assumption A ResNet appearance extractor pretrained on Market1501 and MARS discriminates all ten VisDrone classes, including vehicles.
- ad hoc to paper The displacement of a whole bounding box is captured by optical flow at its top-left and bottom-right corners.
- standard math Hungarian assignment is the appropriate optimal matching for track-detection association.
- domain assumption The VisDrone evaluation protocol (tracklet AP and MOT metrics) measures the intended tracking quality across methods.
Cite this review
Pith. "Pith review of Multiple Object Tracking with Motion and Appearance Cues." pith.science (2026). https://pith.science/paper/6O4ECGTJ
@misc{pith2026190900318,
author = {Pith},
title = {Pith review of: Multiple Object Tracking with Motion and Appearance Cues},
year = {2026},
howpublished = {\url{https://pith.science/paper/6O4ECGTJ}},
note = {Machine review of arXiv:1909.00318}
}
read the original abstract
Due to better video quality and higher frame rate, the performance of multiple object tracking issues has been greatly improved in recent years. However, in real application scenarios, camera motion and noisy per frame detection results degrade the performance of trackers significantly. High-speed and high-quality multiple object trackers are still in urgent demand. In this paper, we propose a new multiple object tracker following the popular tracking-by-detection scheme. We tackle the camera motion problem with an optical flow network and utilize an auxiliary tracker to deal with the missing detection problem. Besides, we use both the appearance and motion information to improve the matching quality. The experimental results on the VisDrone-MOT dataset show that our approach can improve the performance of multiple object tracking significantly while achieving a high efficiency.
Figures
Reference graph
Works this paper leans on
-
[1]
W. Luo, J. Xing, A. Milan, X. Zhang, W. Liu, X. Zhao, and T. K. Kim. Multiple object tracking: A literature review. arXiv preprint arXiv:1409.7618, 2014
arXiv 2014
-
[2]
C. Micheloni, G. L. Foresti, C. Piciarelli, and L. Cinque. An autonomous vehicle for video surveillance of indoor environments. IEEE Transactions on Vehicular Techno - logy, 56(2), 487-498, 2007
work page 2007
-
[3]
E. Bochinski, V. Eiselein, an d T. Sikora. High-speed tracking-by-detection without using image information. In IEEE International Conference on Advanced Video and Sig- nal Based Surveillance (ICAVSS), pages 1–6, 2017
work page 2017
- [4]
-
[5]
L. Chen, H. Ai, Z. Zhuang, and C. Shang. Real-time multiple people tracking with deeply learned candidate selection and person re-identification. In ICME, Vol. 5, p. 8, 2018
work page 2018
-
[6]
Q. Chu, W. Ouyang, H. Li, X. Wang, B. Liu, and N. Yu. Online multi-object tracking using CNN-based single object tracker with spatial -temporal attention mechanism. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4836-4845, 2017
work page 2017
-
[7]
S. Tang, M. Andriluka, B. Andres, and B. Schiele. Multiple people tracking by lifted multicut and person re - identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3539-3548, 2017
work page 2017
-
[8]
D. Sun, X. Yang, M. Y. Liu, and J. Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8934-8943, 2018
work page 2018
Show all 48 references
-
[9]
P. Zhu, L. Wen, X. Bian, H. Ling, and Q. Hu. Vision meets drones: A challenge. arXiv preprint arXiv:1804.07437, 2018
2018 arXiv
-
[10]
Robicquet, A
A. Robicquet, A. Sadeghian, A. Alahi, and S. Savarese. Learning social etiquette: Human trajectory understanding in crowded scenes. In European Conference on Computer Vision (ECVV), pages 549-565, 2016
2016
-
[11]
Li and D
S. Li and D. Y. Yeung. Visual object tracking for unmanned aerial vehicles: A benchmark and new motion models. In Thirty-First AAAI Conference on Artificial Intelligence , 2017
2017
-
[12]
A new approach to linear filtering and prediction problems
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Journal of basic Engineering, 82(1), 35–45, 1960
1960
-
[13]
Milan, S
A. Milan, S . H. Rezatofighi, A . Dick, I . Reid, and K . Schindler. Online multi-target tracking using recurrent neural networks. In Thirty-First AAAI Conference on Artificial Intelligence, 2017
2017
-
[14]
Sadeghian, A
A. Sadeghian, A. Alahi, and S. Savarese. Tracking the untrackable: Learning to track multiple cues with long -term dependencies. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 300-311, 2017
2017
-
[15]
B. D. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the 1981 DARPA Image Understanding Workshop, pages 121-130, 1981
1981
-
[16]
Fischer, A
P. Fischer, A. Dosovitskiy, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2758-2766, 2015
2015
-
[17]
E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2462-2470, 2017
2017
-
[18]
Choi and S
W. Choi and S. Savarese. Multiple target tracking in world coordinate with single, minimally calibrated camera. In European Conference on Computer Vision (ECCV), pages 553-567, 2010
2010
-
[19]
N. Le, A. Heili, and J. M. Odobez. Long-term time-sensitive costs for crf -based tracking by detection. In European Conference on Computer Vision (ECCV), pages 43-51, 2016
2016
-
[20]
Z. Wu, A. Thangali, S. Sclaroff, and M. Betke. Coupling detection and data association for multi ple object tracking. In 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1948-1955, 2012
2012
-
[21]
Pellegrini, A
S. Pellegrini, A. Ess, K. Schindler, and L. Van Gool. You'll never walk alone: Modeling social behavior for multi -target tracking. In 2009 IEEE 12th International Conference on Computer Vision (ICCV), pages 261-268, 2009
2009
-
[22]
L. Zhao, X. Li, Y. Zhuang, and J. Wang. Deeply-learned part- aligned representations for person re -identification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3219-3228, 2017
2017
-
[23]
Leal-Taixé , C
L. Leal-Taixé , C. Canton-Ferrer, and K. Schindler. Learning by tracking: Siamese CNN for robust target association. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 33-40, 2016
2016
-
[24]
Pirsiavash, D
H. Pirsiavash, D. Ramanan, and C. C. Fowlkes. Globally - optimal greedy algorithms for tracking a variable number of objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (C VPR), pages 1201-1208, 2011
2011
-
[25]
S. Tang, B. Andres, M. Andriluka, and B. Schiele. Subgraph decomposition for multi -target tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5033-5041, 2015
2015
-
[26]
S. Tang, B. Andres, M. Andriluka, and B. Schiele. Multi - person tracking by multicut and deep matching. In European Conference on Computer Vision (ECCV), pages 100-111, 2016
2016
-
[27]
J. Munkres. Algorithms for the assignment and transportation problems. Journal of the Society for Industrial and Applied Mathematics, 5(1), 32-38, 1957
1957
-
[28]
Y. Xu, Y. Ban, X. Alameda-Pineda, and R. Horaud. DeepMOT: A Differentiable Framework for Training Multiple Object Trackers. arXiv preprint arXiv:1906.06618, 2019
1906 arXiv
-
[29]
Felzenszwalb, R
P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9), 1627-1645, 2009
2009
-
[30]
S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS), pages 91-99, 2015
2015
-
[31]
Redmon and A
J. Redmon and A. Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018
2018 arXiv
-
[32]
T. Y. Lin, P. Dollá r, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2117-2125, 2017
2017
-
[33]
T. Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollá r. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, 2017
2017
-
[34]
Zhang, L
S. Zhang, L. Wen, X. Bian, Z. Lei, and S. Z. Li. Single-shot refinement neural netw ork for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4203-4212, 2018
2018
-
[35]
Cai and N
Z. Cai and N. Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6154-6162, 2018
2018
-
[36]
Ouyang and X
W. Ouyang and X. Wang. Joint deep learning for pedestrian detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2056-2063, 2013
2013
-
[37]
X. Wang, T. Xiao, Y. Jiang, S. Shao, J. Sun, and C. Shen. Repulsion loss: Detecting pedestrians in a crowd. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7774-7783, 2018
2018
-
[38]
Mayer, E
N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large dataset to train convo - lutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)...
2016
-
[39]
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016
2016
-
[40]
Zheng, L
L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian. Scalable person re -identification: A benchmark. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1116-1124, 2015
2015
-
[41]
Zheng, Z
L. Zheng, Z. Bie, Y. Sun, J. Wang, C. Su, S. Wang, and Q. Tian. Mars: A video benchmark for large-scale person re - identification. In European Conference on Computer Vision (ECCV), pages 868-884, 2016
2016
-
[42]
Milan, S
A. Milan, S. Roth, and K. Schindler. Continuous energy minimization for multitarget tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(1), 58 -72, 2013
2013
-
[43]
L. Wen, W. Li, J. Yan, Z. Lei, D. Yi, and S. Z. Li. Multiple target tracking based on undirected hierarchical relation hypergraph. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1282-1289, 2014
2014
-
[44]
Dicle, O
C. Dicle, O. I. Camps, and M. Sznaier. The way they move: Tracking multiple targets with similar appearance. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2304-2311, 2013
2013
-
[45]
Geiger, M
A. Geiger, M. Lauer, C. Wojek, C. Stiller, and R. Urtasun. 3d traffic scene understanding from movable platforms. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 36(5), 1012-1025, 2013
2013
-
[46]
S. H. Bae and K. J. Yoon. Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1218-1225, 2014
2014
-
[47]
P. Zhu, L. Wen, D. Du, X. Bian, H. Ling, Q. Hu, ... and X. Liu. VisDrone -VDT2018: The Vision Meets Drone Video Detection and Tracking Challenge Results. In Proceedings of the European Conference on Computer Vision (ECCV), pages 496-518, 2018
2018
-
[48]
L. Wen, P. Zhu, D. Du, X. Bian, H. Ling , Q. Hu, and et al. VisDrone-MOT2019: The Vision Meets Drone Multiple Object Tracking Challenge Results. 2019
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.