REVIEW 4 major objections 5 minor 34 references
Tracking tiny wasps for eight minutes defeats modern multi-object trackers even when detections are perfect, and a simple stitching step recovers most lost identities.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 07:46 UTC pith:KWMHAC2Y
load-bearing objection A genuinely new long-duration MOT benchmark with a plausible central finding, but the ground-truth identity labels are undocumented — that has to be addressed before the fragmentation numbers can be taken at face value. the 4 major comments →
WaspMOT: A Benchmark for Long-Term Multi-Object Tracking of Trichogramma Wasps
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The discovery the paper is trying to establish is that identity preservation over thousands of frames is a distinct, unsolved problem that current association mechanisms do not handle, even when detection is not a factor. In the closed-set WaspMOT setting, all ten video sequences keep every wasp present from first to last frame, so a track that breaks is always a failure of association rather than a genuine birth or death. Under oracle detections, the five evaluated trackers achieve IDF1 scores of roughly 45 to 51, meaning they maintain the correct identity less than half the time on average, and all of them break into many shorter tracklets. The paper's second, quantitative claim is that a
What carries the argument
The load-bearing component is the oracle-detection protocol combined with the closed-set, full-length trajectory structure. By stripping identity labels from ground-truth boxes, the benchmark makes the association step the only source of tracking error, and by making every identity persist for the entire 12,000-frame sequence, it rules out re-initialization as a legitimate strategy. The analytical instrument is the spatial tracklet stitching baseline: it represents each fragment by its start and end in space-time, links compatible endpoints using Hungarian assignment, and measures how much identity consistency can be recovered with zero new information beyond position and timing. That baseli
Load-bearing premise
The central claim rests on the manual identity labels being correct: if the 2.57 million annotated bounding-box identities contain systematic errors, then the measured trajectory fragmentation is partly an artifact of the labels rather than a property of the trackers.
What would settle it
Take a random subset of WaspMOT frames and have multiple trained annotators independently re-assign identities, then measure inter-annotator agreement at jump and occlusion moments. If agreement is low, or if a synthetic video with known true identities does not reproduce the same fragmentation rates under oracle detections, the claim that trackers fail at long-term identity preservation is not established.
If this is right
- Every one of the five evaluated trackers fragments trajectories on WaspMOT, so no current tracking-by-detection association mechanism, by itself, preserves identities over minutes-long closed-set videos.
- Because oracle detections are used, the measured IDF1 ceiling of about 51 is a property of association, not detection; improving detectors alone cannot fix it.
- The spatial stitching baseline raises IDF1 by 6.6-9.0 points, so a large share of lost identities is recoverable without appearance information, using only position and temporal order.
- Appearance-based association does not yield a clear advantage over simpler motion-based methods on these visually uniform targets, suggesting appearance cues are of limited value in this regime.
- Long-term identity preservation should be evaluated on sequences long enough that trajectories span the full duration, since the failure mode does not surface on conventional short benchmarks.
Where Pith is reading between the lines
- The size of the recoverable gap suggests a natural architecture: run a short-term tracker online, then apply a global or hierarchical optimization over the whole sequence; the paper's numbers imply that such a second stage could lift IDF1 toward the stitching upper bound even if the online stage is unchanged.
- The same structure — closed-set identities, abrupt jumps, uniform appearance — appears in other ecological and biomedical tracking settings, so one could test whether the fragmentation pattern reproduces on similarly long recordings of ants, bees, or cells; if it does, the benchmark's conclusions generalize beyond wasps.
- The paper does not report annotation-quality statistics, so a direct next experiment is to re-annotate a subset with multiple people and measure identity-label agreement; knowing the label noise floor would let the benchmark separate tracker fragmentation from annotation fragmentation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces WaspMOT, a new benchmark for long-term multi-object tracking consisting of 10 laboratory videos of Trichogramma wasps, each approximately 12,000 frames (roughly 8 minutes at 25 FPS), with dense MOTChallenge-format annotations and oracle detections. The benchmark is closed-set: 15–28 individuals per sequence are claimed to remain present for the entire video, so every trajectory is full-length. Five tracking-by-detection methods (ByteTrack, BoT-SORT, C-BIoU, OC-SORT, McByte) are evaluated under a unified protocol with oracle detections. The central empirical claim is that all methods suffer significant trajectory fragmentation (low IDF1 despite perfect detections) and that a simple spatial tracklet-stitching post-processing baseline consistently recovers 6.6–9.0 IDF1 points, showing that long-term identity preservation is a distinct and underexplored failure mode not well captured by existing short-sequence benchmarks.
Significance. If the ground-truth identities are reliable, WaspMOT fills a genuine gap: it is the only closed-set benchmark with thousands of frames per trajectory, and the oracle-detection design cleanly isolates the association problem. The consistent, sizeable gains from a few lines of stitching post-processing are a concrete, falsifiable demonstration that current association mechanisms lose recoverable identity information over long horizons. The use of TrackEval, publicly available tracker implementations, and a planned dataset release make the benchmark reproducible and likely to be useful to both the MOT and ecological-surveillance communities. The paper also makes a useful secondary point: on these visually homogeneous, rapidly moving targets, appearance-based association (BoT-SORT) does not outperform simpler motion/IoU methods, although this conclusion currently rests on an underspecified ReID adaptation protocol.
major comments (4)
- [Section III-B] The central claim that all trackers fragment even with oracle detections rests entirely on the correctness of the 2,569,527 identity labels. The paper reports no annotation protocol, no double annotation, no inter-annotator agreement, and no quality-control analysis. Because the oracle detections are derived from these same labels (Section IV-A), any annotation-level ID error is counted as a tracking error for all five methods. Given the small size, visual similarity, and abrupt jumps of the wasps, and the 12,000-frame duration, the risk of systematic identity-label noise is material. Without evidence that the ground-truth identities are accurate, the measured 'trajectory fragmentation' and the stitching gains could partly be corrections of annotation errors rather than genuine tracker failures. Please add a detailed annotation protocol, agreement statistics on a re-annotated subset, and
- [Section IV-B] The protocol for BoT-SORT's appearance embeddings is underspecified. The text says the embeddings are 'adapted to WaspMOT' while also claiming that 'all methods are applied using oracle detections and their default parameters without additional tuning.' These two statements are in tension. If the ReID model was trained on the same 10 sequences used for evaluation, then the appearance comparison is contaminated and the paper's conclusion that appearance cues do not help on WaspMOT is uninterpretable. Please specify the exact training split (which sequences/frames were used to adapt the embeddings), and ensure no evaluation frames are used in ReID training. The same transparency is needed for McByte's 'temporally propagated segmentation masks' — their source (e.g., ground-truth masks, a segmentation model, or generated from oracle detections) is not stated, and if they are derived from gro
- [Table I] The MOT17 row is internally inconsistent. With 802.50 average frames and 99.21 average seconds per video, the implied frame rate is approximately 8 FPS, not the '27 (avg.)' listed. This inconsistency undermines the table's credibility and also weakens the statement that WaspMOT sequences are 'an order of magnitude longer' than existing datasets: against the printed MOT17 row, 481 seconds is about 4.9x longer, not 10x; the factor is only an order of magnitude relative to some of the other shorter datasets. Please correct the FPS/frame counts to match the cited sources, recompute the averages, and revise the claim to accurately reflect the distribution across datasets.
- [Section IV-C / Table II] The stitching baseline is described only in qualitative terms: 'spatial proximity and temporal consistency' with a Hungarian assignment. To support the paper's quantitative claim that this simple baseline recovers 6.6–9.0 IDF1 points, the exact thresholds (spatial distance cutoff, temporal gap limit, cost function for the Hungarian matching) must be given. In addition, Table II aggregates across 10 sequences with very different event counts (Figs. 2 and 3), but no per-sequence results, variance, or statistical analysis is reported. The reader cannot tell whether the fragmentation and stitching gains are consistent across all sequences or driven by a few difficult videos. Please include per-sequence HOTA/IDF1 (or at least a scatter/box plot) and, ideally, confidence intervals across sequences.
minor comments (5)
- [References] Reference [11] is cited for the BEE24 dataset, but the listed title 'Topic: A parallel association paradigm for multi-object tracking under complex motions and diverse scenes' does not appear to describe an insect-tracking benchmark. Please check the citation and update it to the correct BEE24 source.
- [Figures 2 and 3] The axes of Figs. 2 and 3 are unlabeled. Add axis titles (e.g., 'number of events' and 'video identifier') and clarify how jump and occlusion events are defined and detected from the ground-truth annotations.
- [Abstract and Section I] The phrases 'not observable on conventional datasets' and 'are not captured by conventional short-duration benchmarks' are stronger than what is shown. The paper demonstrates that fragmentation is present and recoverable on WaspMOT, but it does not prove that the same failure mode is absent from all existing datasets. Please soften these statements to 'not readily observable' or 'under-represented.'
- [Section III-B] The paper says the dataset is closed-set and that all individuals remain present throughout the video, but it does not state what happens during complete occlusions (e.g., when a wasp moves behind an object or to the ceiling). Does the ground truth maintain the bounding box with a predicted location, or is the object considered temporarily absent? This is needed to interpret the oracle-detection setup and the fragmentation metric.
- [Section IV-B] The sentence 'All methods are applied using oracle detections and their default parameters without additional tuning' is contradicted by the BoT-SORT 'adapted to WaspMOT' statement. Even after resolving the underlying issue, the wording should be revised to describe exactly what was adapted and why.
Circularity Check
No significant circularity: the benchmark claims are empirical observations, not derivations, and no prediction reduces to its inputs by construction.
full rationale
WaspMOT is a benchmark paper, not a mathematical derivation. The central claim—that all evaluated trackers suffer trajectory fragmentation even with oracle detections—is an empirical measurement, not a result derived from fitted parameters or from a self-citation chain. The load-bearing assumption is ground-truth identity correctness; the paper does not report annotation protocol, inter-annotator agreement, or quality control, which is a validity or robustness risk, but it is not circularity because there is no equation or fitted quantity that defines the measured fragmentation. Oracle detections are explicitly derived from ground-truth bounding boxes without identity labels, a standard protocol for isolating association. The spatial tracklet stitching baseline is a simple post-processor and is not claimed to be a derivation. BoT-SORT's appearance embeddings are 'adapted to WaspMOT' without a described train/test split, which could affect comparison fairness, but the paper does not present this adaptation as a prediction nor derive its conclusions from it. McByte is co-authored by the paper's authors and cited as an evaluated method; this self-citation is not load-bearing for the main finding because McByte also exhibits low IDF1, and the comparison is externally reproducible once the dataset is released. No uniqueness theorem, ansatz, or renaming is invoked to force a conclusion. Under the required standard, no circular step can be exhibited with a specific quote-and-reduction, so the appropriate finding is no significant circularity.
Axiom & Free-Parameter Ledger
free parameters (3)
- Stitching distance threshold
- Stitching temporal gap limit
- BoT-SORT appearance embedding weights =
trained/adapted to WaspMOT (details omitted)
axioms (4)
- domain assumption Ground-truth identities are correct for all 2,569,527 annotated instances
- domain assumption The closed-set assumption holds: all individuals remain present for the whole video
- standard math TrackEval metrics are applied correctly and are comparable across trackers
- domain assumption Oracle detections (GT boxes without identity labels) isolate association performance
read the original abstract
Multi-object tracking (MOT) has achieved strong performance on benchmarks dominated by short video sequences. However, such datasets do not adequately evaluate long-term identity preservation, where objects must be tracked consistently over extended durations. We introduce WaspMOT, a benchmark designed to address this gap through long-duration tracking of Trichogramma wasps in controlled ecological experiments. The dataset contains 10 sequences of approximately 12,000 frames each (over 8 minutes at 25 FPS), with dense MOTChallenge annotations and oracle detections to isolate association performance. Unlike existing benchmarks, WaspMOT forms a closed-set tracking scenario where all individuals remain present throughout the sequence, requiring consistent identity assignment across thousands of frames despite abrupt jumps, occlusions, and highly similar appearance. We establish a benchmark by evaluating five tracking-by-detection methods, including ByteTrack, BoT-SORT, C-BIoU, OC-SORT, and McByte, under a unified protocol. Results show that all methods suffer from significant trajectory fragmentation, highlighting the difficulty of long-term identity preservation even with perfect detections. A simple spatial tracklet stitching baseline consistently improves performance, indicating that substantial gains remain possible. WaspMOT provides a new benchmark for studying long-term association and reveals limitations of current tracking approaches that are not observable on conventional datasets. The benchmark will be made publicly available at the project repository: https://github.com/tstanczyk95/WaspMOT/ .
Figures
Reference graph
Works this paper leans on
-
[1]
MOT16: A benchmark for multi-object tracking,
A. Milan, L. Leal-Taix ´e, I. Reid, S. Roth, and K. Schindler, “MOT16: A benchmark for multi-object tracking,”arXiv:1603.00831 [cs], Mar. 2016, arXiv: 1603.00831. [Online]. Available: http: //arxiv.org/abs/1603.00831
Pith/arXiv arXiv 2016
-
[2]
Mot20: A benchmark for multi object tracking in crowded scenes,
P. Dendorfer, H. Rezatofighi, A. Milan, J. Shi, D. Cremers, I. Reid, S. Roth, K. Schindler, and L. Leal-Taix ´e, “Mot20: A benchmark for multi object tracking in crowded scenes,” 2020. [Online]. Available: https://arxiv.org/abs/2003.09003
Pith/arXiv arXiv 2020
-
[3]
Sportsmot: A large multi-object tracking dataset in multiple sports scenes,
Y . Cui, C. Zeng, X. Zhao, Y . Yang, G. Wu, and L. Wang, “Sportsmot: A large multi-object tracking dataset in multiple sports scenes,”Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023
2023
-
[4]
Soccernet-tracking: Multiple object tracking dataset and benchmark in soccer videos,
A. Cioppa, S. Giancola, A. Deliege, L. Kang, X. Zhou, Z. Cheng, B. Ghanem, and M. Van Droogenbroeck, “Soccernet-tracking: Multiple object tracking dataset and benchmark in soccer videos,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2022, pp. 3491–3502
2022
-
[5]
Dancetrack: Multi-object tracking in uniform appearance and diverse motion,
P. Sun, J. Cao, Y . Jiang, Z. Yuan, S. Bai, K. Kitani, and P. Luo, “Dancetrack: Multi-object tracking in uniform appearance and diverse motion,”Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021
2021
-
[6]
Bytetrack: Multi-object tracking by associating every detection box,
Y . Zhang, P. Sun, Y . Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu, and X. Wang, “Bytetrack: Multi-object tracking by associating every detection box,” 2022
2022
-
[7]
Bot-sort: Robust associa- tions multi-pedestrian tracking,
N. Aharon, R. Orfaig, and B.-Z. Bobrovsky, “Bot-sort: Robust associa- tions multi-pedestrian tracking,”arXiv preprint arXiv:2206.14651, 2022
Pith/arXiv arXiv 2022
-
[8]
Hard to track objects with irregular motions and similar appearances? make it easier by buffering the matching space,
F. Yang, S. Odashima, S. Masui, and S. Jiang, “Hard to track objects with irregular motions and similar appearances? make it easier by buffering the matching space,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2023, pp. 4799– 4808
2023
-
[9]
Observation- centric sort: Rethinking sort for robust multi-object tracking,
J. Cao, J. Pang, X. Weng, R. Khirodkar, and K. Kitani, “Observation- centric sort: Rethinking sort for robust multi-object tracking,” inPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 9686–9696
2023
-
[10]
No train yet gain: Towards generic multi-object tracking in sports and beyond,
T. Stanczyk, S. Yoon, and F. Bremond, “No train yet gain: Towards generic multi-object tracking in sports and beyond,” inProceedings of the Computer Vision and Pattern Recognition Conference (CVPR) Workshops, June 2025, pp. 6039–6048
2025
-
[11]
Topic: A parallel association paradigm for multi-object tracking under complex motions and diverse scenes,
X. Cao, Y . Zheng, Y . Yao, H. Qin, X. Cao, and S. Guo, “Topic: A parallel association paradigm for multi-object tracking under complex motions and diverse scenes,”IEEE Transactions on Image Processing, vol. 34, pp. 743–758, 2025
2025
-
[12]
A New Approach to Linear Filtering and Prediction Problems,
R. E. Kalman, “A New Approach to Linear Filtering and Prediction Problems,”Journal of Basic Engineering, vol. 82, no. 1, pp. 35–45, 03
-
[13]
Putting the object back into video object segmentation,
H. K. Cheng, S. W. Oh, B. Price, J.-Y . Lee, and A. Schwing, “Putting the object back into video object segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024
2024
-
[14]
Segment anything,
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Doll ´ar, and R. Gir- shick, “Segment anything,”Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023
2023
-
[15]
Scalable person re-identification: A benchmark,
L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian, “Scalable person re-identification: A benchmark,” inProceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015
2015
-
[16]
Towards re-identification for long-term tracking of group housed pigs,
M. Wang, M. Larsen, D. Liu, J. Winters, J.-L. Rault, and T. Norton, “Towards re-identification for long-term tracking of group housed pigs,”Biosystems Engineering, vol. 222, p. 71–81, 10 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S1537511022001799
2022
-
[17]
Individual honey bee tracking in a beehive environment using deep learning and kalman filter,
P. Kongsilp, U. Taetragool, and O. Duangphakdee, “Individual honey bee tracking in a beehive environment using deep learning and kalman filter,”Scientific Reports, vol. 14, no. 11, p. 1061, 1 2024. [Online]. Available: https://www.nature.com/articles/s41598-023-44718-y
2024
-
[18]
Wildlifedatasets: An open-source toolkit for animal re-identification,
V . ˇCerm´ak, L. Picek, A. Luk ´aˇs, and K. Papafitsoros, “Wildlifedatasets: An open-source toolkit for animal re-identification,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 1 2024, pp. 5953–5963
2024
-
[19]
Facenet: A unified embedding for face recognition and clustering,
F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,”2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2015. [Online]. Available: http://dx.doi.org/10.1109/CVPR.2015.7298682
arXiv 2015
-
[20]
Fastreid: A pytorch toolbox for general instance re-identification,
L. He, X. Liao, W. Liu, X. Liu, P. Cheng, and T. Mei, “Fastreid: A pytorch toolbox for general instance re-identification,”arXiv preprint arXiv:2006.02631, 2020
Pith/arXiv arXiv 2006
-
[21]
Omni-scale feature learning for person re-identification,
K. Zhou, Y . Yang, A. Cavallaro, and T. Xiang, “Omni-scale feature learning for person re-identification,” inICCV, 2019
2019
-
[22]
Simple online and realtime tracking with a deep association metric,
N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in2017 IEEE International Conference on Image Processing (ICIP). IEEE, 2017, pp. 3645–3649
2017
-
[23]
Strong- sort: Make deepsort great again,
Y . Du, Z. Zhao, Y . Song, Y . Zhao, F. Su, T. Gong, and H. Meng, “Strong- sort: Make deepsort great again,”IEEE Transactions on Multimedia, 2023
2023
-
[24]
Biological control with trichogramma: Advances, successes, and potential of their use,
S. Smith, “Biological control with trichogramma: Advances, successes, and potential of their use,”Annual Review of Entomology, vol. 41, no. 1, p. 375–406, 1996. [Online]. Available: https://doi.org/10.1146/ annurev.en.41.010196.002111
arXiv 1996
-
[25]
Inundative release of trichogramma dendrolimi at different developmental stages enhances the control efficacy over ostrinia furnacalis,
Y . Wang, Y . Hou, A. Iqbal, S. Wang, L. Monticelli, N. Desneux, and L. Zang, “Inundative release of trichogramma dendrolimi at different developmental stages enhances the control efficacy over ostrinia furnacalis,”Journal of Pest Science, vol. 97, no. 4, p. 1889–1898, 9
-
[26]
Intrinsic inter-and intraspecific competition in parasitoid wasps,
J. Harvey, E. Poelman, and T. Tanaka, “Intrinsic inter-and intraspecific competition in parasitoid wasps,”Annual Review of Entomology, vol. 58, p. 333–351, 2013
2013
-
[27]
Patch exploitation by non- aggressive parasitoids under intra- and interspecific competition,
F.-A. Robert, J. Brodeur, and G. Boivin, “Patch exploitation by non- aggressive parasitoids under intra- and interspecific competition,”Ento- mologia Experimentalis et Applicata, vol. 159, no. 1, p. 92–101, 2016
2016
-
[28]
Trackeval,
A. H. Jonathon Luiten, “Trackeval,” https://github.com/JonathonLuiten/ TrackEval, 2020
2020
-
[29]
The hungarian method for the assignment problem,
H. W. Kuhn, “The hungarian method for the assignment problem,” Naval Research Logistics Quarterly, vol. 2, no. 1-2, pp. 83–97, 1955. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/nav. 3800020109
doi:10.1002/nav 1955
-
[30]
Hota: A higher order metric for evaluating multi-object tracking,
J. Luiten, A. Osep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taix ´e, and B. Leibe, “Hota: A higher order metric for evaluating multi-object tracking,”International Journal of Computer Vision, pp. 1–31, 2020
2020
-
[31]
Performance measures and a data set for multi-target, multi-camera tracking,
E. Ristani, F. Solera, R. Zou, R. Cucchiara, and C. Tomasi, “Performance measures and a data set for multi-target, multi-camera tracking,” in Computer Vision – ECCV 2016 Workshops, G. Hua and H. J ´egou, Eds. Cham: Springer International Publishing, 2016, pp. 17–35
2016
-
[32]
Evaluating multiple object tracking performance: The clear mot metrics,
K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking performance: The clear mot metrics,”EURASIP Journal on Image and Video Processing, vol. 2008, 01 2008
2008
-
[1960]
Available: https://doi.org/10.1115/1.3662552
[Online]. Available: https://doi.org/10.1115/1.3662552
-
[2024]
Available: https://doi.org/10.1007/s10340-023-01731-4
[Online]. Available: https://doi.org/10.1007/s10340-023-01731-4
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.