Pith. sign in

REVIEW 5 major objections 6 minor 28 references

Post-Hoc MOTS: Exploring the Capabilities of Time-Symmetric Multi-Object Tracking

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read This paper shows that a time-symmetric tracker originally built for budding yeast cells transfers zero-shot to pedestrian multi-object tracking and segmentation, matching Tracktor's HOTA (48.56 vs 48.8) while nearly doubling its…

desk verdict The synthetic ablations are solid, but the headline MOTS claim—TS matching Tracktor with far higher AssA—rests on comparing training-set scores with the official test-set score, so it collapses. read the letter →

arxiv 2412.08313 v1 pith:ANUY5NEW submitted 2024-12-11 cs.CV cs.LG

classification cs.CVcs.LG
keywords time-symmetrictrackingMOTSHOTAinstancesegmentationKalmanfilterzero-shottransferpedestriansaliencymaps
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a time-symmetric tracking architecture, designed for budding yeast cells, transfers zero-shot to pedestrian multi-object tracking and segmentation (MOTS) and that its identity-association strength is its main asset. On three MOTS training-set sequences, the TS architecture attains a mean HOTA of 48.56, closely matching the 48.8 published for the Tracktor benchmark, while its mean AssA of 82.39 far exceeds Tracktor's 44.6. The paper interprets this as evidence that the tracking stage is strong and that the Mask R-CNN detection step is what holds overall performance back. Through synthetic scenarios, it also argues that combining positional and morphological cues in the assignment step beats using either alone, and that future visual cues can inform predictions about past motion.

What carries the argument

The load-bearing mechanism is time-symmetric local tracking: a DeepLabV3+ semantic-segmentation model consumes a $2T_R+1$-frame window with a marker channel for the object being tracked, and outputs predicted masks for all frames in that window, both backward and forward. Matching uses mean IoU between local predictions at temporal offsets up to $2T_R$, so the same object can be re-identified after temporary loss; the Hungarian method then makes globally optimal assignments across the sequence, a depth-first search merges IDs to the minimal count, and missed instances are re-interpolated. The comparison machinery is the HOTA family (HOTA, DetA, AssA) plus IoU-50 association scores, evaluated against a Kalman filter baseline and two ablated TS variants: TS-L2, which ignores morphology, and TS-Shape, which ignores position.

What would settle it

Re-run the TS pipeline on the official seven-sequence MOTS test set with the standard evaluation kit and no 10-frame track cutoff; if HOTA falls well below Tracktor's 48.8, the parity claim fails. A second check is to recompute HOTA on the three training clips with all short tracks included, which would reveal how much of the score depends on the cutoff.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated on its own terms, is that the TS architecture's time-symmetric tracking mechanism generalizes far beyond videomicroscopy. With models trained on synthetic pedestrian data (MOTSynth) and evaluated zero-shot on real-world MOTS sequences, the TS tracker reaches overall HOTA parity with Tracktor (48.56 vs 48.8) while achieving a much higher association accuracy (AssA 82.39 vs 44.6). The authors conclude that the architecture's ability to compare local forward and backward predictions in the same state space, and to re-interpolate missed instances, gives it superior identity tracking, and that the remaining gap to a clearly better overall score is in detection and segmentation, not tracking.

Load-bearing premise

The headline parity claim rests on treating HOTA computed on three MOTS training-set clips, with a 10-frame minimum track cutoff, as directly comparable to Tracktor's published score on the official test set.

Editorial extensions

If this is right

  • If the central claim holds, the association half of MOTS can be decoupled from detection: pairing the TS tracker with a stronger detector should push overall HOTA well above the Tracktor benchmark.
  • The large AssA advantage implies the TS tracker maintains stable identities across missed detections and occlusions much better than a forward-only box tracker, which matters for surveillance and crowd analysis on recorded footage.
  • Because the local tracker sees future frames, visual cues such as turn signals can be used to explain or predict earlier motion, an advantage unique to offline, post-hoc processing.
  • The ablation results imply that assignment by both position and morphology is more than the sum of its parts; a tracker that keeps either alone will be substantially weaker in cluttered or morphologically informative scenes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's comparison, the equal-HOTA / doubled-AssA pattern suggests a general recipe: take any forward tracker's boxes and run a time-symmetric re-association pass on the recorded sequence to recover identities, without retraining the detector.
  • The paper evaluates on three training clips; an obvious extension is to submit the same pipeline to the official MOTS test server with the current evaluation kit, which would turn the parity claim into a certified benchmark result.
  • The attention analysis's finding that pretrained temporal channels inherit RGB channel biases could be probed experimentally by fine-tuning with randomized channel orders; the paper itself reports no practical performance difference.
  • The 10-frame minimum-track cutoff inflates DetA and HOTA, so an independent re-evaluation that counts short tracks would clarify how much of the headline parity is tracking skill versus score curation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper evaluates the time-symmetric tracking (TS) architecture introduced in [23] beyond its original yeast-cell setting. It presents an optimized implementation of the tracking pipeline, defines IoU-50 association metrics, and compares four trackers (Kalman filter, full TS, TS-L2, TS-Shape) on synthetic arrow and amoeboid scenarios designed around visual signaling and positional noise. It then reports a zero-shot transfer experiment from the MOTSynth-MOTS-CVPR22 synthetic dataset to three MOTS training-set sequences, together with an attention analysis of the local tracking model. The principal claim is that TS achieves a MOTS HOTA of 48.56 versus 48.8 for Tracktor, with a much higher AssA (82.39 vs. 44.6), implying superior association with detection as the limiting factor.

Significance. The synthetic ablations are thoughtfully designed and provide clean evidence that combining positional and morphological information helps in the tested regimes; the optimized implementation with memory and runtime comparisons is a useful engineering contribution; and the attention analysis is a reasonable first look at the model's temporal and spatial focus. The paper also makes code, data, and models available, which supports reproducibility. If the MOTS benchmark claim were placed on a controlled footing, the result would be of broad interest for offline and post-hoc tracking. As it stands, the headline benchmark is not established because the comparison uses different test sets and different post-processing, and the metric computation is not fully specified.

major comments (5)
  1. [§5.3, §6.2, Fig. 6] The headline comparison of TS with Tracktor is uncontrolled. The reported TS mean HOTA of 48.56 and AssA of 82.39 are computed on three sequences from the MOTS training set, while the quoted Tracktor values of 48.8 and 44.6 are the published official MOTSynth-MOTS-CVPR22 test-set scores. Different image sets, different difficulty levels, and different evaluation protocols make the statement that the two HOTA scores 'closely match' unsupported. The fix is to run Tracktor on exactly the same three sequences with the same track-filtering and scoring code and to report per-sequence numbers; otherwise the benchmark framing in §6.2 and the conclusion should be removed or explicitly downgraded to a non-comparative report.
  2. [§4, §6] The 10-frame minimum track-length filter biases the comparison. The paper applies the filter to all four of its own models, but the quoted Tracktor scores were not produced under this filter. The paper itself states in §6 that excluding short tracks 'inflates DetA and HOTA scores.' This means the HOTA comparison with Tracktor cannot be attributed to tracking quality. Report all metrics both with and without the filter, give the number of removed tracks per model, and ensure that any external benchmark comparison uses identical postprocessing on both sides.
  3. [§4, §5.3] There is an inconsistency in the temporal ranges used in the ablation. Section 4 states that the Kalman filter was allowed a maximum re-interpolation distance of 8 frames to match a local-tracker TR value of 4, with maximum assignment distance 2TR = 8. Section 5.3 then says that for the MOTS experiment the TS model used a local tracking range of 2. If TR differs between the synthetic and MOTS experiments, the Kalman filter's maximum re-interpolation distance should be set to 2TR = 4 for the MOTS comparison; if it was not changed, the Kalman baseline is not matched to the TS model's temporal range. Please clarify and rerun the affected comparison.
  4. [§6, Fig. 6] The paper reports KDEs and 'mean values' for HOTA and AssA without defining the unit over which these quantities are computed. In the HOTA metric family, HOTA and AssA are globally defined per sequence, not per frame. If the KDEs are computed over frames, a per-frame adaptation of the metric must be defined; otherwise the reported mean HOTA of 48.56 is not the standard HOTA and cannot be compared with Tracktor's published value. Please specify the computation unit (frames, sequences, or sliding windows) and provide sequence-level HOTA and AssA values in addition to any frame-level distributions.
  5. [§6.2] The assertion that 'detection is the limiting factor' is not directly supported. No detection-only evaluation of the Mask R-CNN stage on the same three sequences is reported, and no comparison of the detection quality of Tracktor and TS is given. The AssA gap alone cannot separate detection errors from association errors. Please report DetA and detection recall/precision per sequence for both Tracktor and TS, and ideally run a cross-pipeline experiment such as TS tracking applied to Tracktor detections or Tracktor applied to TS detections.
minor comments (6)
  1. [§6.3] The attention analysis is based on one test sample of ten frames; the convergence claim is plausible, but a second sample or per-sample variance would make the conclusions about pretraining effects more robust.
  2. [§2] Please clarify the notation for the temporal window: '2TR + 1' appears as both an input/output channel count and a temporal range; state explicitly whether TR = 2 gives a 5-frame window for the MOTS model.
  3. [§3.1, Eqs. (3)-(5)] The summation symbol is rendered as an italic 'X' in Eq. (3), and the definition of the intersection of IoU-50 indicators should be spelled out as a sum over matched pairs.
  4. [Abstract, §8] The phrase 'state-of-the-art overall segmentation and tracking performance on the MOTS personnel tracking dataset' should be softened until the controlled comparison requested above is available.
  5. [§5.3] The sentence about the official test set is confusing because it first says the official test set has seven samples and then says the paper uses three training-set sequences; please state explicitly that the paper does not use the official test set and explain why.
  6. [Abstract] The code, data, and models link is a Google Drive folder; a versioned archive such as Zenodo or a GitHub release with a commit hash would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical evaluation whose central comparisons rest on independent experiments and an external benchmark, not on a self-referential derivation.

full rationale

The paper's central claims are empirical evaluations of the TS architecture on synthetic datasets and on MOTS-style pedestrian data. The HOTA/AssA numbers for the TS model come from the authors' own forward pass over trained models, and the Tracktor benchmark is an external published result; neither is derived from the paper's inputs by construction. The authors rely on their prior work [23] for the architecture definition and for synthetic scenario baselines, but this is ordinary self-citation to prior implementation, not a load-bearing reduction of the present result. The comparison between TS HOTA on three MOTS training sequences and Tracktor's official test-set HOTA is uncontrolled and likely invalid as a benchmark claim, but that is an experimental-validity flaw, not circularity. Likewise, the omission of tracks shorter than 10 frames is an acknowledged post-processing choice that inflates DetA/HOTA, but the paper explicitly warns against over-interpreting those inflated values; it does not redefine a fitted parameter as a prediction. No equation in the paper reduces to another by definition, and no fitted parameter is renamed as a prediction. The attention analysis and metric definitions are self-contained. Therefore the paper shows no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central tracking architecture is inherited from the authors' prior work, so the ledger mainly contains hand-chosen evaluation choices: TR values, the 10-frame track cutoff, and the test-set substitution. No new entities are proposed.

free parameters (3)
  • Track length cutoff = 10 frames
    Sec 4: "we omitted any tracks shorter than 10 frames in length"; this hand-chosen threshold directly inflates DetA and HOTA, as the authors acknowledge in Sec 6.
  • Local tracking temporal range TR = 4 for synthetic, 2 for MOTS
    Sec 2 and Sec 5.3: TR fixes the 2TR+1 input/output channels and maximum assignment offset. TR=2 for MOTS was chosen for computational reasons, not optimized.
  • Kalman max re-interpolation distance = 8 frames
    Sec 4: set to match 2TR=8; a hyperparameter chosen by the authors for comparison fairness.
assumptions (4)
  • domain assumption HOTA/AssA computed on bounding boxes are appropriate for comparing a segmentation-based tracker to box-based Tracktor.
    Sec 3.2 advises IoU50 as the primary metric and HOTA as a comparative measure; the representation mismatch may obscure segmentation potential.
  • ad hoc to paper The three MOTS training sequences chosen by the authors are a valid proxy for the official MOTS test set when comparing to Tracktor's published score.
    Sec 5.3 replaces the official test set with three training sequences; no evidence of comparability is given.
  • ad hoc to paper Removing tracks shorter than 10 frames does not bias the HOTA comparison in favor of TS.
    Sec 4 introduces the cutoff; Sec 6 concedes it inflates DetA and HOTA, so the comparison is not neutral.
  • domain assumption The MOTSynth-trained Mask R-CNN and local tracker transfer to MOTS without fine-tuning.
    Sec 5.3 uses MOTSynth-trained models for zero-shot evaluation on real MOTS samples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Post-Hoc MOTS: Exploring the Capabilities of Time-Symmetric Multi-Object Tracking." pith.science (2026). https://pith.science/paper/ANUY5NEW

@misc{pith2026241208313,
  author       = {Pith},
  title        = {Pith review of: Post-Hoc MOTS: Exploring the Capabilities of Time-Symmetric Multi-Object Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ANUY5NEW}},
  note         = {Machine review of arXiv:2412.08313}
}
read the original abstract

Temporal forward-tracking has been the dominant approach for multi-object segmentation and tracking (MOTS). However, a novel time-symmetric tracking methodology has recently been introduced for the detection, segmentation, and tracking of budding yeast cells in pre-recorded samples. Although this architecture has demonstrated a unique perspective on stable and consistent tracking, as well as missed instance re-interpolation, its evaluation has so far been largely confined to settings related to videomicroscopic environments. In this work, we aim to reveal the broader capabilities, advantages, and potential challenges of this architecture across various specifically designed scenarios, including a pedestrian tracking dataset. We also conduct an ablation study comparing the model against its restricted variants and the widely used Kalman filter. Furthermore, we present an attention analysis of the tracking architecture for both pretrained and non-pretrained models

Figures

Figures reproduced from arXiv: 2412.08313 by the authors.

Figure 1
Figure 1. Data flow diagram of the TS architecture, illustrating the process from raw input image sequence to finalized track predictions. step influences the final results, it employs a fairly stan￾dard, approach using a Mask R-CNN [11] architecture with a ResNet-X [10] feature pyramid backbone [15], defined and trained in the Detectron 2 environment [28]. Therefore, our analysis will primarily focus on the tracking segment … view at source ↗
Figure 2
Figure 2. A display of memory usage and runtime differences be [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. KDE (top) and mean (bottom) metric results of tracker [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: KDE (top) and mean (bottom) metric results of [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: KDE (top) and mean (bottom) metric results of tracker [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: A display of radial (top) and temporal (bottom) relative [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 19 canonical work pages

  1. [23]

    Enhancing Cell Tracking with a Time-Symmetric Deep Learning Approach

    Gergely Szab ´o, Paolo Bonaiuti, Andrea Ciliberto, and Andr´as Horv ´ath. Enhancing cell tracking with a time- symmetric deep learning approach. arXiv preprint arXiv:2308.03887, 2023. 1, 2, 4, 5, 8

  2. [1]

    An improved forest fire detection method based on the detectron2 model and a deep learning approach

    Akmalbek Bobomirzaevich Abdusalomov, Bappy MD Si- ful Islam, Rashid Nasimov, Mukhriddin Mukhiddinov, and Taeg Keun Whangbo. An improved forest fire detection method based on the detectron2 model and a deep learning approach. Sensors, 23(3):1512, 2023. 7

  3. [2]

    Sanity checks for saliency maps

    Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Good- fellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018. 7

  4. [3]

    Saliency map based data augmentation

    Jalal Al-Afandi, B ´alint Magyar, and Andr ´as Horv ´ath. Saliency map based data augmentation. In 2022 26th Inter- national Conference on Pattern Recognition (ICPR) , pages 4751–4757. IEEE, 2022. 7

  5. [4]

    Statistical inference for probabilistic functions of finite state markov chains

    Leonard E Baum and Ted Petrie. Statistical inference for probabilistic functions of finite state markov chains. The an- nals of mathematical statistics, 37(6):1554–1563, 1966. 4

  6. [5]

    Tracking without bells and whistles

    Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe. Tracking without bells and whistles. In Proceedings of the IEEE/CVF international conference on computer vision, pages 941–951, 2019. 5

  7. [6]

    Encoder-decoder with atrous separable convolution for semantic image segmentation

    Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 2

  8. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. 7

Show all 28 references
  1. [8]

    Motsynth: How can synthetic data help pedestrian detection and tracking? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10849–10859, 2021

    Matteo Fabbri, Guillem Bras ´o, Gianluca Maugeri, Orcun Cetintas, Riccardo Gasparini, Aljo ˇsa Oˇsep, Simone Calder- ara, Laura Leal-Taix´e, and Rita Cucchiara. Motsynth: How can synthetic data help pedestrian detection and tracking? In Proceedings of the IEEE/CVF Internationa...

  2. [9]

    G.D. Forney. The viterbi algorithm. Proceedings of the IEEE, 61(3):268–278, 1973. 4

  3. [10]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 2

  4. [11]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 2

  5. [12]

    Segmentation models pytorch

    Pavel Iakubovskii. Segmentation models pytorch. https: //github.com/qubvel/segmentation_models. pytorch, 2019. 2

  6. [13]

    The hungarian method for the assignment problem

    Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83–97,

  7. [14]

    K ´alm´an

    Rudolf E. K ´alm´an. A new approach to linear filtering and prediction problems. Journal of Basic Engineering , 82(1): 35–45, 1960. 1, 3

  8. [15]

    Feature pyra- mid networks for object detection

    Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyra- mid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2117–2125, 2017. 2

  9. [16]

    Hota: A higher order metric for evaluating multi-object tracking

    Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taix´e, and Bastian Leibe. Hota: A higher order metric for evaluating multi-object tracking. International journal of computer vision, 129:548– 578, 2021. 2, 3

  10. [17]

    Neural kalman filtering

    Beren Millidge, Alexander Tschantz, Anil Seth, and Christo- pher Buckley. Neural kalman filtering. arXiv preprint arXiv:2102.10021, 2021. 3

  11. [18]

    Investiga- tions of object detection in images/videos using various deep learning techniques and embedded platforms—a comprehen- sive review

    Chinthakindi Balaram Murthy, Mohammad Farukh Hashmi, Neeraj Dhanraj Bokde, and Zong Woo Geem. Investiga- tions of object detection in images/videos using various deep learning techniques and embedded platforms—a comprehen- sive review. Applied sciences, 10(9):3280, 2020. 7

  12. [19]

    An image synthesizer

    Ken Perlin. An image synthesizer. ACM Siggraph Computer Graphics, 19(3):287–296, 1985. 5

  13. [20]

    Road damage detection and classification with detectron2 and faster r-cnn

    Vung Pham, Chau Pham, and Tommy Dang. Road damage detection and classification with detectron2 and faster r-cnn. In 2020 IEEE International Conference on Big Data (Big Data), pages 5592–5601. IEEE, 2020. 7

  14. [21]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 4

  15. [22]

    Deep inside convolutional networks: Visualising image classification models and saliency maps

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013. 7

  16. [24]

    Depth-first search and linear graph algo- rithms

    Robert Tarjan. Depth-first search and linear graph algo- rithms. SIAM journal on computing , 1(2):146–160, 1972. 2

  17. [25]

    Variable kernel density estimation

    George R Terrell and David W Scott. Variable kernel density estimation. The Annals of Statistics, pages 1236–1265, 1992. 5

  18. [26]

    Mots: Multi-object tracking and segmen- tation

    Paul V oigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe. Mots: Multi-object tracking and segmen- tation. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 7942–...

  19. [27]

    The unscented kalman filter for nonlinear estimation

    Eric A Wan and Rudolph Van Der Merwe. The unscented kalman filter for nonlinear estimation. In Proceedings of the IEEE 2000 adaptive systems for signal processing, commu- nications, and control symposium (Cat. No. 00EX373), pages 153–158. Ieee, 2000. 3

  20. [28]

    Detectron2

    Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github. com/facebookresearch/detectron2, 2019. 2 9

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.