Polycepta: Object-Centric Appearance Estimation for Multi-Object Tracking
Pith reviewed 2026-06-26 08:33 UTC · model grok-4.3
The pith
Polycepta maintains an independent recursively updated appearance state for each tracked object to estimate future appearances from accumulated observations.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Polycepta constructs and continuously updates an independent appearance state for each tracked object, enabling future appearance representations to be estimated from accumulated observations. The quality of appearance estimation improves as object states evolve during inference. The framework learns the appearance-state construction process rather than memorizing specific appearances, allowing estimation for unseen classes. When integrated into tracking-by-detection systems it produces consistent reductions in identity switches and gains in tracking metrics on KITTI, Waymo Open Dataset, and MOT17 while running at 90.57 Hz.
What carries the argument
The object-centric appearance state, a per-object representation that is recursively estimated and updated from new observations to produce future appearance estimates.
If this is right
- Appearance estimates grow more accurate with each additional observation of the same object rather than staying static.
- Identity switches decline when the evolving states replace conventional appearance descriptors inside tracking-by-detection pipelines.
- Tracking metrics such as MOTA improve on KITTI, Waymo, and MOT17 without requiring heavy pretrained backbones.
- The system runs in real time at 90.57 Hz while delivering state-of-the-art results on KITTI when paired with RobMOT.
- The same state-construction learning enables appearance estimation on object classes absent from the training set.
Where Pith is reading between the lines
- The recursive state could be tested for long-term re-identification across video segments separated by minutes or hours.
- Similar per-object state maintenance might be applied to other sequential vision tasks such as action recognition or video prediction.
- If the learned construction process generalizes, it could reduce reliance on large class-specific training sets in other appearance-based systems.
- The method invites experiments that measure how quickly estimate quality saturates as the number of observations per object increases.
Load-bearing premise
That a learning strategy can be designed so the model learns the appearance-state construction process rather than memorizing specific appearances, allowing generalization to unseen classes and progressive refinement without post-hoc data selection or heavy pretrained backbones.
What would settle it
If adding Polycepta to a standard tracker produces no drop in identity switches or if appearance estimate quality fails to improve when more observations of the same object become available on the reported benchmarks, the central claim would be falsified.
Figures
read the original abstract
The tracking-by-detection paradigm in multi-object tracking (MOT) typically relies on static appearance descriptors to complement motion estimation. However, these descriptors are frame-independent, limiting their robustness as visual cues. Since such descriptors are often obtained from computationally intensive pretrained backbones, real-time MOT systems frequently abandon appearance cues altogether and rely solely on motion prediction and geometric association. In this work, we introduce Polycepta, an object-centric appearance state estimation framework that reformulates appearance modeling as a recursive estimation problem rather than a frame-wise matching task. Polycepta constructs and continuously updates an independent appearance state for each tracked object, enabling future appearance representations to be estimated from accumulated observations. Polycepta is encouraged to learn the appearance-state construction of object-specific representations rather than memorize them through a proposed learning strategy, enabling appearance estimation for unseen classes. A key property of Polycepta is that the quality of appearance estimation improves as object states evolve during inference. While conventional appearance descriptors remain static or degrade over time, Polycepta progressively refines appearance estimates as additional observations are accumulated. Extensive experiments on KITTI, the Waymo Open Dataset, and MOT17 demonstrate consistent reductions in identity switches and improvements in tracking performance when integrated into the tracking-by-detection pipelines. Polycepta operates at 90.57 Hz and delivers state-of-the-art performance on the KITTI benchmark when integrated into the RobMOT framework, achieving a MOTA of 92.27\%.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Polycepta, an object-centric appearance state estimation framework for multi-object tracking that reformulates appearance modeling as a recursive estimation problem. Each tracked object maintains an independent appearance state that is continuously updated from accumulated observations, with a proposed learning strategy intended to induce learning of the state-construction process (rather than memorization) to support generalization to unseen classes. A claimed property is that estimation quality improves as object states evolve during inference. When integrated into tracking-by-detection pipelines, the method yields reductions in identity switches and improved tracking metrics on KITTI, Waymo Open Dataset, and MOT17, achieving SOTA MOTA of 92.27% on KITTI within the RobMOT framework at 90.57 Hz.
Significance. If the central claims hold, Polycepta could provide a practical route to dynamic, observation-accumulating appearance cues in real-time MOT without dependence on heavy pretrained backbones, addressing a known limitation of static descriptors. The reported speed and benchmark gains on standard autonomous-driving and pedestrian datasets suggest potential for deployment impact, particularly if the recursive formulation demonstrably generalizes beyond the training distribution.
major comments (3)
- [Method section] Method section (description of the proposed learning strategy): The mechanism by which the learning strategy encourages recursive process learning rather than memorization of frame-specific descriptors is not formalized with loss equations, training objectives, or explicit anti-memorization components (e.g., contrastive terms or process-level supervision). Without this, the claim that the model generalizes to unseen classes and that estimates improve with accumulated observations cannot be evaluated from the provided evidence.
- [Experiments section] Experiments section: No ablation studies or quantitative analysis isolate the contribution of the recursive appearance-state update versus the base tracker (RobMOT) or standard appearance descriptors. The reported MOTA/ID-switch reductions on KITTI/Waymo/MOT17 could therefore be explained by integration effects rather than the claimed progressive refinement property.
- [Abstract and results] Abstract and results: The performance numbers (92.27% MOTA, 90.57 Hz) are given only for the integrated system; no standalone evaluation of Polycepta’s appearance estimation accuracy (e.g., reconstruction error vs. number of observations) is provided to support the central assertion that quality improves as states evolve.
minor comments (2)
- [Abstract] The abstract states that Polycepta 'operates at 90.57 Hz' but does not clarify whether this includes the full tracking pipeline or only the appearance module; a breakdown would improve clarity.
- [Method section] Notation for the appearance state (e.g., how the recursive update is denoted) is introduced without an accompanying equation or diagram in the visible text, which hinders immediate understanding of the recursive formulation.
Simulated Author's Rebuttal
We thank the referee for the constructive and detailed review. We address each major comment below and indicate the revisions planned for the next version of the manuscript.
read point-by-point responses
-
Referee: [Method section] Method section (description of the proposed learning strategy): The mechanism by which the learning strategy encourages recursive process learning rather than memorization of frame-specific descriptors is not formalized with loss equations, training objectives, or explicit anti-memorization components (e.g., contrastive terms or process-level supervision). Without this, the claim that the model generalizes to unseen classes and that estimates improve with accumulated observations cannot be evaluated from the provided evidence.
Authors: We agree that a more explicit mathematical formulation would improve clarity. The learning strategy described in Section 3 trains the estimator to predict future appearance states from partial observation histories rather than frame-specific descriptors; this is realized by withholding later frames during training and supervising the recursive update. In the revision we will add the precise loss equations, training objective, and any regularization terms used to discourage memorization. revision: yes
-
Referee: [Experiments section] Experiments section: No ablation studies or quantitative analysis isolate the contribution of the recursive appearance-state update versus the base tracker (RobMOT) or standard appearance descriptors. The reported MOTA/ID-switch reductions on KITTI/Waymo/MOT17 could therefore be explained by integration effects rather than the claimed progressive refinement property.
Authors: We recognize that isolating the recursive update is necessary to substantiate the central claim. The current results show end-to-end gains, yet they do not separate the contribution of Polycepta from the base tracker. In the revised manuscript we will insert dedicated ablation tables that compare the full system against RobMOT alone and against RobMOT augmented with conventional static descriptors, thereby quantifying the incremental benefit of the state-evolution mechanism. revision: yes
-
Referee: [Abstract and results] Abstract and results: The performance numbers (92.27% MOTA, 90.57 Hz) are given only for the integrated system; no standalone evaluation of Polycepta’s appearance estimation accuracy (e.g., reconstruction error vs. number of observations) is provided to support the central assertion that quality improves as states evolve.
Authors: The primary evaluation metric is tracking performance because that is the intended deployment setting. Nevertheless, the referee is correct that a direct measurement of appearance-state quality versus observation count would strengthen the progressive-refinement claim. We will add a new results subsection containing quantitative plots of reconstruction or prediction error as a function of the number of accumulated observations on held-out sequences. revision: yes
Circularity Check
No significant circularity; derivation self-contained
full rationale
The abstract and description frame Polycepta as reformulating appearance modeling into a recursive state estimation problem, with a proposed learning strategy intended to induce process learning rather than memorization. No equations, parameter-fitting procedures, self-citations, or uniqueness theorems are quoted that would reduce any claimed prediction or result to its inputs by construction. The central property (improving estimates with accumulated observations and generalization to unseen classes) is asserted as following from the learning strategy design, without evidence that the strategy itself is tautological or that results are statistically forced. This is the common case of an independent methodological claim pending external verification; no load-bearing circular steps are present.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Hybrid-sort: Weak cues matter for robust multi-object tracking,
J. Yang, M. Jiang, Z. Fanet al., “Hybrid-sort: Weak cues matter for robust multi-object tracking,”arXiv preprint arXiv:2303.00766, 2023
-
[2]
Bytetrack: Multi-object tracking by associating every detection box,
Y . Zhang, P. Sun, Y . Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu, and X. Wang, “Bytetrack: Multi-object tracking by associating every detection box,” inProceedings of the European Conference on Computer Vision (ECCV), 2022
2022
-
[3]
Afmtrack: Attention-based feature matching for multiple object tracking,
D. C. Bui and M. Yoo, “Afmtrack: Attention-based feature matching for multiple object tracking,”IEEE Access, vol. 12, 2024
2024
-
[4]
Strong- sort: Make deepsort great again,
Y . Du, Z. Zhao, Y . Song, Y . Zhao, F. Su, T. Gong, and H. Meng, “Strong- sort: Make deepsort great again,”IEEE Transactions on Multimedia, vol. 25, pp. 8725–8737, 2023
2023
-
[5]
Focusing on tracks for online multi-object tracking,
K. Shim, K. Ko, Y . Yang, and C. Kim, “Focusing on tracks for online multi-object tracking,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025
2025
-
[6]
Bag of tricks and a strong baseline for deep person re-identification,
H. Luo, Y . Gu, X. Liao, S. Lai, and W. Jiang, “Bag of tricks and a strong baseline for deep person re-identification,” in2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019, pp. 1487–1495
2019
-
[7]
Robmot: 3d multi-object tracking enhancement through observational noise and state estimation drift mitigation in lidar point clouds,
M. Nagy, N. Werghi, B. Hassan, J. Dias, and M. Khonji, “Robmot: 3d multi-object tracking enhancement through observational noise and state estimation drift mitigation in lidar point clouds,”IEEE Transactions on Intelligent Transportation Systems, vol. 26, no. 10, 2025
2025
-
[8]
Deepfusionmot: A 3d multi-object tracking framework based on camera-lidar fusion with deep association,
X. Wang, C. Fu, Z. Li, Y . Lai, and J. He, “Deepfusionmot: A 3d multi-object tracking framework based on camera-lidar fusion with deep association,”IEEE Robotics and Automation Letters, vol. 7, 2022
2022
-
[9]
Mctrack: A unified 3d multi-object tracking framework for autonomous driving,
X. Wang, S. Qi, J. Zhao, H. Zhou, S. Zhang, G. Wang, K. Tu, S. Guo, J. Zhao, J. Li, and M. Yang, “Mctrack: A unified 3d multi-object tracking framework for autonomous driving,” inProceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025
2025
-
[10]
A multi-modal fusion-based 3d multi-object tracking framework with joint detection,
X. Wang, C. Fu, J. He, M. Huang, T. Meng, S. Zhang, H. Zhou, Z. Xu, and C. Zhang, “A multi-modal fusion-based 3d multi-object tracking framework with joint detection,”IEEE Robotics and Automation Letters, vol. 10, no. 1, pp. 532–539, 2025
2025
-
[11]
Pnas-mot: Multi-modal object tracking with pareto neural architecture search,
C. Peng, Z. Zeng, J. Gao, J. Zhou, M. Tomizuka, X. Wang, C. Zhou, and N. Ye, “Pnas-mot: Multi-modal object tracking with pareto neural architecture search,”IEEE Robotics and Automation Letters, vol. 9, no. 5, pp. 4377–4384, 2024
2024
-
[12]
Fasttracker: Real-time and accurate visual tracking,
H. Hashempoor and Y . D. Hwang, “Fasttracker: Real-time and accurate visual tracking,” 2025. [Online]. Available: https://arxiv.org/abs/2508. 14370
2025
-
[13]
A new approach to linear filtering and prediction problems,
R. E. Kalman, “A new approach to linear filtering and prediction problems,”Journal of Basic Engineering, vol. 82, no. 1, pp. 35–45, 1960
1960
-
[14]
Memot: Multi-object tracking with memory,
J. Cai, M. Xu, W. Li, Y . Xiong, W. Xia, Z. Tu, and S. Soatto, “Memot: Multi-object tracking with memory,” in2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022
2022
-
[15]
Memotr: Long-term memory-augmented trans- former for multi-object tracking,
R. Gao and L. Wang, “Memotr: Long-term memory-augmented trans- former for multi-object tracking,” in2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 9867–9876
2023
-
[16]
Samba: Synchronized set-of-sequences modeling for multiple object tracking,
M. Segu, L. Piccinelli, S. Li, Y .-H. Yang, L. Van Gool, and B. Schiele, “Samba: Synchronized set-of-sequences modeling for multiple object tracking,” inInternational Conference on Learning Representations, Y . Yue, A. Garg, N. Peng, F. Sha, and R. Yu, Eds., vol. 2025, 2025, pp. 30 057–30 070
2025
-
[17]
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,”arXiv preprint arXiv:2312.00752, 2023
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[18]
Searching for mobilenetv3,
A. Howard, M. Sandler, B. Chen, W. Wang, L.-C. Chen, M. Tan, G. Chu, V . Vasudevan, Y . Zhu, R. Pang, H. Adam, and Q. Le, “Searching for mobilenetv3,” in2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 1314–1324
2019
-
[19]
Hippo: Recurrent memory with optimal polynomial projections,
A. Gu, T. Dao, S. Ermon, A. Rudra, and C. R ´e, “Hippo: Recurrent memory with optimal polynomial projections,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020
2020
-
[20]
The design and implementation of FFTW3,
M. Frigo and S. G. Johnson, “The design and implementation of FFTW3,”Proceedings of the IEEE, vol. 93, no. 2, pp. 216–231, 2005
2005
-
[21]
High-speed convolution and correlation,
T. G. Stockham, “High-speed convolution and correlation,”AFIPS ’66 (Spring): Proceedings of the April 26-28, 1966, Spring Joint Computer Conference, pp. 229–233, 1966
1966
-
[22]
Are we ready for autonomous driving? the kitti vision benchmark suite,
A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012
2012
-
[23]
Scalability in percep- tion for autonomous driving: Waymo open dataset,
P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V . Gunagi, C. Chai, B. Caine, V . Vasudevan, W. Han, J. Ngiamet al., “Scalability in percep- tion for autonomous driving: Waymo open dataset,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020
2020
-
[24]
Motchallenge: A benchmark for single- camera multiple target tracking,
P. Dendorfer, A. Osep, A. Milan, K. Schindler, D. Cremers, I. Reid, S. Roth, and L. Leal-Taix ´e, “Motchallenge: A benchmark for single- camera multiple target tracking,”International Journal of Computer Vision, vol. 129, no. 4, pp. 845–881, 2021
2021
-
[25]
Mot20: A bench- mark for multi object tracking in crowded scenes,
P. Dendorfer, H. Rezatofighi, A. Milan, J. Shi, D. Cremers, I. Reid, S. Roth, K. Schindler, and L. Leal-Taix ´e, “Mot20: A bench- mark for multi object tracking in crowded scenes,”arXiv preprint arXiv:2003.09003, 2020
-
[26]
An exponential moving average algorithm,
D. Haynes, S. Corns, and G. K. Venayagamoorthy, “An exponential moving average algorithm,” in2012 IEEE Congress on Evolutionary Computation, 2012, pp. 1–8
2012
-
[27]
Hadamard product arguments and their applications,
K. Lee, H. Ko, D. Oh, J. Kim, and H. Oh, “Hadamard product arguments and their applications,”IEEE Access, vol. 13, pp. 79 736–79 756, 2025
2025
-
[28]
Hota: A higher order metric for evaluating multi-object tracking,
J. Luiten, A. Osep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taix ´e, and B. Leibe, “Hota: A higher order metric for evaluating multi-object tracking,”International Journal of Computer Vision, pp. 1–31, 2020
2020
-
[29]
Trackeval,
A. H. Jonathon Luiten, “Trackeval,” https://github.com/JonathonLuiten/ TrackEval, 2020
2020
-
[30]
Casa: A cascade attention network for 3-d object detection from lidar point clouds,
W. et al., “Casa: A cascade attention network for 3-d object detection from lidar point clouds,”IEEE Transactions on Geoscience and Remote Sensing, 2022
2022
-
[31]
3d multi-object tracking in point clouds based on prediction confidence-guided data association,
H. Wu, W. Han, C. Wen, X. Li, and C. Wang, “3d multi-object tracking in point clouds based on prediction confidence-guided data association,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 6, pp. 5668–5677, 2022
2022
-
[32]
Polarmot: How far can geometric relations take us in 3d multi-object tracking?
A. Kim, G. Bras ´o, A. O ˇsep, and L. Leal-Taix ´e, “Polarmot: How far can geometric relations take us in 3d multi-object tracking?” inProceedings of the European Conference on Computer Vision (ECCV). Springer Nature Switzerland, 2022, pp. 41–58
2022
-
[33]
3d multi-object track- ing with boosting data association and improved trajectory management mechanism,
J. Jin, J. Zhang, K. Zhang, Y . Wang, and D. Pan, “3d multi-object track- ing with boosting data association and improved trajectory management mechanism,”Signal Processing, vol. 218, p. 109367, 2024
2024
-
[34]
3d multi-object tracking based on informatic divergence-guided data association,
J. He, X. Wang, C. Fu, Z. Liet al., “3d multi-object tracking based on informatic divergence-guided data association,”Signal Processing, vol. 222, p. 109521, 2024
2024
-
[35]
Towards Accurate State Estimation: Motion Dynamics Kalman Filter for 3D Multi-Object Tracking
M. Nagy, N. Werghi, B. Hassan, J. Dias, and M. Khonji, “Towards accurate state estimation: Kalman filter incorporating motion dynamics for 3d multi-object tracking,” 2025. [Online]. Available: https://arxiv.org/abs/2505.07254
work page internal anchor Pith review arXiv 2025
-
[36]
Joint multi-object detection and tracking with camera-lidar fusion for autonomous driving,
K. Huang and Q. Hao, “Joint multi-object detection and tracking with camera-lidar fusion for autonomous driving,” in2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021, pp. 6983–6989
2021
-
[37]
Strongfusionmot: A multi-object tracking method based on lidar-camera fusion,
X. Wang, C. Fu, J. He, S. Wang, and J. Wang, “Strongfusionmot: A multi-object tracking method based on lidar-camera fusion,”IEEE Sensors Journal, vol. 23, no. 1, pp. 472–483, 2023
2023
-
[38]
Boost correlation fea- tures with 3d-miiou-based camera-lidar fusion for modt in autonomous driving,
K. Zhang, Y . Liu, F. Mei, J. Jin, and Y . Wang, “Boost correlation fea- tures with 3d-miiou-based camera-lidar fusion for modt in autonomous driving,”Remote Sensing, vol. 15, no. 4, p. 874, 2023
2023
-
[39]
Ultralytics yolov8,
G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics yolov8,” 2023. [Online]. Available: https://github.com/ultralytics/ultralytics
2023
-
[40]
Virtual sparse convolution for multimodal 3d object detec- tion,
W. et al., “Virtual sparse convolution for multimodal 3d object detec- tion,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023
2023
-
[41]
YOLOX: Exceeding YOLO Series in 2021
Z. Ge, S. Liu, F. Wang, Z. Li, and J. Sun, “Yolox: Exceeding yolo series in 2021,”arXiv preprint arXiv:2107.08430, 2021. [Online]. Available: https://arxiv.org
work page internal anchor Pith review Pith/arXiv arXiv 2021
-
[42]
Pv- rcnn: Point-voxel feature set abstraction for 3d object detection,
S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, and H. Li, “Pv- rcnn: Point-voxel feature set abstraction for 3d object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020
2020
-
[43]
Pointrcnn: 3d object proposal generation and detection from point cloud,
S. Shi, X. Wang, and H. Li, “Pointrcnn: 3d object proposal generation and detection from point cloud,” inThe IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
2019
-
[44]
Second: Sparsely embedded convolutional detection,
Y . Yan, Y . Mao, and B. Li, “Second: Sparsely embedded convolutional detection,”Sensors, vol. 18, no. 10, p. 3337, 2018
2018
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.