REVIEW 3 major objections 4 minor 26 references
Probabilistic Trajectory GOSPA: A Metric for Uncertainty-Aware Multi-Object Tracking Performance Evaluation
T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read PTGOSPA extends tracking error metrics to trajectories reported as uncertain Bernoulli densities.
desk verdict A natural and useful extension of TGOSPA to track-level uncertainties, but the central metricity claim is asserted by analogy rather than proven, so the paper needs a serious revision before the main result can be accepted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a trajectory-level assignment problem lifted into the space of Bernoulli densities. Each trajectory is a sequence of Bernoulli densities, each described by an existence probability $r$ and a single-object density $p(\cdot)$; at every time step $k$, a binary assignment vector $\pi^k$ pairs estimated trajectories to true trajectories, and the per-step cost (9) is built from the PGOSPA pair cost $d_P(x_i^k,y_j^k)^p$ with the assignment feasibility condition $d_b(x_i^k(p), y_j^k(p)) < c$, plus expected missed and false detection terms weighted by existence probabilities. Between time steps, the switching cost (11) charges $\gamma^p$ for full assignment changes and $\gamma^p/2$ for assignment-to-unassignment changes. Replacing binary assignments by a doubly stochastic relaxation gives the LP formulation (18), whose cost matrix $D^k_{X,Y}$ encodes the same Bernoulli-level distances and absence costs. This construction is what allows the final error to split into five named components.
What would settle it
Brute-force a small triple of sets of Bernoulli sequences with $K=2$ or $3$ time steps and small $n_X, n_Y$, evaluating Definition 3 by enumerating all assignment vectors $\pi^k$; if for any $c, \gamma, p$ the values violate $d(X,Z) \le d(X,Y) + d(Y,Z)$, the metric claim is false. The same check on Eq. (18) would test the relaxation claim.
Extended reading notes
Core claim
The central claim is that the distance between two sets of trajectory estimates can be defined at the level of Bernoulli densities while still obeying the metric axioms, so that uncertainty enters the evaluation itself rather than being thrown away before scoring. Concretely, PTGOSPA assigns entire estimated trajectories to true trajectories over the evaluation window; at each time step the per-pair cost is the PGOSPA distance between the two Bernoulli densities, with assignment allowed only when the base density distance is below the cutoff $c$, and a switching cost penalizes changes in trajectory-level assignment between consecutive steps. The resulting cost (Definition 3, Eq. (8)) is claimed to be a true metric on the space of sets of time sequences of Bernoulli densities, and its linear programming relaxation (Eq. (18)), which allows soft assignments, is claimed to be a metric as well. The proof is presented as following analogously to the TGOSPA proof, with TGOSPA recovered as the special case where all existence probabilities are one and single-object densities are Dirac deltas. The simulation comparison of TPMBM and TPMB filters shows the two metrics largely agree in trend but diverge when objects die, where PTGOSPA charges the error to existence-probability mismatch instead of missed detection.
Load-bearing premise
The load-bearing premise is that the PTGOSPA construction in Definition 3, and its linear programming relaxation, satisfy the triangle inequality; the paper asserts this by analogy with TGOSPA rather than giving a proof, so if the analogy fails the central claim collapses.
Editorial extensions
If this is right
- Two trackers can be compared on their full posterior output rather than on point estimates extracted from them, so evaluation no longer discards track-level uncertainty.
- The binary and relaxed formulations give a polynomial-time computable lower bound, letting large evaluation problems be solved approximately with soft assignment.
- Decomposed PTGOSPA reports expected localization, existence mismatch, missed and false detection, and track switch errors separately, so a poor score can be traced to a specific failure mode.
- TGOSPA is a special case, meaning existing TGOSPA deployments can be upgraded to PTGOSPA without changing the ground truth representation.
- The simulation shows that uncertainty-aware scoring changes the qualitative diagnosis at object death, shifting the reported error from missed detection to existence probability mismatch.
Reading between the lines
- Because PGOSPA is built on the $p$-Wasserstein distance between Bernoulli densities, PTGOSPA's localization term inherits that Wasserstein interpretation; one testable consequence is that sensitivity to overconfident tracks is governed by the min-existence weighting $\min(r_x,r_y)$.
- The switching cost, applied to trajectory-level assignments through soft matrices $W_k$, could be adapted to penalize identity switches with unequal costs for birth versus swap, a direction the paper only hints at through the $\gamma^p/2$ half-switch.
- The metric could serve as a training loss or reward for learning-based MOT, since the LP relaxation is differentiable in the density parameters; the paper mentions quantifying filter approximation errors, not end-to-end learning.
- Whether the triangle inequality really holds for the full assignment formulation depends on the proof-by-analogy from TGOSPA; a direct verification on random triples of small trajectory sets would settle it before wide deployment.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript defines PTGOSPA, a proposed metric on finite sets of time sequences of Bernoulli densities, by combining trajectory-level assignments with per-time-step PGOSPA costs and a switching cost. It states that PTGOSPA and its linear programming relaxation are metrics (Definition 3, Eq. (8), and Proposition 1, Eq. (18)), formulates an equivalent binary linear program (Lemma 1), derives a five-term decomposition, and illustrates the quantities on a TPMBM/TPMB simulation. The central claim is the metric property of Eqs. (8) and (18), but no proof is given in the paper.
Significance. If the metric property is correct, PTGOSPA would be a valuable uncertainty-aware generalization of TGOSPA, with a clear decomposition and a polynomial-time computable relaxation. The paper deserves credit for the explicit construction, the interpretable decomposition, and the simulation-based demonstration that uncertainty information changes the error profile in a meaningful way. The significance is, however, conditional: the main theorem-like claims, metricity of Definition 3 and Proposition 1, are asserted by analogy and are not demonstrated in the manuscript.
major comments (3)
- [Section III.B, Definition 3 (Eq. (8))] The claim that d(X,Y) is a metric is not supported. The sentence 'The proof that PTGOSPA is a mathematically well-defined metric follows analogously to that of TGOSPA' is not a proof. The TGOSPA proof in [13] relies on a per-time-step cost that is the p-th power of a metric on an augmented object space with a constant dummy distance c/2^{1/p}. In PTGOSPA the unassignment cost is r c^p/2, which depends on the existence probability of the Bernoulli component. This can be interpreted as a distance to a dummy Bernoulli component with r=0, but then the triangle inequality for the augmented space and the gluing argument over time must be verified explicitly. The inequality in (13) is necessary but not sufficient. Please provide a complete proof, or a precise reference to a proof that covers this weighted-dummy case.
- [Section III.C, Proposition 1 (Eq. (18))] The same problem occurs for the LP relaxation. The statement that the proof 'follows analogously' to [13, App. B] is not adequate, because the cost matrix D^k in (16) contains existence-probability-weighted dummy entries and zeros for inactive pairs; the relaxation objective in (18) is not the same object as in [13, Eq. (23)]. The metricity of the relaxation, in particular the triangle inequality for soft assignments, needs to be proved, or the claim should be downgraded to a conjecture.
- [Section III.D / Lemma 1 (Eqs. (15) and (19))] The equivalence between Definition 3 and the binary linear program requires an additional observation that is not stated: if two nonempty Bernoulli densities have d_b >= c, then by (5), d_P(x,y)^p = (x(r)+y(r)) c^p/2, so assigning them costs exactly the same as leaving both unassigned. Without this observation, the restriction in (10) to d_b < c and the unqualified entries d_P(x_i^k,y_j^k)^p in D^k are not obviously consistent. Please add this verification.
minor comments (4)
- [Section III.A, Eq. (9)] The notation X(θ~^k(π^k)) is undefined; earlier only X(π~^k) was defined. Please clarify what the tilde denotes.
- [Section III.C, Eqs. (14a)-(14b) and (16)] In Eqs. (14a)-(14b), 'n nY' and 'n nX' should read n_Y and n_X; in Eq. (16), the condition 'y_i^k' should read 'y_j^k'.
- [Sections II.B and III.A] The paper allows r=0 in the space B, but PGOSPA in Section II.B assumes r>0 for all Bernoulli components. Since components with r=0 are indistinguishable from the empty density, the authors should state whether Bernoulli sequences are restricted to r>0 or whether the metric space is quotiented accordingly, so that identity of indiscernibles holds.
- [Section IV] The qualitative interpretation of the difference between TGOSPA and PTGOSPA missed/existence-mismatch errors near object death is plausible but is supported only by single averaged curves; reporting Monte Carlo variability would strengthen the claim.
Circularity Check
No circularity: PTGOSPA is a new construction from PGOSPA and TGOSPA; the deferred metricity proof is a rigor gap, not a circular reduction.
full rationale
The paper defines PTGOSPA (Definition 3, Eq. 8) by composing the authors' earlier PGOSPA per-step cost (Eq. 5) with the TGOSPA assignment-and-switching structure. The definition is not given in terms of the target distance itself, and the claimed decomposition (Section III-D) is an algebraic rewriting of the objective, not a fitted parameter relabeled as a prediction. No parameter is fitted to a subset of data and then presented as a predicted quantity. The metric proofs are deferred by analogy: 'The proof that PTGOSPA is a mathematically well-defined metric follows analogously to that of TGOSPA' and similarly for the LP relaxation. This is an unverified transfer of a proof from a published prior result with overlapping authors, but the quoted statements do not reduce any equation of this paper to its own input; TGOSPA's proof is independent of the PTGOSPA construction. Likewise, citing PGOSPA [16] supplies the building-block metric rather than assuming the target claim. The self-citations are load-bearing in the sense that the central metricity assertion is not proved self-contained, but that is a correctness/rigor concern, not one of the enumerated circularity patterns. Consequently, no specific circular step can be exhibited, and the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (4)
- ad hoc to paper PTGOSPA as defined in (8) satisfies the triangle inequality and is a metric.
- ad hoc to paper The linear programming relaxation (18) is a metric.
- domain assumption PGOSPA (4) is a metric on multi-Bernoulli densities.
- domain assumption The base metric db on single object densities is a metric (e.g., the 2-Wasserstein distance).
Cite this review
Pith. "Pith review of Probabilistic Trajectory GOSPA: A Metric for Uncertainty-Aware Multi-Object Tracking Performance Evaluation." pith.science (2026). https://pith.science/paper/HBYMIVGO
@misc{pith2026250615148,
author = {Pith},
title = {Pith review of: Probabilistic Trajectory GOSPA: A Metric for Uncertainty-Aware Multi-Object Tracking Performance Evaluation},
year = {2026},
howpublished = {\url{https://pith.science/paper/HBYMIVGO}},
note = {Machine review of arXiv:2506.15148}
}
read the original abstract
This paper presents a generalization of the trajectory general optimal sub-pattern assignment (GOSPA) metric for evaluating multi-object tracking algorithms that provide trajectory estimates with track-level uncertainties. This metric builds on the recently introduced probabilistic GOSPA metric to account for both the existence and state estimation uncertainties of individual object states. Similar to trajectory GOSPA (TGOSPA), it can be formulated as a multidimensional assignment problem, and its linear programming relaxation--also a valid metric--is computable in polynomial time. Additionally, this metric retains the interpretability of TGOSPA, and we show that its decomposition yields intuitive costs terms associated to expected localization error and existence probability mismatch error for properly detected objects, expected missed and false detection error, and track switch error. The effectiveness of the proposed metric is demonstrated through a simulation study.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[13]
´A. F. Garc ´ıa-Fern´andez, A. S. Rahmathullah, and L. Svensson, “A metric on the space of finite sets of trajectories for evaluation of multi- target tracking algorithms,” IEEE Transactions on Signal Processing , vol. 68, pp. 3917–3928, 2020
work page 2020
-
[1]
S. S. Blackman and R. Popoli, Design and analysis of modern tracking systems. Artech House Publishers, 1999
work page 1999
-
[2]
Evaluating multiple object tracking performance: the clear MOT metrics,
K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking performance: the clear MOT metrics,”EURASIP Journal on Image and Video Processing, vol. 2008, pp. 1–10, 2008
work page 2008
-
[3]
HOTA: A higher order metric for evaluating multi- object tracking,
J. Luiten, A. Osep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taix ´e, and B. Leibe, “HOTA: A higher order metric for evaluating multi- object tracking,” International Journal of Computer Vision , vol. 129, pp. 548–578, 2021
work page 2021
-
[4]
A metric for performance evaluation of multi-target tracking algorithms,
B. Ristic, B.-N. V o, D. Clark, and B.-T. V o, “A metric for performance evaluation of multi-target tracking algorithms,” IEEE Transactions on Signal Processing, vol. 59, no. 7, pp. 3452–3457, 2011
work page 2011
-
[5]
Performance evaluation methods for multiple-target-tracking algorithms,
B. E. Fridling and O. E. Drummond, “Performance evaluation methods for multiple-target-tracking algorithms,” in Signal and Data Process- ing of Small Targets 1991 , vol. 1481. SPIE, 1991, pp. 371–383
work page 1991
-
[6]
Ambiguities in evaluating performance of multiple target tracking algorithms,
O. E. Drummond and B. E. Fridling, “Ambiguities in evaluating performance of multiple target tracking algorithms,” in Signal and Data Processing of Small Targets 1992 , vol. 1698. SPIE, 1992, pp. 326–337
work page 1992
-
[7]
A new metric between distributions of point processes,
D. Schuhmacher and A. Xia, “A new metric between distributions of point processes,” Advances in Applied Probability , vol. 40, no. 3, pp. 651–672, 2008
work page 2008
Show all 26 references
-
[8]
A consistent metric for performance evaluation of multi-object filters,
D. Schuhmacher, B.-T. V o, and B.-N. V o, “A consistent metric for performance evaluation of multi-object filters,” IEEE Transactions on Signal Processing, vol. 56, no. 8, pp. 3447–3457, 2008
2008
-
[9]
Gen- eralized optimal sub-pattern assignment metric,
A. S. Rahmathullah, ´A. F. Garc´ıa-Fern´andez, and L. Svensson, “Gen- eralized optimal sub-pattern assignment metric,” in 20th International Conference on Information Fusion (Fusion) . IEEE, 2017, pp. 1–8
2017
-
[10]
Spooky effect in optimal OSPA estimation and how GOSPA solves it,
´A. F. Garc ´ıa-Fem´andez and L. Svensson, “Spooky effect in optimal OSPA estimation and how GOSPA solves it,” in 22th International Conference on Information Fusion (FUSION) . IEEE, 2019, pp. 1–8
2019
-
[11]
An analysis on metric-driven multi-target sensor management: GOSPA versus OSPA,
´A. F. Garc´ıa-Fern´andez, M. Hernandez, and S. Maskell, “An analysis on metric-driven multi-target sensor management: GOSPA versus OSPA,” in 24th International Conference on Information Fusion (FUSION). IEEE, 2021, pp. 1–8
2021
-
[12]
OSPA(2): Using the OSPA metric to evaluate multi-target tracking performance,
M. Beard, B. T. V o, and B.-N. V o, “OSPA(2): Using the OSPA metric to evaluate multi-target tracking performance,” in 2017 International Conference on Control, Automation and Information Sciences (IC- CAIS). IEEE, 2017, pp. 86–91
2017
-
[14]
TGOSPA metric parameters selection and evaluation for visual multi-object tracking,
J. Krej ˇc´ı, O. Kost, O. Straka, Y . Xia, L. Svensson, and ´A. F. Garc´ıa- Fern´andez, “TGOSPA metric parameters selection and evaluation for visual multi-object tracking,” arXiv preprint arXiv:2412.08321, 2024
2024 arXiv
-
[15]
A time-weighted metric for sets of trajectories to assess multi-object tracking algorithms,
´A. F. Garc ´ıa-Fern´andez, A. S. Rahmathullah, and L. Svensson, “A time-weighted metric for sets of trajectories to assess multi-object tracking algorithms,” in 24th International Conference on Information Fusion (FUSION). IEEE, 2021, pp. 1–8
2021
-
[16]
Probabilistic GOSPA: A metric for performance evaluation of multi-object filters with uncertainties,
Y . Xia, ´A. F. Garc ´ıa-Fern´andez, J. Karlsson, T. Yuan, K.-C. Chang, and L. Svensson, “Probabilistic GOSPA: A metric for performance evaluation of multi-object filters with uncertainties,” arXiv preprint arXiv:2412.11482, 2024
2024 arXiv
-
[17]
Joint integrated probabilistic data as- sociation: JIPDA,
D. Musicki and R. Evans, “Joint integrated probabilistic data as- sociation: JIPDA,” IEEE Transactions on Aerospace and Electronic Systems, vol. 40, no. 3, pp. 1093–1099, 2004
2004
-
[18]
The labeled multi- Bernoulli filter,
S. Reuter, B.-T. V o, B.-N. V o, and K. Dietmayer, “The labeled multi- Bernoulli filter,” IEEE Transactions on Signal Processing , vol. 62, no. 12, pp. 3246–3260, 2014
2014
-
[19]
Multiple target tracking based on sets of trajectories,
´A. F. Garc´ıa-Fern´andez, L. Svensson, and M. R. Morelande, “Multiple target tracking based on sets of trajectories,” IEEE Transactions on Aerospace and Electronic Systems , vol. 56, no. 3, pp. 1685–1707, 2020
2020
-
[20]
Poisson multi-Bernoulli mixtures for sets of trajectories,
K. Granstr ¨om, L. Svensson, Y . Xia, J. Williams, and ´A. F. Garc ´ıa- Fern´andez, “Poisson multi-Bernoulli mixtures for sets of trajectories,” IEEE Transactions on Aerospace and Electronic Systems , vol. 61, no. 2, pp. 5178–5194, 2025
2025
-
[21]
Multi-scan implementation of the trajectory Poisson multi- Bernoulli mixture filter,
Y . Xia, K. Granstr ¨om, L. Svensson, ´A. F. Garc´ıa-Fern´andez, and J. L. Williams, “Multi-scan implementation of the trajectory Poisson multi- Bernoulli mixture filter,” Journal of Advances in Information Fusion , vol. 14, no. 2, pp. 213–235, 2019. Fig. 3. TGOSPA error and i...
2019
-
[22]
Trajectory Poisson multi-Bernoulli filters,
´A. F. Garc ´ıa-Fern´andez, L. Svensson, J. L. Williams, Y . Xia, and K. Granstr ¨om, “Trajectory Poisson multi-Bernoulli filters,” IEEE Transactions on Signal Processing , vol. 68, pp. 4933–4945, 2020
2020
-
[23]
Trajectory multi-Bernoulli filters for multi-target tracking based on sets of trajectories,
——, “Trajectory multi-Bernoulli filters for multi-target tracking based on sets of trajectories,” in 23rd International Conference on Informa- tion Fusion (FUSION) . IEEE, 2020, pp. 1–8
2020
-
[24]
R. P. Mahler, Statistical Multisource-Multitarget Information Fusion . Artech House, 2007
2007
-
[25]
On implementing 2D rectangular assignment algo- rithms,
D. F. Crouse, “On implementing 2D rectangular assignment algo- rithms,” IEEE Transactions on Aerospace and Electronic Systems , vol. 52, no. 4, pp. 1679–1696, 2016
2016
-
[26]
Markov chain Monte Carlo multiscan data association for sets of trajectories,
Y . Xia, ´A. F. Garc ´ıa-Fern´andez, and L. Svensson, “Markov chain Monte Carlo multiscan data association for sets of trajectories,” IEEE Transactions on Aerospace and Electronic Systems, vol. 60, no. 6, pp. 7804–7819, 2024
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.