Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

Probabilistic GOSPA: A Metric for Performance Evaluation of Multi-Object Filters with Uncertainties

T0 review · 2 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read P-GOSPA is a metric on multi-Bernoulli densities that extends GOSPA to full probabilistic multi-object representations, reducing to GOSPA when all existence probabilities are one and densities are point masses.

desk verdict A useful, clearly-written extension of GOSPA to multi-Bernoulli densities with a nice alpha=2 decomposition, but the triangle-inequality proof in Appendix A has a real gap that needs fixing before the metric claim can stand. read the letter →

arxiv 2412.11482 v3 pith:PJF2PIBU submitted 2024-12-16 eess.SP

classification eess.SP
keywords multi-objecttrackingperformanceevaluationrandomfinitesetsmulti-BernoulliprocessGOSPAWassersteindistancemetric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-object tracking filters output more than point estimates: Bayesian filters such as the Poisson multi-Bernoulli mixture filter produce entire multi-Bernoulli densities, complete with existence probabilities and per-object uncertainties. Existing metrics like GOSPA compare only deterministic sets of estimated points, discarding this uncertainty. This paper proposes P-GOSPA, a metric on the space of multi-Bernoulli densities, defined by an assignment problem that pairs Bernoulli components and charges a cut-off base distance on single-object densities, an existence-probability mismatch cost, and costs for unassigned components. The paper proves P-GOSPA is a true metric, with the triangle inequality being the nontrivial part, and shows that with parameter $\alpha=2$ it decomposes into expected localization error, existence-probability mismatch error, expected missed detection error, and expected false detection error. If correct, this gives practitioners a tractable, interpretable way to compare filters on their full posterior densities rather than on extracted estimates.

What carries the argument

The central object is the P-GOSPA formula (2), an assignment problem over Bernoulli components. Its two building blocks are the cut-off metric $d^{(c)}(p_x,p_y)=\min(d(p_x,p_y),c)$ on single-object densities and the per-pair cost $\min(r_x,r_y)d^{(c)}(\cdot,\cdot)^p+|r_x-r_y|c^p/\alpha$; unassigned components carry $c^p/\alpha$ times their existence probability. The proof of the triangle inequality hinges on first proving the Bernoulli case (Lemma 1) and then gluing the general case with Minkowski's inequality, using permutations to match component counts. Proposition 2, valid for $\alpha=2$, converts the permutation form into an assignment-set form that yields the four-term error decomposition.

What would settle it

Compute Eq. (2) for three Bernoulli densities in one dimension, e.g., existence probabilities $0.2, 0.5, 0.8$ at point masses $0, 1, 2$, with $c=1$, $\alpha=1$, $p=1$ and Euclidean distance as the base metric; a violation of $d(f_X,f_Y)\le d(f_X,f_Z)+d(f_Z,f_Y)$ would refute the metric claim. The same computation with a non-metric base such as KL divergence would show that the triangle inequality rests on the base being a metric.

Watch

Extended reading notes

Core claim

The central claim is that the P-GOSPA expression in Eq. (2), defined for two multi-Bernoulli densities parameterized by $\{(r_x^i,p_x^i)\}_{i=1}^{n_X}$ and $\{(r_y^j,p_y^j)\}_{j=1}^{n_Y}$, is a metric on the space of such densities. Given a metric $d$ between single-object densities and its cut-off $d^{(c)}=\min(d,c)$, P-GOSPA minimizes over permutations $\pi$ the sum of $\min(r_x^i,r_y^{\pi(i)}) d^{(c)}(p_x^i,p_y^{\pi(i)})^p + |r_x^i-r_y^{\pi(i)}| c^p/\alpha$ for the associated $n_X$ components, plus $c^p/\alpha$ times the sum of the existence probabilities of the $n_Y-n_X$ unassigned components, with the formula symmetrized when $n_X>n_Y$. Non-negativity, definiteness, and symmetry follow directly from the definition; the triangle inequality is proved in Appendix A by first handling the Bernoulli case and then applying an extended Minkowski inequality. For $\alpha=2$ the metric can be rewritten, as Proposition 2, into a 2D assignment problem whose four components are exactly the expected localization error, existence-probability mismatch, expected missed detection, and expected false detection errors. The paper also shows P-GOSPA reduces to the ordinary GOSPA metric when all existence probabilities are one and all single-object densities are Dirac deltas.

Load-bearing premise

For P-GOSPA to be a metric, the user-chosen base distance $d(p_x,p_y)$ between single-object densities must itself be a metric, because the cut-off of a metric is a metric but the cut-off of a non-metric divergence is not guaranteed to satisfy the triangle inequality; the paper's examples all use metric bases such as Wasserstein or Hellinger distances.

Editorial extensions

If this is right

  • With $\alpha=2$, P-GOSPA decomposes into exactly four expected-error terms—localization, existence-probability mismatch, missed detection, and false detection—so evaluation output can be attributed to causes.
  • P-GOSPA is computable in closed analytical form via a 2D assignment problem, whereas the average GOSPA of two independent multi-Bernoulli densities has no general analytical expression.
  • When all existence probabilities equal one and all single-object densities are Dirac deltas, P-GOSPA reduces to GOSPA, making it a strict generalisation that inherits GOSPA's interpretability.
  • P-GOSPA can be used to measure approximation error inside a filter, e.g., the loss incurred by collapsing a Poisson multi-Bernoulli mixture posterior to a single multi-Bernoulli density, which the simulation quantifies.
  • For $\alpha=2$, each unassigned Bernoulli component with existence probability $r$ contributes $r c^p/2$, so missed and false detections are penalized in proportion to the probability that the object exists.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A trajectory-valued analogue is the next natural step: the paper explicitly leaves sets of trajectories as future work, and a P-GOSPA over finite sets of trajectories with existence probabilities would let track-level uncertainty be scored in the same four-term decomposition.
  • The parameter $\alpha$ gives an application-tunable dial between location and existence disagreement: because both the mismatch cost and the unassigned-component cost scale as $c^p/\alpha$, lowering $\alpha$ raises the price of disagreeing about whether an object exists relative to where it is, without leaving the metric family.
  • One could stress-test the metric's usefulness by comparing P-GOSPA rankings against expected losses computed by Monte Carlo sampling of the posterior: if a filter's P-GOSPA error does not correlate with its sampled-average GOSPA error, the closed-form metric would still be a distance but a less informative score for filter selection.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper defines P-GOSPA in Eq. (2), a distance between multi-Bernoulli densities obtained by extending GOSPA's assignment cost to Bernoulli components with existence probabilities and single-object densities. It claims P-GOSPA is a metric (Appendix A), that for alpha=2 it decomposes into expected localization, existence-probability mismatch, missed-detection, and false-detection terms (Proposition 2), and it illustrates the metric on two examples and on PMBM/PMB filtering simulations. The manuscript also proves an upper-bound relationship to the Wasserstein distance between Bernoulli densities (Proposition 1).

Significance. P-GOSPA addresses a real need: GOSPA evaluates point estimates only, while multi-object filters output densities carrying existence and state uncertainty. If the metric property is established, P-GOSPA would be an analytically tractable, parameter-free (modulo c, p, alpha) distance on MB densities that reduces to GOSPA for Dirac/point-mass components and admits the interpretable alpha=2 decomposition. The assignment formulation in Proposition 2 gives a practical computational route, and Proposition 1's Wasserstein connection is a useful sanity check. The main obstacle is that the submitted proof of the triangle inequality contains a false step, so the paper's central claim is not yet established as written.

major comments (2)
  1. [Appendix A, Eq. (32)] The step from Eq. (31) to Eq. (32) is invalid. Denote by C_i the i-th summand on the right of Eq. (31) (the p-th power of the Bernoulli distance between the i-th pair), and by A_i and B_i the analogous summands involving (X,Z) and (Y,Z). The Bernoulli triangle inequality (14) gives C_i^{1/p} <= A_i^{1/p}+B_i^{1/p}, which after raising to the p-th power is C_i <= (A_i^{1/p}+B_i^{1/p})^p, not C_i <= A_i+B_i. The displayed inequality in Eq. (32) is therefore false in general: for p=2, alpha=2, c=1, r_x=r_y=r_z=0.5, and single-object densities with d(p_x,p_z)=d(p_y,p_z)=0.6 and d(p_x,p_y)=1.1 (realizable by three point masses in the plane), one has C_XY=0.5*1^2=0.5 whereas A_XZ+B_YZ=0.5*(0.6^2+0.6^2)=0.36, so Eq. (32) would assert sqrt(0.5) <= 0.6. Since this step is used in all three cases of the multi-Bernoulli proof, the triangle inequality is not proven by the submitted text. A correct argument may be available either by applying Minkowski's inequality to the sequences (A_i^{1/p}) and (B_i^{1/p}) before summing, or by combining Remark 2 with the standard GOSPA metric property, but the proof as written must be revised.
  2. [Section IV, Fig. 7] The PMBM filter's posterior is a mixture of MB densities, not an MB density, so Eq. (2) does not directly apply. The paper computes P-GOSPA between each MB component and the ground truth and then takes a weighted sum. This is an ad hoc extension; a weighted sum of metrics is not automatically a metric, and no definition or properties are given for P-GOSPA on MBM densities. If the authors wish to claim that P-GOSPA evaluates PMBM (and hence 'multi-object filters with uncertainties' generally), this extension must be defined and analyzed; otherwise the PMBM results should be labeled as a heuristic expected-P-GOSPA rather than as the metric itself.
minor comments (4)
  1. [Section II, Definition 1] The metric property of P-GOSPA is only guaranteed when the base distance d(p_x,p_y) is a metric; the paper uses Wasserstein and Hellinger distances, which are metrics, but a sentence explicitly warning against non-metric divergences such as KL divergence would prevent misuse.
  2. [Section II-A] The text refers to 'Section II.II-A' in the second paragraph of Section II; this should be 'Section II-A'.
  3. [References] Reference [12] contains a typo: 'García-Femández' should be 'García-Fernández'.
  4. [Section IV] The simulations report RMS errors over 100 Monte Carlo runs without error bars or variance information; a brief indication of variability would strengthen the comparison between PMBM and PMB.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: P-GOSPA is defined from stated axioms and the claimed metric, decomposition, and Wasserstein-bound properties are derived, not assumed.

full rationale

The paper defines P-GOSPA in Eq. (2) as an explicit formula involving existence probabilities and a user-supplied base metric d(px,py); no parameter is fitted to data and no quantity called a prediction is reused as an input. The reduction to GOSPA in Eq. (3) and the alpha=2 decomposition in Eq. (8) are derived identities rather than renamed inputs: Eq. (7) is an algebraic condition motivating the parameter choice, and Proposition 2 recasts Eq. (2) via assignment functions without importing the conclusion. Lemma 2 and Proposition 1 relate P-GOSPA to Wasserstein distances and explicitly prove an upper bound rather than assuming equality, with appendices deriving these statements from the definition. Self-citations to GOSPA [11], average GOSPA [11, Prop. 2], and the PMBM filter [17] supply background and the base metric being generalized; the triangle-inequality proof relies on the external Minkowski inequality [25] and the already-proved Bernoulli special case, not on a self-citation chain. Remark 2's observation that P-GOSPA can be viewed as GOSPA applied to sets of Bernoulli densities is an equivalence stated after the definition, not the source of the metric property. A possible gap in the multi-Bernoulli triangle-inequality proof at Eq. (32)--where (B_i+C_i)^p appears to be replaced by B_i^p+C_i^p--would be a mathematical correctness issue rather than circularity, and it does not change this assessment.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard random finite set theory, the metric property of the chosen base distance, and standard inequalities such as Minkowski. No constants are fitted to data; c, p, and alpha are user-chosen parameters of the proposed metric.

free parameters (3)
  • Cut-off distance c = Examples use c=5 and c=10; general claim holds for any c>0
    Maximum allowed distance between two single-object densities; user-specified, not fitted to data.
  • Order p = Examples use p=1 and p=2; general claim holds for p in [1, infinity)
    Controls outlier penalty strength; user-specified, not fitted to data.
  • Existence mismatch weighting alpha = alpha=2 recommended and used; definition valid for 0<alpha<=2
    Controls the missed and false detection penalty; alpha=2 follows from the design equation (7), not from data fitting.
assumptions (4)
  • domain assumption The single-object state space X is locally compact, Hausdorff, and second-countable.
    Invoked in Section II to place the work inside Mahler's random finite set framework [6].
  • domain assumption The base distance d(px,py) between single-object densities is a metric, so its cut-off min(d,c) is also a metric.
    Definition 1; the triangle inequality proof in Appendix A relies on d^(c) satisfying the triangle inequality.
  • standard math Minkowski's inequality holds for sequences padded with zeros to equal length.
    Used throughout Appendix A, citing Kubrusly [25].
  • domain assumption PMBM and PMB filters produce the posteriors claimed under the standard dynamic model with Poisson birth.
    Used in Section IV to interpret the simulation comparison, citing [17] and [18].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Probabilistic GOSPA: A Metric for Performance Evaluation of Multi-Object Filters with Uncertainties." pith.science (2026). https://pith.science/paper/PJF2PIBU

@misc{pith2026241211482,
  author       = {Pith},
  title        = {Pith review of: Probabilistic GOSPA: A Metric for Performance Evaluation of Multi-Object Filters with Uncertainties},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PJF2PIBU}},
  note         = {Machine review of arXiv:2412.11482}
}
read the original abstract

This paper presents a probabilistic generalization of the Generalized Optimal Sub-Pattern Assignment (GOSPA) metric, termed P-GOSPA. The GOSPA metric has been widely used to evaluate the distance between finite sets, particularly in multi-object estimation applications. The P-GOSPA extends GOSPA into the space of multi-Bernoulli densities, incorporating inherent uncertainty in probabilistic multi-object representations. Additionally, P-GOSPA retains the interpretability of GOSPA, such as its decomposition into localization, missed detection, and false detection errors in a sound and meaningful manner. Examples and simulations are provided to demonstrate the efficacy of the proposed P-GOSPA metric.

Figures

Figures reproduced from arXiv: 2412.11482 by the authors.

Figure 1
Figure 1. An exemplary scenario with two objects and two MB set densities. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example 1: P-GOSPA versus r and σ 2 . r = 0, P-GOSPA d (5,2) 1 is a constant since for a non-existent object, its single-object density has no effect. We also note that when Gaussian variance σ 2 ≥ 21, the true object becomes missed detected and its estimate becomes a false detection, and thus P-GOSPA d (5,2) 1 becomes invariant to σ 2 . B. Example 2 We consider a 2D example, in which both the ground truth and the e… view at source ↗
Figure 3
Figure 3. Example 2: the true and estimated MB set density. Each Bernoulli [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Ground truth object trajectories used in the simulation. The figure at [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: RMS GOSPA and its decomposition over time. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: RMS P-GOSPA and its decomposition over time. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: RMS P-GOSPA and its decomposition over time, where P-GOSPA is [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Probabilistic Trajectory GOSPA: A Metric for Uncertainty-Aware Multi-Object Tracking Performance Evaluation

    eess.SP 2025-06 conditional novelty 5.0 of 10

    The paper introduces PTGOSPA, a probabilistic trajectory GOSPA metric for evaluating multi-object tracking algorithms that output trajectory estimates with track-level uncertainties.

Reference graph

Works this paper leans on

26 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    S. S. Blackman and R. Popoli,Design and analysis of modern tracking systems. Artech House Publishers, 1999

  2. [2]

    Performance evaluation methods for multiple-target-tracking algorithms,

    B. E. Fridling and O. E. Drummond, “Performance evaluation methods for multiple-target-tracking algorithms,” inSignal and Data Processing of Small Targets 1991, vol. 1481. SPIE, 1991, pp. 371–383

  3. [3]

    Ambiguities in evaluating performance of multiple target tracking algorithms,

    O. E. Drummond and B. E. Fridling, “Ambiguities in evaluating performance of multiple target tracking algorithms,” inSignal and Data Processing of Small Targets 1992, vol. 1698. SPIE, 1992, pp. 326–337

  4. [4]

    A performance assessment environment for radar signal processing and tracking algorithms,

    S. Mabbs, “A performance assessment environment for radar signal processing and tracking algorithms,” inProceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing, vol. 1. IEEE, 1993, pp. 9–12

  5. [5]

    Performance metrics for multiple- sensor multiple-target tracking,

    R. L. Rothrock and O. E. Drummond, “Performance metrics for multiple- sensor multiple-target tracking,” inSignal and Data Processing of Small Targets 2000, vol. 4048. SPIE, 2000, pp. 521–531

  6. [6]

    R. P. Mahler,Advances in statistical multisource-multitarget information fusion. Artech House, 2014

  7. [7]

    Multitarget miss distance via optimal assignment,

    J. R. Hoffman and R. P. Mahler, “Multitarget miss distance via optimal assignment,”IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, vol. 34, no. 3, pp. 327–336, 2004

  8. [8]

    A consistent metric for performance evaluation of multi-object filters,

    D. Schuhmacher, B.-T. V o, and B.-N. V o, “A consistent metric for performance evaluation of multi-object filters,”IEEE Transactions on Signal Processing, vol. 56, no. 8, pp. 3447–3457, 2008

Show all 26 references
  1. [9]

    A new metric between distributions of point processes,

    D. Schuhmacher and A. Xia, “A new metric between distributions of point processes,”Advances in Applied Probability, vol. 40, no. 3, pp. 651–672, 2008

  2. [10]

    Metrics for evaluating feature-based mapping performance,

    P. Barrios, M. Adams, K. Leung, F. Inostroza, G. Naqvi, and M. E. Orchard, “Metrics for evaluating feature-based mapping performance,” IEEE Transactions on Robotics, vol. 33, no. 1, pp. 198–213, 2016

  3. [11]

    Gen- eralized optimal sub-pattern assignment metric,

    A. S. Rahmathullah, ´A. F. Garc´ıa-Fern´andez, and L. Svensson, “Gen- eralized optimal sub-pattern assignment metric,” in20th International Conference on Information Fusion (Fusion). IEEE, 2017, pp. 1–8. 13

  4. [12]

    Spooky effect in optimal OSPA estimation and how GOSPA solves it,

    ´A. F. Garc´ıa-Fem´andez and L. Svensson, “Spooky effect in optimal OSPA estimation and how GOSPA solves it,” in22th International Conference on Information Fusion (FUSION). IEEE, 2019, pp. 1–8

  5. [13]

    An analysis on metric-driven multi-target sensor management: GOSPA versus OSPA,

    ´A. F. Garc´ıa-Fern´andez, M. Hernandez, and S. Maskell, “An analysis on metric-driven multi-target sensor management: GOSPA versus OSPA,” in 24th International Conference on Information Fusion (FUSION). IEEE, 2021, pp. 1–8

  6. [14]

    Incorporating track uncertainty into the OSPA metric,

    S. Nagappa, D. E. Clark, and R. Mahler, “Incorporating track uncertainty into the OSPA metric,” in14th International Conference on Information Fusion. IEEE, 2011, pp. 1–8

  7. [15]

    A track quality based metric for evaluating performance of multitarget filters,

    X. He, R. Tharmarasa, T. Kirubarajan, and T. Thayaparan, “A track quality based metric for evaluating performance of multitarget filters,” IEEE Transactions on Aerospace and Electronic Systems, vol. 49, no. 1, pp. 610–616, 2013

  8. [16]

    An uncertainty-aware performance measure for multi-object tracking,

    J. Pinto, Y . Xia, L. Svensson, and H. Wymeersch, “An uncertainty-aware performance measure for multi-object tracking,”IEEE Signal Processing Letters, vol. 28, pp. 1689–1693, 2021

  9. [17]

    Poisson multi-Bernoulli mixture filter: direct derivation and implementa- tion,

    ´A. F. Garc´ıa-Fern´andez, J. L. Williams, K. Granstr ¨om, and L. Svensson, “Poisson multi-Bernoulli mixture filter: direct derivation and implementa- tion,”IEEE Transactions on Aerospace and Electronic Systems, vol. 54, no. 4, pp. 1883–1901, 2018

  10. [18]

    Marginal multi-Bernoulli filters: RFS derivation of MHT, JIPDA, and association-based MeMBer,

    J. L. Williams, “Marginal multi-Bernoulli filters: RFS derivation of MHT, JIPDA, and association-based MeMBer,”IEEE Transactions on Aerospace and Electronic Systems, vol. 51, no. 3, pp. 1664–1687, 2015

  11. [19]

    Gaussian implementation of the multi-Bernoulli mixture filter,

    ´A. F. Garc´ıa-Fern´andez, Y . Xia, K. Granstr¨om, L. Svensson, and J. L. Williams, “Gaussian implementation of the multi-Bernoulli mixture filter,” in22th International Conference on Information Fusion (FUSION). IEEE, 2019, pp. 1–8

  12. [20]

    The labeled multi- Bernoulli filter,

    S. Reuter, B.-T. V o, B.-N. V o, and K. Dietmayer, “The labeled multi- Bernoulli filter,”IEEE Transactions on Signal Processing, vol. 62, no. 12, pp. 3246–3260, 2014

  13. [21]

    On implementing 2D rectangular assignment algorithms,

    D. F. Crouse, “On implementing 2D rectangular assignment algorithms,” IEEE Transactions on Aerospace and Electronic Systems, vol. 52, no. 4, pp. 1679–1696, 2016

  14. [22]

    The distance between two random vectors with given dispersion matrices,

    I. Olkin and F. Pukelsheim, “The distance between two random vectors with given dispersion matrices,”Linear Algebra and its Applications, vol. 48, pp. 257–263, 1982

  15. [23]

    Metrics for performance evaluation of elliptic extended object tracking methods,

    S. Yang, M. Baum, and K. Granstr ¨om, “Metrics for performance evaluation of elliptic extended object tracking methods,” inInternational Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI). IEEE, 2016, pp. 523–528

  16. [24]

    A metric on the space of finite sets of trajectories for evaluation of multi-target tracking algorithms,

    ´A. F. Garc´ıa-Fern´andez, A. S. Rahmathullah, and L. Svensson, “A metric on the space of finite sets of trajectories for evaluation of multi-target tracking algorithms,”IEEE Transactions on Signal Processing, vol. 68, pp. 3917–3928, 2020

  17. [25]

    C. S. Kubrusly,Elements of operator theory. Springer, 2011

  18. [26]

    Villaniet al.,Optimal transport: old and new

    C. Villaniet al.,Optimal transport: old and new. Springer, 2009, vol. 338

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.