{"id":"463abab1-203c-4a12-b2cb-3585c44d3949","arxiv_id":"2412.08321","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"The paper derives practical rules for setting the switching penalty and cut-off of the TGOSPA tracking metric and recommends three application-specific parameter setups for computer vision.","lead":"This paper explains how to choose the settings of the TGOSPA metric, a mathematically grounded scoring system for multi-object tracking in video. It gives practical parameter recommendations for detector training, online surveillance, and offline scene understanding, and shows how each choice changes the ranking of tracking algorithms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The gamma-calibration rules are proved only for a simplified two-trajectory scenario, and the paper admits they break with more targets or larger c; yet the recommended setups are evaluated under those broken conditions, leaving the central 'tailored switch behavior' claim empirically unverified.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the gamma-selection rules are derived for a stripped-down two-trajectory setup, and the paper itself states that the interpretation breaks when the scenario is more crowded or when c is larger. Observation 11 confirms the break in a real case, so the concern is not hypothetical. This is the most load-bearing issue because the paper's practical contribution is precisely the promise that a user can choose g1 or n instead of guessing gamma; if that mapping is unreliable in realistic multi-target data, the recommended parameter combinations in Section VI become heuristic rather than grounded in the stated guarantees. The concern is not an internal inconsistency or a mathematical error under the stated assumptions: the algebra leading to Eqs. (18) and (24) is sound, the paper is transparent about the limitations, and the computational experiments appear reproducible. For those reasons the paper still merits a conditional acceptance: the method is clearly explained and demonstrably useful in the small scenario, but the claimed generality for everyday MOT evaluation needs stronger empirical support. A concrete sweep of gamma in the BoT SORT scenario would settle whether the calibration error is negligible or large, and would determine whether the verdict should later move toward rejection or toward unconditional acceptance. No independent support fully mitigates this concern: the paper's own Observation 11 is evidence of the failure mode, not of its rarity.","tokens_in":33175,"tokens_out":6648,"duration_ms":79760,"concrete_test":"Use the BoT SORT observation (Section V-B, Observation 11) as a controlled probe: take the two ground-truth trajectories gt2, gt6 and the BoT SORT estimate around frame k=34, where d(gt2, Y1)<c but the distance to the relevant other trajectory exceeds g1=0.255. Sweep gamma over (0, c/2^{1/p}) and compute the exact TGOSPA assignment (using the public implementation) to find the critical gamma at which the optimal assignment switches from 'no switch' to 'two switches'. Compare that critical gamma with the value predicted by Eq. (19) for g1=0.255. If the two disagree by more than 10%, the g1 calibration fails in a scenario the recommended setups actually encounter. Repeat the sweep with a synthetic third trajectory at varying distances to map how the switch threshold shifts with scene density.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that gamma can be set indirectly via a threshold distance g1 (short-term interim changes) or a duration n (long-term changes), giving the user predictable control over what TGOSPA counts as a switch. This control is exact only under the scenario of Sections III-D1 and III-D2: two ground-truth trajectories, one estimated trajectory correctly assigned before the change, and distances beyond c during the change. The authors explicitly state after Eq. (20) that with more trajectories or considerably larger c, 'the interpretation of gamma using g1 described above is no longer valid,' and make a similar qualification after Eq. (24) for hn. Observation 11 is a concrete instance: a BoT SORT estimate with a distance larger than g1=0.255 still produces four switches, because the simplifying condition d(x_t1, y_t1)>c fails. This means the mapping from (g1, n) to switch behavior is not robust in realistic MOT scenes with several trajectories and moderate c. Section VI nevertheless recommends concrete gamma values (e.g., gamma=0.31 for online surveillance and gamma=5 for offline scene understanding) and evaluates them on the full MOT17-09 video with many trajectories, without quantifying how far the actual switch thresholds are from the intended g1 or n. The practical value of the paper depends on this calibration surviving beyond the toy derivation, and that has not been established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the TGOSPA metric to computer-vision multi-object tracking evaluation, argues that common CV scores such as HOTA and 1−HOTA are not mathematical metrics, and proposes principled rules for selecting the TGOSPA parameters (cut-off c, exponent p, and switching penalty γ). The central methodological contribution is an indirect calibration of γ: Eq. (19) sets γ from a short-term threshold distance g1, and Eq. (24) sets γ from a minimum switch length n, with derivations carried out in simplified two-ground-truth, one-estimate scenarios. The paper also proposes a histogram-based c and p selection procedure, recommends three concrete parameter setups for detector training, online surveillance, and offline scene understanding, and evaluates several trackers on the MOT17-09 sequence. The algebraic derivations are presented transparently, and the paper explicitly discloses conditions under which its interpretation breaks, including after Eq. (20), after Eq. (24), and in Observation 11.","tokens_in":33543,"tokens_out":8976,"duration_ms":97560,"significance":"If the proposed calibration survives contact with realistic scenarios, the paper provides a useful bridge between the mathematically principled TGOSPA metric and practical CV evaluation, giving users an intuitive handle on the otherwise unintuitive switching penalty. The paper is honest about the scope of its derivations and ships concrete parameter recommendations with decomposition-style evaluation tables, which is valuable for practitioners. The main significance hinges on whether the recommended g1- and n-based settings actually deliver the intended switch behavior in scenes with multiple trajectories and moderate c; the paper's own Observation 11 shows a violation of the threshold interpretation in exactly such a setting, so the practical claim currently rests on an extrapolation from a toy scenario.","major_comments":[{"comment":"The central claim that γ can be set indirectly through g1 or n is proved only for the simplified scenario with two ground-truth trajectories, one estimated trajectory, and a fixed pre-switch assignment. The paper itself states after Eq. (20) and after Eq. (24) that the interpretation is no longer valid with more trajectories or considerably larger c, and Observation 11 concretely shows a BoT SORT estimate with distance larger than g1=0.255 still producing four switches because d(x^t_1,y^t_1)>c fails. Section VI nevertheless recommends γ=0.31 (from g1=0.17) and γ=5 (from n=10) and evaluates them on the full MOT17-09 video, where the simplifying assumptions are violated. The manuscript therefore does not establish that the recommended γ values realize the intended g1 or n switch thresholds in the evaluated scenes. Please add a quantitative robustness analysis, for example by measuring effective switch thresholds as a function of target count and c, or by calibrating on controlled scenarios with the same trajectory density and showing the deviation from the intended g1/n.","section":"III-D1, III-D2 and VI"},{"comment":"The values of c, p, and γ used in the recommended setups in Section VI are selected using the same MOT17-09 data that is later used for the evaluations in Tables V and VI. In particular, Combinations A-C are read off the histograms in Fig. 11 and the visualization in Fig. 12, which are computed from the algorithms that are subsequently ranked. This creates a mild in-sample selection, and the paper does not report how sensitive the rankings or the metric decompositions are to the chosen parameter values. Please add a sensitivity analysis, for example by varying c and p within interval I2 and reporting the resulting rankings, or by validating the recommended setups on an additional MOT17 sequence, so that the recommendations can be distinguished from an overfit to a single video.","section":"IV-B, IV-C and VI-A"},{"comment":"The statement around Eq. (24) that setting n>K/2 makes TGOSPA behave as if γ→∞ deserves a more careful justification. The maximum possible length of a track-change segment depends on the pre-switch segment length, and the paper's own caveat that the assumption of a sufficiently long pre-switch assignment is 'crucial for the validity of the interpretation of (24)' makes the claim nontrivial. In particular, the text uses both K and K+1 inconsistently in this discussion, and the derivation leading to Eq. (21) does not cover the case where a tracker switches after a very short initial segment and then persists on the wrong target for the remainder of the sequence. Please clarify the exact conditions under which the 'n>K/2 implies no switches' statement holds, or restrict the claim accordingly.","section":"III-D2, Section V"}],"minor_comments":[{"comment":"There is a typo in the phrase 'the funciton dHOTA(X, Y)' near Eq. (1); it should read 'the function dHOTA(X, Y)'.","section":"II-C"},{"comment":"The tracker name is written as 'Bot SORT' in the description 'Bag-of-tricks for simple online and real-time tracking (Bot SORT)', while everywhere else it is 'BoT SORT'; please unify the capitalization.","section":"II-B"},{"comment":"The observations are numbered starting from Observation 2, but there is no Observation 1 in the manuscript; either renumber or add the missing observation.","section":"V"},{"comment":"The caption 'online surveillance and offline scene understanding tracking algorithms evaluation using the IoU metric, MOT17-09 video processing the public FRCNN detections' should be reworded, for example 'evaluation on the MOT17-09 video using the public FRCNN detections', since the current phrasing is grammatically incomplete.","section":"VI-A, Table VI"},{"comment":"The notation 'γ> c 2√p' is ambiguous; it should be written as γ > c/2^{1/p} or, equivalently, γ^p > c^p/2, to make the comparison with the missed/false-object cost cp/2 explicit.","section":"III-C and Observation 7"},{"comment":"The text 'Setting n> K/2 results in TGOSPA behaving as if γ→+∞' should use n>(K+1)/2 consistently with the preceding sentence that refers to the total number of time steps K+1.","section":"III-D2"}],"recommendation":"major_revision","confidential_remarks":"The paper's analytic core is sound and the authors are transparent about the limitations of the switching-penalty interpretation. My main concern is not with the derivations but with the gap between the toy-scenario calibration and the full-video evaluations used to support the practical recommendations. This gap is fixable with additional experiments or a more careful hedging of the claims, so I recommend major revision rather than rejection. The in-sample selection of c and p from the same video used for the final ranking is a secondary issue that should also be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is the paper that actually tells you how to set the TGOSPA switching penalty in practice, and it does so with derivations rather than guesses. The gamma-selection formulas, Eqs. (19) through (25), are new relative to the cited TGOSPA papers, and the three recommended parameter setups (detector training, online surveillance, offline scene understanding) are concrete and clearly motivated. The authors also correctly note that HOTA and 1-HOTA are not metrics, though that result is already in [15], so there it is a restatement rather than a contribution.\n\nThe core algebra is sound under the stated scenario: two ground-truth trajectories, one estimate, distances beyond c during the change. The threshold derivations for both the interim and long-term cases check out. The paper is also admirably transparent about the limits. Immediately after Eq. (20) and Eq. (24) it says the g1 and hn interpretations are no longer valid with more trajectories or considerably larger c, and Observation 11 is a concrete instance of that failure: a BoT SORT estimate with distance larger than g1=0.255 still produces four switches because the simplifying condition d(x_t1,y_t1)>c does not hold.\n\nThat transparency is also where the soft spot sits. The central practical claim is that a user can choose g1 or n and get predictable switch behavior. That predictability is proven only in the toy scenario, and the paper itself says it breaks in realistic scenes. Section VI then recommends concrete gamma values and evaluates them on the full MOT17-09 video with many trajectories, but never quantifies how far the actual switch thresholds are from the intended g1 or n. The main selling point, tailored switch behavior, is therefore empirically unverified beyond the toy derivation. There is also mild in-sample circularity: c and p are chosen from histograms of MOT17-09 and then evaluated on the same video. The paper is open about this, and it is not disqualifying, but it does weaken the generality of the specific recommended values.\n\nWho benefits: anyone in CV or signal processing who wants to use TGOSPA seriously, and anyone designing MOT benchmarks. The HOTA comparison in Appendix C is useful for teaching and for clarifying application-dependent penalties.\n\nBottom line: this deserves a serious referee. It is a solid, honest paper with a genuinely new handle on a parameter that was previously just \"choose something.\" The main revision ask should be a robustness check: either more videos or a sensitivity analysis showing how the switch threshold behaves when the assumptions are violated.","headline":"Useful parameter-selection rules for TGOSPA, honestly bounded but empirically thin where it matters most: the switch-calibration claim is proven only in a toy scenario.","tokens_in":34026,"tokens_out":2426,"would_cite":true,"duration_ms":26063,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A distance threshold or minimum duration sets the TGOSPA track-switch penalty.","keywords":["multi-object tracking","TGOSPA metric","track switches","performance evaluation","switching penalty","IoU metric","computer vision","metric axioms"],"falsifier":"Construct the exact two-trajectory scenario, set $\\gamma$ by (19) for a fixed $g_1$, and vary the wrong-track distance $d$ around $g_1$: the optimal TGOSPA assignment should flip from no switch to two switches exactly at $d = g_1$; if the boundary shifts when a third trajectory is added or when $c$ is enlarged, the threshold interpretation is refuted.","tokens_in":33050,"feed_emoji":"🎯","tokens_out":8208,"duration_ms":81865,"temperature":0.7,"pith_summary":"The paper tries to establish that TGOSPA, a metric on finite sets of trajectories, is the right evaluation tool for visual multi-object tracking because its parameters can be chosen to match an application. Its main result is that the switching penalty $\\gamma$ does not have to be guessed: choosing a threshold distance $g_1 < c$ and setting $\\gamma = ((c^p - g_1^p)/2)^{1/p}$ makes short interim track changes count as switches, while setting $\\gamma = n^{1/p} c$ makes TGOSPA ignore switches shorter than $n+1$ time steps. These rules turn a counterintuitive hyperparameter into an interpretable tolerance for track changes. The paper also proves that HOTA and $1 - \\text{HOTA}$ are not mathematical metrics, and it recommends three concrete parameter triples for detector training, online surveillance, and offline scene understanding.","feed_headline":"Distance thresholds set TGOSPA track-switch penalties exactly","feed_subtitle":"A small derivation turns gamma into a distance or a duration, so users can tailor tracking evaluation to each application.","key_machinery":"The carrying object is the TGOSPA metric $d^{(c,\\gamma)}_p$ (Definition 1), a metric between finite sets of discrete-time trajectories that minimizes over time-indexed assignment vectors and decomposes (equation 14) into a localization term, a missed-objects term, a false-alarms term, and a switch term. The parameter-selection rules are produced by equating the cost of a no-switch assignment with the cost of a switch assignment in a simplified two-trajectory, one-estimate scenario; the equality gives closed-form formulas for $\\gamma$ in terms of $g_1$ or $n$. Because exact TGOSPA is NP-hard, the numerical evaluations use the LP relaxation from [16], which is itself a metric and agreed with exact TGOSPA in all cases tested in the paper.","core_discovery":"The central claim is that the TGOSPA switching penalty $\\gamma$ can be set through threshold distances. In a scenario where one estimate briefly tracks the wrong ground-truth trajectory, TGOSPA counts the episode as two switches exactly when $\\gamma < ((c^p - d^p)/2)^{1/p}$, where $d$ is the estimate's distance to the wrong trajectory at the switched time step; hence a user-provided threshold $g_1$ yields $\\gamma = ((c^p - g_1^p)/2)^{1/p}$. For permanent changes, TGOSPA counts the change as one switch when $\\gamma < (\\ell c^p - \\sum d_k^p)^{1/p}$, and setting $\\gamma = n^{1/p} c$ guarantees that changes lasting at most $n$ time steps are ignored. The authors validate the interpretation on a two-pedestrian occlusion segment of MOT17-09, show that different $\\gamma$ choices reorder BoT SORT and Tracktor++v2, and propose the three recommended setups. They further show that HOTA and $1 - \\text{HOTA}$ violate the metric axioms, and that the TGOSPA decomposition into localization, missed, false, and switch terms gives reasons for algorithm orderings that CV scores cannot provide.","pith_inferences":["My inference (not the paper's claim): in crowded scenes the effective threshold will be stricter than $g_1$ because nearby third trajectories compete for the same assignment; a practical workaround would be to set $g_1$ from the observed distribution of inter-object distances and verify switch counts on validation data.","A testable extension the paper does not pursue: use the same cost-equality argument to derive thresholds for other event types, such as a track that splits into two estimates for a few frames, rather than only for temporary or permanent switches.","Since the paper's $\\gamma = 0$ case reduces TGOSPA to per-frame GOSPA and forfeits the metric property, any use of TGOSPA as a true metric between two algorithms' outputs requires a positive switching penalty; this makes the parameter-selection rules not just a convenience but a prerequisite for the metric interpretation."],"forward_implications":["Users can replace the opaque switching penalty $\\gamma$ with a distance tolerance $g_1$ or a minimum switch duration $n$, making evaluation results interpretable before the metric is computed.","Setting $\\gamma = n^{1/p} c$ implements 'ignore switches shorter than $n+1$ time steps' as a hard specification, which is directly relevant for scene-understanding applications that only care about long track fragmentations.","Detector training should use $\\gamma = 0$, because with any positive $\\gamma$ the temporary disconnections of a per-frame detector are counted as switches and the detector ranking becomes misleading.","The three recommended parameter triples produce different orderings of the same trackers, so application-specific scoring is not cosmetic: the same algorithm can be the right choice for online surveillance and the wrong choice for offline scene understanding under TGOSPA."],"supporting_citations":[{"why":"Supplies the TGOSPA metric definition and its proof that it is a metric on finite sets of trajectories.","marker":"[16]"},{"why":"Supplies the time-weighted TGOSPA extension used for the simplified $\\gamma = 0$ and $\\gamma \\to \\infty$ computations.","marker":"[17]"},{"why":"Supplies the GOSPA metric, the per-time-step metric used when $\\gamma = 0$ and the special case underlying the distance $d^{(c)}_p$.","marker":"[28]"},{"why":"Supplies the IoU-induced metric and the c-selection guideline functions that the paper extends for choosing $c$ and $p$.","marker":"[27]"},{"why":"Supplies the HOTA score, its definition, and the claim that HOTA addresses earlier score drawbacks; the paper shows HOTA and $1-\\text{HOTA}$ are not metrics and compares orderings.","marker":"[10]"},{"why":"Supplies the demonstration that common CV scores, including $\\text{dHOTA} = 1-\\text{HOTA}$, fail the metric axioms.","marker":"[15]"}],"fun_headline_variants":["Set TGOSPA switch penalties with distance thresholds","Distance thresholds pin down TGOSPA track-switch cost","Exact gamma from distance thresholds in TGOSPA","Choose TGOSPA switch penalty via a distance","TGOSPA gamma as a distance threshold, exactly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the two-trajectory, one-estimate derivation describes real scenes: the estimate is correctly assigned before the change, the wrong-track distance at the change is below the relevant threshold, and no third trajectory competes for the assignment; the paper itself flags that this interpretation is no longer valid with more trajectories or larger $c$, and its Observation 11 records a real BoT SORT case where the predicted threshold does not hold.","fun_headline_variants_meta":{"raw":{"variants":["Set TGOSPA switch penalties with distance thresholds","Distance thresholds pin down TGOSPA track-switch cost","Exact gamma from distance thresholds in TGOSPA","Choose TGOSPA switch penalty via a distance","TGOSPA gamma as a distance threshold, exactly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1518,"prompt_tokens":970,"completion_tokens":548,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":473}},"tokens_in":586,"tokens_out":548,"duration_ms":5804,"temperature":1.0,"reasoning_tokens":473,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:57:30.501708+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct the exact two-trajectory scenario, set $\\gamma$ by (19) for a fixed $g_1$, and vary the wrong-track distance $d$ around $g_1$: the optimal TGOSPA assignment should flip from no switch to two switches exactly at $d = g_1$; if the boundary shifts when a third trajectory is added or when $c$ is enlarged, the threshold interpretation is refuted.","supporting_citations":[{"cited_title":"A metric on the space of finite sets of trajectories for evaluation of multi-target tracking algorithms,","cited_arxiv_id":null,"evidence_quote":"Supplies the TGOSPA metric definition and its proof that it is a metric on finite sets of trajectories."},{"cited_title":"A time- weighted metric for sets of trajectories to assess multi-object tracking algorithms,","cited_arxiv_id":null,"evidence_quote":"Supplies the time-weighted TGOSPA extension used for the simplified $\\gamma = 0$ and $\\gamma \\to \\infty$ computations."},{"cited_title":"Gener- alized optimal sub-pattern assignment metric,","cited_arxiv_id":null,"evidence_quote":"Supplies the GOSPA metric, the per-time-step metric used when $\\gamma = 0$ and the special case underlying the distance $d^{(c)}_p$."},{"cited_title":"Bounding box detection in visual tracking: Measurement model parameter estimation,","cited_arxiv_id":null,"evidence_quote":"Supplies the IoU-induced metric and the c-selection guideline functions that the paper extends for choosing $c$ and $p$."},{"cited_title":"HOTA: A higher order metric for evaluating multi-object tracking,","cited_arxiv_id":null,"evidence_quote":"Supplies the HOTA score, its definition, and the claim that HOTA addresses earlier score drawbacks; the paper shows HOTA and $1-\\text{HOTA}$ are not metrics and compares orderings."},{"cited_title":"How trustworthy are performance evaluations for basic vision tasks?,","cited_arxiv_id":null,"evidence_quote":"Supplies the demonstration that common CV scores, including $\\text{dHOTA} = 1-\\text{HOTA}$, fail the metric axioms."}],"review_version":1}