{"id":"099e5a36-5adc-489d-86f5-21d795194983","arxiv_id":"2603.28029","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An open-loop perception-evaluation framework converts false positives and false negatives into physical braking/steering effort metrics (FSR, MDR, LEA) and demonstrates their complementarity on nuScenes and Argoverse 2.","lead":"The paper proposes three new metrics—FSR, MDR, and LEA—that translate 3D perception errors in autonomous driving into the physical avoidance effort (braking or steering) they would require of the ego vehicle. It evaluates these metrics on nuScenes and Argoverse 2 across three detection/tracking pipelines, reporting that 65–93% of perception errors are non-critical and that the metrics capture information not covered by existing criticality measures.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Complementarity evidence in Tab. V is confounded by RSB gate admitting 98% of FN tracks; low ρ may reflect non-threat off-path objects, not safety-relevant information.","rationale":"The reader's verdict (CONDITIONAL) already identifies the RSB gate's conservatism as the weakest assumption and requires threshold sensitivity analysis before endorsing quantitative conclusions. My concern sharpens this: the specific statistical evidence for non-redundancy (Tab. V, Sec. IV-D) is potentially an artifact of scoring laterally irrelevant tracks with a 1D longitudinal metric. This does not change the verdict—the paper still warrants conditional acceptance with the added condition that correlations be re-examined on a planner-relevant subset. The concrete test is straightforward and would settle whether the complementarity claim survives a meaningful threat filter. I therefore see no need to move the verdict; the reader's conditional stance is appropriate.","tokens_in":14344,"tokens_out":4967,"duration_ms":52397,"concrete_test":"Recompute the track-level Spearman correlations of Tab. V (RSB column) and the TET/TTC percentages in Sec. IV-D using only tracks that pass the SAT gate (or satisfy |dy| ≤ wc, Eq. 7). If ρ(MDR,DRAC) rises above ~0.6 and the fraction of MDR>3 m/s² tracks with TET=0 falls below ~15%, the claimed complementarity is an artifact of the conservative gate. Report the same numbers for the RSB-admitted but SAT-rejected subset separately.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that MDR, FSR, and LEA capture safety-relevant information inaccessible to established metrics—rests on Spearman correlations computed under the RSB collision filter (Sec. IV-D, Tab. V). RSB admits 98% of FN and 93% of FP tracks (Sec. V), including laterally distant objects such as opposite-road vehicles (Fig. 4, FN2/FN3). For these, MDR is computed from a 1D longitudinal model (Eq. 5) that ignores lateral geometry, yet the gate admits them because ellipsoidal reachable sets overlap under bounded lateral acceleration. Consequently, the low correlations (e.g., MDR–DRAC ρ=0.30 under RSB vs 0.68 under SAT) and the claim that 42% of MDR>3 m/s² FNs have TET=0 may be driven by tracks a planner would never brake for. The paper acknowledges the gate's conservatism but still presents the resulting correlation structure as evidence of non-redundancy. The load-bearing premise—that the scored threat set is planner-relevant—is unvalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a family of effort-based criticality metrics for open-loop evaluation of 3D perception errors in autonomous driving. False Speed Reduction (FSR) translates persistent false positives into a cumulative braking-impulse-like quantity, Maximum Deceleration Rate (MDR) translates missed objects into a peak required longitudinal deceleration, and Lateral Evasion Acceleration (LEA) quantifies the minimum lateral acceleration to steer around a predicted collision. The metrics are gated by an ellipsoidal reachability filter that decides which object–ego pairs are scored. The framework is evaluated on nuScenes and Argoverse 2 with three perception pipelines (AB3DMOT, CenterPoint, BEVFusion), reporting severity distributions, cumulative effort, and Spearman correlations against established metrics such as TTC, DRAC, THW, and TET. The central claims are that 65–93% of perception errors are non-critical and that the proposed metrics capture safety-relevant information not provided by conventional measures.","tokens_in":14657,"tokens_out":4024,"duration_ms":44555,"significance":"If the central claim is substantiated, the paper offers a practically useful way to rank perception failures by the collision-avoidance effort they would impose, with clearly interpretable units (m/s and m/s^2) and explicit error-type separation. The work is strengthened by self-contained kinematic derivations (Eqs. 2 and 5), a modular collision filter that can be exchanged, a broad evaluation across datasets and pipelines, and scenario-level analyses that illustrate how the metrics behave. The paper also honestly lists several limitations, including the heuristic nature of the metrics and the conservatism of the reachability gate. However, the central claim of non-redundancy with established metrics rests on correlation evidence that may be confounded by the very conservatism the authors acknowledge. The paper therefore requires additional validation of the planner-relevance of the scored threat set before the strong abstract claim can be accepted.","major_comments":[{"comment":"The complementarity claim is built on Spearman correlations computed under the RSB gate, which admits 98% of FN and 93% of FP tracks. As the paper shows in Fig. 4, many of these are opposite-road or off-road objects that a planner would never brake for. The low MDR–DRAC correlation (ρ=0.30) and the statistic that 42% of MDR>3 m/s² FNs have TET=0 may therefore be dominated by non-threat tracks, not by safety-relevant information. The authors acknowledge the gate's conservatism but still present the resulting correlation structure as evidence of non-redundancy. Please report the correlation analysis on a planner-relevant subset (e.g., ego-lane objects, or the SAT-gated set) and show that the metrics retain their complementary value there. This is load-bearing for the abstract claim.","section":"§IV-D, Table V, and §V (Role of the collision filter)"},{"comment":"FSR is defined as (N_frames × T_cycle) × a_avg, i.e., the sum of per-frame required decelerations multiplied by the frame interval. The per-frame deceleration is the constant braking that would be needed to avoid the phantom if the ego started braking at that frame. Summing these values over a persistent phantom is not the accumulated velocity loss of a single braking maneuver; it can double-count the same threat across frames and can exceed the physical speed differential (e.g., Fig. 4 FP1: 17.5 m/s). If FSR is intended as a heuristic 'effort' score, the paper should state this more precisely and justify why summation rather than, say, the maximum required speed reduction is the right aggregation. As written, the physical interpretation of 'cumulative velocity loss' is not fully supported.","section":"§III-B, Eq. (4)"},{"comment":"The abstract states that Spearman correlation analysis 'confirms that all three metrics capture safety-relevant information inaccessible to established time-based, deceleration-based, or normalized criticality measures.' Given that the paper itself describes the metrics as 'heuristic approximations' (§V) and that no closed-loop planner validation is performed, 'confirms' overstates the evidence. A correlation analysis at track level shows statistical non-redundancy, but it does not establish safety relevance. Please temper the wording or add a validation step (e.g., replaying high-scoring errors in a planner or comparing against human-judged criticality) to support the safety-relevance claim.","section":"Abstract and §VI"}],"minor_comments":[{"comment":"Reference [24] is listed as 'Argoverse 2' but the text says 'Argoverse [24]' at one point in Section IV-A; please make the citation consistent.","section":"§II and References"},{"comment":"The table and figure captions would benefit from explicitly stating the dataset split (validation set) and the number of scenes used. The current text mentions nuScenes has 1000 scenes, but the exact split is not stated.","section":"Table II and Fig. 3"},{"comment":"The reachable-set semi-axis growth is written as σ(τ)=d/2 + 0.5*a_max*τ², but the justification for the linear-in-τ term (from the initial position uncertainty) is not given. Please add a short explanation or reference.","section":"§III-D, Eq. (5)"},{"comment":"The text says 'All pairwise |ρ| among established metrics exceed 0.69 (RSB)' but does not list the TET–THW correlation explicitly. A supplementary table with all pairwise correlations would be helpful for reproducibility.","section":"§IV-D"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its derivations and evaluation breadth, but the central claim of non-redundant safety-relevant information is currently supported only by correlations under an overly permissive gate. The skeptic's concern that the RSB-admitted non-threat tracks confound Table V lands: the paper's own scenario analysis (Fig. 4) provides concrete examples. I would encourage the editor to require the authors to either re-run the correlation analysis on a planner-relevant subset or substantially weaken the claim in the abstract. The FSR aggregation issue is also worth addressing, but it is a modeling choice that can be clarified. Overall, the contribution is publishable after revision, but not in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is good: FSR and MDR give interpretable physical meaning to false positives and false negatives, and the evaluation is broad—three pipelines, two datasets, and an honest discussion of the RSB/SAT trade-off. The kinematic derivations in Eqs. (2) and (5) are elementary but sound, and FSR's persistence-sensitivity is a genuinely useful complement to single-frame measures. I'd want to see this paper engaged with seriously.\n\nThe main problem is the central empirical claim. The Spearman correlations in Tab. V are computed under RSB, which admits 98% of FN tracks. The paper itself shows (Fig. 4) that this gate scores opposite-road vehicles and other laterally distant objects that a planner would ignore. Under SAT—where the threat set is actually plausible—MDR–DRAC correlation jumps from 0.30 to 0.68. That doesn't tell me MDR is capturing unique safety information; it tells me the low RSB correlations are largely an artifact of scoring a pile of non-threats. The 42% of high-MDR FNs with TET=0 needs to be recomputed on the SAT-gated or lane-filtered set before it can support the non-redundancy claim.\n\nAlso, the 65–93% non-critical fractions depend entirely on hand-set thresholds in Tab. III, with no sensitivity analysis. And the \"braking impulse\" phrasing for FSR is a misnomer—it's a cumulative speed reduction, not an impulse.\n\nThat said, the paper is careful in its limitations discussion, acknowledges the metrics are heuristic, and doesn't oversell them as replacements for closed-loop testing. The FSR/MDR definitions themselves are clean and reproducible enough to be worth testing on other datasets.\n\nVerdict: worth sending to peer review, but the authors should be pushed to separate the filter's effect from the metrics' properties—report both gates throughout, or restrict the main correlation claims to the SAT subset. If the complementarity survives that, it's a real contribution. If not, the paper still stands as a practical screening tool, just with a more modest conclusion.","headline":"Useful new metrics, but the headline non-redundancy claim is inflated by an over-permissive collision filter.","tokens_in":15184,"tokens_out":2134,"would_cite":true,"duration_ms":24890,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Perception errors become braking and steering effort, not just counts","keywords":["effort-based criticality","false speed reduction","maximum deceleration rate","lateral evasion acceleration","reachability analysis","3D perception evaluation","autonomous driving safety","false positives and false negatives"],"falsifier":"Run a closed-loop simulation or real-vehicle test where a planner responds to the same false positive and false negative scenarios; compare the actual braking or steering effort to the FSR/MDR/LEA predictions. If the predicted effort does not correlate with the planner's actual commanded deceleration or lateral acceleration across a set of diverse traffic scenes, the metric's claim to represent avoidance effort collapses. A simpler check: vary treact from 0.2 to 0.5 s and amax bounds by ±50% and observe whether the ranking of three pipelines by cumulative FSR/MDR changes materially—if it flips","tokens_in":14179,"feed_emoji":"🚗","tokens_out":1938,"duration_ms":22346,"temperature":0.7,"pith_summary":"This paper tries to establish that the safety relevance of false-positive and false-negative perception errors in autonomous driving can be quantified as physical collision-avoidance effort: false positives are scored by the cumulative speed reduction they would force (FSR), and false negatives by the peak braking demand they would require (MDR). A third metric, lateral evasion acceleration (LEA), covers cases where steering is the cheaper avoidance channel. The authors argue that these effort-based metrics capture safety-relevant information that conventional time-based or deceleration-based criticality measures like TTC and DRAC miss, and that they can rank and mine the most dangerous perception failures without needing a full planning stack. If true, this would give developers a way to compare detection and tracking pipelines on safety impact rather than raw accuracy.","feed_headline":"New metrics turn perception errors into braking and steering effort","feed_subtitle":"False positives and false negatives each get a physical cost, exposing risks that time-to-collision misses.","key_machinery":"The central mechanism is a pair of longitudinal kinematic equations: Eq. (2) computes the instantaneous deceleration caused by a persistent false positive, and Eq. (5) computes the braking demand for a missed object with its own acceleration. FSR aggregates the per-frame braking over the error's duration; MDR takes the per-frame maximum. These are complemented by a reachability-based ellipsoidal collision filter that grows vehicle shapes quadratically under bounded acceleration and predicts collision time as the earliest reachable-set overlap, ensuring that only dynamically plausible threats are scored. LEA, adapted from lateral-evasion kinematics, computes the minimum lateral acceleration n","core_discovery":"The paper's central claim is that each perception error can be translated, via kinematic equations, into the avoidance effort it would impose on the ego vehicle. For false positives, the required deceleration is derived by equating the distance the ego vehicle travels to the phantom object's distance plus separation until speeds match, yielding a per-frame braking value that accumulates over the phantom's lifetime into FSR. For false negatives, a constant-acceleration model accounts for the missed object's own motion, and the maximum per-frame braking demand becomes MDR. Both are gated by a reachability-based collision filter that predicts whether the ego and object reachable sets overlap wi","pith_inferences":["The reachability gate admits 98% of FN and 93% of FP tracks, while a tighter SAT gate admits only 4% and 1%; this suggests the reported non-critical fractions and correlation structure depend heavily on choosing a conservative filter, and a different gate could change which errors are considered critical.","Because the metrics use fixed scalar parameters (reaction time 0.3 s, acceleration bounds from naturalistic driving), the rank ordering of pipelines might shift if those parameters were varied; a sensitivity analysis over these values would test the robustness of the cross-pipeline conclusions.","The metrics are derived for longitudinal and lateral evasion separately; combining them into a single avoidance cost (as the paper mentions via a steer-threat number) could produce a more actionable safety score, though the paper intentionally keeps them separate to avoid discounting converging lateral threats.","If the metrics were applied at multiple confidence thresholds in a precision-recall sweep, they could yield criticality-weighted detection costs, effectively turning the framework into a safety-aware benchmark that does not require closed-loop simulation."],"forward_implications":["Raw detection error counts substantially overestimate safety risk: the paper finds that 65–93% of errors are non-critical, so effort-based metrics could serve as a risk filter before expensive closed-loop testing.","False positives that persist for many frames, even with moderate per-frame braking, accumulate high FSR; this penalizes phantom tracks that no single-frame TTC-based metric would flag.","False-negative severity is nearly independent of the perception pipeline: mean MDR stays at 2.0–2.5 m/s² across all pipelines and datasets, suggesting a geometry-driven safety floor that detector improvements alone cannot remove.","LEA is almost uncorrelated with longitudinal metrics, meaning lateral evasion effort is a distinct risk axis; combining it with MDR can distinguish an in-lane missed vehicle (high MDR, low LEA) from a laterally distant one (elevated MDR, low LEA).","Cumulative effort—the sum of FSR or MDR over all error tracks—can disambiguate pipelines that have similar precision or recall but very different total safety burden."],"fun_headline_variants":["Perception errors priced in braking and steering effort","False positives cost deceleration, false negatives cost braking","How much effort to avoid a phantom? New metrics answer","Turn perception errors into required avoidance effort","Effort-based metrics: from error to avoidance cost"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a single longitudinal, constant-deceleration model with fixed scalar parameters (reaction time, acceleration bounds, reachable-set growth) is a faithful proxy for what a real planner would do in response to a perception error.","fun_headline_variants_meta":{"raw":{"variants":["Perception errors priced in braking and steering effort","False positives cost deceleration, false negatives cost braking","How much effort to avoid a phantom? New metrics answer","Turn perception errors into required avoidance effort","Effort-based metrics: from error to avoidance cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1366,"prompt_tokens":702,"completion_tokens":664,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":591}},"tokens_in":446,"tokens_out":664,"duration_ms":7068,"temperature":1.0,"reasoning_tokens":591,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T17:06:45.824596+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a closed-loop simulation or real-vehicle test where a planner responds to the same false positive and false negative scenarios; compare the actual braking or steering effort to the FSR/MDR/LEA predictions. If the predicted effort does not correlate with the planner's actual commanded deceleration or lateral acceleration across a set of diverse traffic scenes, the metric's claim to represent avoidance effort collapses. A simpler check: vary treact from 0.2 to 0.5 s and amax bounds by ±50% and observe whether the ranking of three pipelines by cumulative FSR/MDR changes materially—if it flips","supporting_citations":[],"review_version":1}