{"id":"9f11017e-e30a-4634-a287-c0bd5e9a0da8","arxiv_id":"1908.06368","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Average delay (AD), a new video object detection metric built from clipped per-instance detection delays under false-positive-ratio controls, reveals that DFF, FGFA, and CaTDet preserve mAP while increasing detection delay.","lead":"Many video object detectors that preserve average precision (AP) are actually slower to notice new objects. This paper introduces average delay (AD), a metric that measures how quickly detectors find objects in video, and shows that AP alone misses this behavior.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AD metric's exponential, constant-per-frame-probability model is contradicted by the paper's own heavy-tail and confidence-buildup evidence; whether reported delay rankings survive a model-free aggregation is untested.","rationale":"The reader's weakest_assumption identifies the same core issue: the constant-p exponential model is asserted in Section 3.1 and contradicted by the paper's own evidence of heavy tails and gradual confidence buildup. My stress-test sharpens this into a concrete testable threat to the central claim. AD is not simply 'average delay'; Eq. 3 averages the reciprocal of (clipped mean + 1), a convex transform that weights low-delay operating points more heavily. If the paper's rankings are robust to replacing this transform with ordinary averaging of clipped delays, then the exponential framing is cosmetic and the empirical conclusion stands. If not, then the claim that 'most methods drastically increase the detection delay but still preserve AP' is not established independent of the metric's assumed model. This concern is load-bearing but not disqualifying: the paper is transparent about the heavy-tail deviation and provides code, so the proposed re-analysis is feasible. The 3-fold consistency check in Table 5 is useful but only tests variance of the chosen AD, not sensitivity to the metric definition. I therefore keep the reader's CONDITIONAL verdict unchanged: the paper's contribution is plausible and reproducible, but the central empirical conclusion should be verified against a model-free delay aggregation before being accepted as a robust finding.","tokens_in":11388,"tokens_out":6106,"duration_ms":72696,"concrete_test":"Using released VMetrics code and detection outputs, recompute AD for all headline methods (R-FCN, Faster R-CNN, DFF at key-frame intervals, FGFA, CaTDet) in two ways: (a) replace Eq. 3's p-averaging with the arithmetic mean of clipped delays over the six FP ratios, AD' = (1/R)Σ_r D*_r; (b) repeat both original AD and AD' with W=60 instead of W=30. If rank order and relative gaps remain, the exponential assumption is harmless; if any adjacent pair swaps or gaps shrink below fold variability, the central claim is metric-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that video detectors preserve AP but drastically worsen detection delay—depends on AD being a faithful summary of temporal detection behavior. AD is built in Section 3.1 on the assumption that each frame offers an independent detection with constant probability p, so delay follows a discrete exponential distribution and p can be inferred from the clipped mean via Eq. 2. Eq. 3 then averages these inferred p values across false-positive ratios. The paper's own evidence contradicts the constant-p model: Table 3 shows off-window rates far above the exponential expectation, and Figure 7 shows FGFA confidence building gradually over frames rather than appearing with fixed probability. Under time-varying per-frame detection probability, the quantity 1/(D*+1) is a nonlinear transform of the clipped mean, not a latent detection rate, and averaging it across FP ratios imposes a specific, arbitrary weighting of delays. Consequently, the reported AD gaps (e.g., R-FCN 9.0 vs. DFF 1:10 11.6 vs. FGFA 10.2 in Figure 6) may reflect this weighting and the manual clipping at W=30 rather than robust temporal differences. Table 5 checks fold-to-fold consistency of the chosen AD numbers but does not test whether the rankings survive alternative, equally reasonable delay aggregations. If rankings change under a simpler model-free metric, the paper's main empirical conclusion is an artifact of the exponential assumption; if they do not, the assumption is a harmless wrapper. This is the load-bearing uncertainty because the paper's contribution is precisely a new metric, and the empirical claim is evaluated only with that metric.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that average precision (AP) is insensitive to the temporal behavior of video object detectors and proposes a new metric, average delay (AD), to measure detection delay. AD is defined in Eq. (3) as a function of per-frame detection probabilities inferred from clipped mean delays, with a detection window W=30 and six false-positive ratios. To support the evaluation, the authors introduce VIDT, a subset of ImageNet VID with multiple tracklets. Experiments with DFF, FGFA, CaTDet, and standard image detectors show that methods can preserve or improve mAP while substantially worsening AD, and the authors conclude that video object detection should be evaluated with a delay metric, especially for latency-critical applications.","tokens_in":11715,"tokens_out":2874,"duration_ms":32568,"significance":"If the main empirical claim holds, the paper makes a useful methodological contribution: it identifies a blind spot in mAP-based video object detection evaluation, releases code and a curated subset (VIDT), and provides a concrete way to quantify algorithmic delay. The toy-case experiments in Table 2 are a nice sanity check of the metric's behavior under controlled manipulations. The central claim, however, rests on AD being a faithful summary of temporal detection behavior, and the paper's own evidence casts doubt on the stochastic model used to derive AD. The contribution is therefore significant but currently not fully validated.","major_comments":[{"comment":"The constant-p, independent-per-frame detection model underlying Eqs. (1)-(2) is contradicted by the paper's own evidence. Table 3 shows off-window percentages roughly double the exponential expectation (e.g., 10.2% vs. 5.3% for R-FCN), and Fig. 7 shows FGFA confidence gradually building over frames rather than appearing with constant probability. Under a time-varying detection probability, the quantity 1/(D*+1) is not a latent detection rate, and Eq. (3) becomes an arbitrary nonlinear weighting of clipped delays. The authors should test whether the reported AD rankings survive a model-free aggregation, for instance mean/median clipped delay, hazard-rate estimates, or time to 50% detection, and report rank correlations. Without such a robustness check, the main conclusion that 'most methods drastically increase detection delay' may be an artifact of the exponential assumption.","section":"Sec. 3.1, Eqs. (1)-(2), Fig. 7, Table 3"},{"comment":"The two key design choices, W=30 and the FP ratio set {0.1, 0.2, 0.4, 0.8, 1.6, 3.2}, are introduced without justification, yet AD is a function of both. Because the paper's headline comparisons (e.g., R-FCN 9.0 vs. DFF 11.6 vs. FGFA 10.2 in Fig. 6) are point estimates of AD, it is important to show that rankings are stable over reasonable variations in W (e.g., 10, 20, 50, 100) and over alternative FP-ratio grids, including an unweighted average of per-ratio delays. If rankings change, the metric's comparisons are not robust.","section":"Sec. 3.3, Eq. (3)"},{"comment":"What is called a 'test of significance' is not a significance test. The 3-fold validation reports point estimates only and does not provide variances, confidence intervals, or any p-value for the differences between AD of R-FCN, DFF, and FGFA; moreover, folds are constructed from the dataset and are not independent samples of the underlying video distribution. The alternative dataset VIDT-2017 is a helpful additional check, but it also reports point estimates without uncertainty. The authors should add per-instance bootstrap or paired-difference confidence intervals to establish that the observed AD gaps are unlikely to arise from the small number of instances.","section":"Sec. 6.4, Table 5"},{"comment":"The rule that an instance disappearing for more than 10 frames is counted as a new instance is an arbitrary definition that directly affects the measured delay: a detector that recognizes re-appearance quickly will appear to have low delay, while one that does not re-detect will be penalized. Since this threshold is a free parameter of the dataset construction, a sensitivity analysis (e.g., thresholds of 5, 10, 20 frames) is needed to show that the main conclusions do not hinge on this choice.","section":"Sec. 4.2"}],"minor_comments":[{"comment":"'A verage' in the title is a typo; also 'descend' should be 'decent' in the conclusion.","section":"Abstract"},{"comment":"The caption contains a typo: 'delay dstribution' should be 'delay distribution'.","section":"Sec. 3.1, Fig. 2"},{"comment":"In the sentence 'we derive the corresponding AD fom the averaged p', 'fom' should be 'from'.","section":"Sec. 3.3"},{"comment":"The notation ̄D*_r implicitly depends on the window W and the FP ratio r; it would be clearer to write ̄D*_r(W) or define the dependence explicitly.","section":"Sec. 3.3, Eq. (3)"},{"comment":"The reference to 'CaTDET [25]' in the background section should be capitalized consistently as 'CaTDet' throughout.","section":"Sec. 2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely of interest to the video object detection community, but the main empirical claim depends on AD being a faithful delay summary. The missing robustness analysis against model-free delay aggregations and the absence of uncertainty quantification in Sec. 6.4 are load-bearing gaps. If the authors add those analyses and the rankings hold, I would support acceptance; if the rankings change substantially, the conclusion may need to be reframed. The relation to the authors' prior CaTDet paper [25] is an extension, not a circularity, but should be positioned clearly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: read this before you next benchmark a video object detector. The paper's claim is that mAP hides detection delay, and it backs it with a metric (AD) and a curated subset (VIDT). I think the claim is right even though the metric's statistical model is shaky.\n\nWhat's new: previous delay measures—CaTDet's single-precision delay and NAB's anomaly scoring—don't generalize. AD averages per-frame detection probability (inferred from clipped mean delay) across six false-positive ratios, which gives a more complete picture. The paper also shows, in clean experiments, that DFF, FGFA, and CaTDet preserve mAP while increasing delay compared to frame-by-frame detectors. The toy-case analysis in Table 2 is especially nice: it demonstrates that mAP and NAB can be gamed by boosting late detections, while AD is much less sensitive.\n\nThe soft spot is the exponential assumption. The paper models delay as discrete exponential with constant per-frame detection probability p, then infers p from the clipped mean. But its own Table 3 shows off-window rates roughly double the exponential expectation, and Figure 7 shows FGFA confidence building gradually—both contradict constant p. Under a time-varying detection probability, AD is still a reasonable summary statistic (it penalizes long delays via the harmonic-style averaging), but the paper's derivation is a justification, not a derivation. The real test is whether the rankings survive alternative aggregations, e.g., raw clipped means, median delay, or different W. The paper doesn't provide that, and its 'test of significance' is just fold-to-fold consistency, not a statistical test. I also note W=30 and the six FP ratios are presented without sensitivity analysis. These are weaknesses, but they are fixable and don't undermine the central observation—the mAP-vs-AD scatter in Figure 1 shows a decoupling that any reasonable delay metric would likely expose.\n\nWho should read it: anyone proposing a video detection method or designing benchmarks for latency-critical perception. It deserves serious peer review; I'd ask for a robustness section as a condition.","headline":"mAP really does hide detection delay, and AD is a credible metric to expose it; just don't take the exponential model literally.","tokens_in":12229,"tokens_out":2860,"would_cite":true,"duration_ms":30194,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Average precision hides detection delay in video object detection","keywords":["average delay","video object detection","evaluation metric","detection delay","average precision","ImageNet VIDT","false positive ratio","temporal dynamics"],"falsifier":"Compare each detector's empirical delay distribution against the fitted discrete exponential model on VIDT, for example by computing the Kaplan-Meier survival curve of first-detection times and testing whether detection probability per frame is constant. The paper's own Figure 7, showing FGFA confidence rising steadily over frames, already suggests the constant-p model fails there; a quantitative test would estimate p(t) and show whether re-ranking detectors by a nonparametric mean delay changes the AD ordering.","tokens_in":11203,"feed_emoji":"⏱️","tokens_out":4714,"duration_ms":45520,"temperature":0.7,"pith_summary":"This paper argues that average precision (AP), the standard accuracy metric for video object detection, is blind to when a detector first recognizes an object, and proposes a complementary metric called average delay (AD) that measures how many frames pass before an instance is detected. AD is defined over six false-positive ratios and averages the inferred per-frame detection probability, so it rewards early detection without rewarding a detector that simply emits many false positives. Experiments on a new subset of ImageNet VID, named VIDT, show that several video-specific detectors, including key-frame, feature-aggregation, and cascaded methods, keep mAP roughly unchanged while substantially increasing detection delay. For latency-critical uses such as autonomous driving, the paper concludes, accuracy metrics should be accompanied by a delay metric.","feed_headline":"Detection delay metric reveals what mAP hides","feed_subtitle":"New average delay score shows video detectors can preserve accuracy while detecting objects far too late.","key_machinery":"The load-bearing object is the AD formula, $\\mathrm{AD} = 1 / ((1/R) \\sum_r 1/(\\bar{D}^*_r + 1)) - 1$, where $\\bar{D}^*_r$ is the clipped mean delay to first detection at false-positive ratio $r$. The argument treats each detection as a Bernoulli event with constant probability $p$ per frame, so delay follows a discrete exponential distribution and $p$ is estimated as $1/(\\text{mean clipped delay} + 1)$. Averaging $p$ rather than raw delay makes the metric a harmonic-style mean that weights small delays more heavily, preventing a detector with many very late detections from looking good, and the false-positive ratio controls the false-alarm budget so a detector cannot reduce delay by emitting many false positives.","core_discovery":"The central claim is that mAP cannot distinguish detectors with different temporal behavior: a detector that recognizes objects only late in their trajectory can match a detector that recognizes them immediately, as long as the total number of true positives over frames is similar. The paper introduces AD as the delay counterpart to AP: for each of six false-positive ratios it measures the clipped mean delay to first detection, converts that mean to a per-frame detection probability under a discrete exponential model, averages those probabilities, and converts back to frames. This design makes AD sensitive to retardation (suppressing early detections) and nearly insensitive to tail boost (raising confidence of late detections), behaviors that mAP, NAB scores, and the CaTDet delay metric do not share. On the VIDT subset, DFF with sparse key frames, FGFA with feature aggregation, and CaTDet all preserve mAP while raising AD relative to frame-by-frame baselines, and the paper interprets this as evidence that temporal information is often used to refine late detections rather than to detect earlier.","pith_inferences":["An inference beyond the paper: AD as defined is one summary statistic of the whole delay distribution; a natural extension would be a delay-recall curve or a time-aware precision-recall surface that exposes not just mean delay but the full distribution over instances.","The paper's exponential assumption is a convenience; a detector whose confidence ramps up gradually, as FGFA visibly does, has time-dependent detection probability, so future work could replace the single p with a time-varying hazard model and still use the same false-positive-ratio framework.","The paper evaluates established detectors but does not propose a training objective; a testable extension is to train with a delay-aware loss that penalizes late first detections, and check whether AD can be improved without sacrificing mAP.","Because AD is computed from tracklet-level first detections, it implicitly assumes clean track identities; extending the metric to tracking-by-detection systems with identity switches would require a policy for when a detection belongs to an existing instance."],"forward_implications":["If mAP is retained as the headline accuracy number, video object detection papers should report AD alongside it; a method that improves or preserves mAP while worsening AD is not a strict improvement.","Latency-critical deployment, such as autonomous vehicle perception, should prefer detectors with low AD even when their mAP is slightly lower, because detections arriving after the decision deadline are effectively useless.","Methods that aggregate features over many frames (FGFA style) or use sparse key frames (DFF style) should be expected to trade delay for accuracy; the paper's measurements quantify that trade.","Anchor-scale choices affect delay as well as accuracy: adding small anchor scales improves AD for small instances, so delay evaluation can guide architecture choices beyond mAP.","AD's class-agnostic aggregation on VIDT means future evaluation datasets for delay should be built to contain instances entering at random times and locations, as VIDT does."],"supporting_citations":[{"why":"Provides the ImageNet VID dataset from which the new VIDT subset is selected for delay evaluation.","marker":"[6]"},{"why":"Defines the mAP metric that the paper argues is insensitive to temporal detection behavior.","marker":"[7]"},{"why":"Supplies the NAB benchmark and scoring approach that AD is compared against in the toy-case experiments.","marker":"[16]"},{"why":"Introduces an earlier delay metric (CaTDet) evaluated at a single precision, which AD generalizes and improves with multiple false-positive ratios.","marker":"[25]"},{"why":"Provides Faster R-CNN as a frame-by-frame baseline with low AD that video methods are contrasted against.","marker":"[31]"},{"why":"Describes FGFA, the feature-aggregation method whose increased AD despite improved mAP is a central piece of evidence.","marker":"[38]"},{"why":"Describes DFF, the key-frame method whose sparse key frames worsen AD, another central piece of evidence.","marker":"[39]"},{"why":"Provides R-FCN, the backbone detector used for baselines and for the implementations of the video methods evaluated.","marker":"[5]"}],"fun_headline_variants":["AP ignores detection delay in videos","New delay metric outshines mAP for video","Why mAP fails to catch late detections","Average Delay: the metric AP can't see","Video detectors: AP hides the delay problem"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The metric's numbers rest on the assumption that on each frame a detector detects an object with the same fixed probability, so delay follows a single exponential distribution; if a detector's per-frame detection probability changes with time since appearance, the reported AD values and rankings may partly reflect that assumption rather than the true temporal behavior.","fun_headline_variants_meta":{"raw":{"variants":["AP ignores detection delay in videos","New delay metric outshines mAP for video","Why mAP fails to catch late detections","Average Delay: the metric AP can't see","Video detectors: AP hides the delay problem"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1511,"prompt_tokens":894,"completion_tokens":617,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":549}},"tokens_in":510,"tokens_out":617,"duration_ms":6724,"temperature":1.0,"reasoning_tokens":549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:47:29.551887+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare each detector's empirical delay distribution against the fitted discrete exponential model on VIDT, for example by computing the Kaplan-Meier survival curve of first-detection times and testing whether detection probability per frame is constant. The paper's own Figure 7, showing FGFA confidence rising steadily over frames, already suggests the constant-p model fails there; a quantitative test would estimate p(t) and show whether re-ranking detectors by a nonparametric mean delay changes the AD ordering.","supporting_citations":[{"cited_title":"The pascal visual object classes (VOC) challenge","cited_arxiv_id":null,"evidence_quote":"Defines the mAP metric that the paper argues is insensitive to temporal detection behavior."},{"cited_title":"Evaluating real-time anomaly detection algorithms–the numenta anomaly bench- mark","cited_arxiv_id":null,"evidence_quote":"Supplies the NAB benchmark and scoring approach that AD is compared against in the toy-case experiments."},{"cited_title":"CaTDet: Cascaded Tracked Detector for Efficient Object Detection from Video","cited_arxiv_id":"1810.00434","evidence_quote":"Introduces an earlier delay metric (CaTDet) evaluated at a single precision, which AD generalizes and improves with multiple false-positive ratios."},{"cited_title":"Flow-guided feature aggregation for video object de- tection","cited_arxiv_id":null,"evidence_quote":"Describes FGFA, the feature-aggregation method whose increased AD despite improved mAP is a central piece of evidence."},{"cited_title":"Deep feature flow for video recognition","cited_arxiv_id":null,"evidence_quote":"Describes DFF, the key-frame method whose sparse key frames worsen AD, another central piece of evidence."},{"cited_title":"R-FCN: Object detection via region-based fully convolutional networks","cited_arxiv_id":null,"evidence_quote":"Provides R-FCN, the backbone detector used for baselines and for the implementations of the video methods evaluated."}],"review_version":1}