{"id":"1515fe7a-474c-4c43-8e04-daaf398f19d4","arxiv_id":"2611.17190","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":7,"one_line_summary":"A benchmark for reference-frame UAV video stabilization that measures and mitigates consensus capture by moving objects in dynamic urban scenes.","lead":"This paper measures how moving vehicles in drone footage can trick a video stabilization algorithm into aligning frames to the vehicles instead of the road, a failure called consensus capture. It provides a benchmark with static and dynamic tests to evaluate ten registration pipelines and shows that object masking prevents this capture.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified. The benchmark's capture measurements are internally consistent and GCP-audited where they matter; the main limitations are acknowledged and bounded.","rationale":"The reader's weakest assumption — the pseudo-ground-truth reference and the 1.00 px GCP repeatability — is a real limitation and is honestly acknowledged in b659 and b275/b276. I agree that it is the most fragile element of the dynamic protocol. However, I do not find it load-bearing for the central claim: the capture fraction is defined against the reference estimator, but a spurious capture would require the reference's sub-pixel error to be coherently aligned with group motion over tens of pixels, and the GCP-fitted transforms confirm the captured estimates they observe. The existence claim 'coherently moving objects can capture the consensus' is additionally supported by the theoretical mechanism (Stewart 1997; Chum et al. 2005 discussion in Section 4.1) and by the masked-vs-unmasked contrast on identical pairs. The remaining caveat I would check is whether the real-time ORB captures are robust to the ratio-test threshold chosen in Part II tuning (r=0.99, a corner of the grid where the ratio test rejects almost nothing). This does not by itself threaten the central argument — masked ORB at the same threshold shows no capture, so coherent motion is still necessary — but reporting the capture counts at the shipped r=0.90 value would make the applied claim more precise. Since the paper releases code, data, and every campaign configuration, this is a small regeneration rather than new annotation effort. On the evidence as presented, the reader's ACCEPT verdict is appropriate.","tokens_in":61488,"tokens_out":16562,"duration_ms":127738,"concrete_test":"Rerun the real-time unmasked-ORB campaign on the 304 moving-group pairs at the Part I/shipped ratio-test threshold r=0.90, keeping all other stored campaign parameters, seeds, and settings fixed, and compare the set of pairs with capture fraction lambda >= 0.1 against the r=0.99 tuned run. If the 17 short-offset pulls disappear or shrink materially, the headline capture result should be qualified as conditional on the near-disabled ratio test; if they persist, the threshold sensitivity concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim survives scrutiny. The capture fraction (Eq. 18) is a direct geometric measure, and the GCP-fitted homographies independently confirm the few captured estimates that fall on audited frames (median 4.3 px real-time, 4.7 px offline, versus sub-1.2 px for masked ORB), so the phenomenon is not an artifact of the reference estimator. The pseudo-ground-truth dependence flagged by the reader exists and is explicitly acknowledged in the paper (b275, b276, b659): the reference shares a detector with two candidates, and covariance with candidates is not measured, but the audit residual is 0.06 px median / 0.38 px 95th percentile, an order of magnitude below the captured estimates, and the masking comparison is supported by reference-free and GCP-based outcomes. The acknowledged restrictions (one recording day, at most 1.94% moving foreground, nine cases, b653) bound the generality but do not undercut an existence claim about consensus capture. The inlier-statistics claim is supported by within-case, within-offset-bin rank comparisons rather than pooled aggregates. I find no load-bearing flaw. The one check I would still run is a sensitivity of the real-time ORB captures to the Part II tuning move of the ratio-test threshold to r=0.99, a grid corner the paper itself notes disables most of the ratio test.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper presents a two-part benchmark for reference-based UAV video stabilization, introducing and measuring 'consensus capture'—a failure mode where coherently moving objects (e.g., a platoon of vehicles) supply mutually consistent correspondences that draw the robust homography estimate toward their motion, while inlier statistics remain normal. Part I evaluates ten pipelines under exactly known homographies on static 4K aerial frames, establishing baseline accuracy and the effects of compute budget, threshold tuning, and object masking. Part II evaluates the same pipelines on nine hovering-drone cases over busy intersections, using a GCP-audited reference estimator and metrics that measure background alignment, object position error, and the capture fraction (the share of a moving group's displacement the estimate follows). The key finding is that unmasked ORB is captured on 28 of 304 real-time pairs and 6 offline pairs, with inlier statistics indistinguishable from non-captured estimates, and that object masking removes capture for all pipelines. Code, data, and results are publicly released.","tokens_in":69055,"tokens_out":1517,"duration_ms":212008,"significance":"The paper addresses a genuine and under-measured failure mode in UAV video stabilization that has been recognized as a risk but not systematically benchmarked. The methodology is thorough: pre-specified analysis plan, site-disjoint tuning/evaluation split, GCP audit of the reference estimator, and four complementary reference-free alignment measures. The finding that inlier statistics cannot detect capture is important for practitioners who rely on these statistics for pipeline selection or quality control. The benchmark provides a reusable framework for evaluating registration pipelines on dynamic scenes, and the public release of code, data, and results enables extension to other pipelines, altitudes, and traffic conditions. The paper's main limitations (one recording day, at most 1.94% moving foreground, nine cases) are explicitly acknowledged and bounded.","major_comments":[{"comment":"The real-time tuning moves ORB's ratio-test threshold to r=0.99, which the paper notes 'lies next to r=1, where the ratio test no longer rejects anything.' This is a significant departure from ORB's default configuration (r=0.9) and from the offline regime where ORB keeps r=0.9 and still shows 6 captures. The paper's main capture claim (28 real-time pairs across 5 cases) depends on this tuning choice. The paper should discuss whether capture occurs at stricter ratio-test settings (e.g., r=0.9) in the real-time regime, or at least acknowledge that the real-time capture count may be inflated by the disabled ratio test. Without this analysis, it is unclear whether the real-time capture results reflect a property of ORB or a property of the specific tuning that admits many more false matches.","section":"Section 7.1, Tables C.1-C.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The captured SIFT and RootSIFT estimates (2 each) have CTE_GCP50 of 1.0 px, which equals the GCP repeatability at the 95th percentile (1.00 px, Table 6). The paper acknowledges that 'the GCPs separate almost nothing' (Section 7.4), but the claim that SIFT and RootSIFT 'are captured as well' (Section 7.3) should be qualified given that these measurements are at the noise floor of the annotation. The GCPs cannot independently confirm capture for these pipelines, and the reference-relative scores for SIFT and RootSIFT are the least independent of the reference estimator (as acknowledged in b276). The paper should either provide additional evidence (e.g., capture fraction values for these estimates) or explicitly note that the SIFT/RootSIFT captures are at the resolution limit of the audit.","section":"Section 7.3, Table 7","authors_acknowledge":"They separate a captured estimate from a correct one and give a masking contrast independent of the reference, but they cannot rank pipelines whose errors differ by less than a pixel, and the reference-relative columns, which can, are limited by the audited residual.","authors_acknowledge_p":0.77}],"minor_comments":[{"comment":"The capture threshold lambda >= 0.1 is justified against the identity transform (0.07) and reference estimator error (0.08). Was this threshold pre-specified or chosen post-hoc? The paper should state this explicitly.","section":"Section 4.5, Eq. (18)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The case selection procedure uses an affine ECC fit to remove platform motion. The validation against published trajectories (Table C.3) shows high correlation, but since the validation uses the same recordings, it tests implementation fidelity rather than independent accuracy. The paper should note this distinction.","section":"Section 7.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The capture fraction plots use a non-standard axis (linear within [0,0.1], logarithmic beyond). While the rationale is explained, the discontinuity at 0.1 makes visual comparison of values near the threshold difficult. Consider adding a zoomed panel or using a standard axis with a horizontal line at the threshold.","section":"Figure 8","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The GCP repeatability is reported for only 2 of 9 cases. The paper should state why the other 7 cases were not repeated, and whether the reported repeatability (1.00 px at 95th percentile) is assumed to hold for all cases.","section":"Table 6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The distinction between 'failure' (bottom, no estimate) and 'breakdown' (large but finite error) is important for interpreting the results. The paper states that no pipeline fails on any pair, but unmasked ORB has breakdowns with CTE up to 7085 px. This distinction should be clarified earlier and more prominently, perhaps with a definition in Section 4.6.","section":"Section 4.6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The claim that 'the statistics that describe a fit's self-consistency would then describe the wrong fit' (b668) could be misinterpreted as meaning inlier statistics are always uninformative. The paper shows that within each pipeline and case, the inlier ratio does order estimates by accuracy (median Spearman rho = -0.38 to -0.14, Fig. 10), with capture being the exception. Consider qualifying the conclusion to reflect that inlier statistics fail specifically for the capture case, not universally.","section":"Section 9 (Conclusion)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The claim that 'homography estimation does not change its output across different global random seeds' is strong and depends on the specific OpenCV build. The paper should note that this was verified for the build recorded in the result manifests, and may not hold for other builds or versions.","section":"Appendix A","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This is a solid paper. The headline is that it measures a failure mode—consensus capture—that has been known anecdotally for years but never systematically benchmarked on real footage. The authors build a careful two-part benchmark: a static protocol with exact ground truth and a dynamic protocol on real hovering-drone video, with a GCP audit that keeps the pseudo-ground-truth from being circular. The central finding—that unmasked ORB gets captured by coherently moving vehicles, that inlier statistics don't flag it, and that object masking prevents it—is well supported by the data.\n\nWhat the paper does well: the methodology is unusually thorough. Pre-specified analysis plan, site-disjoint tuning split, equivalence margins, cluster-level inference with sign-flip tests, public code and data. The GCP audit is a real strength: they measure the reference estimator's error against independent control points and report the resolution floor, so you know what differences are resolvable. The capture fraction (Eq. 18) is a direct geometric measure that doesn't depend on the reference estimator. The paper acknowledges its limitations (single recording day, altitude band, at most 1.94% moving foreground, nine cases) and doesn't overclaim.\n\nSoft spots: the main one is that the primary masking contrast (OBB vs. none for the average pipeline) doesn't meet the pre-specified claim rule in either regime. The effect is real for ORB but diluted by averaging over pipelines and cases, most of which never present a moving group. The paper is honest about this, but it means the headline claim about masking is supported descriptively rather than inferentially. The GCP resolution (1.00 px at the 95th percentile) limits comparisons between pipelines that differ by less than a pixel, which is most of them. The reference estimator shares a detector (SIFT) with two candidate pipelines, so their reference-relative scores are the least independent; the paper acknowledges this and relies on the GCP-based scores for the masking comparison. The stress-test note's suggestion about the ratio-test threshold (r=0.99 for real-time ORB, which disables most of the ratio test) is worth checking, but it's a tuning choice made on a site-disjoint split, so it's not a flaw—just a sensitivity worth exploring.\n\nWho this is for: anyone working on UAV video stabilization, trajectory extraction from drone footage, or robust estimation with coherent outliers. The benchmark is reusable and the released code and data are a real contribution.\n\nRecommendation: this paper deserves a serious referee. The methodology is sound, the claims are supported, and the limitations are honestly bounded. I would accept it with minor comments about the ratio-test sensitivity and perhaps a note about the inferential status of the masking claim.","headline":"Solid benchmark paper that measures a real failure mode (consensus capture) in UAV video stabilization. The methodology is thorough, the evidence supports the claims, and the limitations are honestly stated. Worth engaging with.","tokens_in":62394,"tokens_out":673,"duration_ms":54993},"pith_extraction":null,"created_at":"2026-11-13T10:42:58.583987+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":null,"supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}