{"id":"9331d280-295f-456d-b7c0-10347eef2e7e","arxiv_id":"2505.16740","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Conformalized YOLO boxes on the LARD runway dataset show empirical containment of 74-77% at a 70% target, and the proposed full-containment metric C-mAP jumps from under 2% to roughly 53-57%.","lead":"This paper applies conformal prediction to runway detection in aerial landing images, expanding YOLO bounding boxes so they statistically cover the true runway. It also introduces a new metric, C-mAP, that scores detections only when a predicted box fully contains the ground truth.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Calibration/evaluation split for conformal prediction is unspecified; if the same test subset is used for both, the reported 74–77% coverage is in-sample and the central out-of-sample guarantee is untested.","rationale":"The reader's weakest assumption identifies exactly the condition that must hold for the central claim: exchangeability between calibration and test. The manuscript's split description is too ambiguous to verify this. This is not an indictment of the method—split-conformal CP is standard, PUNCC is used, and the code is released—but the empirical coverage and C-mAP results in Tables 3 and 5 are the only evidence for the headline claim, and their validity depends on the hidden protocol. A clean split and error bars would settle it. I therefore keep CONDITIONAL and do not recommend rejection, because the issue is addressable by clarifications and a rerun.","tokens_in":12634,"tokens_out":8807,"duration_ms":80036,"concrete_test":"Inspect the released code and split indices (github.com/alyasltd/conformal_runway_detection) or rerun the pipeline with an explicit disjoint partition: randomly split the 2,315 test images into 80% calibration and 20% evaluation, compute the Bonferroni quantiles once on the calibration split, then measure coverage, C-mAP, and mAP on the disjoint 20%, repeating over 5 random splits and reporting 95% Clopper-Pearson intervals. If the reported coverage (≥74%) is not reproduced on held-out images, the empirical guarantee is not out-of-sample; if it is reproduced, the ambiguity is resolved and the claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 says 'the experiments on conformal prediction are done on the remaining 20% of the test set' but never states where the calibration nonconformity quantiles are computed. A split-conformal coverage claim of at least 1−α requires that calibration and test boxes be exchangeable and that the quantiles be fixed before seeing the evaluation examples. If the 80% of the test set not mentioned as 'remaining' is used for calibration and the 20% for evaluation, the protocol is standard but should be stated; if instead the same 20% is used to both fit and evaluate the quantiles, the empirical coverage in Table 5 is an in-sample check and gives no evidence for the out-of-sample guarantee advertised in Section 4.2 ('theoretically guaranteed to contain the true bounding box ... at least 1−α'). The paper's own conclusion concedes that 'guarantees rely on calibration distributions matching test conditions.' Because real images appear only in the test set, the choice of calibration source (test vs. validation) also determines whether the real-image coverage is out-of-distribution. This ambiguity is load-bearing: the central numerical claims—coverage 73.93–77.06% at α=0.3 and the resulting C-mAP gains—cannot be interpreted without it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies conformal prediction to runway detection in aerial imagery, using fine-tuned YOLOv5 and YOLOv6 models on the LARD dataset. It uses coordinate-wise additive and multiplicative nonconformity scores with a Bonferroni correction, following de Grancey et al., and reports empirical coverage between 73.93% and 77.06% at alpha=0.3. It also introduces a new metric, Conformal mAP (C-mAP), in which a detection is a true positive only if IoU is above a threshold and IoA equals 1, and reports that conformalized models raise C-mAP from below 2% to roughly 53-57% while keeping standard mAP above 92%. The paper argues that conformal prediction provides statistical guarantees useful for vision-based landing systems.","tokens_in":12960,"tokens_out":7030,"duration_ms":59530,"significance":"If the calibration/evaluation protocol is properly specified, the paper is a useful application case study of conformal prediction in aerospace perception: it follows an established methodology, uses a public dataset, and releases code and trained models. The reported coverage values are internally consistent with the Bonferroni-corrected target, and the introduction of C-mAP is a simple but application-relevant evaluation metric for downstream tasks that require full containment of the runway. The incremental methodological novelty is modest, since the conformal procedure is imported from prior work and the C-mAP metric is a direct modification of mAP, but the application context and the empirical comparison of additive versus multiplicative penalties add value for the COPA community.","major_comments":[{"comment":"The calibration split for computing the conformal quantiles is never specified. The text states that \"the experiments on conformal prediction are done on the remaining 20% of the test set,\" but it does not state where the nonconformity-score quantiles in Section 4.2 are computed. A split-conformal coverage guarantee requires that calibration and evaluation samples be exchangeable and that the quantiles be fixed before seeing the evaluation boxes. If the same 20% subset was used for both calibration and evaluation, the coverage numbers in Table 5 are an in-sample check and provide no evidence for the out-of-sample guarantee advertised in Section 4.2; if the other 80% of the test set was used for calibration, that should be stated with sample sizes. The choice also determines whether the real-image results are in-distribution, since real images appear only in the test set. This ambiguity is load-bearing for the central numerical claims.","section":"Section 5, Dataset paragraph; Tables 3 and 5"},{"comment":"Coverage is reported as \"the proportion of predicted boxes that cover their associated ground truth,\" which conditions on a detection existing. The formal statement in Section 4.2, P(b_gt_{n+1} is contained in b_conf_{n+1}) >= 1-alpha, applies to a paired prediction and ground-truth box; if the detector outputs no box for an image, there is no conformalized box and that ground truth is excluded from the denominator. The paper should state this conditioning explicitly and, ideally, report the detection failure rate and unconditional coverage over all test ground truths, since the downstream VLS pipeline needs a box for every approach image.","section":"Section 5.2, coverage definition; Section 4.2, formal guarantee"},{"comment":"The C-mAP gain is partly by construction. The C-AP true-positive condition is IoU >= t and IoA = 1, and conformalization is precisely a procedure that expands boxes to enforce containment; therefore the jump in C-mAP from 0.77-1.31% to 52.71-56.86% is expected. The paper should not present this increase as independent evidence of improved reliability. The primary evidence should be empirical coverage relative to the target 1-alpha, together with the cost in standard mAP and box inflation. The metric remains useful as an application-specific evaluation criterion, but the paper should discuss how its definition aligns with the conformalization objective when interpreting the results.","section":"Section 4.2 and Table 3"}],"minor_comments":[{"comment":"The sentence \"Our next study will compare the Conformal Average precision of all models\" appears to be a leftover from an outline; it should be revised to \"We next compare...\".","section":"Section 5.2"},{"comment":"The header \"squared root of box area\" should be \"square root of box area\".","section":"Table 5"},{"comment":"The captions use \"YOLO'S predictions\"; the possessive should be \"YOLO's predictions\" or, more simply, \"YOLO predictions\".","section":"Figures 8 and 9 captions"},{"comment":"The notation \"mAP@50:95\" should be \"mAP@0.5:0.95\" to match the table and the text.","section":"Section 2.1"},{"comment":"The phrase \"across three metrics: standard mean Average Precision (mAP), Conformal mean Average Precision (C-mAP, naturally extended from C-mAP)\" contains a typo; the second occurrence should refer to C-AP or a correctly named extension.","section":"Section 5.2, metric definitions"},{"comment":"The abstract's claim about \"paving the way for certification\" is stronger than the evidence presented; the conclusion appropriately limits the claims, so the abstract should be aligned with that more cautious framing.","section":"Abstract and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of COPA and the central ambiguity about the calibration split is likely fixable. If the authors confirm that a proper split-conformal protocol was used, add confidence intervals, and clarify the conditioning in the coverage definition, the main empirical claims would be supported. The novelty is modest but acceptable for an application-oriented venue. I would not reject on novelty grounds, but the current manuscript does not yet allow the reader to verify the advertised out-of-sample guarantee."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. The core message is simple: take de Grancey et al.'s additive/multiplicative conformal boxes, apply them to YOLOv5/v6 on LARD runway images, and report coverage plus a new metric, C-mAP, that counts a detection as a true positive only when it fully contains the ground truth (IoA=1). That is a direct reuse of known machinery, but the C-mAP metric is new and makes sense for the downstream pose-estimation problem, where partial overlap is not enough.\n\nThe paper does several things well. It is transparent about the method being an application of existing conformal tools, it releases code and trained models, and it is honest in the conclusion about the limitations: single-object scenes, and guarantees depending on calibration/test distribution matching. The reported coverage numbers (74–77% at alpha=0.3) are internally consistent with a Bonferroni-corrected 70% target. The large jump in C-mAP from below 2% to around 53–57% is plausible given that conformalization is designed to produce containing boxes.\n\nThe soft spot is real and load-bearing: the calibration/evaluation split is not specified. The text says conformal experiments are done on 'the remaining 20% of the test set' but never says where the nonconformity quantiles are computed. If the quantiles are fit on the same 20% used for evaluation, the reported coverage is an in-sample check, not evidence for the advertised out-of-sample guarantee. If they are fit on the validation set (or the other 80% of test), that should be stated. This matters even more because real images appear only in the test set; if calibration is purely synthetic, the real-image results are out-of-distribution. This is fixable with a paragraph and a clear protocol description.\n\nMinor issues: no confidence intervals or multiple seeds, and the comparison is just conformalized vs. vanilla YOLO without a stronger baseline like more aggressive NMS thresholds. These are not fatal.\n\nBottom line: the paper is a competent, honest application of known conformal prediction to an aerospace task, plus a simple but well-motivated metric. It deserves a serious referee, provided the experimental protocol is clarified. I'd recommend asking the authors to specify exactly how calibration and evaluation sets were split, and to report error bars. The paper is a reasonable candidate for publication after that revision.","headline":"Standard conformal prediction applied to runway detection; the new C-mAP metric is sensible but the calibration/evaluation split is under-specified and that ambiguity is load-bearing.","tokens_in":13462,"tokens_out":2476,"would_cite":true,"duration_ms":20326,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Conformal prediction gives runway detectors a statistical containment guarantee, raising a new strict C-mAP metric from under 2% to over 53% while keeping mAP above 92%.","keywords":["Conformal Prediction","Object Detection","Runway Detection","Uncertainty Quantification","Vision-Based Landing","C-mAP","YOLOv5","YOLOv6"],"falsifier":"Split a held-out runway set into two disjoint halves, calibrate the per-side quantiles on one half, then evaluate containment on the other half at alpha = 0.3; if the empirical coverage falls below 70% on the untouched half, the reported guarantee is not reproducible under a clean calibration/evaluation separation.","tokens_in":12484,"feed_emoji":"✈️","tokens_out":6723,"duration_ms":49577,"temperature":0.7,"pith_summary":"The paper argues that conformal prediction can supply the statistical uncertainty guarantees missing from standard object-detection confidence scores, specifically for runway detection in vision-based landing systems. Using fine-tuned YOLOv5 and YOLOv6 models on the LARD runway dataset, the authors apply coordinate-wise nonconformity scores with a Bonferroni correction to enlarge predicted boxes, and report that the resulting conformal boxes contain the ground-truth runway in about 74–77% of test images at a user-set target of 70% (alpha = 0.3). They also introduce C-mAP, a precision-recall metric that counts a detection only when it both overlaps (IoU threshold) and fully encloses (IoA = 1) the runway; on this metric conformalized models jump from below 2% to roughly 53–57% while standard mAP stays above 92%. The significance is practical: if these guarantees hold, downstream pose estimation can rely on a proven containment probability rather than on raw confidence scores.","feed_headline":"Conformal boxes lift runway detection containment from 1% to 55%","feed_subtitle":"New C-mAP metric requires full enclosure; conformalized YOLO models reach 53–57% while mAP stays above 92%.","key_machinery":"The load-bearing object is the conformal bounding box, constructed from coordinate-wise nonconformity scores. For each side of a matched predicted box, the signed difference (additive) or relative difference (multiplicative) to the ground-truth box is recorded over the calibration set, and the Bonferroni-corrected quantile at level 1 - alpha/4 is used to expand that side at inference. This expansion is what converts a raw YOLO box into a conformal box with a containment guarantee. The companion object is the C-AP/C-mAP matching rule, which requires both IoU >= t and IoA = 1, where IoA is the intersection divided by the ground-truth area; this forces a detection to fully enclose the runway, matching the downstream need for complete runway crops in pose estimation.","core_discovery":"The central claim is that conformal prediction, applied to the four coordinates of a detected bounding box, produces a conformal box that contains the true runway box with probability at least 1 - alpha, without retraining the underlying detector. The method computes per-side nonconformity scores (additive or multiplicative) between matched predicted and ground-truth boxes on a calibration set, takes Bonferroni-corrected quantiles at level alpha/4 for each side, and expands new predictions accordingly. At alpha = 0.3, the paper reports empirical containment rates of 73.93–77.06% across the four conformalized YOLO variants. The same procedure raises the proposed C-mAP score from 0.77–1.31% for the base YOLO models to 52.71–56.86% for the conformalized versions, while standard mAP remains above 92% in every case. The paper also reports that conformalization expands predicted boxes modestly, with a Stretch ratio between 1.11 and 1.16 and average side margins under 19 pixels.","pith_inferences":["If the calibration pool matches real landing approaches, the same coordinate-wise conformalization could be applied to other critical detections in the aircraft pipeline, such as obstacles or ground markers, where full containment is more important than tight boundaries.","The reported coverage (74–77%) exceeds the 70% target, which suggests the Bonferroni correction is conservative; a method that controls the joint coverage risk more efficiently could obtain the same guarantee with smaller boxes.","The paper's experimental description says conformal experiments are run on 'the remaining 20% of the test set,' which leaves it unclear whether calibration and evaluation are disjoint; if both are drawn from the same test pool, the advertised out-of-sample coverage could be overstated.","A natural next test is multi-object scenes: with several runways or false alarms per image, independent per-box conformalization may produce overlapping or inflated boxes that distort C-mAP, exactly the regime the paper names as a limitation."],"forward_implications":["At the reported alpha = 0.3, a vision-based landing system can treat each conformal runway box as containing the true runway with about 74–77% empirical reliability, giving pose estimation a quantitative safety margin.","Operators can trade coverage against box size by lowering alpha: smaller alpha raises both coverage and C-mAP but enlarges the conformal boxes further.","The C-mAP metric provides a single-number evaluation that rewards full enclosure alongside localization, so detector comparisons for landing tasks can reflect downstream requirements rather than IoU overlap alone.","Because conformalization is applied after training, existing YOLOv5 and YOLOv6 runway detectors can be retrofitted with statistical guarantees without retraining or architectural changes."],"supporting_citations":[{"why":"Supplies the coordinate-wise nonconformity scores and the Bonferroni correction used to build conformal boxes from object-detector outputs.","marker":"de Grancey et al. (2022)"},{"why":"Establishes the distribution-free conformal prediction framework and the exchangeability assumption on which the coverage guarantee rests.","marker":"Vovk et al. (2005)"},{"why":"Provides the conformal risk control formulation that connects user-specified error levels to coverage guarantees in structured prediction.","marker":"Angelopoulos et al. (2024)"},{"why":"Gives the Stretch metric and an earlier conformal object-detection application on railway signaling that this work extends to aerial runway detection.","marker":"Andéol et al. (2023)"},{"why":"Introduces the LARD dataset of synthetic and real runway approach images on which all experiments are run.","marker":"Ducoffe et al. (2023)"},{"why":"Supplies the pre-trained YOLOv5-small model that is fine-tuned and conformalized in the experiments.","marker":"Jocher et al. (2020)"},{"why":"Supplies the pre-trained YOLOv6-small model that provides the second base detector for comparison.","marker":"Li et al. (2022)"}],"fun_headline_variants":["Conformal boxes lift runway C-mAP from 1% to 55%","New C-mAP metric raises runway detection containment to 55%","Conformal prediction adds statistical guarantee: 1% to 55% C-mAP","Runway detection: conformal boxes push containment metric from 1% to 55%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The coverage guarantee depends on the calibration and test images being exchangeable, i.e., drawn from the same runway-approach conditions, and the paper does not clearly demonstrate a disjoint calibration/evaluation split on its 20% test subset.","fun_headline_variants_meta":{"raw":{"variants":["Conformal boxes lift runway C-mAP from 1% to 55%","New C-mAP metric raises runway detection containment to 55%","Conformal prediction adds statistical guarantee: 1% to 55% C-mAP","Runway detection: conformal boxes push containment metric from 1% to 55%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000688,"raw_usage":{"total_tokens":3086,"prompt_tokens":878,"completion_tokens":2208,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":2121}},"tokens_in":494,"tokens_out":2208,"duration_ms":14575,"temperature":1.0,"reasoning_tokens":2121,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:55:46.112073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split a held-out runway set into two disjoint halves, calibrate the per-side quantiles on one half, then evaluate containment on the other half at alpha = 0.3; if the empirical coverage falls below 70% on the untouched half, the reported guarantee is not reproducible under a clean calibration/evaluation separation.","supporting_citations":[{"cited_title":"Object detection with probabilistic guarantees","cited_arxiv_id":null,"evidence_quote":"Supplies the coordinate-wise nonconformity scores and the Bonferroni correction used to build conformal boxes from object-detector outputs."},{"cited_title":"Conformal risk control","cited_arxiv_id":null,"evidence_quote":"Provides the conformal risk control formulation that connects user-specified error levels to coverage guarantees in structured prediction."}],"review_version":1}