{"id":"c9ce913d-9904-4ab2-8409-48069cd22131","arxiv_id":"2504.17655","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Conformal prediction gives valid and compact label sets on a seven-class aerial event recognition task with fewer than 400 training samples, and temperature scaling sometimes increases set size.","lead":"This paper tests a statistical method that converts a single image label into a small list of possible labels, using few training examples. It shows the lists stay reliable on aerial drone images and that temperature scaling does not always shrink them.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temperature scaling fitted on the calibration set used for the conformal quantile can break score exchangeability, so the coverage guarantee is not logically protected and the empirical coverage numbers may be biased.","rationale":"The reader identified exactly the same load-bearing weakness: fitting temperature on Dcal and then using the same Dcal for conformal quantile computation violates the standard exchangeability condition underlying split conformal prediction. This concern is substantive because it attacks the only theoretical basis for the paper's coverage claims. The empirical results may still be valid as descriptive observations, but the paper presents them as conformal guarantees, and the LAC margins are thin enough that the double-use bias could matter. The paper is otherwise honest and the experimental design is reasonable; the issue can be addressed by re-running with a separate temperature-fitting set or by explicitly justifying why the double-use does not break exchangeability in this setting. Therefore, the reader's conditional verdict remains appropriate and no additional adjustment is needed.","tokens_in":13570,"tokens_out":4486,"duration_ms":48312,"concrete_test":"Repeat the MobileNet LAC pipeline with a three-way split: train, temperature-validation, and calibration-quantile subsets. Optimize T only on the temperature-validation subset, then compute all nonconformity scores and the conformal quantile on the untouched calibration subset. Run the same 50 random splits used in the paper and compare the distribution of empirical coverage and average prediction set size against the paper's Table 1 and Figure 3. If the LAC coverage at alpha=0.2 systematically drops below 0.8 or the set sizes shift by more than 0.05, the double use of Dcal is a material violation; if the results are statistically indistinguishable, the concern is theoretical rather than practical for this dataset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the split-conformal coverage guarantee (Equation 5) applies to the reported results. In Section 4, the author states that the temperature T is optimized by minimizing cross-entropy on the calibration set Dcal, and the same Dcal is then used to compute nonconformity scores and the conformal quantile. Split-conformal validity rests on exchangeability of calibration and test scores, which requires the score function to be independent of Dcal. Once T is fit to the labels in Dcal, the calibration scores are not exchangeable with test scores: the fitted T can exploit the calibration labels, making calibration scores stochastically smaller than test scores, which would inflate reported coverage. The paper itself notes in Section 2 that temperature scaling 'often shares the same calibration dataset Dcal', but it does not verify that the guarantee survives this double use. Because the LAC coverage at alpha=0.2 is only 0.81 with a standard deviation of 0.05, any bias from this double use could move the true coverage below the nominal 0.8 level. The descriptive claims about set sizes remain plausible, but the central claim of achieving the nominal coverage is not protected by the finite-sample conformal guarantee as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical evaluation of split-conformal prediction methods—LAC, APS, and RAPS—on a seven-class subset of the ERA aerial video dataset. The author fine-tunes three pretrained CNNs (MobileNetV2, DenseNet-121, ResNet-152) on a small training set (386 samples), uses a calibration set of 261 samples, and a test set of 112 samples, repeating the full pipeline over 50 random splits. Two calibration pipelines are compared: one with and one without temperature scaling. The main empirical findings are that LAC yields small prediction sets roughly at the nominal coverage, APS and RAPS exceed the nominal coverage with larger sets, and temperature scaling does not consistently reduce set size, sometimes increasing it for ResNet. The paper concludes that conformal prediction can provide useful uncertainty estimates in data-scarce, unconstrained environments even with lightweight models and simple nonconformity scores, and it advocates for future work on noisy labels and model compression.","tokens_in":13800,"tokens_out":7636,"duration_ms":70171,"significance":"If the empirical results are valid, the paper offers a useful and realistic benchmark for conformal prediction in a data-scarce remote-sensing setting. The repeated random splits, the comparison of three architectures, and the use of a public dataset are strengths. The observation that temperature scaling can increase set size for some architectures is practically relevant. However, the central validity claim is currently undermined by the reuse of the calibration set for both temperature fitting and conformal quantile computation, so the reported coverage numbers are not protected by the finite-sample conformal guarantee. Reproducibility is also weakened by the omission of RAPS hyperparameters. The paper does not provide code, though the use of public libraries (MAPIE, skorch) partially mitigates this concern.","major_comments":[{"comment":"The temperature parameter T is optimized by minimizing cross-entropy on Dcal, and the same Dcal is then used to compute the nonconformity scores and the conformal quantile. Split-conformal validity (Eq. 5) requires the score function to be fixed independently of the calibration data. Because T is fitted to the labels in Dcal, the calibration scores are not exchangeable with test scores, so the finite-sample coverage guarantee asserted in the abstract and Section 2 is not valid. The paper notes in Section 2 that temperature scaling 'often shares the same calibration dataset Dcal' but does not address the consequences. This is especially concerning for LAC at α=0.2, where mean coverage is 0.81±0.05 and the first quartile falls below 0.8 (Table 1, Fig. 3a); a bias from fitting T on Dcal could move true coverage below the nominal level. The authors should either reserve a separate held-out subset of Dcal for temperature fitting or explicitly state that the coverage results are heuristic and not protected by split-conformal theory.","section":"Section 4 (TS calibration) and Section 2, Eq. (10)"},{"comment":"The RAPS score in Eq. (9) depends on two hyperparameters, the regularization strength λ and the cutoff rank k_reg. These values are never reported, nor is the selection procedure described, in the experimental section. Because RAPS prediction set sizes and coverage can vary substantially with λ and k_reg, the RAPS results in Tables 1–2 and the qualitative comparison with LAC/APS are not reproducible without this information. The authors should state the parameter values used (or the default choices made by the MAPIE library) and, ideally, justify them.","section":"Section 2, Eq. (9); Section 4 experiments"}],"minor_comments":[{"comment":"'ßConstructing' appears to be a typo for 'Constructing'.","section":"Section 3, category list"},{"comment":"'the the highest-logit class' contains a duplicated article.","section":"Section 2, text after Eq. (10)"},{"comment":"The claim that the results 'highlight the significant potential of model compression techniques' is not directly supported by the experiments, since no compression or distillation is performed; the paper only compares architectures of different sizes. This wording should be softened or supported by additional experiments.","section":"Abstract and Section 5"},{"comment":"No code or data availability statement is provided; releasing the pipeline would aid reproducibility.","section":"General"},{"comment":"The paper does not perform any statistical significance testing for the difference in set sizes with and without temperature scaling; a paired test across the 50 splits would make the claim that temperature scaling has an inconsistent effect more rigorous.","section":"Tables 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for an applied ML journal. The main concern is the calibration-set reuse, which is fixable but requires substantial revision and likely re-running of experiments. The absence of RAPS hyperparameters and code availability should also be addressed. No concerns about citation integrity; the author's self-citations are relevant prior work. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a clean, honest empirical study of split-conformal methods on a seven-class subset of the ERA aerial dataset. The application is new, and the setup is sensible: three pretrained backbones, only 386 training samples, 50 random splits, and coverage/set-size reported side by side. The paper does a good job showing that a lightweight model like MobileNet matches ResNet in prediction-set quality, and the temperature-scaling observation—that T<1 sharpens and shrinks sets while T>1 softens and enlarges them—is a useful practical takeaway. I would trust the descriptive claims about set sizes and relative trends across architectures.\n\nThe soft spot is real and load-bearing. Temperature T is optimized by cross-entropy on Dcal, and then the same Dcal is used to compute nonconformity scores and the conformal quantile. That breaks the exchangeability assumption behind the split-conformal guarantee. The scores on Dcal are no longer distributed like scores on fresh test data, so the reported coverage values are not protected by the finite-sample theorem. The paper even notes the same calibration set is used, but it never addresses the consequence. This does not make the empirical results worthless—the coverage numbers are close to nominal and the qualitative conclusions likely hold—but the sentence “achieves the nominal coverage” is not justified as stated. A nested split or a sensitivity check would fix it.\n\nSmaller issues: RAPS hyperparameters (lambda and kreg) are never reported, so those results are not reproducible; the seven-class subset selection is described in terms of application areas but not documented as a selection procedure; and no code or seeds are provided, which is unfortunate for a 50-trial study. These are minor-to-moderate and fixable.\n\nThe central argument—that conformal prediction can produce useful prediction sets with scarce and messy data—holds up fine. The paper is aimed at practitioners deploying uncertainty estimates on drones or other constrained platforms, and it deserves a serious referee. I would send it to review with a request to address the calibration-data reuse and reporting gaps.\n\nRecommendation: accept for peer review, major revision.","headline":"Useful empirical benchmark of conformal prediction on a scarce aerial dataset, but the coverage guarantee is unproven because temperature scaling is fitted on the same calibration set used for the quantile.","tokens_in":14307,"tokens_out":1876,"would_cite":false,"duration_ms":20674,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Split-conformal prediction delivers calibrated, small prediction sets on scarce, unconstrained aerial imagery, and temperature scaling does not reliably shrink them.","keywords":["conformal prediction","prediction sets","aerial image classification","temperature scaling","uncertainty quantification","transfer learning","event recognition","coverage guarantee"],"falsifier":"Rerun the 50-trial MobileNet/LAC pipeline but split $D_{\\mathrm{cal}}$ into two disjoint parts, fitting the temperature on one part and computing the conformal quantile on the other, then compare empirical coverage with the reported numbers. If coverage falls below the nominal $1-\\alpha$, the reported validity depends on tuning and thresholding on the same calibration set; if coverage holds with similarly small sets, the double use is harmless.","tokens_in":13371,"feed_emoji":"🛰️","tokens_out":11573,"duration_ms":96999,"temperature":0.7,"pith_summary":"This paper asks whether conformal prediction—a post-hoc procedure that turns a classifier's scores into a set of labels guaranteed to contain the true label with probability at least $1-\\alpha$—still delivers on that promise when training data are scarce and images are visually messy. Using a seven-class subset of an aerial event-recognition dataset, with only 386 training and 261 calibration images per trial, it tests three nonconformity scores (LAC, APS, and RAPS) on small fine-tuned pretrained classifiers. The paper's central finding is that even the lightest model and the simplest score produce informative prediction sets: LAC on MobileNet attains the nominal 80% or 90% coverage with median set sizes around 1.1 and 1.7, respectively, while APS and RAPS over-cover with sets of two to three labels. A second finding is that temperature scaling does not consistently improve efficiency—it shrinks sets when the fitted temperature is below one and enlarges them when it is above one, as happened with ResNet.","feed_headline":"Conformal prediction holds up on scarce aerial imagery","feed_subtitle":"Simple scores meet 90% coverage with sets near size two; temperature scaling doesn't shrink them reliably.","key_machinery":"The carrying mechanism is split-conformal prediction built on a nonconformity score. On a calibration set $D_{\\mathrm{cal}}$, one computes scores $s(x_i,y_i)$, takes $\\hat{q}$ as the $\\lceil(n+1)(1-\\alpha)\\rceil/n$ empirical quantile of those scores, and outputs $C(x)=\\{y\\in\\mathcal{Y}:s(x,y)\\le\\hat{q}\\}$. Three scores are compared: LAC uses $1-\\pi_y(x)$, the softmax probability of the true class; APS accumulates sorted softmax probabilities up to the true class's rank; RAPS adds a rank penalty $\\lambda(L_y-k_{\\mathrm{reg}})_+$ to that cumulative sum. Temperature scaling enters by replacing logits $z_c(x)$ with $z_c(x)/T$ before the softmax. The paper's empirical point is that the direction of the temperature adjustment matters: $T<1$ sharpens the distribution and shrinks prediction sets, while $T>1$ softens it and enlarges them, which is why temperature scaling helped MobileNet and DenseNet but hurt ResNet.","core_discovery":"The paper claims that split-conformal prediction, applied post hoc to a fine-tuned pretrained classifier, gives statistically valid and practically informative prediction sets on a seven-class subset of the ERA aerial event dataset, even though only a few hundred labeled images are available and the scenes are highly variable. In 50 repeated trials, the LAC score on a lightweight MobileNet attained the nominal 0.8 and 0.9 coverage levels with average prediction-set sizes of roughly 1.1 and 1.7, while the APS and RAPS scores systematically exceeded the nominal coverage, reaching about 0.90 at $\\alpha=0.2$ and 0.95 at $\\alpha=0.1$, with larger sets between roughly 1.7 and 2.8. The paper also contends that temperature scaling is not a reliable way to reduce set size: optimizing $T$ on the calibration set shrank sets for MobileNet and DenseNet, whose fitted temperatures were below one, and enlarged them for ResNet, whose fitted temperatures were above one. The conclusion is that in this scarce-data regime, simple scores and compact models are sufficient for useful uncertainty quantification, and that calibration choices rather than backbone capacity drive the trade-off between coverage and set size.","pith_inferences":["A testable extension the paper does not run: split $D_{\\mathrm{cal}}$ so that $T$ is fitted on one half and the conformal quantile on the other, separating the effect of temperature scaling from the finite-sample coverage guarantee.","Because the paper finds that $T<1$ sharpens scores and shrinks sets while $T>1$ softens and enlarges them, one could optimize $T$ directly for expected set size subject to coverage instead of for calibration cross-entropy and measure whether the efficiency gain persists.","The authors' suggested direction on noisy labels could be quantified by injecting label noise into the calibration set at increasing rates and tracking coverage and set size for LAC versus APS and RAPS, since the adaptive scores rely differently on rank information."],"forward_implications":["A lightweight classifier and the LAC score can deliver prediction sets with average size near 1.1 at 80% coverage and near 1.7 at 90% coverage on this seven-class task, so single-label deployment can be augmented with a small uncertainty set at little compute cost.","Temperature scaling should be evaluated per architecture: it reduces set sizes when the fitted $T$ is below one (MobileNet, DenseNet) and increases them when $T$ is above one (ResNet), so it is not a universally beneficial calibration step.","Because APS and RAPS systematically over-cover while LAC tracks the nominal level, users who must not miss the true label may prefer the adaptive scores even at the price of larger sets.","The coverage and set-size results are similar across classifiers that differ by an order of magnitude in parameters, indicating that the calibration stage, not the backbone, governs the reliability-efficiency trade-off in this regime."],"supporting_citations":[{"why":"Supplies the seven-class subset of the Event Recognition in Aerial Videos dataset used in all experiments.","marker":"[24]"},{"why":"Defines the LAC nonconformity score that produces the smallest prediction sets.","marker":"[32]"},{"why":"Defines the APS score whose cumulative-probability construction yields over-coverage and larger sets.","marker":"[31]"},{"why":"Defines the RAPS score with rank-based regularization, the third method evaluated.","marker":"[3]"},{"why":"Introduces temperature scaling, the post-hoc calibration step whose effect on set size the paper measures.","marker":"[16]"},{"why":"Supplies the conformal prediction framework and the construction of prediction sets from a calibrated quantile.","marker":"[2]"},{"why":"Supplies the empirical quantile formula used to set the conformal threshold at the desired coverage level.","marker":"[42]"},{"why":"Defines MobileNetV2, the lightweight classifier used in the primary set of experiments.","marker":"[33]"},{"why":"Defines DenseNet-121, one of the three pretrained backbones compared.","marker":"[4]"},{"why":"Defines ResNet-152, the backbone whose fitted temperature above one enlarged the APS and RAPS sets.","marker":"[44]"}],"fun_headline_variants":["Conformal prediction shines on scarce aerial data","Tiny data, tiny sets: conformal prediction delivers","Temperature scaling fails to shrink aerial prediction sets","Simple scores beat scaling in scarce aerial classification","Conformal sets hit coverage on challenging aerial scenes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that tuning the temperature parameter on the calibration set and then computing the conformal quantile from that same calibration set preserves the exchangeability on which split-conformal coverage guarantees rest.","fun_headline_variants_meta":{"raw":{"variants":["Conformal prediction shines on scarce aerial data","Tiny data, tiny sets: conformal prediction delivers","Temperature scaling fails to shrink aerial prediction sets","Simple scores beat scaling in scarce aerial classification","Conformal sets hit coverage on challenging aerial scenes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1516,"prompt_tokens":1067,"completion_tokens":449,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":379}},"tokens_in":683,"tokens_out":449,"duration_ms":4329,"temperature":1.0,"reasoning_tokens":379,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:34:11.678660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the 50-trial MobileNet/LAC pipeline but split $D_{\\mathrm{cal}}$ into two disjoint parts, fitting the temperature on one part and computing the conformal quantile on the other, then compare empirical coverage with the reported numbers. If coverage falls below the nominal $1-\\alpha$, the reported validity depends on tuning and thresholding on the same calibration set; if coverage holds with similarly small sets, the double use is harmless.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the seven-class subset of the Event Recognition in Aerial Videos dataset used in all experiments."},{"cited_title":"Sadinle, J","cited_arxiv_id":null,"evidence_quote":"Defines the LAC nonconformity score that produces the smallest prediction sets."},{"cited_title":"Romano, M","cited_arxiv_id":null,"evidence_quote":"Defines the APS score whose cumulative-probability construction yields over-coverage and larger sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces temperature scaling, the post-hoc calibration step whose effect on set size the paper measures."},{"cited_title":"Angelopoulos and S","cited_arxiv_id":null,"evidence_quote":"Supplies the conformal prediction framework and the construction of prediction sets from a calibrated quantile."},{"cited_title":"Tibshirani","cited_arxiv_id":null,"evidence_quote":"Supplies the empirical quantile formula used to set the conformal threshold at the desired coverage level."},{"cited_title":"Sandler, A","cited_arxiv_id":null,"evidence_quote":"Defines MobileNetV2, the lightweight classifier used in the primary set of experiments."},{"cited_title":"Arulananth, S","cited_arxiv_id":null,"evidence_quote":"Defines DenseNet-121, one of the three pretrained backbones compared."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ResNet-152, the backbone whose fitted temperature above one enlarged the APS and RAPS sets."}],"review_version":1}