{"id":"91184846-743f-49a8-9608-5a3ec7ce05d9","arxiv_id":"2412.11801","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"For Gravity Spy glitch classification, the optimal conformal prediction nonconformity measure depends on the chosen metric: baseline wins for F1 and average set size, maxscore2 wins for singleton count.","lead":"Conformal prediction adds a mathematically guaranteed uncertainty estimate to machine learning outputs, and this study applies it to the Gravity Spy classifier that labels noise glitches in gravitational wave detectors. The authors show that the best nonconformity measure (the function converting raw scores into prediction sets) depends on which performance metric matters: small prediction sets, correct unique labels, or balanced F1.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical ranking of nonconformity measures rests on the Gravity Spy final_label, which is not an independent ground truth; because final_label includes the ML classifier's own output, this circularity could change which NCM is optimal.","rationale":"The reader's verdict is CONDITIONAL and identifies the same ground-truth concern; I agree that this is the principal threat. The paper is a well-structured proof of concept: separate optimisation and evaluation sets, five repeated splits, and an honest discussion of the label problem. The metric-dependence part of the central claim is robust — Table II shows a singleton-count advantage for maxscore2 over baseline (0.271 vs 0.179) that is many standard errors from zero, while F1 and average set size effectively tie. The application-dependence part is supported by per-class results (Fig. 10) and by comparison with prior work. However, the label circularity is deeper than ordinary label noise: because final_label includes the ML classifier's own predictions, the evaluation can reward an NCM that happens to align with the label-generation process. This does not invalidate the general claim that NCM choice is metric- and problem-dependent, but it does mean the specific 'best' NCMs reported for Gravity Spy are not established unless the results are shown to be stable under a more independent label. The lack of released code for the optimisation section (Section III) compounds this by preventing independent verification, but the label issue is the more substantive scientific concern. A re-analysis with citizen-only labels is the cleanest way to settle it. No other assumption in the paper appears to threaten the central claim: the conformal formalism is applied correctly and the evaluation protocol is sound, so the verdict should remain CONDITIONAL.","tokens_in":19250,"tokens_out":17409,"duration_ms":160159,"concrete_test":"Re-run the optimisation and evaluation of Table II using an alternative truth label constructed solely from citizen-scientist votes (e.g., majority vote of volunteer classifications) for the subset of glitches where this label is unambiguous (say, at least three agreeing volunteers). If the ranking of NCMs changes — especially if maxscore2 no longer maximises singletons — the central claim's empirical illustration is not robust to the label definition. If that subset is too small for full calibration, use the already-calibrated quantiles and evaluate only on the high-confidence subset, or add controlled label noise to final_label to test the stability of the ranking.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that the optimal nonconformity measure (NCM) depends on the application and the chosen metric — is supported by comparisons of optimised NCMs on the Gravity Spy 'retired' dataset. However, the 'true label' used for calibration and evaluation is the final_label, defined in Section II.B.1 as a combination of citizen-scientist votes and the Gravity Spy ML classifier's own classifications. The ML algorithm that produced that label is the same one whose softmax scores are being recalibrated by conformal prediction. Consequently, the evaluation is not anchored to an independent physical class: a confident-but-wrong ML prediction can be propagated into the final_label, and CP then calibrates on precisely those errors. The authors acknowledge only label inaccuracy ('Our true label might thus not always be accurate'), not this circularity, and they do not quantify its effect on the relative ranking of NCMs. In particular, the large singleton-count advantage of maxscore2 over the baseline (0.271 vs 0.179 in Table II) could be partly an artifact of how final_label is generated rather than a genuine property of the underlying glitch classes. Since the paper's conclusion is about which NCM is optimal, this non-independence of the labels is the most load-bearing unquantified threat to the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies split conformal prediction (CP) to the Gravity Spy glitch-classification CNN. It introduces a family of parameterized nonconformity measures (baseline, softmax, entropy, entropy softmax, margin2, maxscore2, CNN, Brier), optimizes their parameters on a dedicated optimization split under three metrics (macro-F1, singleton rate, average set size), and evaluates on a held-out split. The main finding is that the optimal nonconformity measure depends on the metric: the baseline is best or tied under F1 and average set size, whereas maxscore2 yields the highest singleton count, with some per-class exceptions. The validity guarantee of Eq. (1) is verified empirically.","tokens_in":19477,"tokens_out":6137,"duration_ms":56350,"significance":"The paper is a useful proof of concept for applying CP to a real scientific ML pipeline, and the optimization protocol (separate optimization/evaluation sets, five repeats, public code) is a methodological strength. The central qualitative claim that optimal NCM choice is application- and metric-dependent is plausible and consistent with the literature. However, because the 'true label' is the Gravity Spy final_label, which incorporates the same ML classifier's output, the quantitative ranking of NCMs is not anchored to an independent physical classification; this limits the astrophysical conclusions and requires a robustness check or a substantially stronger caveat.","major_comments":[{"comment":"The calibration and evaluation labels are the Gravity Spy final_label, which is a combination of citizen-scientist votes and the ML classifier's classifications, and the ML classifier is the same one whose softmax scores are transformed by the conformal procedure. The paper acknowledges that this true label may be inaccurate but does not address the circularity: a confidently wrong ML prediction can propagate into final_label, and CP then calibrates on exactly those errors. Equation (1)'s coverage guarantee is with respect to these labels, not to the physical glitch class. The claimed singleton advantage of maxscore2 over baseline (0.271 vs 0.179 in Table II) could be partly an artifact of how final_label is generated. Please add a robustness check (e.g., restrict to glitches with high citizen-scientist consensus, or remove the ML contribution from final_label) or, at minimum, state explicitly that all results are conditional on final_label and cannot be read as optimality for the underlying physical classes.","section":"II.B.1 and Table II"},{"comment":"The statistical significance of the NCM comparison is not established. The evaluation scores in Table II are computed on a single fixed evaluation set, and the quoted uncertainties reflect only the variation of the optimized parameters over the five splits of the optimization set; the finite-sample variance of the evaluation set is not included. As a result, small differences such as F1 = 0.456 (baseline) vs 0.455 (margin2, softmax) cannot be distinguished from noise, and the statement that the baseline is best under F1 needs either an estimate of evaluation-set uncertainty or a paired significance test across the five repeats. This is important for the central claim that the optimum depends on the metric.","section":"III.C-III.D and Table II"}],"minor_comments":[{"comment":"In the Discussion, 'average set seize' should be 'average set size'.","section":"Section IV"},{"comment":"The phrase 'it becomes visibly discreet' should read 'visibly discrete'.","section":"Section III.C"},{"comment":"The word 'adaptions' should be 'adaptations'.","section":"Section III.A"},{"comment":"The caption of Fig. 2 states that the y-axis is the predicted label, but the plotted y-values are the labels in the prediction set; please clarify the distinction.","section":"Section II.B.2"},{"comment":"Reference [34] contains 'et al.' in the middle of the title, which is a formatting error.","section":"References"},{"comment":"The text says 'The code for Section II is openly available', but it is unclear whether the optimization code for Section III is also released; please clarify which parts of the analysis are reproducible.","section":"Section II and III"},{"comment":"The notation for the classification-score vector f and its component f_y is used in the table but only implicitly defined; please define both in the table caption or the main text.","section":"Table I"},{"comment":"The ROC comparison between CP and Gravity Spy sweeps different quantities (alpha for CP, classification threshold for Gravity Spy); please state this explicitly in the caption to avoid an unfair comparison.","section":"Fig. 5"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and should be of interest to the gravitational-wave data-analysis community. The conformal-prediction machinery itself is correctly imported, and the optimization/evaluation split is careful. The main obstacle is the label circularity: final_label includes the same ML classifier's output that CP recalibrates. If the authors can add a robustness check with more independent labels, or explicitly reframe the conclusions as conditional on final_label, the paper would be publishable. I do not see circularity in the derivation of the CP guarantees themselves."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, honest application paper. It is the first to apply conformal prediction to the multi-class Gravity Spy classifier, it introduces a new maxscore2 nonconformity measure, and it parameterises several published measures to optimise them under three metrics. The evaluation protocol is a real strength: separate optimisation and evaluation splits, five repeats for parameter uncertainty, and an honest discussion of degeneracies and regularisation. The coverage plots match the stated validity guarantee, and the paper correctly reports that for F1 and average set size most optimised measures reduce to the simple baseline, while maxscore2 gives the best singleton count. The central claim—that the optimal nonconformity measure depends on the metric and the dataset—is supported for this study.\n\nThe soft spot is the ground truth. The 'true label' is Gravity Spy's final_label, which combines citizen-scientist votes with the ML classifier's own output. That means the calibration and evaluation are not anchored to an independent physical class. The authors note that the label may be inaccurate, but they do not address the stronger point: the label is partly produced by the same algorithm whose softmax scores are being recalibrated. This dependence could change the ranking of nonconformity measures, especially the singleton advantage of maxscore2. It is not fatal, because the paper is explicitly a proof of concept and the absolute differences between measures are small, but it should be quantified or at least stated as non-independence rather than just noise. A sensitivity check using a subset with human-only labels, or treating final_label as a noisy proxy, would answer the question. Also, only the Section II code is on Zenodo; the optimisation code is not released, which limits reproducibility of the main comparison.\n\nOverall: worth a serious referee. The authors know the limits and the methodology is clean enough that the conclusions hold as a demonstration. I would accept it with a request to make the label-dependence explicit and release the optimisation code. For a reader who wants to deploy CP on Gravity Spy as it exists, this is a useful recipe; for claims about underlying glitch physics, treat the ranking with caution.","headline":"A clean, honest proof-of-concept for conformal prediction on Gravity Spy; the empirical nonconformity-measure ranking is real within the study, but the label it calibrates against is partly produced by the very classifier being recalibrated.","tokens_in":20046,"tokens_out":3918,"would_cite":true,"duration_ms":38074,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":["04.80.Nn","07.05.Mh"],"model":"deepseek-v4-flash","headline":"Conformal prediction's optimal uncertainty score for glitch classification is not fixed but depends on the metric being optimised.","keywords":["conformal prediction","nonconformity measure","Gravity Spy","glitch classification","gravitational waves","classification uncertainty","machine learning","prediction sets"],"falsifier":"Re-run the same optimisation on a glitch sample whose labels have been independently vetted by human experts (or on the 'golden' images) and check whether the baseline still ties for F1 and average set size and maxscore2 still maximises singletons; a change in the ranking would show the result is an artefact of using the final_label as ground truth. Alternatively, repeat the analysis with the improved multi-view Gravity Spy CNN and see if the metric-dependent ranking persists.","tokens_in":1713,"feed_emoji":"🌌","tokens_out":3078,"duration_ms":88419,"temperature":0.7,"pith_summary":"Conformal prediction wraps any classifier's point predictions in a prediction set with a guaranteed coverage probability, but it requires a chosen score function (the nonconformity measure) to do so. This paper asks which nonconformity measure is best for classifying transient noise artefacts (glitches) in gravitational-wave data with the Gravity Spy classifier, and shows that the answer depends on how 'best' is measured. On the full dataset, the standard baseline measure—one minus the classifier's probability for a label—is optimal or tied for optimal under the F1 score and the average prediction-set size, while a new measure the authors call maxscore2 maximises the number of uniquely classified glitches (singletons). For several individual glitch classes, other measures beat the baseline, so the paper concludes that the optimal nonconformity measure is specific to both the application and the metric of interest. The result is practical: choosing a score function is a decision that should be matched to the downstream use of the uncertainty estimates.","feed_headline":"Best glitch uncertainty score is metric-dependent","feed_subtitle":"On Gravity Spy data, the simple baseline wins for F1 and set size; maxscore2 wins on singletons.","key_machinery":"The central object is the nonconformity measure $A(x,y)$, a score function that converts the classifier's heuristic probabilities into a rigorous uncertainty via a calibration quantile. The paper uses Mondrian (label-conditional) conformal prediction, where a separate quantile $\\hat{q}_y$ is computed for each glitch class, guaranteeing coverage per class as well as marginally. Eight parameterised families of nonconformity measures are compared: baseline, softmax, entropy, entropy softmax, margin2, maxscore2 (introduced here), CNN-based, and Brier; parameters are optimised by grid search plus L-BFGS-B, with regularisation to break degeneracy along axes. The machinery does the work of turning the question 'which uncertainty score should we use?' into a well-defined optimisation problem that can be solved separately for each performance metric.","core_discovery":"The central discovery is an empirical answer to an optimisation question in conformal prediction. The authors parameterise eight families of nonconformity measures, optimise their parameters separately for three metrics (F1 score, average prediction-set size, and singleton count) on the Gravity Spy 'retired' dataset, and find that the baseline measure $A(x,y)=1-f_y(x)$ is optimal or comparable under F1 and average set size, whereas all optimised alternatives produce more singletons, with maxscore2 best. This shows that the optimal nonconformity measure is not intrinsic to the algorithm or dataset but is relative to the metric of interest; class-level results confirm this by showing cases where the baseline is not the best. The paper therefore demonstrates both a concrete application of conformal prediction to multi-class glitch classification and a general methodology for optimising nonconformity measures.","pith_inferences":["An implication the authors leave implicit is that the near-tie between F1 and average set size means these two metrics are largely redundant; a user should choose the one that matches the downstream decision rather than optimising both.","Because the calibration labels are a fusion of citizen and machine classifications, not ground truth, the coverage guarantee is conditional on those labels; extending calibration to propagate label noise (e.g., confidence-weighted labels) is a natural next test that could alter the observed rankings.","A practical extension would be to rerun this optimisation on the improved multi-view Gravity Spy classifier to see whether the metric-dependent ranking persists across model generations; if it changes, the optimal measure is model-dependent as well as metric-dependent."],"forward_implications":["A practitioner using Gravity Spy who cares about F1 or small prediction sets can safely default to the baseline nonconformity measure; no optimised alternative beats it on the full dataset.","A practitioner who values unique, high-purity classifications should choose the maxscore2 measure, which returns the highest singleton count at alpha=0.1 and across error rates.","Because the rankings change for individual glitch classes, per-class optimisation or mixing of nonconformity measures could improve class-specific F1 scores by up to about 0.2 over the baseline.","Conformal prediction converts the CNN's scores into guaranteed-coverage prediction sets; for example, with alpha=0.1 a set of Tomte glitches can be built where each glitch is in the set with 90% certainty relative to the chosen labels.","The optimisation recipe transfers to any point-prediction algorithm, but the optimised parameter values themselves do not transfer to other datasets or classifiers."],"supporting_citations":[{"why":"Supplies the conformal-prediction validity statement and quantile construction that the paper's coverage guarantee rests on.","marker":"[25]"},{"why":"Defines the nonconformity measure framework and the monotonic-invariance property used to compare measures.","marker":"[26]"},{"why":"Introduces the Gravity Spy citizen-science and CNN classifier whose scores are wrapped by conformal prediction.","marker":"[20]"},{"why":"Provides the 'retired' dataset with ml_label, classification scores, and final_label used for calibration and evaluation.","marker":"[34]"},{"why":"Supplies the CNN nonconformity measure family and its parameterisation, one of the optimised candidates.","marker":"[28]"},{"why":"Gives the earlier comparison of baseline and Brier nonconformity measures under different metrics that this work extends.","marker":"[29]"},{"why":"The prior application of conformal prediction to gravitational-wave search pipelines that this multi-class, optimisation-focused work builds on.","marker":"[27]"},{"why":"Documents the imperfect calibration of the Gravity Spy CNN probabilities, motivating the external calibration that conformal prediction provides.","marker":"[22]"}],"fun_headline_variants":["Best glitch uncertainty score depends on metric","Conformal prediction: baseline wins for F1, maxscore2 for singletons","For Gravity Spy, choose your uncertainty metric first","Glitch classification: optimal uncertainty measure is metric-specific"],"cache_read_input_tokens":22144,"weakest_assumption_plain":"The paper treats the Gravity Spy 'final_label'—a blend of volunteer and computer classifications—as the true glitch class; if that label is wrong for a glitch, the coverage guarantee does not protect the actual physical class.","fun_headline_variants_meta":{"raw":{"variants":["Best glitch uncertainty score depends on metric","Conformal prediction: baseline wins for F1, maxscore2 for singletons","For Gravity Spy, choose your uncertainty metric first","Glitch classification: optimal uncertainty measure is metric-specific"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000688,"raw_usage":{"total_tokens":3111,"prompt_tokens":934,"completion_tokens":2177,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":2111}},"tokens_in":550,"tokens_out":2177,"duration_ms":17504,"temperature":1.0,"reasoning_tokens":2111,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:33:40.494383+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same optimisation on a glitch sample whose labels have been independently vetted by human experts (or on the 'golden' images) and check whether the baseline still ties for F1 and average set size and maxscore2 still maximises singletons; a change in the ranking would show the result is an artefact of using the final_label as ground truth. Alternatively, repeat the analysis with the improved multi-view Gravity Spy CNN and see if the metric-dependent ranking persists.","supporting_citations":[{"cited_title":"Model-agnostic nonconformity functions for conformal classification","cited_arxiv_id":null,"evidence_quote":"Gives the earlier comparison of baseline and Brier nonconformity measures under different metrics that this work extends."},{"cited_title":"Classification uncertainty for transient gravitational-wave noise artefacts with optimised conformal prediction","cited_arxiv_id":"2412.11801","evidence_quote":"Documents the imperfect calibration of the Gravity Spy CNN probabilities, motivating the external calibration that conformal prediction provides."}],"review_version":1}