{"id":"0ff6cbfe-1e5d-479f-9b67-4105cc7cc6b9","arxiv_id":"2508.12085","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Unified conformal multiple testing framework that leverages full data for improved power and FDR control with automatic procedure selection.","lead":"The paper proposes a unified framework for conformalized multiple testing that uses all available data including null, alternative, and unlabelled samples to build non-conformity scores and calibrate p-values via full permutation. This is intended to boost power while controlling false discovery rate and to enable automatic selection of the best procedure without extra data splits.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Full-permutation FDR control may fail when non-conformity scores are fit on combined null+alternative data","rationale":"The reader's weakest assumption directly identifies the same exchangeability/permutation validity issue that is load-bearing for the 'uses all data while rigorously controlling FDR' claim. Because the original verdict was formed from the abstract alone, confirming or refuting the assumption in the full proof and experiments would move the paper from UNVERDICTED to CONDITIONAL (valid under stated conditions) or lower if the bound does not hold.","tokens_in":1614,"tokens_out":380,"duration_ms":26998,"concrete_test":"Locate the main FDR theorem; extract the exact exchangeability or permutation-invariance assumption it invokes. Then re-run the paper's primary simulation setting (e.g., the Gaussian or mixture model with 20% alternatives) but replace the full-data score construction with a version that uses only null-labelled data for fitting; if the empirical FDR at nominal 0.05 rises above 0.07 while power drops, the full-data construction is responsible for the claimed gain and its validity must be re-proven.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that constructing scores from the entire dataset (null, alternative, and unlabelled) and then applying a full permutation strategy still yields valid p-values that control FDR. Standard conformal guarantees rely on exchangeability between the test point and calibration points under the null. When the score function itself is trained on data that includes alternatives, this symmetry can be broken for true-null hypotheses because the fitted score now depends on the presence and strength of alternatives. The abstract asserts rigorous control, but the weakest link is whether the proof (presumably in the methods or theorem section) explicitly restores exchangeability or derives a bound that accounts for this dependence.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes a unified framework for conformalized multiple testing that uses all available data (null, alternative, and unlabelled) to construct non-conformity scores and calibrate p-values via a full permutation strategy. This is claimed to improve power through better score quality and larger calibration sets while rigorously controlling the false discovery rate, and to enable automatic selection of the optimal conformal procedure among candidates without additional data splitting. Numerical experiments are presented to demonstrate superior efficiency and adaptability.","tokens_in":1733,"tokens_out":346,"duration_ms":27798,"significance":"If the FDR control is rigorously established, the framework could meaningfully advance conformal multiple testing by addressing data efficiency, a common practical limitation in existing methods. The systematic design principle and automatic selection feature would be valuable contributions, particularly if the approach generalizes across diverse scenarios as suggested by the experiments.","major_comments":[{"comment":"The central FDR control claim depends on the full permutation strategy remaining valid when non-conformity scores are constructed from the combined dataset including alternatives. The proof (in the theoretical guarantees section) must explicitly show how exchangeability or a suitable bound is preserved for true-null hypotheses despite the score function's dependence on alternative data; without this, the symmetry required for valid p-values may not hold.","section":null}],"minor_comments":[{"comment":"The abstract and introduction could more explicitly reference the specific prior conformal multiple testing methods being unified, to better situate the contribution.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to fit the scope of a methodology journal in statistics, but the citation list should be checked for completeness regarding foundational conformal prediction and FDR-controlling permutation tests."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback and for recognizing the potential significance of the unified conformalized multiple testing framework. We address the major comment below in detail.","responses":[{"response":"We thank the referee for this precise observation on the theoretical foundation. We agree that greater explicitness is warranted to clarify how exchangeability is preserved for true-null hypotheses when the score function depends on the full dataset (including alternatives). In the revised manuscript we will expand the theoretical guarantees section with a new dedicated paragraph and supporting lemma. The argument proceeds by conditioning on the alternative observations (which are fixed) and showing that the full-permutation distribution over the remaining indices still induces uniform ranks for the true-null p-values, because the score function is applied symmetrically across all permuted assignments. This establishes the required marginal validity without relying on full joint exchangeability of the entire sample. We believe this addition will fully resolve the concern while leaving the main results unchanged.","revision_made":"yes","referee_comment":"The central FDR control claim depends on the full permutation strategy remaining valid when non-conformity scores are constructed from the combined dataset including alternatives. The proof (in the theoretical guarantees section) must explicitly show how exchangeability or a suitable bound is preserved for true-null hypotheses despite the score function's dependence on alternative data; without this, the symmetry required for valid p-values may not hold."}],"tokens_in":1179,"tokens_out":303,"duration_ms":97245,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is a framework that uses every observation—nulls, alternatives, and unlabelled points—to build non-conformity scores and then calibrates p-values with a full permutation scheme. The goal is tighter FDR control plus higher power by avoiding the usual data splits that shrink either the score or the calibration set.","headline":"The paper unifies conformal multiple testing around full-data scoring and permutation calibration for better power, but the FDR validity when scores are fit on mixed null-alternative data is the part that needs checking.","tokens_in":2197,"tokens_out":148,"would_cite":false,"duration_ms":26744,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A unified framework uses all available data to raise power in conformalized multiple testing while controlling the false discovery rate.","keywords":["conformalized multiple testing","false discovery rate","full permutation","data efficiency","non-conformity scores","unlabelled data","power improvement"],"falsifier":"Generate data that deliberately violates exchangeability, such as a sequence with a clear time trend, run the procedure at a nominal false-discovery-rate level of 0.05, and check whether the realized proportion of false discoveries exceeds 0.05 by a statistically significant margin.","tokens_in":2513,"feed_emoji":"📊","tokens_out":646,"duration_ms":29685,"temperature":0.7,"pith_summary":"The paper presents a single framework that folds null, alternative, and unlabelled observations together when building non-conformity scores and when calibrating p-values. It replaces partial-data splits with a full permutation strategy that keeps every observation in play for both steps. This produces stronger scores and a larger effective calibration set, which in turn lifts detection power while still guaranteeing false-discovery-rate control. The same construction supplies a rule for picking the strongest procedure from a list of candidates without setting aside extra data.","feed_headline":"Full-data conformal method lifts power in multiple testing","feed_subtitle":"Pooling null, alternative and unlabelled points for scores and calibration raises discoveries while holding false discoveries at the target.","key_machinery":"the full permutation strategy applied to non-conformity scores built from the pooled dataset, which simultaneously improves score quality and enlarges the calibration sample.","core_discovery":"By constructing non-conformity scores from the entire collection of null, alternative, and unlabelled points and then calibrating p-values through a full permutation of the combined scores, the procedure controls the false discovery rate at the target level and simultaneously increases the number of discoveries that can be made.","pith_inferences":["The same full-permutation logic could be tested on streaming or sequentially arriving data to see whether power remains stable as the sample grows.","Replacing the hand-crafted scores with scores learned from a neural network trained on the pooled data might further enlarge the power advantage.","The automatic selection rule inside the framework could be applied to other conformal tasks such as constructing prediction sets rather than testing."],"forward_implications":["The same dataset yields more true discoveries at any fixed false-discovery-rate target.","A practitioner can compare several candidate score functions on the full data and retain the one that produces the largest number of rejections without extra splitting.","The method applies directly to mixtures of labelled and unlabelled observations without requiring separate calibration samples.","Power gains appear across both low- and high-dimensional settings provided the exchangeability condition on the scores is preserved."],"fun_headline_variants":["Conformal testing using all data for calibration","Full data permutation for conformal p values","Unified conformalized testing with full data","All available data in conformal multiple testing","Full data efficiency for conformal multiple testing"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The full permutation of scores built from the combined data is assumed to deliver valid false-discovery-rate control, which holds only when the observations satisfy exchangeability or the chosen scores obey the required symmetry properties.","fun_headline_variants_meta":{"raw":{"variants":["Conformal testing using all data for calibration","Full data permutation for conformal p values","Unified conformalized testing with full data","All available data in conformal multiple testing","Full data efficiency for conformal multiple testing"]},"model":"grok-4.3","cost_usd":0.00924,"raw_usage":{"total_tokens":3999,"prompt_tokens":552,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":92403000,"prompt_tokens_details":{"text_tokens":552,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3387,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":552,"tokens_out":60,"duration_ms":38303,"temperature":1.0,"reasoning_tokens":3387,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-22T12:56:08.412620+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Generate data that deliberately violates exchangeability, such as a sequence with a clear time trend, run the procedure at a nominal false-discovery-rate level of 0.05, and check whether the realized proportion of false discoveries exceeds 0.05 by a statistically significant margin.","supporting_citations":[],"review_version":1}