{"id":"93da7cb6-25bf-4714-abc9-b1192579d2b6","arxiv_id":"2608.02063","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A primary-preserving retain-or-correct policy, trained from out-of-fold residual utility, improves modulation classification accuracy on RadioML and Hisar benchmarks by 1.0 to 2.7 percentage points.","lead":"This paper adds a decision layer on top of an automatic modulation classifier: it learns when to keep the default prediction and when to switch to one of several auxiliary predictors, using out-of-fold training and a validation-frozen policy. On three standard radio-signal benchmarks, the combined system raises accuracy by 1 to 3 percentage points, with positive paired gains under all tested channel perturbations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retraining-seed sensitivity is the central unresolved threat: all gains come from one training run, so the causal claim that the complete policy drives improvement is not yet established.","rationale":"I read the central claim as a causal statement about the decision-layer design. To establish it, the observed improvement must be attributable to the design rather than to a lucky draw of model training. The protocol controls for many confounds: same OOF records, same validation budget, same held-out sample order. But it does not control for the random initialization and training dynamics of the neural models. A single run is an n=1 experiment over training randomness. The paired significance tests are over samples, not over runs, so they cannot certify that the complete policy beats the isolated objective reliably. The paper's own Section VIII admits this. In contrast, the reader's chosen weakest assumption (OOF estimator calibration under full-training deployment) is substantially mitigated by the validation-selection step: because the validation split is scored using full-training model outputs, threshold selection can compensate for systematic OOF/full shifts; and the frozen-policy stress test provides direct empirical evidence that the deployed system works under distribution shift. Hence the seed-sensitivity gap is more load-bearing. The proposed retraining-seed study would settle it; without it, the verdict should remain CONDITIONAL, requiring the additional evidence.","tokens_in":15958,"tokens_out":8809,"duration_ms":147089,"concrete_test":"Retrain the full pipeline with 5 different random seeds for all neural models (primary and candidates), keeping the candidate pool and validation/holdout splits fixed. For each seed, re-run OOF utility estimation, validation policy selection, and held-out evaluation. Report the seed-level distribution of (i) complete-system gain over primary and (ii) Full ERU-RC minus Isolated ERU. If the 95% intervals (over seeds) for (i) exclude zero and (ii) is positive in at least 4/5 seeds, the causal claim holds; otherwise, the observed consistency is seed-luck and the verdict should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is causal: the complete residual-evidence and validation-frozen action-policy design, not the isolated utility objective, produces the consistent held-out gains (2.700/1.008/2.098 pp on RML2016.10A/B and Hisar). But every number in Tables II-V and the stress test comes from a single training run of the neural base models (KAN-Fourier, IQCC-Former, long-spectral-lite). The paired bootstrap/McNemar tests condition on frozen prediction records and quantify sampling variation over held-out samples only; they cannot capture variation over random seeds and training stochasticity. If the single primary seed happened to leave unusually correctable errors, or the candidates happened to be unusually complementary, the measured gains and even the ranking Full ERU-RC > Isolated ERU might not replicate. Section VIII explicitly lists the absence of a controlled retraining-seed study as a limitation but does not bound its impact. Since the paper's headline conclusion is about design causality rather than a single artifact, this gap is load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post-inference decision-correction layer for automatic modulation classification. A structured KAN-Fourier classifier acts as the primary/trusted predictor; several heterogeneous neural and non-neural candidates provide probability evidence. Candidate-specific residual utility is learned from out-of-fold predictions (Eqs. 17-19), and a disjoint validation split is used to freeze action thresholds, approved transitions, blending coefficients, and a risk mask before held-out evaluation. The complete system is reported to improve primary accuracy by 2.700, 1.008, and 2.098 percentage points on RML2016.10A, RML2016.10B, and HisarMod2019.1, respectively. The paper argues that the consistent gain comes from the complete residual-evidence and validation-frozen action-policy design, not from the isolated residual-utility objective, and supports this with controlled decision-level baselines, paired bootstrap and Holm-corrected McNemar tests, action-family attribution, a partition sensitivity analysis, and an 11-condition frozen-policy stress test under CFO, I/Q imbalance, and synthetic fading.","tokens_in":16196,"tokens_out":4329,"duration_ms":43408,"significance":"The protocol is unusually careful for a decision-layer paper: OOF probabilities, disjoint validation for threshold selection, held-out evaluation, paired significance tests with family-wise correction, ablations that isolate the utility objective from the action policy, and an honest discussion of what is not established. The formal propositions are explicitly labeled as accounting identities or conditional optimality statements, not as generalization guarantees. If the results are reproducible, the paper makes a useful, appropriately scoped contribution to post-hoc decision correction in AMC and to the broader literature on safe model overriding. The main open threat is that all headline numbers come from a single training run, so the causal claim about design choice is not yet fully secured.","major_comments":[{"comment":"Retraining-seed sensitivity is load-bearing and not resolved. All accuracies, rankings, and stress-test gains are computed from prediction records produced by one training run of each neural model. The paired bootstrap and McNemar tests condition on those frozen records and therefore quantify only sampling variability over held-out samples; they do not capture variability over neural training seeds. The central claim is a causal comparison of decision designs, and several margins are small (e.g., Full ERU-RC vs. OOF linear stacking on RML2016.10B is -0.003 pp with CI crossing zero; vs. OOF XGBoost stacking on HisarMod2019.1 is +0.123 pp). Absent a multi-seed study or a bound on seed sensitivity, the ranking 'complete policy > isolated objective' could change under retraining. Section VIII lists this as a limitation but does not quantify its impact; since the headline conclusion depends o","section":"Section VI-B and Eqs. (17)-(19)"},{"comment":"The ERU estimators are trained on out-of-fold predictions produced by models trained on subsets of the training data, but at deployment they are applied to predictions from models trained on the full training set. This distributional transfer is assumed rather than established. The frozen-policy stress test perturbs the input channel, which is a useful but different check: it does not directly measure the OOF-to-full-training calibration gap. The paper should provide an empirical comparison between OOF and full-training prediction distributions on the validation split (e.g., calibration curves or a stability check of the frozen thresholds), or explicitly state this as an unverified assumption that could cause net harm under distribution shift. This issue is central to the leakage-control claim, so it needs a specific response.","section":"Section VI-B and Eqs. (17)-(19)"},{"comment":"The statistical significance analysis relies on a 'frozen analysis plan' with a pre-specified family of 15 cross-dataset comparisons, but the manuscript gives no evidence that this plan was fixed before held-out results were observed. The Holm correction is meaningful only if the comparison family is genuinely pre-specified. Since the paper emphasizes reproducibility and releases code and artifacts, it should include the analysis plan, comparison list, and timing evidence in the released package. Without this, the adjusted p-values should be treated as exploratory rather than confirmatory.","section":"Section IV, Eq. (12), and Section VII-D"}],"minor_comments":[{"comment":"Equation (8) defines U_cond without a factor of 100, but Table II reports conditional utility in percent. Please add the scaling explicitly or state that reported values are percentages.","section":"Eq. (8) and Table II"},{"comment":"The clean-control re-inference gives 77.898% for the Primary and 80.002% for Frozen RC, while the archived benchmark results are 77.769% and 79.867%. The explanation in terms of FP32 and CUDA nondeterminism is plausible, but the absolute differences of 0.129 and 0.135 pp are not negligible relative to some of the reported gains. Please state the number of repeat runs and the observed run-to-run variance, or mark the clean control as a single non-archived execution.","section":"Section VI-B"},{"comment":"The 'implementation-sensitivity check' reports three replacement backbones, but only one run each. The differences among 63.523%, 63.668%, and 63.632% are within a range that could easily be seed noise; the paper appropriately avoids component-level claims, but it would be helpful to state explicitly that these values are single-run and therefore only suggestive.","section":"Section VII-E"},{"comment":"The phrase 'unified risk mask' is described as including the former harm-guard logic, but the independent contribution of the mask is not separately measurable on two datasets. The text acknowledges this, but a reader may still find the term 'risk mask' deceptively modular. Consider renaming it 'validation-selected rejection rule' or adding a one-sentence explanation of why it is treated as part of the complete policy rather than an independent module.","section":"Section V-F"},{"comment":"The leave-one-storage-block sensitivity analysis is described as post-hoc, which is good. Please make clear in the figure caption or text that the five intervals are not corrected for multiple comparisons; they are descriptive stability checks.","section":"Section VII-H"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and technically careful, and the experimental protocol is above the norm for this area. The retraining-seed issue is the main gate: because the claimed design-level superiority rests on small margins from single runs, I would condition acceptance on a seed-sensitivity analysis or a suitably weakened claim. The transfer assumption from OOF to full-training predictions is also worth an explicit empirical check. The manuscript is otherwise publishable after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper is a careful, honestly delimited empirical study of a post-inference decision layer for AMC, and the protocol is the real contribution. The math is trivial (deliberately labeled accounting identities), but the OOF residual-utility target, the explicit retain action, and the validation-frozen action policy form a combination I have not seen in this subfield.\n\nWhat is genuinely good: train-split OOF predictions feed candidate-specific residual utility; a disjoint validation split freezes thresholds, approved transitions, and risk masks before held-out evaluation; and the action audit is mutually exclusive, which lets them show which modules do zero work. The baselines are fair—same probability pool, same validation budget—and they explicitly report the uncomfortable result that isolated ERU is not uniformly best and that linear stacking ties on RML2016.10B. Code and frozen artifacts are promised, which matters. The 11-condition frozen-policy stress test is useful supporting evidence.\n\nSoft spots, in proportion. The main one is exactly what Section VIII names: every headline number comes from a single training run of the neural base models. Paired bootstrap and McNemar condition on frozen prediction records; they cannot speak to seed sensitivity. That makes the central causal claim—complete policy, not isolated objective—stronger than the evidence. This is not a fatal flaw, but it is load-bearing, and a retraining-seed study with even three to five seeds would bound it. Second, the candidate pool is engineered per dataset; that is acknowledged, but it limits transfer claims. Third, the 15-comparison family is stated as pre-specified but I cannot verify that from the text; I would ask for the frozen analysis plan. Fourth, they do not engage the learning-to-defer or selective classification literature, which is adjacent and would sharpen the novelty claim. Minor: the Hisar complexity is not lightweight.\n\nVerdict: worth refereeing. A serious editor should send this out, not desk-reject it, and the referee should ask for the seed study and the frozen analysis plan. I would cite it for the decision-correction protocol in AMC, but I would not yet cite the accuracy gains as established.","headline":"A careful, honestly delimited decision-correction study for AMC; the protocol is the real contribution, but the central causal claim still needs a retraining-seed run before the gains should be taken at face value.","tokens_in":16674,"tokens_out":2778,"would_cite":true,"duration_ms":26049,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A retain-or-correct decision layer lifts modulation classification by up to 2.7 percentage points, with gains that survive a frozen-policy stress test.","keywords":["automatic modulation classification","cognitive receiver","residual utility","out-of-fold learning","decision correction","retain-or-correct policy","validation-freezing","risk-aware routing"],"falsifier":"Run the same frozen policy on a new dataset or a new channel condition where the out-of-fold distribution used to train the utility estimators differs substantially from the held-out distribution, and show that the paired gain falls below zero or that the 95% confidence interval of the gain crosses zero; alternatively, a full retraining-seed study in which the primary's error surface varies could reveal that the gains are not stable across primary retrains.","tokens_in":15813,"feed_emoji":"📡","tokens_out":2630,"duration_ms":22433,"temperature":0.7,"pith_summary":"The paper argues that a cognitive receiver should not blindly trust its default classifier; it should learn, from cross-fitted out-of-fold predictions, when an alternative predictor is likely to rescue an error without causing harm, and only then override the default. The complete system improves held-out accuracy by 2.700, 1.008, and 2.098 percentage points on three standard automatic modulation classification benchmarks. Controlled comparisons show that the isolated residual-utility objective does not uniformly beat other out-of-fold meta-learners; the consistent gains come from the full policy design: an explicit retain action, validation-frozen thresholds, approved transitions, and a risk mask. A frozen-policy stress test under carrier-frequency offset, I/Q imbalance, and synthetic fading reports positive paired gains in all 11 conditions, with every 95% confidence interval above zero.","feed_headline":"Reserve-correction layer adds up to 2.7 points to radio classifier","feed_subtitle":"A retain-or-correct policy, learned out-of-fold and frozen on validation, beats the isolated utility objective across three benchmarks.","key_machinery":"The central mechanism is cross-fitted residual utility (ERU): for each candidate predictor k, a per-sample utility label u_ik = +1 if the candidate rescues a primary error, -1 if it harms a correct primary, and 0 otherwise, learned from out-of-fold predictions to avoid leakage. A validation-frozen action policy then chooses among retain, candidate adoption, probability blending, approved transitions, and conditional routes by maximizing estimated conditional gain, tied to the accounting identity that accuracy improvement equals (N_rescue - N_harm) / N. This identity makes the decision objective transparent and the retain action explicit.","core_discovery":"On its own terms, this paper establishes that a post-inference recover-or-correct policy—built from candidate-specific residual utility learned on out-of-fold predictions and frozen on a disjoint validation split—produces consistent held-out accuracy improvements across three AMC benchmarks. The central result is that the complete system (KAN-Fourier-RC) raises accuracy from 63.632% to 66.332% on RML2016.10A, from 65.161% to 66.168% on RML2016.10B, and from 77.769% to 79.867% on HisarMod2019.1. The paper carefully shows that the isolated rescue-minus-harm utility target does not uniformly dominate OOF linear stacking, OOF XGBoost stacking, or OOF candidate competence; the repeatable advantag","pith_inferences":["Editorial: The same OOF residual-utility scheme could transfer to other domains where overriding a trusted default (e.g., a medical diagnosis or fraud alert) must be justified by evidence, and the retain action is critical to avoid harm.","Editorial: The frozen-policy stress test suggests that decision-layer gains may be robust to deployment-time distribution shifts, but the paper's synthetic perturbations do not replace over-the-air validation; a natural next experiment is to run the same frozen policy on software-defined-radio captures.","Editorial: The finding that the isolated utility objective does not dominate baselines implies that future work should focus on policy design—such as risk masks and action routing—rather than on fine-tuning the meta-objective.","Editorial: The coverage–utility decomposition could be used as a model-agnostic monitoring tool: if a deployed policy's conditional utility drops below its validation-time estimate, that is a signal to retrain or re-freeze the thresholds."],"forward_implications":["A weaker auxiliary model can still be valuable if its residual utility is positive on a recognizable subset, so decision-layer design should focus on conditional utility rather than standalone accuracy.","The explicit retain action and validation-freezing are necessary for consistent gains; the isolated utility objective alone is not sufficient, meaning the full policy architecture is the load-bearing contribution.","The frozen policy retains positive paired utility under CFO, I/Q imbalance, and synthetic Rayleigh/Rician fading without retraining, although absolute accuracy still degrades in severe conditions.","Because residual utility is primary-relative, replacing or improving the primary classifier requires rebuilding the OOF records, residual estimators, and validation policy.","The coverage–utility decomposition (gain = correction rate times conditional utility) provides a diagnostic for when to act: optimizing either factor alone is suboptimal."],"fun_headline_variants":["Recover-or-correct gains up to 2.7 points in AMC","Post-inference rescue rule adds 2.7 accuracy points","Out-of-fold override policy improves AMC in all stress tests","Isolated utility target lags; full override policy wins"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The residual-utility estimators learned on three-fold out-of-fold records remain calibrated when applied to held-out predictions from full-training models, including under carrier-frequency offset, I/Q imbalance, and synthetic fading—an assumption the stress test supports empirically but does not guarantee for arbitrary deployment conditions.","fun_headline_variants_meta":{"raw":{"variants":["Recover-or-correct gains up to 2.7 points in AMC","Post-inference rescue rule adds 2.7 accuracy points","Out-of-fold override policy improves AMC in all stress tests","Isolated utility target lags; full override policy wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001013,"raw_usage":{"total_tokens":4145,"prompt_tokens":804,"completion_tokens":3341,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":3279}},"tokens_in":548,"tokens_out":3341,"duration_ms":20331,"temperature":1.0,"reasoning_tokens":3279,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T15:54:50.515409+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same frozen policy on a new dataset or a new channel condition where the out-of-fold distribution used to train the utility estimators differs substantially from the held-out distribution, and show that the paired gain falls below zero or that the 95% confidence interval of the gain crosses zero; alternatively, a full retraining-seed study in which the primary's error surface varies could reveal that the gains are not stable across primary retrains.","supporting_citations":[],"review_version":1}