{"id":"57425c30-bfa8-47f8-bed0-70a52e576841","arxiv_id":"2506.17872","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"FedNAM+ couples Neural Additive Models with a gradient-based Dynamic Level Adjustment scheme for uncertainty in federated learning, but the coverage guarantee is not established.","lead":"FedNAM+ combines Neural Additive Models with a gradient-based Dynamic Level Adjustment scheme to produce uncertainty estimates in federated learning. The paper claims conformal coverage guarantees, but the method is a heuristic that by its own appended critique lacks a theoretical foundation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing assumption fails: Algorithm 2's Dynamic Level Adjustment never computes a non-conformity score or a calibration quantile, so it does not inherit Eq. (1)'s coverage guarantee; Section 11 concedes there is no proof.","rationale":"The reader and I identify the same single point: the conformal coverage guarantee is the paper's central advertised property, and Algorithm 2 is the mechanism. The paper itself supplies the decisive admission in Section 11. I also checked whether anything else could rescue the claim: Theorem 1 concerns smooth convergence, not predictive coverage; the reported accuracy drop is inconsistent between the abstract (0.1%) and Table 2 (0.8734 to 0.8521, about a 2.1% drop); and no empirical coverage numbers appear anywhere. The DLA heuristic could still be useful as an uncertainty visualization, but calling it conformal without a validity proof is unsupported. The correct verdict is REJECT; a revision would need a real conformal calibration step or a theorem showing gradient-magnitude thresholds yield coverage.","tokens_in":9900,"tokens_out":2748,"duration_ms":29299,"concrete_test":"Run a controlled MNIST experiment with the model used in Section 5: reserve a calibration split and a test split; execute Algorithm 2 exactly as written with alpha=0.1; for each test sample, record whether the true label falls in the returned set/interval; compute empirical marginal coverage and mean set size over 10 seeds. If coverage is substantially below 0.9 or does not exceed 0.9 with high probability, Eq. (1)'s guarantee fails. As a positive control, run split conformal with the same model and softmax non-conformity scores; it should reach about 90% coverage, demonstrating the gap is due to DLA.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that FedNAM+ outputs prediction sets satisfying P[y_{n+1} in C(x_{n+1})] >= 1-alpha. Standard split conformal obtains this by thresholding a non-conformity score at the (1-alpha) empirical quantile of a held-out calibration set. Algorithm 2 does neither: lines 5-9 compute per-sample gradient magnitudes, normalize them, compare them to the median, and return C = alpha*beta*(1+Gmag) or alpha*(1+Gmag). No calibration scores, no quantile, no exchangeability argument, and no statement of what set of labels C contains. The alpha is modified per sample, so even if the raw widths used a quantile, the target level changes per point. The manuscript's Section 11 explicitly concedes: 'there is no formal proof that correlates gradient magnitudes with reliable uncertainty estimates.' Thus the only route to the coverage claim is absent. Without Eq. (1), the contribution reduces to a heuristic for adaptive interval width, not a conformal prediction method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes FedNAM+, a federated learning framework that combines Neural Additive Models (NAMs) with a 'Dynamic Level Adjustment' (DLA) mechanism claimed to be a novel conformal prediction method. It asserts that the framework produces prediction sets satisfying the conformal coverage guarantee P[y_{n+1} in C(x_{n+1})] >= 1-alpha (Eq. 1), while providing pixel-level interpretability and computational efficiency relative to Monte Carlo Dropout. The experimental sections report results on CT-scan, MNIST, and CIFAR data, with visual comparisons of uncertainty intervals and feature attributions. The paper includes appended critical sections (Sections 11-13) that explicitly state that DLA lacks a theoretical foundation, that the convergence proof assumes IID data, and that the conformal prediction application is questionable. The central algorithmic object, Algorithm 2, adapts interval widths from gradient magnitudes without constructing non-conformity scores or calibration sets.","tokens_in":10171,"tokens_out":6398,"duration_ms":62340,"significance":"If the claimed coverage guarantee and efficiency results were valid, a method that combines interpretable NAMs with distribution-free prediction sets in federated learning would be a useful contribution. The paper does not, however, ship machine-checked proofs, reproducible code, or calibration-based coverage measurements, and the manuscript's own appended sections concede that the theoretical basis for the core method is missing. The DLA scheme may be worth exploring as an empirical heuristic for adaptive interval widths, but as presented it is not a conformal prediction method and does not establish the paper's central claims.","major_comments":[{"comment":"Algorithm 2 does not implement conformal prediction and cannot inherit the coverage guarantee in Eq. (1). Lines 5 through 9 compute per-sample gradient magnitudes, normalize them, compare them to the median, and return interval widths C = alpha*beta*(1+Gmag) or alpha*(1+Gmag). No non-conformity score S(x,y) is computed, no calibration set D_cal is formed, no empirical quantile Q_{1-alpha} is taken, and no exchangeability argument is supplied; the output is a scalar width rather than a prediction set over labels. Because alpha is modified per sample, the target coverage level changes pointwise, so the resulting intervals have no stated coverage level. Section 11 explicitly concedes that there is no formal proof correlating gradient magnitudes with reliable uncertainty estimates, and Section 13 states that the conformal application casts doubt on the validity of the results. The central claim of calibrated conformal coverage is therefore unsupported.","section":"Section 3.5, Algorithm 2"},{"comment":"The convergence proof is tautological. The asserted inequality L(w_{t+1}) <= L(w_t) - eta*||grad L(w_t)||^2 + O(delta_t) introduces O(delta_t) as the residual of exactly this inequality, so the statement cannot fail and provides no information about convergence or about the effect of Dynamic Level Adjustment. Section 12 itself states that the proof assumes IID client data, which is inconsistent with the federated setting and with the paper's own Section 8 limitation. No non-IID convergence analysis or bound on delta_t is provided. This theorem does not provide a theoretical foundation for FedNAM+.","section":"Section 3.2.1 and Section 12"},{"comment":"Coverage is listed among the evaluation metrics, but no experiment reports empirical coverage. Table 1 reports average uncertainties, Table 2 reports accuracy and prediction bounds, and Figures 3-9 display interval widths and pixel maps. The abstract's claim that the method achieves desired coverage levels is therefore unverified; the key quantitative guarantee of the paper is never measured.","section":"Section 4.2 and Table 2"},{"comment":"The abstract states a 'small drop in accuracy (0.1%) for MNIST,' but Table 2 reports model accuracy before the framework as 0.8734 and after as 0.8521, a drop of about 2.1 percentage points. The discrepancy is not explained. In addition, Algorithm 1 line 11 calls an undefined function ConformalPrediction and never constructs a calibration set or specifies how prediction sets are formed, so the experimental procedure does not match the claimed method.","section":"Table 2 and Section 5"}],"minor_comments":[{"comment":"The claimed complexity reduction to O(N log N + delta) is asserted without derivation; the median computation in Algorithm 2 can explain the N log N term, but because no conformal procedure is specified, the claim is not meaningful.","section":"Section 3.2.2"},{"comment":"The ablation studies refer to a clustering mechanism that is not otherwise described, and Section 4.1 lists the UCI Diabetes dataset although no experiment on it appears in the results.","section":"Section 9.1 and Section 4.1"},{"comment":"The manuscript contains an empty Section 10 titled 'Theoretical Contributions' followed by Sections 11-13 that acknowledge the lack of theoretical foundation, the IID assumption problem, and the questionable conformal application. If these sections are intended as limitations, they are inconsistent with the paper's claims; if they are intended as notes to reviewers, they should be removed or fully integrated into a revised framing.","section":"Sections 10-13"},{"comment":"Several citations appear mismatched or implausible: reference [20] cites Simonyan and Zisserman for EfficientNet, which is not the correct source, and reference [6] uses placeholder-style author names. The manuscript should be checked for citation accuracy.","section":"References"},{"comment":"There are numerous typographical and grammatical errors (e.g., 'uncertainity', 'Ciph ar10', 'LIME ... falls short in estimating uncertainty') that require careful proofreading.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"To the editor: The manuscript's own Sections 11-13 concede that the core method lacks a theoretical foundation and that the conformal application is questionable. My reading of Algorithm 2 confirms that it is not a conformal prediction method: it never computes non-conformity scores or calibration quantiles, so Eq. (1) cannot be inherited. The empirical coverage is never measured, and the reported accuracy drop contradicts the abstract. These are load-bearing failures that cannot be fixed by local edits; the paper would need a fundamentally different method and evaluation. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nBottom line: don't trust the coverage guarantee. The one genuinely new thing is Algorithm 2's Dynamic Level Adjustment—scaling alpha by normalized input-gradient magnitudes and thresholding at the median. I checked the cited literature and that recipe is not in FedNAM or the conformal prediction papers. That is a legitimate contribution, though a modest one. The underlying instinct (adaptive interval width, gradient-driven sensitivity, federated interpretability) is a reasonable direction, and the NAM + FL combination is sensible.\n\nWhat the paper does well: it identifies a real gap—interpretable, uncertainty-aware federated learning—and the proposed architecture is plausible. The authors also leave an unusual trail of honesty: Sections 10–13 of the submitted text explicitly state that DLA has no theoretical foundation, that the convergence proof assumes IID data and is insufficient, and that the conformal application \"lacks proper citation and understanding.\" You rarely see that in a paper, but here it reads like an accidental admission rather than a structured limitation section.\n\nNow the soft spots, in order of severity. First, the central claim fails. Standard split conformal achieves Eq. (1) by thresholding a non-conformity score at the 1−α quantile of a calibration set. Algorithm 2 does neither: it computes gradient magnitudes, normalizes, compares to a median, and returns C = α·β·(1+Gmag) or α·(1+Gmag). No quantile, no calibration set, no exchangeability argument. Worse, α is modified per sample, so the target coverage level is not even fixed. There is no route from this algorithm to Eq. (1). The paper's own Section 11 concedes exactly this. Second, the convergence proof in Section 3.2.1 is vacuous: O(δ_t) is defined as the residual of the inequality, so the \"proof\" cannot fail. Third, the empirical section never reports coverage. The abstract says \"only 0.1% on MNIST\" but Table 2 shows accuracy dropping from 0.8734 to 0.8521—a 2.4% drop. Fourth, reference [6] is \"Jane Doe and Alice Smith\"—a placeholder, effectively a fabricated citation. These are not minor issues; together they sink the paper as a research contribution.\n\nWho this is for: a reader interested in the specific DLA heuristic as a preprocessing trick might get a starting point, but anyone looking for calibrated uncertainty in federated learning will be misled. The paper should not be accepted in current form. If the authors rebuilt it around a real conformal calibration step—held-out set, proper non-conformity score, reported coverage—the DLA heuristic could be a useful addition. As it stands, my recommendation: sending it to peer review is defensible only if the editors expect heavy revision and reject the current version; the coverage claim and the placeholder citation must be fixed.","headline":"A genuinely new gradient-based heuristic buried under an unsupported conformal coverage claim; the paper contradicts itself and should be rejected, though the DLA idea merits a proper rework.","tokens_in":10623,"tokens_out":2914,"would_cite":false,"duration_ms":28958,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A federated learning pipeline using Neural Additive Models and a gradient-adjusted conformal predictor claims prediction sets with coverage guarantee and only 0.1% MNIST accuracy loss.","keywords":["federated learning","neural additive models","conformal prediction","uncertainty quantification","dynamic level adjustment","pixel-wise uncertainty","model interpretability","gradient sensitivity"],"falsifier":"Run Algorithm 2 on a held-out test split with target 1-alpha = 0.9, compute empirical coverage across all clients, and check whether the observed coverage falls materially below 0.9 on a large calibration set; if it does, the coverage guarantee is not met and the central claim fails. A cleaner test uses synthetic data with known label noise, where the true uncertainty per sample is fixed, and checks whether the Dynamic Level Adjustment widths actually order samples by that noise.","tokens_in":1380,"feed_emoji":"📊","tokens_out":2256,"duration_ms":80791,"temperature":0.7,"pith_summary":"The paper aims to make federated learning both interpretable and uncertainty-aware. It proposes FedNAM+, which combines Neural Additive Models with a conformal-prediction scheme called Dynamic Level Adjustment: gradient magnitudes from each input determine how wide each prediction set should be. The intended payoff is that a clinician or analyst sees not just a prediction but a pixel-level map of where the model is unsure, calibrated to a user-chosen coverage level, without the repeated forward passes that Monte Carlo Dropout requires. On MNIST the reported accuracy loss is 0.1%, and the method also produces class- and pixel-level uncertainty visualizations.","feed_headline":"Prediction sets with coverage for only 0.1% accuracy loss","feed_subtitle":"FedNAM+ attaches per-pixel uncertainty intervals to federated models with minimal accuracy loss.","key_machinery":"The load-bearing mechanism is the additive decomposition of Neural Additive Models, $f_m(x) = \\sum_j f_{m,j}(x_j)$, together with the Dynamic Level Adjustment algorithm. The NAM's separated per-feature networks make each pixel's contribution explicit, and the algorithm's per-sample scaling of $\\alpha$ by normalized input-gradient magnitudes is what turns ordinary conformal sets into adaptive ones. This mechanism works by using gradient magnitude as a proxy for predictive uncertainty, then applying that proxy to adjust interval width while retaining the conformal calibration threshold computed on each client's data.","core_discovery":"The central discovery claimed is an integration rather than a new theory: a federated global model, built by averaging client-trained parameters, is wrapped in a Neural Additive Model whose per-feature networks expose contributions, and a conformal predictor then produces prediction sets whose widths are modulated per sample by the input-gradient magnitude. Algorithm 2 normalizes gradient magnitudes, thresholds them at the median, and scales the base $\\alpha$ by $\\beta(1+G_{mag})$ on high-gradient samples and by $(1+G_{mag})$ otherwise. The claim is that these sets still satisfy the coverage guarantee $P[y_{n+1} \\in C(x_{n+1})] \\geq 1-\\alpha$, while being narrower than quantile-regression conformal prediction's uniformly wide intervals and cheaper than Monte Carlo Dropout. The paper also claims this yields meaningful pixel-wise uncertainty estimates that identify low-confidence regions.","pith_inferences":["A direct extension would replace the median-gradient threshold in Algorithm 2 with a held-out calibration split, so the coverage guarantee of Equation (1) is restored by construction rather than assumed; this is not what the paper proves.","If gradient magnitude is a reliable uncertainty signal, the same adjustment could be ported to non-additive architectures, giving adaptive conformal sets for any differentiable model without retraining.","The per-sample scaled sets could be merged across clients by aggregating nonconformity scores rather than weights, which would make the global coverage statement more robust to client heterogeneity; the paper does not test this.","The gap between gradient-based and Bayesian uncertainty in the paper's own table suggests the two measures are not interchangeable, so a practitioner should treat these intervals as relative confidence scores unless they are empirically calibrated on held-out labels."],"forward_implications":["If the coverage guarantee holds, practitioners can choose a target like 90% or 95% and get prediction sets that include the true label at that rate on each client's data.","Uncertainty heatmaps become a byproduct of training, so clinicians can inspect which pixels drive a medical diagnosis and how confident the model is in that region.","The reported 0.1% MNIST accuracy drop means interpretability and uncertainty would come almost for free in accuracy terms, making the method attractive where Monte Carlo Dropout's repeated forward passes are too costly.","Class-level uncertainty summaries, such as digit 8 being most uncertain and digit 6 least, could guide targeted data collection for the classes the federated model handles worst.","Because the framework keeps data local and only exchanges weights, the uncertainty estimates are available without centralizing sensitive data."],"supporting_citations":[{"why":"It supplies the Neural Additive Model architecture whose per-feature networks make pixel-level contributions interpretable.","marker":"[1]"},{"why":"It supplies FedNAM, the prior federated NAM framework that FedNAM+ extends with conformal uncertainty.","marker":"[18]"},{"why":"It is cited as the conformal prediction foundation from which the coverage guarantee is taken.","marker":"[25]"},{"why":"It provides the quantile-regression conformal baseline whose uniformly wide intervals the Dynamic Level Adjustment method is compared against.","marker":"[22]"},{"why":"It provides the CT-scan medical dataset used to demonstrate pixel-wise uncertainty interpretation.","marker":"[9]"}],"fun_headline_variants":["Federated learning gets interpretable uncertainty via conformal sets","Pixel-wise confidence intervals for federated models at minimal loss","FedNAM+ merges NAM interpretability with conformal prediction","Gradient-guided conformal sets for federated interpretability","Federated learning with reliable uncertainty at tiny accuracy cost"],"cache_read_input_tokens":12800,"weakest_assumption_plain":"The framework's central promise rests on the assumption that per-sample alpha values scaled by normalized gradient magnitudes still yield prediction sets meeting the stated coverage guarantee, which the paper's own limitations acknowledge is unproven.","fun_headline_variants_meta":{"raw":{"variants":["Federated learning gets interpretable uncertainty via conformal sets","Pixel-wise confidence intervals for federated models at minimal loss","FedNAM+ merges NAM interpretability with conformal prediction","Gradient-guided conformal sets for federated interpretability","Federated learning with reliable uncertainty at tiny accuracy cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000482,"raw_usage":{"total_tokens":2389,"prompt_tokens":956,"completion_tokens":1433,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1349}},"tokens_in":572,"tokens_out":1433,"duration_ms":11046,"temperature":1.0,"reasoning_tokens":1349,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:57:44.306154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 2 on a held-out test split with target 1-alpha = 0.9, compute empirical coverage across all clients, and check whether the observed coverage falls materially below 0.9 on a large calibration set; if it does, the coverage guarantee is not met and the central claim fails. A cleaner test uses synthetic data with known label noise, where the true uncertainty per sample is fixed, and checks whether the Dynamic Level Adjustment widths actually order samples by that noise.","supporting_citations":[{"cited_title":"Neural additive models: Interpretable machine learning with neural nets","cited_arxiv_id":null,"evidence_quote":"It supplies the Neural Additive Model architecture whose per-feature networks make pixel-level contributions interpretable."},{"cited_title":"A clustered federated learn- ing method of user behavior analysis based on non-iid data","cited_arxiv_id":null,"evidence_quote":"It is cited as the conformal prediction foundation from which the coverage guarantee is taken."},{"cited_title":"Im- proved conformalized quantile regression","cited_arxiv_id":null,"evidence_quote":"It provides the quantile-regression conformal baseline whose uniformly wide intervals the Dynamic Level Adjustment method is compared against."},{"cited_title":"Chest ct-scan images dataset","cited_arxiv_id":null,"evidence_quote":"It provides the CT-scan medical dataset used to demonstrate pixel-wise uncertainty interpretation."}],"review_version":2}