Pith. sign in

REVIEW 3 major objections 4 minor 17 references

Federated conformal risk control that pools calibration data protects the average hospital while violating per-site coverage at 40% of institutions; this paper shows that shrinkage-based per-site thresholds restore near-nominal coverage at

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 10:46 UTC pith:IMJPLH2R

load-bearing objection Solid empirical demonstration of a real per-site calibration failure in pooled federated CRC, but the proposed fix's per-site guarantee is a tuned heuristic, not a theorem. the 3 major comments →

arxiv 2606.20115 v3 pith:IMJPLH2R submitted 2026-06-18 cs.LG cs.CV

When Average Calibration Fails: Site-Conditional Federated Conformal Risk Control

Nafis Fuad Shahid This is my paper
classification cs.LG cs.CV
keywords federated learningconformal risk controlsegmentationcoverage guaranteessite-conditional coveragerisk-curve shrinkagebrain tumor MRIuncertainty quantification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that naive pooled federated conformal risk control — calibrating one threshold on all sites' data together — gives a marginal false-negative guarantee while failing at the level of individual hospitals: on 1,251 brain-tumor volumes from 20 real institutions, 40% of sites exceed the target α=0.10, with the worst at 0.178. Per-site local calibration fixes coverage but inflates prediction sets by 83x, making them clinically unusable. The proposed shrinkage-based protocol blends each site's empirical risk curve with the pooled curve, weighted by n0/(n_k+n0), plus a finite-sample correction; with n0=19 chosen by leave-one-site-out sensitivity, it achieves 2.7/20 violations at only 2.0x stretch, a 97.5% reduction in stretch versus local CRC. The marginal guarantee is preserved by construction, while per-site coverage is empirically near-nominal across tested targets and seeds.

Core claim

The central claim is that the coverage-efficiency trade-off in federated conformal risk control can be steered by a single hyperparameter n0 that shrinks each site's empirical risk curve toward the pooled curve: R̂_shrink_k(λ) = w_k R̂_k(λ) + (1−w_k) R̂_global(λ) + corr_k, with w_k = n_k/(n_k + n0). The paper shows on FeTS-2022 that this recovers per-site coverage that naive pooling misses — 8/20 sites violated at α=0.10 under pooling, versus 2.7/20 under shrinkage at n0=19 — while keeping prediction-set stretch at 2.0x instead of the 83x of fully local calibration. It also shows the finite-sample correction corr_k is load-bearing: removing it triples violations to the naive-pooling level, a

What carries the argument

Risk-curve shrinkage with an interpolated finite-sample correction. Each site sends only its G-point empirical risk curve. The server forms R̂_shrink_k(λ) = w_k R̂_k(λ) + (1−w_k) R̂_global(λ) + corr_k, where w_k = n_k/(n_k+n0) is the standard empirical-Bayes shrinkage weight and corr_k = w_k·B/(n_k+1) + (1−w_k)·B/(N+1) interpolates the CRC finite-sample corrections between site and pooled. The threshold for each site is the smallest λ where this shrunken curve drops to α. This mechanism turns a single scalar n0 into a continuous dial between local CRC (n0→0) and pooled CRC (n0→∞).

Load-bearing premise

The load-bearing premise is that the heuristic finite-sample correction corr_k = w_k·B/(n_k+1) + (1−w_k)·B/(N+1) validly corrects the shrunken risk curve — the paper says it is 'motivated by but not formally derived from CRC theory,' and removing it triples violations — so if this interpolation is not generally valid, the per-site coverage of the proposed method is unsupported.

What would settle it

Run the identical protocol on a second heterogeneous multi-site dataset (or a simulated mixture with known site distributions) and check held-out per-site mean FNR against α. If any site's shrunken curve crosses α at a threshold whose true FNR is above α, the heuristic correction fails. A sharper test: derive the finite-sample distribution of R̂_shrink_k(λ) under the site-mixture assumption and compute coverage; if the true risk at the selected threshold exceeds α by more than the discretization error, the method's core correction is invalid.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • In federated medical segmentation, safety metrics should be reported per site, not only as averages; pooled calibration can hide 40% of hospitals failing the target.
  • Hospitals can obtain site-specific thresholds by transmitting only G scalars per site (about 0.8 KB), with no patient-level images, masks, or per-volume scores leaving the site.
  • A single global threshold that satisfies the marginal guarantee is too conservative for practice (67x stretch at n0=9); per-site thresholds are needed for clinical utility.
  • The empirical-Bayes shrinkage form w_k = n_k/(n_k+n0) transfers naturally to other monotone losses and other segmentation targets.
  • Leave-one-site-out sensitivity analysis provides a way to pick n0 as the lowest-stretch point with at most a target number of violations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same shrinkage-of-risk-curves mechanism could be applied to other monotone calibration targets (e.g., Dice loss, false-positive rate, or regression interval width) whenever calibration data are split across sites with varying sample sizes.
  • A formal derivation of corr_k — or a bound showing it is conservative for the blended estimator — would turn the empirical per-site guarantee into a finite-sample statement; until then, the method's principal risk is the heuristic correction.
  • If the deployment site-mixture differs from the calibration proportions used in Theorem 1, the marginal guarantee may fail; a testable extension is to reweight the pooled curve by expected deployment proportions and re-derive the shrinkage weights.
  • The paper's finding that direct Lagrangian budget optimization 'cheats' the marginal constraint by over-risking small sites suggests that coverage-fairness constraints need to be explicit in any optimization-based federated calibration scheme.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies federated conformal risk control (CRC) for medical image segmentation. On FeTS-2022 (1,251 subjects, 20 institutions), it shows that naive pooled CRC meets the marginal false-negative-rate target but violates site-level coverage at 8/20 institutions, while per-site local CRC restores coverage at 83x stretch. The authors propose a shrinkage-based protocol, Eq. (1), in which each site transmits only its empirical risk curve (G scalars) to a server; the server computes per-site thresholds by blending each site's risk curve with the pooled curve and adding a finite-sample correction corrk. They report that with n0=19, selected via leave-one-site-out (LOSO) sensitivity analysis, the method achieves 2.7/20 site-level violations at 2.0x stretch. They also show that direct Lagrangian budget optimization fails and that removing corrk triples violations. Theorem 1 provides a marginal guarantee for a conservative global threshold (λhat_fed), not for the deployed per-site thresholds.

Significance. If the empirical claims hold, the paper makes a useful practical contribution: it quantifies the marginal-versus-conditional coverage gap in a real federated medical segmentation setting, identifies an important failure mode of pooled CRC, and proposes a communication-efficient heuristic with a favorable coverage-efficiency trade-off. The paper is honest in stating limitations, and the ablations (especially the corrk ablation) are informative. However, the formal contribution is much narrower than the headline: the only theorem covers a conservative threshold that is not what the paper deploys, and the per-site guarantees rest on an unproven correction term. Moreover, the reported operating point (n0=19) is selected on the same dataset, making the headline numbers a tuned fit. The core ideas are plausible, but the current evidence is insufficient to support the central claim as a general method.

major comments (3)
  1. [Sec. 3.4, Eq. (1); Sec. 4.5; Theorem 1] The deployed per-site thresholds have no distribution-free coverage guarantee. The correction term corrk is essential—Table 2 and Sec. 4.5 show that removing it raises violations from 1.3–2.7 to 8.0–9.3—yet it is explicitly called 'motivated by but not formally derived from CRC theory.' Theorem 1 provides a marginal guarantee only for the conservative λhat_fed, which the paper reports has 67x stretch and is not the deployed method. Consequently, the headline claim of 2.7/20 violations at 2.0x stretch rests entirely on a single empirical benchmark with a heuristic correction. The authors should either provide a finite-sample analysis of R_shrink_k, or reposition the paper as an empirical heuristic and validate it on additional datasets/anatomies.
  2. [Sec. 4.3, Table 1] The choice of n0=19 is made via leave-one-site-out evaluation on the same FeTS-2022 dataset used to report the final numbers. Thus the row 'Ours (n0=19, LOSO)' in Table 1 is a post-selection result, not an out-of-sample prediction. The reported 2.7 violations and 2.0x stretch are therefore tuned on the test data. A proper evaluation would use a nested split: select n0 on an inner loop (e.g., LOSO on a training subset) and evaluate on held-out sites/patients, or explicitly present the result as within-sample sensitivity analysis. In addition, with only three seeds, the violation counts (e.g., 2.7 ± 1.7) have wide error bars; for small sites the mean test FNR has high variance, so the reported count may be within noise of the null of perfect calibration.
  3. [Sec. 3.6, Remark] Theorem 1 relies on the constructed site-mixture assumption P(H=k)=(n_k+1)/(N+K). The authors acknowledge that if the deployment mixture differs, the marginal guarantee may not transfer. This is correct and should be prominently stated. However, the remark also notes that the theorem applies 'for any choice of per-site thresholds' as long as the max includes λhat_pool—which means the theorem provides no information about the per-site behavior of the proposed method. The paper's theoretical content is thus limited to a marginal, mixture-dependent guarantee for a non-deployed threshold. This is a load-bearing gap because the paper's stated contribution is site-conditional control.
minor comments (4)
  1. [Table 2] The table appears to have a formatting error: rows titled 'No corr k' show '99.3' and '158.3' in the Violations column; from the text, these should likely be '9.3' and '8.3' for n0=9 and n0=15. Please correct the table.
  2. [Sec. 3.1] The loss ℓ is defined as the per-volume false-negative rate and B=1, but the paper should state explicitly that B=1 because ℓ ∈ [0,1] and that the correction in Eq. (1) uses this bound. The notation is clear but a brief justification would improve readability.
  3. [Sec. 3.4, Algorithm 1] Algorithm 1 enforces monotonicity of R_k(λ) via cumulative minimum, but the text in Sec. 3.1 says the risk is non-increasing in λ. The paper should clarify whether the cumulative minimum is applied to the raw empirical curve before computing R_global and R_shrink, and whether the same monotonicity enforcement is applied to R_global.
  4. [Sec. 5, Limitations] The limitations paragraph is appropriately candid, but the paper should also mention the absence of code/data release as a reproducibility limitation. Given that the central result is empirical, making the evaluation pipeline available would strengthen the contribution.

Circularity Check

1 steps flagged

Headline n0=19 trade-off is a LOSO-selected value on the same benchmark; the reported 2.7 violations / 2.0x stretch is the selection objective, so the central performance claim is tuned, not a prediction.

specific steps
  1. fitted input called prediction [Sec. 4.3 (The n0 Dial and LOSO Sensitivity Analysis); Table 1 row 'Ours (n0=19, LOSO)']
    "As a post-hoc sensitivity analysis, we perform leave-one-site-out evaluation: for each candidate n0, hold out one site, calibrate shrinkage on the remaining K−1 sites’ calibration data, and evaluate violations and stretch on the held-out site. LOSO identifies n0=19 as the lowest-stretch operating point with ≤3 mean violations, achieving 2.7 violations at 2.0× stretch, squarely within the sweet spot of Fig. 4."

    The headline operating point is chosen on the same FeTS-2022 benchmark by optimizing the exact metrics that are then reported (violations and stretch at n0=19). n0=19 is selected as the lowest-stretch point whose mean violations are ≤3, so reporting '2.7 violations at 2.0× stretch' for that n0 is restating the selection objective, not an out-of-sample validation. A true prediction would need n0 fixed before evaluating on a held-out split/site not used in the LOSO sweep. The pooled-CRC failure and corr_k ablation remain informative, but the central trade-off number is a tuned value.

full rationale

The paper's formal Theorem 1 is a valid (if narrow) monotonicity argument: defining λhat_fed as the max with λhat_pool makes the marginal guarantee follow by construction, and the paper explicitly states that per-site validity is empirically validated and that corr_k is heuristic. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled via citation. The central circularity is the use of leave-one-site-out evaluation on the same dataset both to select n0 and to report the headline performance at n0=19. Because n0 is chosen as the lowest-stretch point with ≤3 mean violations, the reported '2.7 violations at 2.0× stretch' is the selection objective value, not an independent prediction. This fits the 'fitted input called prediction' pattern and makes the paper's strongest performance claim partially circular, though the failure-mode analysis of naive pooled CRC, the privacy properties, and the ablations remain independent empirical content. Score 6 reflects one central claim partially reducing to a tuned selection on the same data.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central claim depends on two unproven elements beyond standard CRC machinery: the data-dependent selection of n0 and the heuristic correction term. The marginal guarantee is only available under a specific site-mixture assumption and for a conservative threshold that the paper does not deploy.

free parameters (1)
  • n0 (shrinkage prior strength) = 19 (selected via LOSO on FeTS-2022)
    Controls the weight w_k = n_k/(n_k+n_0) in Eq. (1). The paper sweeps n0 and chooses 19 via leave-one-site-out evaluation on the same dataset. This is a data-fitted hyperparameter that determines the reported violations/stretch trade-off.
axioms (4)
  • domain assumption Within each site, calibration and test volumes are exchangeable (partial exchangeability).
    Standard assumption in federated conformal prediction (Lu et al. [7]); invoked in Sec. 3.2 and Theorem 1 to extend marginal CRC guarantees.
  • ad hoc to paper Test-site mixture satisfies P(H=k)=(n_k+1)/(N+K).
    Used in Theorem 1 for the marginal guarantee. The authors note in Limitations that if the deployed mixture differs, the guarantee may not transfer. This is a strong assumption not derived from the federated deployment.
  • ad hoc to paper The heuristic correction term corr_k = w_k*B/(n_k+1)+(1-w_k)*B/(N+1) is a valid finite-sample correction for the shrunken risk curve.
    The paper states it is 'motivated by but not formally derived from CRC theory' (Sec. 3.4). Ablation (Sec. 4.5) shows removing it triples violations, so the method's validity rests on this unproven interpolation.
  • standard math Loss ℓ is non-increasing in λ and bounded in [0,1] with B=1.
    Required for CRC inversion; standard in Angelopoulos et al. [1], stated in Sec. 3.1.

pith-pipeline@v1.3.0-alltime-deepseek · 10407 in / 16585 out tokens · 131111 ms · 2026-08-02T10:46:41.128928+00:00 · methodology

0 comments
read the original abstract

Conformal risk control (CRC) provides distribution-free segmentation guarantees by calibrating a prediction-set threshold on held-out data. In federated deployments, the standard approach pools calibration scores into a single threshold. We quantify, on real multi-institutional brain tumor data (FeTS-2022, 1,251 subjects, 20 institutions), a critical failure: naive pooled CRC protects the average hospital but violates coverage at 40% of individual institutions, with the worst site exceeding the target false-negative rate by 7.8 percentage points. We trace this failure to a hidden design choice: the aggregation weights implicitly determine whose coverage is protected. Sample-size weighting optimizes patient-level validity but can sacrifice institution-level reliability; equal-site weighting improves institution-level reliability on this benchmark at comparable efficiency, using only a single scalar per site. We propose risk-curve shrinkage as a principled mechanism: each site transmits its empirical risk curve (G scalars) and a single hyperparameter n0 smoothly interpolates between site-specific local calibration and sample-size-weighted pooled calibration. Leave-one-site-out sensitivity analysis identifies n0=19, achieving 2.7/20 violations at 2.0x stretch. Direct Lagrangian budget optimization fails by concentrating risk on vulnerable hospitals; the finite-sample correction term is essential: removing it triples violations. No patient-level images, masks, or per-volume scores leave any site.

Figures

Figures reproduced from arXiv: 2606.20115 by Nafis Fuad Shahid.

Figure 1
Figure 1. Figure 1: Protocol overview. Each site transmits only its empirical risk curve (G scalars) to the server, which computes shrinkage-regularized per-site thresholds and broadcasts them back. No patient-level images, masks, or per-volume scores leave any site. Algorithm 1 Shrinkage-Based Federated CRC Require: Sites k ∈ [K]; target α; grid Λ; prior strength n0 1: Each site k: compute Rˆk(λ) for λ ∈ Λ; enforce monotonic… view at source ↗
Figure 2
Figure 2. Figure 2: Representative empirical risk upper bounds Rˆk(λ) + corrk from one calibration split (seed 42), showing two FeTS-2022 institutions: easy (inst. 7, nk=6) and hard (inst. 4, nk=23). At λˆpool, the hard site’s risk bound exceeds α=0.10, the failure mode our method addresses. Shrinkage (n0=19) selects λˆours between λˆpool and λˆlocal, achiev￾ing near-nominal coverage without extreme stretch. Dataset averages:… view at source ↗
Figure 3
Figure 3. Figure 3: Per-institution FNR (left) and prediction-set stretch (right) on FeTS-2022. B3 (blue) violates α=0.10 at 8/20 sites; B2 (orange) largely restores coverage at 83× stretch. Ours (red, n0=15): practical stretch with 2–3 borderline violations. LOSO￾selected n0=19 is reported in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of n0 on violations (left), worst-site FNR (center), and stretch (right). Dashed: B3 (blue) and B2 (orange) references. Shaded: sweet spot n0 ∈ [10, 20] [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 3 linked inside Pith

  1. [1]

    In: ICLR (2024)

    Angelopoulos, A.N., Bates, S., Fisch, A., Lei, L., Schuster, T.: Conformal risk control. In: ICLR (2024)

    2024

  2. [2]

    arXiv preprint arXiv:2211.02701 (2022)

    Cardoso, M.J., et al.: MONAI: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701 (2022)

    Pith/arXiv arXiv 2022

  3. [3]

    In: MICCAI (2024)

    Chen, J., Ma, B., Cui, H., Xia, Y.: FedEvi: Improving federated medical image segmentation via evidential weight aggregation. In: MICCAI (2024)

    2024

  4. [4]

    Information and Inference10(2), 455–482 (2021)

    Foygel Barber, R., Candès, E.J., Ramdas, A., Tibshirani, R.J.: The limits of distribution-free conditional predictive inference. Information and Inference10(2), 455–482 (2021)

    2021

  5. [5]

    In: AAAI Bridge Workshop (2025)

    Gupta, S., Jangid, N., Sethi, A.: FedStein: Enhancing multi-domain federated learning through James-Stein estimator. In: AAAI Bridge Workshop (2025)

    2025

  6. [6]

    In: ICML (2024)

    Kang, M., Lin, Z., Sun, J., Xiao, C., Li, B.: Certifiably Byzantine-robust federated conformal prediction. In: ICML (2024)

    2024

  7. [7]

    In: ICML (2023)

    Lu, C., Yu, Y., Karimireddy, S.P., Jordan, M.I., Raskar, R.: Federated conformal predictors for distributed uncertainty quantification. In: ICML (2023)

    2023

  8. [8]

    arXiv preprint arXiv:2504.07611 (2025)

    Luo, R., Zhou, Z.: Conditional conformal risk adaptation. arXiv preprint arXiv:2504.07611 (2025)

    Pith/arXiv arXiv 2025

  9. [9]

    In: AISTATS (2017)

    McMahan, B., et al.: Communication-efficient learning of deep networks from de- centralized data. In: AISTATS (2017)

    2017

  10. [10]

    In: MICCAI (2025)

    Mossina, L., Friedrich, C.: Conformal prediction for image segmentation using mor- phological prediction sets. In: MICCAI (2025)

    2025

  11. [11]

    arXiv preprint arXiv:2602.23296 (2026)

    Nguyen, Q.H., Wang, J., Ku, W.S.: Conformalized neural networks for fed- erated uncertainty quantification under dual heterogeneity. arXiv preprint arXiv:2602.23296 (2026)

    arXiv 2026

  12. [12]

    Nature Communications13, 7346 (2022)

    Pati, S., et al.: Federated learning enables big data for rare cancer boundary de- tection. Nature Communications13, 7346 (2022)

    2022

  13. [13]

    In: ICML (2023)

    Plassier, V., Makni, M., Rubashevskii, A., Moulines, E., Panov, M.: Conformal pre- diction for federated uncertainty quantification under label shift. In: ICML (2023)

    2023

  14. [14]

    In: MICCAI (2025)

    Teneggi, J., Stayman, J.W., Sulam, J.: Conformal risk control for semantic uncer- tainty quantification in computed tomography. In: MICCAI (2025)

    2025

  15. [15]

    In: MICCAI (2024)

    Tölle, M., Navarro, F., Eble, S., Wolf, I., Menze, B., Engelhardt, S.: FUNAvg: Fed- erated uncertainty weighted averaging for datasets with diverse labels. In: MICCAI (2024)

    2024

  16. [16]

    Machine Learning 92(2–3), 349–376 (2012)

    Vovk, V.: Conditional validity of inductive conformal predictors. Machine Learning 92(2–3), 349–376 (2012)

    2012

  17. [17]

    arXiv preprint arXiv:2603.14198 (2026)

    Wen, H., Simeone, O., Xing, H.: Efficient federated conformal prediction with group-conditional guarantees. arXiv preprint arXiv:2603.14198 (2026)

    Pith/arXiv arXiv 2026