REVIEW 4 major objections 4 minor 2 cited by
FedCF: Fair Federated Conformal Prediction
T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read Fairness in federated conformal prediction can be controlled by client-side counts alone, without centralizing calibration data.
desk verdict The core finite-sample bound is invalid—exchangeability does not yield the claimed deterministic interval—but the federated decomposition is useful and worth a careful look. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central objects are the filter function FM that selects group/label slices; client-side counts α_k^{(g,y~);λ} (number of sliced calibration points whose non-conformity score for y~ is ≤ λ) and n_k^{(g,y~)} (slice size); and the exchangeability events E_k used to decompose the coverage probability via the law of total probability. Lemma B.1 supplies interval bounds for each client's conditional coverage, Lemma B.2 bounds the slice-membership probability, and Lemma B.3 aggregates these into L and U; Theorem 3.1 assembles them into L_cov and U_cov. The descent-based fair-optimization procedure (Algorithm 1), analogous to gradient descent with momentum, searches the threshold space until the
What would settle it
Construct two clients with opposite score distributions within the same group/label slice—say client A's scores all below 0.2 and client B's all above 0.8—and draw test points mostly from client A. Compute the empirical fairness-specific coverage at λ=0.5 and compare with the L_cov and U_cov from Equation 4: if the empirical coverage falls outside the stated interval, the conditional-exchangeability assumption underlying Lemma B.1 is falsified.
Extended reading notes
Core claim
The paper proves that the fairness-specific coverage level Pr[s(x_test, y~) <= λ | FM(x_test, y_test, g, y~)=1] admits upper and lower bounds L_cov <= Pr[...] <= U_cov computed from per-client quantities. Client k contributes α_k^{(g,y~);λ} n_k^{(g,y~)} / ((n_k^{(g,y~)}+1)(n_k+1) U^{(g,y~)}) to the lower bound and (α_k^{(g,y~);λ}+1)/((n_k+1) L^{(g,y~)}) to the upper bound, weighted by γ_k=(n_k+1)/(N+K), where α counts the client's sliced calibration scores below λ, n is the client's calibration size, n^{(g,y~)} is the slice size, and L,U bound the prior probability of belonging to the slice. The bounds follow from decomposing the target probability into client-exchangeability events and appl
Load-bearing premise
The bounds hold only if, after filtering to a given group and positive label, the test point's non-conformity score is exchangeable with each client's filtered calibration scores (Lemma B.1); the paper inherits this conditional exchangeability from the prior Conformal Fairness result rather than proving it in the federated setting, and if it fails the coverage-gap bounds collapse.
Editorial extensions
If this is right
- A regulator can audit a deployed federated conformal predictor by asking clients to send the same counts for the deployed threshold; if the computed gap is below c, the predictor can be certified fair without any client revealing calibration data.
- Clients need only transmit O(|G||Y+|) summary values per server round; a communication-efficient protocol is available when bandwidth is tight, while an enhanced-privacy protocol hides more distributional detail at quadratic communication cost.
- Because the bounds rely only on exchangeability within slices, FedCF applies to tabular, graph, and image data and to different non-conformity scores such as APS, RAPS, and DAPS.
- Choosing interval bounds yields a finite-sample fairness guarantee at the price of larger prediction sets; choosing MLE point estimates gives tighter sets but no guarantee, so the realized gap can exceed c even though it improves on the fairness-agnostic baseline.
- The descent reformulation reduces the number of client-server communication rounds needed to construct a fair conformal predictor compared with a discretized search over thresholds.
Reading between the lines
- The audit procedure in Appendix G suggests a cheap compliance loop: a regulator could periodically re-run the count aggregation against the current deployed threshold as new calibration data arrive, as long as the test stream still follows the mixture of clients that participate.
- The conditional-exchangeability requirement is the spot to test first: on a deliberately heterogeneous synthetic federation, comparing empirical slice coverage with L_cov/U_cov would show whether the guarantee needs repair rather than tuning.
- Because the server only needs summed counts, the same bounds could be combined with secure aggregation or shuffling protocols without changing the interval formulas, potentially giving a differentially private fairness audit.
- A natural strengthen would be a version of Theorem 3.1 that drops conditional exchangeability within group-label slices and relies only on marginal partial exchangeability, since that would make the guarantee portable to more heterogeneous federations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper extends the Conformal Fairness (CF) framework of Vadlamani et al. (2025) to federated learning. The main theoretical result (Theorem 3.1, Eq. 4) claims finite-sample interval bounds on the group- and label-conditional coverage probability Pr[s(x_test, y_tilde) <= lambda | FM=1] from client-computed counts, together with IID point estimates. The paper also introduces a momentum-descent variant of CF for federated threshold search, two client-server aggregation protocols with different privacy/communication trade-offs, an optional differential-privacy extension, and an auditing procedure. Experiments are reported on ACSIncome/ACSEducation, Pokec-{n,z}, and Fitzpatrick with APS, RAPS, and DAPS non-conformity scores.
Significance. If the theorem were correct, FedCF would give a useful, communication-efficient fairness audit for federated conformal predictors. The experimental breadth, the explicit treatment of communication vs. privacy protocols, and the availability of source code are strengths. However, the finite-sample guarantee is the paper's main contribution, and the proof of Theorem 3.1 rests on an invalid exchangeability argument in Lemmas B.1 and B.2. The counterexample in the major comments shows that the claimed deterministic bounds can exclude the true conditional coverage with non-negligible probability even in the IID setting. The empirical results may still be useful as heuristics, but they do not compensate for the false central claim.
major comments (4)
- [Appendix B.1.1, Lemma B.1 (Eq. 9)] The asserted interval is not a valid consequence of exchangeability. Counterexample: take a client slice with n_k^{(g,y_tilde)}=1 calibration point and let Z_i = 1[s(x_i,y_tilde) <= lambda]. If Z_i ~ Bernoulli(0.9) (IID, hence exchangeable) and the calibration point has Z_1=0, then P(Z_test=1 | Z_1=0)=0.9, whereas Eq. (9) gives upper bound (0+1)/(1+1)=0.5. Conditioning on the observed count does not make the test point a uniform draw from the calibration bag. Since this lemma enters directly into Theorem 3.1 (Eq. 4), the claimed finite-sample bound on the conditional coverage level is false.
- [Appendix B.1.2, Lemma B.2 (Eqs. 11-13)] The same error appears in the bound for the filter probability. Equation (13) treats D_calib^(k)+ as a finite bag and states that the test point's FM value is determined by the empirical frequency of that bag. But the test value is not observed; after conditioning on the n_k calibration points, it is an additional exchangeable draw. For example, with n_k=1, if the calibration point has FM=0 and the true FM probability is 0.9, Lemma B.2's upper bound (0+1)/(1+1)=0.5 excludes the true value. Thus L^(g,y_tilde), U^(g,y_tilde) in Eq. (15) are not valid, and the server-side aggregation in Theorem 3.1 collapses.
- [Theorem 3.1 and Corollary B.1 (Eqs. 4, 19)] Because both Lemmas B.1 and B.2 are invalid, the central theorem and its tightened corollary are unsupported. The proof is a formal substitution into Eq. (3), so no independent argument is supplied. In addition, Eq. (4) divides by L^(g,y_tilde) or U^(g,y_tilde), which can be zero in small samples, so the expressions are not even well-defined in all cases. The advertised finite-sample control of the fairness coverage gap therefore does not hold.
- [Appendix B.1.1, proof of Lemma B.1] The proof of Lemma B.1 does not derive the bound; it cites 'the conditional coverage guarantees given in CF (Vadlamani et al., 2025).' Since Lemma B.1 is load-bearing for Theorem 3.1 and the cited result is not proved here, the manuscript is not self-contained on its central claim. Moreover, the counterexample above shows the imported lemma is false, so the missing proof is not merely a presentation issue.
minor comments (4)
- [Section 3.2, Algorithm 1] Line 4 computes cg_t from lambda_0 rather than lambda_t, so the loop does not evaluate the current threshold; b_t is never initialized; and update_lr is unspecified. No convergence or correctness analysis is provided for the momentum descent, so the claim that FedCF finds the minimal lambda satisfying the fairness constraint is not established.
- [Section 5, 'On Interval Bounds'] The text suggests using binomial proportion confidence intervals such as Wallis (2013), but this is not what Lemmas B.1-B.3 actually do. If the intended method is a binomial CI, the theorem statements and Algorithm 2-3 would need to be revised accordingly; as written, the suggestion is inconsistent with the proof.
- [Table 1] In the definition of alpha_k^{(g,y_tilde);lambda}, the sum is written as sum over (x_i, ) in S_k, omitting y_i. Please correct the notation.
- [Appendix B] There are several typographical issues, e.g., 'defintion', 'equivilantly', and the repeated 'i.e.' constructions. These do not affect the technical content.
Circularity Check
Theorem 3.1's finite-sample fairness bounds reduce to local empirical counts: Lemma B.1 is imported from the authors' own CF paper and Lemma B.2 defines the target probability as the calibration-slice frequency.
-
self citation load bearing
[Appendix B.1.1, Lemma B.1 proof (used in Theorem 3.1, Eq. 4)]
"The interval bounds follow from the conditional coverage guarantees given in CF (Vadlamani et al., 2025)."
Lemma B.1 bounds Term I of Eq. 3, the client-side conditional coverage probability that the paper needs for its main finite-sample guarantee. Its proof is not given here; it is deferred to a prior paper by the same authors (Vadlamani, Srinivasan, Maneriker, Payani, Parthasarathy). Theorem 3.1's proof then 'substitutes' the bounds from Lemmas B.1, B.2, and B.3 into Equation 3, so the headline coverage-gap bound is inherited from a same-author citation rather than derived in this manuscript. The cited conditional-exchangeability premise is stronger than the partial-exchangeability assumption of FCP and is not proved from the assumptions stated here, making this self-citation load-bearing.
-
self definitional
[Appendix B.1.2, Lemma B.2 proof]
"Treating this as a finite 'bag' of covariates, we have ∀(x,y)∈D_calib^{(k)+}, Pr[FM(x,y,g,˜y)=1|Ek] = Σ_{(xi,yi)∈D_calib^{(k)+}} FM(xi,yi,g,˜y) / (nk+1). In other words, we have defined the probability of randomly selecting a covariate with FM(x,y,g,˜y)=1."
The lemma claims finite-sample interval bounds for Pr[FM=1|E_k], but the proof first defines that probability as the empirical frequency in the known calibration bag augmented with the test point. The bounds in Eq. 11 are then just the observed count n_k^{(g,˜y)}/(n_k+1) and (n_k^{(g,˜y)}+1)/(n_k+1), with the '+1' representing the unknown test value. Thus the claimed 'guarantee' is the client's local count by construction, not a consequence of exchangeability. Since Theorem 3.1 (Eq. 4) multiplies these same counts into L_cov and U_cov, the fairness coverage-gap bound reduces to the client-computed histogram summaries rather than a derived finite-sample probability bound.
full rationale
The paper's federated decomposition (Eq. 3) is a law-of-total-probability identity, and the client-server protocols, communication/privacy discussion, and experiments are substantive independent work. The circularity is concentrated in the theoretical guarantee advertised as the main contribution. Lemma B.1, which bounds Term I of Eq. 3, is not proved; it is imported from the same authors' Conformal Fairness paper (Vadlamani et al. 2025), and Theorem 3.1's proof is just substitution of that lemma. Lemma B.2 then 'proves' the Term II bounds by defining Pr[FM=1|E_k] as the empirical frequency in the finite bag, so the bounds in Eq. 4 are the client-computed counts n_k^{(g,˜y)} and α_k^{(g,˜y);λ} divided by n_k+1. That is the fitted input renamed as a finite-sample guarantee, not a consequence of exchangeability. Because these counts are exactly what the clients send, the headline coverage-gap control is equivalent to the local summaries by construction. The experiments are real and show the method works in practice, but they do not rescue the theoretical claim from reducing to its inputs. Score is 7 rather than 10 because the federated aggregation algebra and protocol design are independent contributions and the empirical evaluation is not itself circular.
Assumptions & free parameters
free parameters (4)
- learning rate η for descent-based CF
- momentum constant μ
- number of descent rounds (num_rounds)
- RAPS regularization (ν, k_reg) and DAPS diffusion δ =
fixed a priori
assumptions (4)
- domain assumption Partial exchangeability: within each client, the calibration multiset together with the test point is exchangeable, and the test point is assigned to client k with probability γ_k ∝ n_k+1.
- domain assumption Conditional exchangeability of filtered subpopulations: restricted to a group and positive-label slice (FM=1), the test point is exchangeable with the client's sliced calibration points, so the CF interval bounds in Lemma B.1 hold.
- standard math Bernoulli-trial exchangeability for binary events in Lemmas B.2 and B.4.
- ad hoc to paper The coverage gap as a function of λ behaves well enough for the momentum descent to make progress toward the constraint c.
Cite this review
Pith. "Pith review of FedCF: Fair Federated Conformal Prediction." pith.science (2026). https://pith.science/paper/J7HMVRE6
@misc{pith2026250922907,
author = {Pith},
title = {Pith review of: FedCF: Fair Federated Conformal Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/J7HMVRE6}},
note = {Machine review of arXiv:2509.22907}
}
read the original abstract
Conformal Prediction (CP) is a widely used technique for quantifying uncertainty in machine learning models. In its standard form, CP offers probabilistic guarantees on the coverage of the true label, but it is agnostic to sensitive attributes in the dataset. Several recent works have sought to incorporate fairness into CP by ensuring conditional coverage guarantees across different subgroups. One such method is Conformal Fairness (CF). In this work, we extend the CF framework to the Federated Learning setting and discuss how we can audit a federated model for fairness by analyzing the fairness-related gaps for different demographic groups. We empirically validate our framework by conducting experiments on several datasets spanning multiple domains, fully leveraging the exchangeability assumption.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 2 Pith papers
-
Multi-Agent Conformal Prediction with Personalized Statistical Validity
PFWCP achieves personalized asymptotic marginal and calibration-conditional coverage in federated conformal prediction via density ratio weighting and quantile aggregation under one-shot communication.
-
Beyond Procedure: Substantive Fairness in Conformal Prediction
Equalizing prediction-set size, not coverage, correlates with fairer downstream decisions, and label-clustered conformal prediction achieves this balance.
Reference graph
Works this paper leans on
-
[1]
In other words, we can check whether, 23 Preprint
(nk + 1)L(ga,˜y)− α(gb,˜y);λ k n(gb,˜y) k (n(gb,˜y) k + 1)(nk + 1)U(gb,˜y) ) | {z } true coverage gap + KX k=1 γkX (ga,gb,˜y) k | {z } Guassian R.V (23) =cov gap(λ, Fm, ga, gb,˜y) +X, X∼ N 0, KX k=1 γ2 kσ2 k;(ga,gb,˜y) (24) Using a prespecified probabilityβwe can accept or reject the statement cov gap est(λ, Fm, ga, gb,˜y)≤c. In other words, we can check ...
2014
-
[3]
We present bounds and point estimates for each of the terms in Equation 3 across Lemmas B.1, B.2, and B.3, leading to a proof of Theorem 3.1
In doing so, the computation of the coverage level is split between the clients and the server. We present bounds and point estimates for each of the terms in Equation 3 across Lemmas B.1, B.2, and B.3, leading to a proof of Theorem 3.1. B.1.1 CLIENT-SIDEESTIMATES Since each client operates independently with its own dataset, we can derive interval bounds...
2025
-
[9]
doi: 10.1109/CVPR.2016.90. Dennis Hirsch, Timothy Bartley, Aravind Chandrasekaran, Davon Norris, Srinivasan Parthasarathy, and Piers Norris Turner. Business Data Ethics: Emerging Models for Governing AI and Advanced Analytics. Springer Nature,
-
[12]
URLhttps: //ojs.aaai.org/index.php/AAAI/article/view/21459
doi: 10.1609/aaai.v36i11.21459. URLhttps: //ojs.aaai.org/index.php/AAAI/article/view/21459. Charles Lu, Yaodong Yu, Sai Praneeth Karimireddy, Michael Jordan, and Ramesh Raskar. Federated confor- mal predictors for distributed uncertainty quantification. InInternational Conference on Machine Learning, pp. 22942–22964. PMLR,
-
[13]
Association for Computing Machinery. ISBN 9798400701030. doi: 10.1145/3580305.3599454. URLhttps://doi.org/10.1145/3580 305.3599454. Pranav Maneriker, Aditya T Vadlamani, Anutam Srinivasan, Yuntian He, Ali Payani, et al. Conformal pre- diction: A theoretical note and benchmarking transductive node classification in graphs. Transactions on Machine Learning ...
-
[14]
ISSN 2150-8097. doi: 10.14778/3352063.3352135. URLhttps://doi.org/10.14778/3352063.3352135. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference on Artificial Inte...
-
[15]
Local law 4 of 2021: Fair chance act, 7
New York City Council. Local law 4 of 2021: Fair chance act, 7
2021
-
[16]
11 Preprint
URLhttps://www.nyc.gov/ site/cchr/law/fair-chance-act.page. 11 Preprint. New York City Council. Local law 144 of 2021: Prohibiting automated employment decision tools. New York City Register, 7
2021
Show all 21 references
-
[18]
Yanfei Zhou and Matteo Sesia
URLhttps: //arxiv.org/abs/1905.12022. Yanfei Zhou and Matteo Sesia. Conformal classification with equalized coverage for adaptively selected groups. Advances in Neural Information Processing Systems, 37:108760–108823,
1905 arXiv
-
[20]
On closer inspection, we observe that Terms I and II can be combined and bound together
can be bounded as Lcov(λ, FM , g,˜y)≤Pr[s(xtest,˜y)≤λ|F M (xtest, ytest, g,˜y) = 1]≤Ucov(λ, FM , g,˜y), where Lcov(λ, FM , g,˜y) = KP k=1 γkα(g,˜y);λ k n(g,˜y) k (n(g,˜y) k +1)(nk+1)U (g,˜y)andU cov(λ, FM , g,˜y) = KP k=1 γk(α(g,˜y);λ k +1) (nk+1)L(g,˜y) .(4) If the data is II...
2021
-
[21]
C.5 NON-CONFORMITYSCORES Adaptive Prediction Sets (APS)The most popular CP method for classification problems is APS (Romano et al., 2020b)
for both base model training and the FedCF framework. C.5 NON-CONFORMITYSCORES Adaptive Prediction Sets (APS)The most popular CP method for classification problems is APS (Romano et al., 2020b). The scoring function first sorts the softmax logits in descending order and accumu...
2022
-
[22]
K×G×G ▷Used for private formulations 5:forclientk∈ Kin parallel do 6:ifformulations==COMM EFFICIENTthen 7: Receive(l k, uk, nk) =CLIENTCG COMM EFFICIENT(k, λ0, FM ,˜y,G) 8:l list[k]←l k,u list[k]←u k 9:else 10: Receive(pw cgk, nk) =CLIENTCG PRIVATE(k, λ0, FM ,˜y,G) 11:pw cg li...
2006
-
[2013]
URLhttps://doi.org/10.1080/09296174.2013.799918
doi: 10.1080/09296174.2013.799918. URLhttps://doi.org/10.1080/09296174.2013.799918. Jie Wen, Zhixia Zhang, Yang Lan, Zhihua Cui, Jianghui Cai, and Wensheng Zhang. A survey on federated learning: challenges and applications. International journal of machine learning and cyberne...
2013
-
[2016]
ISBN 9781450342322
Association for Computing Machinery. ISBN 9781450342322. doi: 10.1145/ 2939672.2939785. URLhttps://doi.org/10.1145/2939672.2939785. John Cherian and Lenny Bronner. How the washington post estimates outstanding votes for the 2020 presi- dential election. Retrieved September, 13:2023,
2020
-
[2017]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun
URLhttp://arxiv.org/abs/1706.02216. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778,
2016 arXiv
-
[2018]
doi: https://doi.org/10.1016/j
ISSN 1084-8045. doi: https://doi.org/10.1016/j. jnca.2018.05.003. URLhttps://www.sciencedirect.com/science/article/pii/S108 4804518301590. 10 Preprint. Soroush H. Zargarbashi, Simone Antonelli, and Aleksandar Bojchevski. Conformal prediction sets for graph neural networks. In ...
2018 doi
-
[2019]
European Parliament and Council of the European Union. Regulation (EU) 2024/1689 of the Euro- pean Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intel- ligence (Artificial Intelligence Act) and amending Regulations (EC) No 300/2008, (...
2024
-
[2020]
CR Komala, Ashok Kumar, N Hema, S Nagarani, Ajay Singh Yadav, M Rajendiran, R Srinivasan, and V Vijayan
URLhttps://arxiv.org/abs/2001.08361. CR Komala, Ashok Kumar, N Hema, S Nagarani, Ajay Singh Yadav, M Rajendiran, R Srinivasan, and V Vijayan. Fair-automl: Enhancing fairness in machine learning predictions through automated machine learning and bias mitigation techniques. In A...
2001 arXiv
-
[2021]
doi: https://doi.org/10.1016/j.simpa.2020.100049
ISSN 2665-9638. doi: https://doi.org/10.1016/j.simpa.2020.100049. URLhttps://www.sciencedirec t.com/science/article/pii/S2665963820300403. Cynthia Dwork. Differential privacy. InInternational colloquium on automata, languages, and programming, pp. 1–12. Springer,
2020
-
[2022]
Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Hei Li Kwing, Titouan Parcollet, Pedro PB de Gusm˜ao, and Nicholas D Lane
URLhttps://arxiv.org/abs/2009.14193. Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Hei Li Kwing, Titouan Parcollet, Pedro PB de Gusm˜ao, and Nicholas D Lane. Flower: A friendly federated learning research framework. ar...
2009 arXiv
-
[2024]
Arya Fayyazi, Mehdi Kamal, and Massoud Pedram
URLhttps: //eur-lex.europa.eu/eli/reg/2024/1689/oj/eng. Arya Fayyazi, Mehdi Kamal, and Massoud Pedram. Facter: Fairness-aware conformal thresholding and prompt engineering for enabling fair llm-based recommender systems. In Forty-second International Conference on Machine Learning,
2024
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.