REVIEW 5 major objections 5 minor 26 references
Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcare
T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read In federated medical-image learning, a server-free co-distillation scheme that shares only each client's majority-class feature average keeps minority-class accuracy higher and more stable than FedAvg, FedProto, FedAMP, and FedDistill as…
desk verdict A simple co-distillation variant for imbalanced federated learning with a plausible mechanism, but the missing test split and feature/logit inconsistency leave the central claim unverified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the teacher–student co-distillation loop centered on each client's 'expertise class,' the majority class in that client's local data. A teacher client samples k images from its expertise class, computes their average feature representation, and shares only that vector as a soft target; the student adds an MSE loss between its own features for that class and the received average, weighted by a coefficient λ, on top of cross-entropy. Uniform random selection of the teacher each round lets every client's expertise propagate to every other client, so minority-class knowledge spreads across hospitals without exchanging model parameters or raw images.
What would settle it
Re-run the same 4-client and 6-client skew experiments with an explicitly documented held-out test set per client, and check whether co-distillation's minority-class accuracy advantage over FedAvg, FedProto, FedAMP, and FedDistill persists on unseen images; if the gap shrinks or reverses, the claimed advantage is an artifact of training-set evaluation.
Extended reading notes
Core claim
The authors' central claim is that a server-free co-distillation protocol is a reliable way to counter class imbalance in federated healthcare learning. In their binary diseased-versus-non-diseased setup, each client's majority class is its expertise; each round a student client samples one teacher uniformly at random, receives the averaged feature representation of that teacher's expertise class, and trains with the weighted combination of cross-entropy loss and an MSE loss that pulls the student's own features for that class toward the teacher's average. They report that this beats FedAvg, FedProto, FedAMP, and FedDistill in most tested settings, degrades less as skew grows (standard deviations of 0.03–0.10 for co-distillation versus 0.38–0.46 for FedAvg), and holds up when training images are scarce, such as 19.3% minority accuracy on COVID at 50 images and 60% skew while other methods fall to 0–4%.
Load-bearing premise
The central claim stands on the unstated assumption that the reported accuracies are computed on images the model has not already trained on; the paper never describes a train/test split, so a reader cannot rule out that the tables measure memorization rather than generalization.
Editorial extensions
If this is right
- In a federated healthcare deployment with severe class skew, client models can preserve minority-class accuracy better than parameter-aggregation methods, which the paper reports FedAvg collapsing to near 0% at 40–60% skew.
- Because only averaged feature vectors are exchanged, communication per round is smaller than full model updates, a practical advantage for bandwidth-limited hospital settings.
- The reported advantage holds for both 4-client and 6-client setups, indicating the method does not depend on one particular cohort size.
- At 60% skew with limited images, co-distillation keeps minority accuracy in the 79–86% range on APTOS and 19–90% on COVID depending on image count, while FedAvg records 0% in several configurations.
Reading between the lines
- Editorial inference: the design implies the method's benefit should scale with the complementarity of client specialties; a testable prediction is that co-distillation gains the most when hospitals specialize in different majority classes and gains less when their class distributions are similar.
- Editorial inference: the reported stability under increasing skew suggests a follow-up study could test whether the random teacher selection can be replaced by a teacher-selection policy that prioritizes confident or balanced teachers, potentially lowering the standard deviation further.
- Editorial inference: averaged feature vectors are shared rather than raw data, but the paper does not quantify how much class information those averages leak; a privacy audit of the soft targets is a natural next step before clinical deployment.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a federated co-distillation framework for class-imbalanced healthcare image classification. Each client is assigned a majority 'expertise' class; during training, a student client randomly selects a teacher client and receives the teacher's average feature representation of its expertise class, then adds an MSE regularizer on its own samples of that class. The authors evaluate minority-class accuracy on APTOS and COVID-19 datasets with 4 and 6 clients, skew levels of 0–60%, and varying dataset sizes, comparing against FedAvg, FedProto, FedAMP, and FedDistill. They claim co-distillation outperforms these baselines and exhibits the lowest standard deviation as imbalance increases.
Significance. If the empirical claims are correct, the framework offers a simple, communication-efficient mechanism for private collaborative learning under class imbalance, with potential applicability to medical imaging. The paper includes a code release and provides algorithmic descriptions, which aid reproducibility. However, the significance is presently limited by a missing evaluation protocol and internal inconsistencies about what information is shared. The method is close in spirit to existing federated distillation approaches, and its contribution reduces to the specific choice of sharing only the teacher's expertise-class feature average; this is a modest conceptual step. The headline claims of 'outperforming' and 'least standard deviation' are not fully supported by the paper's own tables, and the lack of a described held-out test split means the quantitative conclusions are not yet established.
major comments (5)
- [§3.2, §3.3, Algorithms 1–2, Figure 1] The evaluation section does not specify any train/test split. Section 4.2 states only that 'We train all models for 100 epochs' and defines accuracy as 'total minority class images correctly classified / total minority class images', without saying whether the images are from a held-out test set, a validation set, or the training set. Because the reported numbers are the sole evidence for every central claim (outperformance, robustness, low-resource behavior), the paper must specify the exact data split, how clients' local splits are constructed, and confirm that the reported accuracies are computed on held-out data. Otherwise the results may reflect training-set memorization rather than generalization.
- [§5.1, Tables 1 and 2] There is a direct contradiction about what is communicated between clients. Section 3.2 says the teacher 'calculates the probabilistic outputs for these samples which are then transmitted' and Section 3.3 refers to 'soft targets' and 'logits', but Algorithm 1 returns only an average feature vector (AVG(feature(χ))) and Algorithm 2 computes an MSE loss between the student's and teacher's feature representations. Figure 1 likewise shows averaged representations. This is not merely a wording issue: sharing feature averages versus soft labels changes the privacy properties, the communication volume, and the actual training objective. The authors must state which protocol was actually implemented and make the text, algorithms, and figure consistent.
- [§3.3, Algorithms 1–2] The abstract's blanket claim that 'co-distillation outperforms other federated methods in handling class imbalance' is not supported by the paper's own results at low skew. In Table 1 (4 clients, APTOS), CD is below FedProto and FedAMP at 0% skew and below FedProto at 40% skew. In Table 2 (6 clients, APTOS), CD is below FedProto, FedAMP, and FedDistill at both 0% and 20% skew. The paper should qualify the claim to high-skew settings (e.g., 40–60%) or state the regimes where co-distillation is not the best, rather than asserting general outperformance.
- [§5, Tables 1–4] The hyperparameters λ (co-distillation coefficient) and k (number of teacher samples) are never given numerical values or a selection procedure. These parameters control the strength of the distillation signal and the communication cost, so the experiments are not reproducible without them. Moreover, if these values were chosen using the test data, the reported comparisons would be biased. The authors must report the values used and describe how they were selected.
- [§5.1] The paper states that 'We report the average of three independent iterations' but reports only a single sd per method computed across skew levels. No per-condition means or standard deviations are provided, so it is impossible to assess whether the accuracy differences between CD and the baselines (e.g., CD vs FedProto at 40% skew in Table 1) are statistically meaningful or within run-to-run noise. The authors should report per-cell averages and standard deviations (or confidence intervals) for each experimental condition.
minor comments (5)
- [§4.1 and Table captions] There is a typo: 'completely loosing classification ability' should read 'completely losing classification ability'.
- [Introduction and §3.2] The relationship between 'non-expertise class' in the table captions and 'minority class' in Section 4.2 should be made explicit, and the captions should clarify whether the sample sizes (e.g., '600 images for APTOS and 200 images for COVID per each class') refer to per-client or total dataset sizes.
- [References] The paper claims reduced communication overhead relative to model-parameter sharing but provides no communication-cost measurements. Either include such measurements or soften the claim.
- [Figure 1] Reference [12] and [13] are duplicate entries for the same paper, and references [22] and [23] also appear to be the same work (FedKD / Communication-efficient federated learning via knowledge distillation). The citation list should be deduplicated.
- [§3.3] Figure 1 is too low-resolution to read the text; the diagram should be redrawn with legible labels.
Circularity Check
No significant circularity: the comparison is empirical against external baselines, with no equation or fitted parameter reducing to the claimed result.
full rationale
The paper makes no formal derivation claim. Its central assertion—that co-distillation achieves higher minority-class accuracy and lower variance than FedAvg, FedProto, FedAMP, and FedDistill under increasing skew—is supported by Tables 1–4, which report accuracies for all methods under identical skew and image-count conditions. The method's objective (Algorithm 2: CE(Y,M(X)) + lambda*MSE(feature(x), R)) is not defined in terms of the reported outcome; the teacher representation R is computed from a sampled subset of the teacher's own majority class and is external to the student's loss. No parameter is fitted to the headline metric, no prior result by these authors is invoked to forbid alternatives, and the baseline comparisons are external to the paper. The most substantial concerns—the absence of any stated train/test split in Section 4.2 and the inconsistency between the text (soft targets/logits) and Algorithms 1–2 (averaged feature vectors with MSE)—bear on validity and reproducibility rather than circularity, because they do not establish that any reported result is equivalent to its input by construction. Accordingly, the circularity score is 0.
Assumptions & free parameters
free parameters (2)
- co-distillation coefficient lambda =
not reported
- teacher sample count k =
not reported
assumptions (3)
- domain assumption Each client's majority class is its 'expertise class', and its averaged feature representation is a reliable distillation target for other clients.
- domain assumption Sharing averaged feature representations (or soft labels) preserves privacy relative to sharing model parameters.
- domain assumption The reported accuracy numbers come from a held-out test set.
Cite this review
Pith. "Pith review of Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcare." pith.science (2026). https://pith.science/paper/ZX3LZN7D
@misc{pith2026241110383,
author = {Pith},
title = {Pith review of: Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcare},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZX3LZN7D}},
note = {Machine review of arXiv:2411.10383}
}
read the original abstract
Federated Learning (FL) is a pioneering approach in distributed machine learning, enabling collaborative model training across multiple clients while retaining data privacy. However, the inherent heterogeneity due to imbalanced resource representations across multiple clients poses significant challenges, often introducing bias towards the majority class. This issue is particularly prevalent in healthcare settings, where hospitals acting as clients share medical images. To address class imbalance and reduce bias, we propose a co-distillation driven framework in a federated healthcare setting. Unlike traditional federated setups with a designated server client, our framework promotes knowledge sharing among clients to collectively improve learning outcomes. Our experiments demonstrate that in a federated healthcare setting, co-distillation outperforms other federated methods in handling class imbalance. Additionally, we demonstrate that our framework has the least standard deviation with increasing imbalance while outperforming other baselines, signifying the robustness of our framework for FL in healthcare.
Figures
Reference graph
Works this paper leans on
-
[1]
APTOS 2019 Blindness Detection. 2019. APTOS 2019 Blindness Detection . https://www.kaggle.com/competitions/aptos2019-blindness-detection/data
work page 2019
-
[2]
Theodora S Brisimi, Ruidi Chen, Theofanie Mela, Alex Olshevsky, Ioannis Ch Paschalidis, and Wei Shi. 2018. Federated learning of predictive models from federated electronic health records. International journal of medical informatics 112 (2018), 59–67
work page 2018
-
[3]
Xingjian Cao, Gang Sun, Hongfang Yu, and Mohsen Guizani. 2023. PerFED- GAN: Personalized Federated Learning via Generative Adversarial Networks. IEEE Internet of Things Journal 10, 5 (2023), 3749–3762. https://doi.org/10. 1109/JIOT.2022.3172114
arXiv 2023
-
[4]
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer
-
[5]
COVID-19 Radiography Database. 2020. COVID-19 Radiography Data- base. https://www.kaggle.com/datasets/tawsifurrahman/covid19-radiography- database
work page 2020
-
[6]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems 27 (2014)
2014
-
[7]
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. 2021. Personalized Cross-Silo Federated Learning on Non-IID Data. Proceedings of the AAAI Conference on Artificial Intelligence 35, 9 (May 2021), 7865–7873. https://doi.org/10.1609/aaai.v35i9.16960
-
[8]
Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324. https: //doi.org/10.1109/5.726791
doi:10.1109/5.726791 1998
Show all 26 references
-
[9]
Lin Li, Jianping Gou, Baosheng Yu, Lan Du, and Zhang Yiand Dacheng Tao. 2024. Federated Distillation: A Survey. arXiv preprint arXiv:2404.08564 (2024)
2024 arXiv
-
[10]
Mingchen Li, Xuechen Zhang, Christos Thrampoulidis, Jiasi Chen, and Samet Oymak. 2021. AutoBalance: Optimized Loss Functions for Imbalanced Data. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan (Eds.). https:/...
2021
-
[11]
Songtao Lu, Yawen Zhang, Yunlong Wang, and Christina Mack. 2019. Learn electronic health records by fully decentralized federated learning. arXiv preprint arXiv:1912.01792 (2019)
2019 arXiv
-
[12]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (Proceedings of ...
2017
-
[13]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep net- works from decentralized data. (2017), 1273–1282
2017
-
[14]
Phani Srivatsav Paladugu, Joshua Ong, Nicolas Nelson, Sharif Amit Kamran, Ethan Waisberg, Nasif Zaman, Rahul Kumar, Roger Daglius Dias, Andrew Go Lee, and Alireza Tavakkoli. 2023. Generative adversarial networks in medicine: important considerations for this emerging innovatio...
2023
-
[15]
Le, Avi Deb Raha, Apurba Adhikary, and Choong Seon Hong
Yu Qiao, Chaoning Zhang, Huy Q. Le, Avi Deb Raha, Apurba Adhikary, and Choong Seon Hong. 2023. Knowledge Distillation in Federated Learning: Where and How to Distill?. In 2023 24st Asia-Pacific Network Operations and Manage- ment Symposium (APNOMS). 18–23
2023
-
[16]
Hyowoon Seo, Jihong Park, Seungeun Oh, Mehdi Bennis, and Seong-Lyun Kim
-
[17]
Xian Shuai, Yulin Shen, Siyang Jiang, Zhihe Zhao, Zhenyu Yan, and Guoliang Xing. 2022. BalanceFL: Addressing class imbalance in long-tail federated learning. In 2022 21st ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 271–284
2022
-
[18]
Aliya Tabassum, Aiman Erbad, Wadha Lebda, Amr Mohamed, and Mohsen Guizani. 2022. Fedgan-ids: Privacy-preserving ids using gan and federated learn- ing. Computer Communications 192 (2022), 299–310
2022
-
[19]
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. 2022. Fedproto: Federated prototype learning across heteroge- neous clients. 36, 8 (2022), 8432–8440
2022
-
[20]
Lixu Wang, Shichao Xu, Xiao Wang, and Qi Zhu. 2021. Addressing class imbal- ance in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, V ol. 35. 10165–10173
2021
-
[21]
Yanhan Wang, Wenting Wang, Xin Wang, Heng Zhang, Xiaoming Wu, and Ming Yang. 2024. FedTweet: Two-fold Knowledge Distillation for non-IID Federated Learning. Computers and Electrical Engineering 114 (03 2024), 109067. https: //doi.org/10.1016/j.compeleceng.2023.109067
2024
-
[22]
Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. 2022. Communication-efficient federated learning via knowledge distillation. Nature communications 13, 1 (2022), 2032
2022
-
[23]
Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. 2022. Communication-efficient federated learning via knowledge distillation. Nature Communications 13 (04 2022), 2032. https://doi.org/10.1038/s41467-022-29763- x
2022 doi
-
[24]
Raneen Younis and Marco Fisichella. 2022. FLY-SMOTE: Re-balancing the non- IID iot edge devices data in federated learning system. IEEE Access 10 (2022), 65092–65102
2022
-
[2002]
Journal of artificial intelligence research 16 (2002), 321–357
SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16 (2002), 321–357
2002
-
[2022]
Machine Learning and Wireless Communications (2022), 457
16 federated knowledge distillation. Machine Learning and Wireless Communications (2022), 457
2022
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.