Pith. sign in

REVIEW 5 major objections 5 minor 26 references

Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcare

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read In federated medical-image learning, a server-free co-distillation scheme that shares only each client's majority-class feature average keeps minority-class accuracy higher and more stable than FedAvg, FedProto, FedAMP, and FedDistill as…

desk verdict A simple co-distillation variant for imbalanced federated learning with a plausible mechanism, but the missing test split and feature/logit inconsistency leave the central claim unverified. read the letter →

arxiv 2411.10383 v1 pith:ZX3LZN7D submitted 2024-11-15 cs.LG

classification cs.LG
keywords FederatedLearningCo-distillationClassImbalanceHealthcareMedicalImagingKnowledgeDistillationDataHeterogeneityMinorityAccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that co-distillation, a federated learning scheme with no central server, handles class imbalance better than established federated baselines in medical-image settings. Each client acts as both teacher and student: a student randomly picks a teacher client, receives an averaged feature representation of the teacher's majority ('expertise') class, and adds a mean-squared-error distillation term to its own cross-entropy loss. The authors report that on APTOS retinal images and COVID-19 chest X-rays, this protocol keeps minority-class accuracy higher than FedAvg, FedProto, FedAMP, and FedDistill as class skew increases from 0% to 60%, with the lowest standard deviation across runs. The practical stake is that hospitals could collaborate on imbalanced private imaging data without sharing raw images or model weights.

What carries the argument

The load-bearing mechanism is the teacher–student co-distillation loop centered on each client's 'expertise class,' the majority class in that client's local data. A teacher client samples k images from its expertise class, computes their average feature representation, and shares only that vector as a soft target; the student adds an MSE loss between its own features for that class and the received average, weighted by a coefficient λ, on top of cross-entropy. Uniform random selection of the teacher each round lets every client's expertise propagate to every other client, so minority-class knowledge spreads across hospitals without exchanging model parameters or raw images.

What would settle it

Re-run the same 4-client and 6-client skew experiments with an explicitly documented held-out test set per client, and check whether co-distillation's minority-class accuracy advantage over FedAvg, FedProto, FedAMP, and FedDistill persists on unseen images; if the gap shrinks or reverses, the claimed advantage is an artifact of training-set evaluation.

Watch

Extended reading notes

Core claim

The authors' central claim is that a server-free co-distillation protocol is a reliable way to counter class imbalance in federated healthcare learning. In their binary diseased-versus-non-diseased setup, each client's majority class is its expertise; each round a student client samples one teacher uniformly at random, receives the averaged feature representation of that teacher's expertise class, and trains with the weighted combination of cross-entropy loss and an MSE loss that pulls the student's own features for that class toward the teacher's average. They report that this beats FedAvg, FedProto, FedAMP, and FedDistill in most tested settings, degrades less as skew grows (standard deviations of 0.03–0.10 for co-distillation versus 0.38–0.46 for FedAvg), and holds up when training images are scarce, such as 19.3% minority accuracy on COVID at 50 images and 60% skew while other methods fall to 0–4%.

Load-bearing premise

The central claim stands on the unstated assumption that the reported accuracies are computed on images the model has not already trained on; the paper never describes a train/test split, so a reader cannot rule out that the tables measure memorization rather than generalization.

Editorial extensions

If this is right

  • In a federated healthcare deployment with severe class skew, client models can preserve minority-class accuracy better than parameter-aggregation methods, which the paper reports FedAvg collapsing to near 0% at 40–60% skew.
  • Because only averaged feature vectors are exchanged, communication per round is smaller than full model updates, a practical advantage for bandwidth-limited hospital settings.
  • The reported advantage holds for both 4-client and 6-client setups, indicating the method does not depend on one particular cohort size.
  • At 60% skew with limited images, co-distillation keeps minority accuracy in the 79–86% range on APTOS and 19–90% on COVID depending on image count, while FedAvg records 0% in several configurations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the design implies the method's benefit should scale with the complementarity of client specialties; a testable prediction is that co-distillation gains the most when hospitals specialize in different majority classes and gains less when their class distributions are similar.
  • Editorial inference: the reported stability under increasing skew suggests a follow-up study could test whether the random teacher selection can be replaced by a teacher-selection policy that prioritizes confident or balanced teachers, potentially lowering the standard deviation further.
  • Editorial inference: averaged feature vectors are shared rather than raw data, but the paper does not quantify how much class information those averages leak; a privacy audit of the soft targets is a natural next step before clinical deployment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a federated co-distillation framework for class-imbalanced healthcare image classification. Each client is assigned a majority 'expertise' class; during training, a student client randomly selects a teacher client and receives the teacher's average feature representation of its expertise class, then adds an MSE regularizer on its own samples of that class. The authors evaluate minority-class accuracy on APTOS and COVID-19 datasets with 4 and 6 clients, skew levels of 0–60%, and varying dataset sizes, comparing against FedAvg, FedProto, FedAMP, and FedDistill. They claim co-distillation outperforms these baselines and exhibits the lowest standard deviation as imbalance increases.

Significance. If the empirical claims are correct, the framework offers a simple, communication-efficient mechanism for private collaborative learning under class imbalance, with potential applicability to medical imaging. The paper includes a code release and provides algorithmic descriptions, which aid reproducibility. However, the significance is presently limited by a missing evaluation protocol and internal inconsistencies about what information is shared. The method is close in spirit to existing federated distillation approaches, and its contribution reduces to the specific choice of sharing only the teacher's expertise-class feature average; this is a modest conceptual step. The headline claims of 'outperforming' and 'least standard deviation' are not fully supported by the paper's own tables, and the lack of a described held-out test split means the quantitative conclusions are not yet established.

major comments (5)
  1. [§3.2, §3.3, Algorithms 1–2, Figure 1] The evaluation section does not specify any train/test split. Section 4.2 states only that 'We train all models for 100 epochs' and defines accuracy as 'total minority class images correctly classified / total minority class images', without saying whether the images are from a held-out test set, a validation set, or the training set. Because the reported numbers are the sole evidence for every central claim (outperformance, robustness, low-resource behavior), the paper must specify the exact data split, how clients' local splits are constructed, and confirm that the reported accuracies are computed on held-out data. Otherwise the results may reflect training-set memorization rather than generalization.
  2. [§5.1, Tables 1 and 2] There is a direct contradiction about what is communicated between clients. Section 3.2 says the teacher 'calculates the probabilistic outputs for these samples which are then transmitted' and Section 3.3 refers to 'soft targets' and 'logits', but Algorithm 1 returns only an average feature vector (AVG(feature(χ))) and Algorithm 2 computes an MSE loss between the student's and teacher's feature representations. Figure 1 likewise shows averaged representations. This is not merely a wording issue: sharing feature averages versus soft labels changes the privacy properties, the communication volume, and the actual training objective. The authors must state which protocol was actually implemented and make the text, algorithms, and figure consistent.
  3. [§3.3, Algorithms 1–2] The abstract's blanket claim that 'co-distillation outperforms other federated methods in handling class imbalance' is not supported by the paper's own results at low skew. In Table 1 (4 clients, APTOS), CD is below FedProto and FedAMP at 0% skew and below FedProto at 40% skew. In Table 2 (6 clients, APTOS), CD is below FedProto, FedAMP, and FedDistill at both 0% and 20% skew. The paper should qualify the claim to high-skew settings (e.g., 40–60%) or state the regimes where co-distillation is not the best, rather than asserting general outperformance.
  4. [§5, Tables 1–4] The hyperparameters λ (co-distillation coefficient) and k (number of teacher samples) are never given numerical values or a selection procedure. These parameters control the strength of the distillation signal and the communication cost, so the experiments are not reproducible without them. Moreover, if these values were chosen using the test data, the reported comparisons would be biased. The authors must report the values used and describe how they were selected.
  5. [§5.1] The paper states that 'We report the average of three independent iterations' but reports only a single sd per method computed across skew levels. No per-condition means or standard deviations are provided, so it is impossible to assess whether the accuracy differences between CD and the baselines (e.g., CD vs FedProto at 40% skew in Table 1) are statistically meaningful or within run-to-run noise. The authors should report per-cell averages and standard deviations (or confidence intervals) for each experimental condition.
minor comments (5)
  1. [§4.1 and Table captions] There is a typo: 'completely loosing classification ability' should read 'completely losing classification ability'.
  2. [Introduction and §3.2] The relationship between 'non-expertise class' in the table captions and 'minority class' in Section 4.2 should be made explicit, and the captions should clarify whether the sample sizes (e.g., '600 images for APTOS and 200 images for COVID per each class') refer to per-client or total dataset sizes.
  3. [References] The paper claims reduced communication overhead relative to model-parameter sharing but provides no communication-cost measurements. Either include such measurements or soften the claim.
  4. [Figure 1] Reference [12] and [13] are duplicate entries for the same paper, and references [22] and [23] also appear to be the same work (FedKD / Communication-efficient federated learning via knowledge distillation). The citation list should be deduplicated.
  5. [§3.3] Figure 1 is too low-resolution to read the text; the diagram should be redrawn with legible labels.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the comparison is empirical against external baselines, with no equation or fitted parameter reducing to the claimed result.

full rationale

The paper makes no formal derivation claim. Its central assertion—that co-distillation achieves higher minority-class accuracy and lower variance than FedAvg, FedProto, FedAMP, and FedDistill under increasing skew—is supported by Tables 1–4, which report accuracies for all methods under identical skew and image-count conditions. The method's objective (Algorithm 2: CE(Y,M(X)) + lambda*MSE(feature(x), R)) is not defined in terms of the reported outcome; the teacher representation R is computed from a sampled subset of the teacher's own majority class and is external to the student's loss. No parameter is fitted to the headline metric, no prior result by these authors is invoked to forbid alternatives, and the baseline comparisons are external to the paper. The most substantial concerns—the absence of any stated train/test split in Section 4.2 and the inconsistency between the text (soft targets/logits) and Algorithms 1–2 (averaged feature vectors with MSE)—bear on validity and reproducibility rather than circularity, because they do not establish that any reported result is equivalent to its input by construction. Accordingly, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper's central empirical claim relies on several unstated modeling and evaluation assumptions: that teacher feature averages are useful distillation targets, that sharing them preserves privacy, and that the reported accuracies reflect held-out performance. No free parameters are fitted to the test set, but hyperparameters lambda and k are unspecified.

free parameters (2)
  • co-distillation coefficient lambda = not reported
    Weights the MSE distillation loss in Algorithm 2; its value can substantially change the teacher signal strength, yet no value or tuning protocol is given.
  • teacher sample count k = not reported
    Number of images sampled to compute the teacher's average representation in Algorithm 1; affects stability and information content of the distilled target.
assumptions (3)
  • domain assumption Each client's majority class is its 'expertise class', and its averaged feature representation is a reliable distillation target for other clients.
    Introduced in Section 3.2 and Algorithm 1; no evidence is given that class-conditional feature means are stable across clients.
  • domain assumption Sharing averaged feature representations (or soft labels) preserves privacy relative to sharing model parameters.
    Claimed in Sections 1 and 3.3 with no analysis; averaged features can still encode patient-specific information.
  • domain assumption The reported accuracy numbers come from a held-out test set.
    Section 4.2 describes training for 100 epochs and then reports accuracy without describing any split; the paper silently assumes generalization is measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcare." pith.science (2026). https://pith.science/paper/ZX3LZN7D

@misc{pith2026241110383,
  author       = {Pith},
  title        = {Pith review of: Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcare},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZX3LZN7D}},
  note         = {Machine review of arXiv:2411.10383}
}
read the original abstract

Federated Learning (FL) is a pioneering approach in distributed machine learning, enabling collaborative model training across multiple clients while retaining data privacy. However, the inherent heterogeneity due to imbalanced resource representations across multiple clients poses significant challenges, often introducing bias towards the majority class. This issue is particularly prevalent in healthcare settings, where hospitals acting as clients share medical images. To address class imbalance and reduce bias, we propose a co-distillation driven framework in a federated healthcare setting. Unlike traditional federated setups with a designated server client, our framework promotes knowledge sharing among clients to collectively improve learning outcomes. Our experiments demonstrate that in a federated healthcare setting, co-distillation outperforms other federated methods in handling class imbalance. Additionally, we demonstrate that our framework has the least standard deviation with increasing imbalance while outperforming other baselines, signifying the robustness of our framework for FL in healthcare.

Figures

Figures reproduced from arXiv: 2411.10383 by the authors.

Figure 1
Figure 1. Student Client randomly select another client as a teacher which shares averaged representations of their expertise class, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 15 canonical work pages

  1. [1]

    APTOS 2019 Blindness Detection. 2019. APTOS 2019 Blindness Detection . https://www.kaggle.com/competitions/aptos2019-blindness-detection/data

  2. [2]

    Theodora S Brisimi, Ruidi Chen, Theofanie Mela, Alex Olshevsky, Ioannis Ch Paschalidis, and Wei Shi. 2018. Federated learning of predictive models from federated electronic health records. International journal of medical informatics 112 (2018), 59–67

  3. [3]

    Xingjian Cao, Gang Sun, Hongfang Yu, and Mohsen Guizani. 2023. PerFED- GAN: Personalized Federated Learning via Generative Adversarial Networks. IEEE Internet of Things Journal 10, 5 (2023), 3749–3762. https://doi.org/10. 1109/JIOT.2022.3172114

  4. [4]

    Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer

  5. [5]

    COVID-19 Radiography Database. 2020. COVID-19 Radiography Data- base. https://www.kaggle.com/datasets/tawsifurrahman/covid19-radiography- database

  6. [6]

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems 27 (2014)

  7. [7]

    Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. 2021. Personalized Cross-Silo Federated Learning on Non-IID Data. Proceedings of the AAAI Conference on Artificial Intelligence 35, 9 (May 2021), 7865–7873. https://doi.org/10.1609/aaai.v35i9.16960

  8. [8]

    Lecun, L

    Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324. https: //doi.org/10.1109/5.726791

Show all 26 references
  1. [9]

    Lin Li, Jianping Gou, Baosheng Yu, Lan Du, and Zhang Yiand Dacheng Tao. 2024. Federated Distillation: A Survey. arXiv preprint arXiv:2404.08564 (2024)

  2. [10]

    Mingchen Li, Xuechen Zhang, Christos Thrampoulidis, Jiasi Chen, and Samet Oymak. 2021. AutoBalance: Optimized Loss Functions for Imbalanced Data. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan (Eds.). https:/...

  3. [11]

    Songtao Lu, Yawen Zhang, Yunlong Wang, and Christina Mack. 2019. Learn electronic health records by fully decentralized federated learning. arXiv preprint arXiv:1912.01792 (2019)

  4. [12]

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (Proceedings of ...

  5. [13]

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep net- works from decentralized data. (2017), 1273–1282

  6. [14]

    Phani Srivatsav Paladugu, Joshua Ong, Nicolas Nelson, Sharif Amit Kamran, Ethan Waisberg, Nasif Zaman, Rahul Kumar, Roger Daglius Dias, Andrew Go Lee, and Alireza Tavakkoli. 2023. Generative adversarial networks in medicine: important considerations for this emerging innovatio...

  7. [15]

    Le, Avi Deb Raha, Apurba Adhikary, and Choong Seon Hong

    Yu Qiao, Chaoning Zhang, Huy Q. Le, Avi Deb Raha, Apurba Adhikary, and Choong Seon Hong. 2023. Knowledge Distillation in Federated Learning: Where and How to Distill?. In 2023 24st Asia-Pacific Network Operations and Manage- ment Symposium (APNOMS). 18–23

  8. [16]

    Hyowoon Seo, Jihong Park, Seungeun Oh, Mehdi Bennis, and Seong-Lyun Kim

  9. [17]

    Xian Shuai, Yulin Shen, Siyang Jiang, Zhihe Zhao, Zhenyu Yan, and Guoliang Xing. 2022. BalanceFL: Addressing class imbalance in long-tail federated learning. In 2022 21st ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 271–284

  10. [18]

    Aliya Tabassum, Aiman Erbad, Wadha Lebda, Amr Mohamed, and Mohsen Guizani. 2022. Fedgan-ids: Privacy-preserving ids using gan and federated learn- ing. Computer Communications 192 (2022), 299–310

  11. [19]

    Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. 2022. Fedproto: Federated prototype learning across heteroge- neous clients. 36, 8 (2022), 8432–8440

  12. [20]

    Lixu Wang, Shichao Xu, Xiao Wang, and Qi Zhu. 2021. Addressing class imbal- ance in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, V ol. 35. 10165–10173

  13. [21]

    Yanhan Wang, Wenting Wang, Xin Wang, Heng Zhang, Xiaoming Wu, and Ming Yang. 2024. FedTweet: Two-fold Knowledge Distillation for non-IID Federated Learning. Computers and Electrical Engineering 114 (03 2024), 109067. https: //doi.org/10.1016/j.compeleceng.2023.109067

  14. [22]

    Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. 2022. Communication-efficient federated learning via knowledge distillation. Nature communications 13, 1 (2022), 2032

  15. [23]

    Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. 2022. Communication-efficient federated learning via knowledge distillation. Nature Communications 13 (04 2022), 2032. https://doi.org/10.1038/s41467-022-29763- x

  16. [24]

    Raneen Younis and Marco Fisichella. 2022. FLY-SMOTE: Re-balancing the non- IID iot edge devices data in federated learning system. IEEE Access 10 (2022), 65092–65102

  17. [2002]

    Journal of artificial intelligence research 16 (2002), 321–357

    SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16 (2002), 321–357

  18. [2022]

    Machine Learning and Wireless Communications (2022), 457

    16 federated knowledge distillation. Machine Learning and Wireless Communications (2022), 457

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.