{"id":"7c5eac7f-1a12-46c7-a49d-b1d5b63426e8","arxiv_id":"2411.10383","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A co-distillation federated learning variant sharing majority-class feature averages is reported to keep minority-class accuracy higher than FedAvg, FedProto, FedAMP, and FedDistill on two medical imaging datasets under class skew.","lead":"A federated learning variant for hospitals is tested, where each client shares averaged feature representations of its majority class so other clients can learn that class better. The paper reports better minority-class accuracy and lower variance than four baselines on two medical image datasets, but the evaluation lacks a described held-out test split and error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.2 never defines a held-out test split; Tables 1–4 may report training-set minority-class accuracy, so the central 'co-distillation outperforms' claim is unsupported. This must be resolved before the headline result can stand.","rationale":"The paper aims to show that a co-distillation variant of federated learning improves minority-class accuracy and stability under increasing class imbalance. The evidence is entirely empirical, so the precondition for the central claim is that Tables 1–4 report generalization accuracy on data not used for training. Section 4.2 never establishes this; no train/test split is described, and the reported accuracy formula is compatible with training-set accuracy. This is the same weakness the reader identified, so I agree with the reader's assessment. I also note a secondary methodological inconsistency: the prose says soft labels are shared, while Algorithms 1–2 and Figure 1 share averaged feature vectors; this should be corrected but is not the primary blocker. I did not find independent support, such as machine-checked proofs or reproducible scripts with a clear split, that would offset the missing evaluation protocol. Therefore the reader's REJECT verdict should stand, and the proposed concrete check would confirm whether the concern actually lands.","tokens_in":8383,"tokens_out":4459,"duration_ms":44676,"concrete_test":"Clone the released repository (https://github.com/humairafirdowse/codistillation) and inspect the data-loading and training loop for a held-out test partition or split seed. If no such split exists, rerun the APTOS 4-client experiment at skew 0/20/40/60 with a stratified 80/20 train/test split and the paper's stated LeNet-5/100-epoch setup, recomputing minority-class accuracy on the untouched test folds. If CD no longer beats FedProto/FedDistill at the reported skew levels or its sd ranking changes, the central claim fails; if the code already contains a proper split, verify that Tables 1–4 report test-fold numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim rests entirely on the accuracy numbers in Tables 1–4. Section 4.2 states only 'We train all models for 100 epochs. We report the classification accuracy on minority class' and gives the formula total minority-class images correctly classified / total minority-class images, but it never says whether those images come from a held-out test set, a validation split, or the training set itself. Without an explicit split, the reported accuracies can measure memorization of the training distribution rather than generalization; the low standard deviations that support the robustness claim are then an artifact of fitting the same data. The associated GitHub repository is mentioned but not inspected, so this ambiguity is unresolved in the manuscript. A secondary issue is internal consistency: Section 3.2 and the Abstract say clients share soft targets/logits, while Algorithms 1–2 and Figure 1 share averaged feature vectors and compute MSE in feature space; this makes it unclear what was actually communicated and evaluated. But the missing held-out split is the most load-bearing concern because every quantitative conclusion depends on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a federated co-distillation framework for class-imbalanced healthcare image classification. Each client is assigned a majority 'expertise' class; during training, a student client randomly selects a teacher client and receives the teacher's average feature representation of its expertise class, then adds an MSE regularizer on its own samples of that class. The authors evaluate minority-class accuracy on APTOS and COVID-19 datasets with 4 and 6 clients, skew levels of 0–60%, and varying dataset sizes, comparing against FedAvg, FedProto, FedAMP, and FedDistill. They claim co-distillation outperforms these baselines and exhibits the lowest standard deviation as imbalance increases.","tokens_in":8565,"tokens_out":6796,"duration_ms":59253,"significance":"If the empirical claims are correct, the framework offers a simple, communication-efficient mechanism for private collaborative learning under class imbalance, with potential applicability to medical imaging. The paper includes a code release and provides algorithmic descriptions, which aid reproducibility. However, the significance is presently limited by a missing evaluation protocol and internal inconsistencies about what information is shared. The method is close in spirit to existing federated distillation approaches, and its contribution reduces to the specific choice of sharing only the teacher's expertise-class feature average; this is a modest conceptual step. The headline claims of 'outperforming' and 'least standard deviation' are not fully supported by the paper's own tables, and the lack of a described held-out test split means the quantitative conclusions are not yet established.","major_comments":[{"comment":"The evaluation section does not specify any train/test split. Section 4.2 states only that 'We train all models for 100 epochs' and defines accuracy as 'total minority class images correctly classified / total minority class images', without saying whether the images are from a held-out test set, a validation set, or the training set. Because the reported numbers are the sole evidence for every central claim (outperformance, robustness, low-resource behavior), the paper must specify the exact data split, how clients' local splits are constructed, and confirm that the reported accuracies are computed on held-out data. Otherwise the results may reflect training-set memorization rather than generalization.","section":"§3.2, §3.3, Algorithms 1–2, Figure 1"},{"comment":"There is a direct contradiction about what is communicated between clients. Section 3.2 says the teacher 'calculates the probabilistic outputs for these samples which are then transmitted' and Section 3.3 refers to 'soft targets' and 'logits', but Algorithm 1 returns only an average feature vector (AVG(feature(χ))) and Algorithm 2 computes an MSE loss between the student's and teacher's feature representations. Figure 1 likewise shows averaged representations. This is not merely a wording issue: sharing feature averages versus soft labels changes the privacy properties, the communication volume, and the actual training objective. The authors must state which protocol was actually implemented and make the text, algorithms, and figure consistent.","section":"§5.1, Tables 1 and 2"},{"comment":"The abstract's blanket claim that 'co-distillation outperforms other federated methods in handling class imbalance' is not supported by the paper's own results at low skew. In Table 1 (4 clients, APTOS), CD is below FedProto and FedAMP at 0% skew and below FedProto at 40% skew. In Table 2 (6 clients, APTOS), CD is below FedProto, FedAMP, and FedDistill at both 0% and 20% skew. The paper should qualify the claim to high-skew settings (e.g., 40–60%) or state the regimes where co-distillation is not the best, rather than asserting general outperformance.","section":"§3.3, Algorithms 1–2"},{"comment":"The hyperparameters λ (co-distillation coefficient) and k (number of teacher samples) are never given numerical values or a selection procedure. These parameters control the strength of the distillation signal and the communication cost, so the experiments are not reproducible without them. Moreover, if these values were chosen using the test data, the reported comparisons would be biased. The authors must report the values used and describe how they were selected.","section":"§5, Tables 1–4"},{"comment":"The paper states that 'We report the average of three independent iterations' but reports only a single sd per method computed across skew levels. No per-condition means or standard deviations are provided, so it is impossible to assess whether the accuracy differences between CD and the baselines (e.g., CD vs FedProto at 40% skew in Table 1) are statistically meaningful or within run-to-run noise. The authors should report per-cell averages and standard deviations (or confidence intervals) for each experimental condition.","section":"§5.1"}],"minor_comments":[{"comment":"There is a typo: 'completely loosing classification ability' should read 'completely losing classification ability'.","section":"§4.1 and Table captions"},{"comment":"The relationship between 'non-expertise class' in the table captions and 'minority class' in Section 4.2 should be made explicit, and the captions should clarify whether the sample sizes (e.g., '600 images for APTOS and 200 images for COVID per each class') refer to per-client or total dataset sizes.","section":"Introduction and §3.2"},{"comment":"The paper claims reduced communication overhead relative to model-parameter sharing but provides no communication-cost measurements. Either include such measurements or soften the claim.","section":"References"},{"comment":"Reference [12] and [13] are duplicate entries for the same paper, and references [22] and [23] also appear to be the same work (FedKD / Communication-efficient federated learning via knowledge distillation). The citation list should be deduplicated.","section":"Figure 1"},{"comment":"Figure 1 is too low-resolution to read the text; the diagram should be redrawn with legible labels.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The reader's report recommends rejection, and I agree that the evaluation is seriously incomplete. However, I see the issues as fixable within the manuscript's scope: the authors can rerun experiments with a proper held-out test split, clarify the shared representation, qualify the claims, and report hyperparameters. These are substantial changes but not a fundamental flaw in the proposed idea. I would suggest the editor ask for a major revision with a clear requirement that the evaluation protocol be described in detail and all tables recomputed on held-out data; if the authors cannot provide a test split, rejection would be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper has a simple, plausible idea — in federated learning with per-client class imbalance, have each student client randomly pick a teacher and align only on the teacher's majority-class averaged features. That is a modest but real twist on FedDistill/prototype methods, and the skew sweep on APTOS and COVID-19 is a reasonable way to evaluate it. There's a code link, and the communication cost is genuinely lower than parameter averaging. If the effect is real, it's useful for hospital collaborations.\n\nBut as it stands, the headline claim is not supported. Section 4.2 never says the reported accuracies come from a held-out test set. \"We report the classification accuracy on minority class\" with a formula over \"minority class images\" could mean training images. Without an explicit split, the numbers in Tables 1–4 might measure memorization, and the low standard deviations (which the paper sells as robustness) would be an artifact. This is the load-bearing issue, and it's fixable, but it has to be fixed before anyone can trust the results.\n\nThere's also an internal inconsistency: the text and abstract say clients share soft labels/logits, but Algorithms 1–2 and Figure 1 share averaged feature vectors and compute MSE in feature space. That's not a small wording issue — it changes what was actually communicated and evaluated. Hyperparameters λ and k are never reported, so the distillation strength and teacher sample count can't be checked. And the abstract's \"outperforms other federated methods\" is contradicted by the paper's own tables: FedProto wins on APTOS at 0% and 20% skew with 6 clients, and at 40% with 4 clients. The honest claim would be \"better on average at higher skew.\" The related work mentions class-imbalance-specific FL baselines (BalanceFL, FEDGAN-IDS, Ratio Loss) but none are compared.\n\nThe method isn't broken in concept, and the high-skew regime where co-distillation consistently wins is the one that matters most in practice. But the missing split and the feature/logit mismatch mean I can't tell whether the central result is real. This is a paper that needs a serious revision, not a desk-reject-then-forget: the protocol is concrete, the comparison is broad, and the open questions are answerable.\n\nFor a reading group, it's a good case study in how evaluation hygiene can sink an otherwise reasonable idea. I wouldn't cite it in its current form. If I were the editor I'd send it to review — a referee can ask for the split, the hyperparameters, and honest comparisons, and the authors seem capable of providing them.","headline":"A simple co-distillation variant for imbalanced federated learning with a plausible mechanism, but the missing test split and feature/logit inconsistency leave the central claim unverified.","tokens_in":9140,"tokens_out":3224,"would_cite":false,"duration_ms":30393,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In federated medical-image learning, a server-free co-distillation scheme that shares only each client's majority-class feature average keeps minority-class accuracy higher and more stable than FedAvg, FedProto, FedAMP, and FedDistill as…","keywords":["Federated Learning","Co-distillation","Class Imbalance","Healthcare","Medical Imaging","Knowledge Distillation","Data Heterogeneity","Minority Class Accuracy"],"falsifier":"Re-run the same 4-client and 6-client skew experiments with an explicitly documented held-out test set per client, and check whether co-distillation's minority-class accuracy advantage over FedAvg, FedProto, FedAMP, and FedDistill persists on unseen images; if the gap shrinks or reverses, the claimed advantage is an artifact of training-set evaluation.","tokens_in":8174,"feed_emoji":"🩺","tokens_out":8825,"duration_ms":71761,"temperature":0.7,"pith_summary":"The paper sets out to show that co-distillation, a federated learning scheme with no central server, handles class imbalance better than established federated baselines in medical-image settings. Each client acts as both teacher and student: a student randomly picks a teacher client, receives an averaged feature representation of the teacher's majority ('expertise') class, and adds a mean-squared-error distillation term to its own cross-entropy loss. The authors report that on APTOS retinal images and COVID-19 chest X-rays, this protocol keeps minority-class accuracy higher than FedAvg, FedProto, FedAMP, and FedDistill as class skew increases from 0% to 60%, with the lowest standard deviation across runs. The practical stake is that hospitals could collaborate on imbalanced private imaging data without sharing raw images or model weights.","feed_headline":"Co-distillation beats 4 federated baselines as class skew grows","feed_subtitle":"Minority-class accuracy holds up better and stays steadier as data skew grows.","key_machinery":"The load-bearing mechanism is the teacher–student co-distillation loop centered on each client's 'expertise class,' the majority class in that client's local data. A teacher client samples k images from its expertise class, computes their average feature representation, and shares only that vector as a soft target; the student adds an MSE loss between its own features for that class and the received average, weighted by a coefficient λ, on top of cross-entropy. Uniform random selection of the teacher each round lets every client's expertise propagate to every other client, so minority-class knowledge spreads across hospitals without exchanging model parameters or raw images.","core_discovery":"The authors' central claim is that a server-free co-distillation protocol is a reliable way to counter class imbalance in federated healthcare learning. In their binary diseased-versus-non-diseased setup, each client's majority class is its expertise; each round a student client samples one teacher uniformly at random, receives the averaged feature representation of that teacher's expertise class, and trains with the weighted combination of cross-entropy loss and an MSE loss that pulls the student's own features for that class toward the teacher's average. They report that this beats FedAvg, FedProto, FedAMP, and FedDistill in most tested settings, degrades less as skew grows (standard deviations of 0.03–0.10 for co-distillation versus 0.38–0.46 for FedAvg), and holds up when training images are scarce, such as 19.3% minority accuracy on COVID at 50 images and 60% skew while other methods fall to 0–4%.","pith_inferences":["Editorial inference: the design implies the method's benefit should scale with the complementarity of client specialties; a testable prediction is that co-distillation gains the most when hospitals specialize in different majority classes and gains less when their class distributions are similar.","Editorial inference: the reported stability under increasing skew suggests a follow-up study could test whether the random teacher selection can be replaced by a teacher-selection policy that prioritizes confident or balanced teachers, potentially lowering the standard deviation further.","Editorial inference: averaged feature vectors are shared rather than raw data, but the paper does not quantify how much class information those averages leak; a privacy audit of the soft targets is a natural next step before clinical deployment."],"forward_implications":["In a federated healthcare deployment with severe class skew, client models can preserve minority-class accuracy better than parameter-aggregation methods, which the paper reports FedAvg collapsing to near 0% at 40–60% skew.","Because only averaged feature vectors are exchanged, communication per round is smaller than full model updates, a practical advantage for bandwidth-limited hospital settings.","The reported advantage holds for both 4-client and 6-client setups, indicating the method does not depend on one particular cohort size.","At 60% skew with limited images, co-distillation keeps minority accuracy in the 79–86% range on APTOS and 19–90% on COVID depending on image count, while FedAvg records 0% in several configurations."],"supporting_citations":[{"why":"Defines FedAvg, the standard parameter-aggregation baseline whose collapse at high skew anchors the comparison.","marker":"[12]"},{"why":"Defines FedAMP, the personalized pairwise-cooperation baseline compared against co-distillation.","marker":"[7]"},{"why":"Defines FedProto, the prototype-aggregation baseline that occasionally beats co-distillation at low skew.","marker":"[19]"},{"why":"Defines federated knowledge distillation, the closest baseline; the key difference is it averages features over all classes, not the teacher's expertise class.","marker":"[16]"},{"why":"Supplies the LeNet-5 architecture used for every client and every baseline.","marker":"[8]"},{"why":"Supplies the APTOS retinal image dataset used for the imbalance experiments.","marker":"[1]"},{"why":"Supplies the COVID-19 Radiography dataset used for the imbalance experiments.","marker":"[5]"}],"fun_headline_variants":["Server-free co-distillation tames class skew in federated healthcare","Co-distillation stays steady as federated class imbalance grows","Co-distillation keeps minority accuracy steady as skew grows","Federated co-distillation proves robust to growing class imbalance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim stands on the unstated assumption that the reported accuracies are computed on images the model has not already trained on; the paper never describes a train/test split, so a reader cannot rule out that the tables measure memorization rather than generalization.","fun_headline_variants_meta":{"raw":{"variants":["Server-free co-distillation tames class skew in federated healthcare","Co-distillation stays steady as federated class imbalance grows","Co-distillation keeps minority accuracy steady as skew grows","Federated co-distillation proves robust to growing class imbalance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000896,"raw_usage":{"total_tokens":3833,"prompt_tokens":890,"completion_tokens":2943,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":2874}},"tokens_in":506,"tokens_out":2943,"duration_ms":20473,"temperature":1.0,"reasoning_tokens":2874,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:40:43.572873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 4-client and 6-client skew experiments with an explicitly documented held-out test set per client, and check whether co-distillation's minority-class accuracy advantage over FedAvg, FedProto, FedAMP, and FedDistill persists on unseen images; if the gap shrinks or reverses, the claimed advantage is an artifact of training-set evaluation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FedProto, the prototype-aggregation baseline that occasionally beats co-distillation at low skew."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines federated knowledge distillation, the closest baseline; the key difference is it averages features over all classes, not the teacher's expertise class."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the APTOS retinal image dataset used for the imbalance experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the COVID-19 Radiography dataset used for the imbalance experiments."}],"review_version":1}