{"id":"b7213efa-b838-4cab-b4ba-ce734e9e1a78","arxiv_id":"2411.15403","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Weak classes that are intrinsically confusable persist under class-balanced federated learning; a partial knowledge distillation method that distills from class-specific experts improves their accuracy.","lead":"This paper shows that in federated learning, some classes are consistently harder to recognize than others, even when every class has the same number of training samples. It proposes a selective knowledge-distillation method that improves accuracy for these weak classes by up to 10.7% without hurting overall accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FashionMNIST PKD gain may be offset: Table 1 arithmetic shows the other 8 classes lose ~10 points total at #clients=100; per-class accuracies needed to support the ICD claim.","rationale":"Reader's weakest assumption (expert proficiency) is plausible, but the more immediate checkable risk is the per-class trade-off implied by Table 1. The reported average and min/max are insufficient to support the claim that dominant classes are unchanged. Our arithmetic is robust: sum of class accuracies is 10*Ave; we isolate the eight other classes. For FashionMNIST #clients=100, these eight classes lose 10.0 points in aggregate. If confirmed, the method's benefit is a redistribution, not an absolute gain. This strengthens the need for the conditional verdict: authors must release per-class accuracy data and code. We therefore keep the reader's CONDITIONAL verdict (UNCHANGED) but add a specific acceptance criterion. Agreement is partial, since our concern is a concrete failure mode of the distillation transfer the reader flagged.","tokens_in":18287,"tokens_out":10450,"duration_ms":92156,"concrete_test":"Obtain the per-class accuracy table for the local-class-balanced setups in Table 1 (FashionMNIST 10 and 100 clients, FedAvg vs PKD), e.g., from the authors' code or by re-running the released experiments. Compute the change in each class. If the sum of the eight non-reported classes decreases by more than, say, 2 points in the #clients=100 setup while the min gains 10.7, then the 'inherent ICD effectively alleviated' claim fails and the paper should report this trade-off explicitly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that PKD improves the weakest class without degrading others (Section 4.2: 'the accuracy of dominant classes remains largely unchanged or is even slightly improved'). The aggregate numbers in Table 1 contradict this for the headline FashionMNIST result. For #clients=100, FedAvg has Ave 84.66 (sum 846.6) and PKD Ave 84.72 (sum 847.2), total change +0.6 points. The min class gains 10.7 (51.6→62.3) and the max class loses 0.1 (96.7→96.6). Therefore the remaining eight classes together must change by +0.6 - 10.7 + 0.1 = -10.0 points, an average loss of about 1.25 points per class. The same pattern appears at #clients=10: the eight non-extreme classes lose about 6.4 points total while min gains 8.2. Thus the reported 10.7% weak-class improvement is offset by a comparable aggregate drop in the other classes. This raises the possibility that PKD is not resolving the inherent inter-class discrepancy but merely shifting which classes are worst, exactly the failure mode the authors attribute to re-weighting in Section 4.3. The paper does not report per-class accuracies, so this trade-off is hidden. This is more directly load-bearing than the expert-proficiency assumption: even if the expert is perfectly proficient, the selective KD loss can over-index on the weakest class and degrade others.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies class-wise accuracy disparities in federated learning under globally and locally balanced class distributions. The authors show that certain \"weak classes\" are consistently harder to classify across architectures and partitioning strategies, attribute this to high-level feature similarity among classes, and propose a two-expert partial knowledge distillation (PKD) pipeline: after a warmup stage, weak-class groups are identified from local confusion patterns, experts are trained on samples from those groups via FL, and during subsequent FL training a selective KL-divergence distillation loss is applied only when the student misclassifies a sample as another class in the same weak-class group. Experiments on MNIST, FashionMNIST, CIFAR-10/100, and FEMNIST report improvements in the minimum class-wise accuracy (up to 10.7% on FashionMNIST) with roughly preserved average accuracy.","tokens_in":18617,"tokens_out":6491,"duration_ms":52050,"significance":"The phenomenon of inherent inter-class discrepancy, if confirmed, is a genuinely interesting observation with practical implications for fairness in FL. The paper's strengths include: the phenomenon is documented across multiple datasets, network architectures, and partition strategies; the proposed PKD is simple and the overhead analysis (Section 3.3, Tables 9-10) suggests a favorable compute-accuracy trade-off; and the comparison against re-weighting and data augmentation (Table 6) supports the claim that standard imbalance remedies do not solve the problem. The main weakness is that the headline claim--improving weak classes without hurting others--is not supported by the aggregate numbers in Table 1; per-class results and variance estimates are needed. As presented, the evidence is consistent with the alternative explanation that PKD merely shifts which classes are worst rather than resolving the inherent discrepancy.","major_comments":[{"comment":"The aggregate numbers in Table 1 contradict the claim that dominant classes remain largely unchanged. For FashionMNIST with 100 clients, FedAvg has an average accuracy of 84.66 and PKD 84.72, a total change of +0.6 points across the ten classes; the minimum class gains 10.7 points (51.6 to 62.3) while the maximum loses 0.1 points. The remaining eight classes must therefore lose 10.0 points cumulatively, roughly 1.25 points per class on average. The same arithmetic holds for the 10-client row: total change +2.1 = +0.3 (max) + 8.2 (min) + (sum of the other eight), so the other eight classes lose about 6.4 points. This directly contradicts the statement in Section 4.2 that \"the accuracy of dominant classes remains largely unchanged or is even slightly improved.\" Because the paper reports only max/average/min and never per-class accuracies, it does not rule out that PKD is shifting the inter-class discrepancy rather than reducing it. Please report full per-class accuracy tables, with standard deviations, for at least the main setups in Tables 1 and 4.","section":"Section 4.2, Table 1"},{"comment":"The magnitude of the headline discrepancy is reported inconsistently. The abstract says \"over 36.9%\", the introduction says \"up to 45.4% on the FashionMNIST dataset and 36.9% on the CIFAR-10 dataset\", Fig. 2 shows Delta_max = 45.4 for FashionMNIST under FL with local class balance, but Table 1 (10 clients) gives max - min = 97.30 - 68.40 = 28.9%, and Fig. 3 gives Delta = 45.1 for the 100-client case. It appears that 45.4 comes from a different architecture (ConvNet-L) than the LeNet-5 used in Table 1; please state explicitly which configuration each reported discrepancy refers to and present one consistent set of numbers for the main setup.","section":"Section 2.2, Fig. 2 vs Table 1 and Fig. 3"},{"comment":"The method's central motivation is that \"an expert trained specifically on a group of confounding weak classes shows greater proficiency in differentiating among them\" (Section 1). This premise is never directly demonstrated: no experiment reports the expert models' class-wise accuracy on the weak-class groups, nor the quality of the distilled soft labels (p_e in Algorithm 2, Line 17). If the expert is not more accurate than the global model on these classes, the reported improvements cannot be attributed to the claimed mechanism. Please add expert accuracy results and/or an ablation that replaces the trained expert with temperature-raised soft labels obtained from the global model itself.","section":"Section 1 and Section 3.1, Algorithm 1 Stage 2"},{"comment":"The method introduces several hyperparameters (lambda, T, T1, T2, T3, and G for CIFAR-100) that are set to fixed values without a sensitivity study. Given that the reported average accuracy changes are small (e.g., +0.06 on FashionMNIST with 100 clients in Table 1), it is important to know whether the weak-class gains persist across a reasonable range of these hyperparameters and whether the trade-off identified in the first major comment remains. Please include an analysis varying at least lambda, T, and the warmup/expert-training durations.","section":"Section 4.1, Hyperparameters"}],"minor_comments":[{"comment":"The phrase \"without comprising the global average accuracy\" should read \"without compromising the global average accuracy.\"","section":"Section 4.2"},{"comment":"The text says \"Fig. 4(a) assesses the similarities in high-level features between each pair of classes,\" but in the figure the pairwise distance matrix appears to be panel (b), not (a). Please correct the cross-reference.","section":"Section 2.2, Fig. 4"},{"comment":"The manuscript contains an inserted Chinese annotation in the Figure 3 caption area (\"这张图和table1 现在134图都可以对应上...\") as well as other small annotations such as \"加了class\" and \"50轮\". These appear to be leftover internal notes and must be removed before submission; as they stand, they are unprofessional and could confuse readers.","section":"Figure 3 caption area"},{"comment":"The layout of Table 8, with \"FedAvg PKD\" followed by separate Delta columns for G = 1 and G = 2, is unclear; consider presenting G = 1 and G = 2 as separate method columns or rows.","section":"Table 8"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be formatted as an ACM conference submission (the first page includes 'Conference acronym XX' and an anonymous author block). The editor may wish to ask the authors to reformat for the journal and to remove the leftover internal annotations. No code or data release is mentioned; given the arithmetic concerns about Table 1, the authors should be asked to provide the full per-class results and standard deviations. The phenomenon and method are interesting enough to warrant a major revision rather than rejection, provided the per-class trade-off is resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth a look, but not for the reason the authors advertise. The real contribution is the demonstration that certain classes stay weak under class-balanced federated training, and that the confusion is structured (T-shirt/shirt, pullover/coat/shirt). That part is credible and consistent across architectures and partitions. The proposed PKD method — train class-group experts and use their soft labels only on misclassifications within the group — is neat, cheap, and moves the worst class up by 10.7% on FashionMNIST and 6.3% on CIFAR-10. The overhead analysis is honest and useful.\n\nNow the soft spots. First, the claim in Section 4.2 that 'the accuracy of dominant classes remains largely unchanged or is even slightly improved' does not survive the arithmetic of Table 1. For FashionMNIST at #clients=100, the average moves +0.06, the min class +10.7, the max -0.1. That forces the other eight classes to lose about 10 points combined, roughly 1.25 points each. So PKD is not a free lunch; it appears to trade accuracy of the stronger classes for the weakest. That doesn't kill the method — reducing the max-min gap from 45.1 to 34.3 is still something — but the paper must report per-class numbers and own the trade-off. The current text hides it.\n\nSecond, no code, no seeds, no standard deviations. Five runs averaged is fine if the variance is small, but they don't show it. Third, the manuscript has an internal Chinese note in the Figure 3 caption and some inconsistency in headline discrepancy values (45.4% vs 28.9% vs 36.9%) because they come from different configurations. That's fixable but sloppy.\n\nThe expert-proficiency assumption is a real dependency, but it's tested on four datasets and a natural partition, so I don't see it as the load-bearing problem. The hidden trade-off is.\n\nWho is this for? FL researchers working on fairness and class imbalance. The phenomenon deserves a serious referee. I'd send it to review, but with a strong request: give per-class accuracy, error bars, and code. Until then, treat the headline improvement as an upper bound on the benefit, not the full picture.\n\nMy take: engage with it, but don't cite the 'no degradation' claim yet.","headline":"A real FL fairness phenomenon and a plausible fix, but the claim of 'no degradation elsewhere' is contradicted by the paper's own Table 1 arithmetic.","tokens_in":19134,"tokens_out":3403,"would_cite":false,"duration_ms":30929,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Even with perfectly balanced federated data, certain classes stay consistently weak; selective expert distillation raises worst-class accuracy by up to 10.7% without lowering the global average.","keywords":["federated learning","inter-class accuracy discrepancy","weak classes","knowledge distillation","class imbalance","partial knowledge distillation","class-balanced learning"],"falsifier":"Run a controlled experiment where the expert is trained on the same weak classes plus an equal number of samples from other classes, then distill it in the same PKD pipeline; if the student's weak-class improvement matches the original PKD, the 'class-specific expert' premise is unnecessary. Alternatively, find a benchmark where the set of worst classes changes across random seeds or architectures, which would contradict the claimed consistency of weak classes.","tokens_in":58,"feed_emoji":"🎯","tokens_out":6340,"duration_ms":99193,"temperature":0.7,"pith_summary":"This paper establishes that certain classes are consistently hard for federated learning models even when the training data is perfectly balanced, both globally across clients and locally within each client. These 'weak classes' are not minority classes: they are inherent to the dataset, and the gap between the best and worst class accuracy can exceed 36% on FashionMNIST and CIFAR-10. The authors trace the cause to high-level visual feature similarity among specific class pairs, which makes the classifier confuse them. They then propose a partial knowledge distillation method in which class-specific expert models, trained only on the weak classes, correct the global model's predictions, but only when a misclassification among those classes actually occurs. The result is a 10.7% improvement in worst-class accuracy on FashionMNIST and 6.3% on CIFAR-10 without reducing global average accuracy.","feed_headline":"10.7% accuracy gain for weak classes via partial distillation","feed_subtitle":"Class-balanced FL still has weak classes; PKD recovers them without lowering average accuracy.","key_machinery":"The central mechanism is Partial Knowledge Distillation (PKD), a three-stage federated pipeline. Stage 1 is standard FedAvg warmup; Stage 2 identifies weak-class groups from the confusion matrix of softmax outputs and trains one expert model per group on only that group's samples, using the same federated averaging; Stage 3 runs FedAvg again but, whenever the student's prediction for a sample is a misclassification among classes within a known weak group, the corresponding expert's temperature-scaled softmax on those classes is used as a target, and the student updates with an added KL-divergence loss between its own and the expert's probabilities. The trigger condition and the restriction to one expert per sample are what make the distillation 'partial', and the loss is zero for all other samples, keeping the added computation small.","core_discovery":"The central claim is that the inter-class accuracy discrepancy in federated learning is not merely a symptom of imbalanced sample sizes; a set of 'weak' classes persists even under globally and locally balanced class distributions, and this set is stable across network architectures, learning paradigms, and data partitioning schemes. The weak classes—such as T-shirt/shirt and pullover/coat/shirt in FashionMNIST, or bird/cat/deer/dog in CIFAR-10—share similar high-level feature representations, so the model's mistakes are concentrated among them, with up to 88% of a weak class's misclassifications landing on other weak classes. The paper argues that an expert model trained only on the samples of one such weak-class group learns to separate those confusable classes better than a model trained on all classes, and that selectively distilling this expert's soft labels into the global student model—only when the student misclassifies a sample within that group—raises the weakest class accuracy while leaving average accuracy essentially unchanged.","pith_inferences":["Because the weak classes are defined by feature similarity rather than sample count, the same PKD trigger logic could be applied in centralized learning, where the paper's own measurements show a smaller but still present discrepancy.","The expert-in-the-loop design is a form of conditional ensemble: the cost is low because the expert is consulted only on misclassified weak-group samples, so the approach could be combined with existing heterogeneity-robust FL optimizers rather than replacing them.","One testable extension is to replace the misclassification trigger with a confidence threshold, which would let the method fire on correct-but-uncertain predictions and possibly improve calibration rather than only accuracy.","If the observed confusion pairs reflect intrinsic dataset structure, then a pretrained feature extractor could predict the weak-class groups in a new dataset before any federated training begins."],"forward_implications":["If the inherent ICD is real, then class-rebalancing and augmentation methods that assume sample scarcity will not fix weak-class performance; methods must target the feature confusion itself.","PKD can be added to any FedAvg-like federated training loop and improves the worst-performing class by 8.2–12.3 points under balanced partitions and by larger margins under pathological partitions, without lowering average accuracy.","The same weak-class groups appear across different networks and data splits, so the identification step can be done once after a short warmup and reused.","The method also transfers to realistic non-IID partitions (Dirichlet, FEMNIST native) and to larger label sets (CIFAR-100), where selecting 1–2 worst groups still yields 3–5 point gains in minimum class accuracy."],"supporting_citations":[{"why":"Supplies the FedAvg baseline and federated averaging framework that PKD builds on.","marker":"[19]"},{"why":"Provides the temperature-scaled softmax and KL-divergence formulation used in the partial distillation loss.","marker":"[10]"},{"why":"FashionMNIST is the primary benchmark where the weak-class phenomenon and the 10.7% gain are demonstrated.","marker":"[28]"},{"why":"CIFAR-10 is the second main benchmark, with weak classes bird/cat/deer/dog and a 6.3% gain.","marker":"[12]"},{"why":"Defines MNIST and the LeNet-5 architecture used for the digit experiments.","marker":"[13]"},{"why":"Focal loss serves as a representative class-imbalance method that PKD outperforms on weak-class accuracy.","marker":"[16]"},{"why":"BalanceFL, a feature-level augmentation technique, is the closest prior ICD mitigation baseline and is surpassed by PKD.","marker":"[24]"},{"why":"FedCodl, a knowledge-distillation method for heterogeneity, is compared under the Dirichlet partition.","marker":"[21]"},{"why":"FedProx, a proximal-update heterogeneity method, is a comparison baseline under the Dirichlet partition.","marker":"[23]"},{"why":"FEMNIST provides the realistic native writer-based partition used for evaluation with partial client participation.","marker":"[2]"}],"fun_headline_variants":["Partial distillation lifts weak-class accuracy by 10.7% in FL","Weak classes persist in balanced FL—selective distillation fixes them","PKD: Boost weak classes in federated learning without average loss","Federated learning's hidden weak spots: a 10.7% boost via distillation","Distillation targets inherent weak classes in federated learning"],"cache_read_input_tokens":21248,"weakest_assumption_plain":"The method works only if an expert model trained solely on a few confusing classes is genuinely better at telling those classes apart than the full model, and if that skill transfers back through the selective distillation loss during federated training.","fun_headline_variants_meta":{"raw":{"variants":["Partial distillation lifts weak-class accuracy by 10.7% in FL","Weak classes persist in balanced FL—selective distillation fixes them","PKD: Boost weak classes in federated learning without average loss","Federated learning's hidden weak spots: a 10.7% boost via distillation","Distillation targets inherent weak classes in federated learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000369,"raw_usage":{"total_tokens":1970,"prompt_tokens":930,"completion_tokens":1040,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":947}},"tokens_in":546,"tokens_out":1040,"duration_ms":9308,"temperature":1.0,"reasoning_tokens":947,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:20:24.486913+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled experiment where the expert is trained on the same weak classes plus an equal number of samples from other classes, then distill it in the same PKD pipeline; if the student's weak-class improvement matches the original PKD, the 'class-specific expert' premise is unnecessary. Alternatively, find a benchmark where the set of worst classes changes across random seeds or architectures, which would contradict the claimed consistency of weak classes.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CIFAR-10 is the second main benchmark, with weak classes bird/cat/deer/dog and a 6.3% gain."},{"cited_title":"Lecun, L","cited_arxiv_id":null,"evidence_quote":"Defines MNIST and the LeNet-5 architecture used for the digit experiments."}],"review_version":1}