{"id":"bef528f1-db19-4036-9f12-7f9dfad7182c","arxiv_id":"2505.09733","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A federated learning pipeline that combines confidence-based label cleaning, collaborative conditional GAN training, and FedProx improves macro-F1 over noisy FedAvg and FedProx baselines on MNIST and Fashion-MNIST, with cleaning alone responsible for most of the improvement.","lead":"This paper proposes a three-stage federated learning pipeline that cleans noisy labels locally, trains a shared conditional GAN across clients, and fills in missing classes with synthetic images before training a global classifier. Reported experiments on MNIST and Fashion-MNIST indicate that confidence-based cleaning alone drives most of the F1 gains, while the GAN-based step adds little in most settings.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1) adds raw entropy to a 'confidence' score, so the cleaning stage preferentially retains the uncertain samples it is supposed to remove; this is load-bearing because cleaning drives the reported gains.","rationale":"The paper's central empirical claim is that the three-stage pipeline, especially local confidence-based cleaning, makes FedAvg and FedProx robust to label noise and missing classes. The results show that the cleaning-only variants (CleanAvg and CleanProx) are the top performers and that the GAN stage is frequently not beneficial, so the filtering stage must carry the argument. The only precise definition of the filter is Algorithm 1 with Eq. (1). In that definition, C_ent is not a confidence; it is an uncertainty. Since the aggregate score is a sum and retention is C_agg ≥ T, the procedure described would preferentially keep high-entropy samples. That is exactly the opposite of the noise-cleaning hypothesis. The issue is not a minor typo in prose: it is in the formal algorithm and the mathematical equation, and no code or ablation is supplied to show which version was actually run. Therefore the empirical support for the strongest claim is presently equivocal: either the implementation silently corrected the sign (then the paper misdescribes its method), or it did not (then the improvement mechanism is not the one claimed). Both possibilities require revision. This justifies the reader's conditional verdict and does not change it.","tokens_in":19707,"tokens_out":3659,"duration_ms":37623,"concrete_test":"Implement Algorithm 1 exactly as written on one MNIST client with 30% injected asymmetric label noise. For each sample record whether its true label was corrupted and compute C_agg from Eq. (1). If the mean C_agg of corrupted samples is not below that of clean samples, or if the retained set is enriched for corrupted labels, the formula is inverted; rerun the comparison with C_ent replaced by 1 - H(p)/log C to see whether Table II's 'cleaning-only' results are sensitive to the sign. This directly settles whether the reported gains come from the described confidence filter.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The cleaning stage is the main source of reported improvement, and Equation (1) is its only formal scoring rule. As written, C_ent(x_i) = -Σ_c p_c log p_c is raw entropy, which is large for uncertain predictions; C_agg is the arithmetic mean of entropy, margin, and silhouette score, and Algorithm 1 retains samples with C_agg ≥ T. High-entropy samples therefore receive a higher aggregate score and are preferentially kept, while confident samples are more likely to be discarded. This inverts the stated purpose of removing noisy or ambiguous instances. Because Table II and the conclusion attribute most of the gain to cleaning (the GAN component is often neutral or harmful), the headline empirical claim depends on either an uncorrected implementation of Eq. (1) or a mechanism opposite to the one described. The threshold in Eq. (2) does not repair the sign error.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-stage federated learning pipeline designed to mitigate label noise, missing classes, and class imbalance. Stage 1 is a client-level confidence-based noise cleaning step that computes entropy, margin, and silhouette scores for each sample, aggregates them, and filters out samples below an adaptive threshold. Stage 2 is a federated conditional GAN (cGAN) training procedure intended to generate synthetic samples for missing classes. Stage 3 completes local datasets with these synthetic samples and trains a global classifier with FedAvg or FedProx. The authors evaluate six model variants (with/without cleaning, with/without GAN augmentation, with FedAvg/FedProx) on MNIST and Fashion-MNIST under varying noise ratios and missing class sizes, reporting macro-F1 scores and claiming significant improvements over noisy baselines.","tokens_in":19899,"tokens_out":3987,"duration_ms":45297,"significance":"If the results were sound, the proposed modular framework would be a useful practical contribution to robust federated learning, particularly the idea of combining local filtering with synthetic data completion to address simultaneous data quality challenges. The paper provides clear algorithmic skeletons, a reasonable evaluation design across two datasets, and an ablation structure that isolates cleaning and augmentation effects. However, the central cleaning equation is internally inconsistent, the GAN training protocol is critically underspecified, and the reported numbers are presented without variance or significance information. As a result, the headline empirical claims are not currently backed by a technically coherent description of the method.","major_comments":[{"comment":"The aggregate confidence score in Eq. (1) is computed as the arithmetic mean of raw entropy, margin, and silhouette score. Raw entropy is largest for the most uncertain predictions, so high-entropy samples receive high aggregate scores and are retained by the C_agg ≥ T criterion, while confident samples are more likely to be discarded. This is the opposite of the stated goal of \"retaining high quality data\" and \"isolating and removing mislabeled instances,\" and the threshold in Eq. (2) does not repair the sign inversion. Because the cleaning stage is the main source of the reported gains in Tables II and V, this is a load-bearing error: the implementation either contradicts the written algorithm or the described mechanism would not produce the claimed improvements.","section":"III-B, Eq. (1), Algorithm 1"},{"comment":"The federated GAN training stage never defines the adversarial objective. Algorithm 2 repeatedly calls a function train_one_epoch(G_global, D_global, μ) and averages the returned weights, but the paper does not specify the generator loss, the discriminator loss, the conditioning mechanism (how the label y is fed to the generator and discriminator beyond a concatenation mentioned in Section IV), or the gradient updates. Weight averaging alone does not define a GAN. This makes the \"collaborative conditional GAN\" unimplementable and unreproducible, and it undermines all claims about synthetic data quality in Section V-C.","section":"III-C, Algorithm 2"},{"comment":"The text in Section IV-B states that asymmetric label noise is introduced by \"assigning them to semantically related but incorrect classes,\" but Algorithm 5 assigns new labels by \"sampling from the label distribution of the valid set,\" which is a uniform/random relabeling among the available classes and is not semantically related. Since the experimental results depend on which noise model is actually used, this mismatch must be resolved and the experiments described accordingly.","section":"IV-B, Algorithm 5"},{"comment":"The paper states in Section IV that each experiment was repeated 50 times and average results were reported, yet Table II and the figures present only point estimates with no standard deviations, confidence intervals, or significance tests. Given that the central claim is \"significant improvements in federated model performance,\" the lack of any variance information makes it impossible to assess whether the differences between models are meaningful, especially for close comparisons such as CleanAvg versus GenCleanAvg.","section":"V, Table II"}],"minor_comments":[{"comment":"The abstract and conclusion claim the framework \"rigorously maintains data privacy\" and \"incorporates differential privacy mechanisms,\" but Section VII states that differential privacy mechanisms were not explicitly applied in the current experiments. These statements should be reconciled.","section":"Abstract / Section VII"},{"comment":"The paper mentions \"confidence weighted aggregation\" in the introduction and related work, but Algorithm 4 aggregates client updates by sample count only. No mechanism for weighting updates by confidence scores is described, so the terminology is misleading.","section":"III-A / IV-C"},{"comment":"The noise injection equation in Algorithm 5 has two branches that appear to be labeled inconsistently with the surrounding text; please clarify when each branch is used and how the final noise ratio is guaranteed for small client datasets.","section":"IV-B"},{"comment":"The formatting of Table II runs model names and numeric values together, making it very hard to read. Clear column separators or a per-dataset layout would be necessary for a journal publication.","section":"Table II"},{"comment":"The citation for entropy-based confidence is about knowledge entropy decay during language model pretraining, which is not the standard reference for prediction confidence or calibration. Please cite a standard source on softmax entropy or confidence estimation.","section":"II, reference [28]"},{"comment":"The paper does not report the number of synthetic samples added per missing class in the actual experiments, although Algorithm 3 has a parameter s. Without this information, the effect of the GAN augmentation cannot be quantitatively interpreted.","section":"Section V-B"}],"recommendation":"reject","confidential_remarks":"The paper has a central technical inconsistency in the cleaning equation and an underspecified GAN algorithm; the reported experiments cannot be reproduced as written. The authors should be encouraged to correct the scoring rule, specify the GAN training objective, and provide variance information in a future revision, but as it stands the evidence does not support the claimed contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this one. First, it is a combination of existing ideas — confidence-based filtering, federated cGAN training, and FedProx — applied to MNIST and Fashion-MNIST under label noise and missing classes. Nothing is conceptually new. Second, the paper’s reported gains come almost entirely from the cleaning stage, and the cleaning equation has a sign error. Equation (1) adds raw entropy to margin and silhouette scores; the text itself says lower entropy means higher confidence, so the aggregate favors uncertain samples and Algorithm 1 preferentially keeps them. The adaptive threshold in Equation (2) does not repair the direction. This is load-bearing, because Table II and the conclusion attribute most of the improvement to cleaning.\n\nTo the paper’s credit, the empirical evaluation is fairly thorough: six model variants, two datasets, 24 noise-and-missing-class conditions, and 50 repetitions per condition. The tables and figures are readable, and the authors honestly acknowledge that the GAN component is often neutral or harmful, so cleaning alone is the main driver. That is a useful negative result about generative augmentation in this simple-image setting. The related work is broad and the survey of robust-FL methods is competent.\n\nThe soft spots are more than cosmetic. The sign error in the central equation is the biggest one, and since no code or error bars are provided, the reader cannot tell whether the implementation matches the text or does something else. The GAN training stage never defines the generator or discriminator objective — it only describes weight averaging — so the cGAN claim is under-specified. There is also a mismatch in the data-completion story: Section III-D mentions a centralized validation classifier, but Algorithm 3 just adds generated samples with no validation step. The single self-citation [5] is unrelated to the method; that is not a problem on its own.\n\nWho is this for? Possibly a reader who wants a quick empirical look at a naive confidence-filtering heuristic on small grayscale tasks. But the internal contradiction undermines the central claim, and the novelty is only in the combination. I would not cite it, and I do not think it deserves referee time as it stands. If the authors fix the equation, provide code, and report error bars, it could become a modest workshop paper.","headline":"A workmanlike pipeline of known components whose main empirical claim rests on a cleaning rule that, as written, does the opposite of what the text says.","tokens_in":20408,"tokens_out":2492,"would_cite":false,"duration_ms":25052,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a three-stage federated learning pipeline—local confidence-weighted noise cleaning, federated conditional GAN training, and synthetic-data completion—makes federated classifiers robust to label noise, missing…","keywords":["federated learning","label noise","conditional GAN","data cleaning","non-IID data","class imbalance","FedProx","macro-F1 score"],"falsifier":"Run the cleaning stage on MNIST with 50% known label noise and compare the label-correctness rates of retained versus discarded samples; if retained samples are not clearly more often correct, or if the printed score formula preferentially keeps high-entropy samples, the reported gains cannot be attributed to the stated cleaning mechanism.","tokens_in":19480,"feed_emoji":"🧹","tokens_out":7402,"duration_ms":66340,"temperature":0.7,"pith_summary":"The paper claims that a three-stage federated learning pipeline can make standard aggregation rules robust to label noise, missing classes, and class imbalance without sharing raw client data. The first stage removes locally mislabeled samples using a confidence score built from prediction entropy, prediction margin, and silhouette-based cluster fit; the second trains a conditional GAN collaboratively; the third fills missing classes with synthetic samples before FedAvg or FedProx training. On MNIST and Fashion-MNIST with asymmetric label noise and missing classes, the cleaned variants are reported to beat noisy FedAvg and FedProx baselines by large macro-F1 margins, sometimes reaching 0.98. The paper's most consequential practical claim is that the cleaning stage alone, without GANs, provides most of the robustness gain, making the approach feasible on resource-constrained clients.","feed_headline":"Cleaning lifts noisy federated F1 to 0.98","feed_subtitle":"Confidence filtering plus optional GAN completion beats noisy FedAvg and FedProx on MNIST and Fashion-MNIST.","key_machinery":"The central object is the confidence filter's aggregate score in Algorithm 1: $C_{\\mathrm{agg}}(x_i)=\\frac{1}{3}(C_{\\mathrm{ent}}(x_i)+C_{\\mathrm{margin}}(x_i)+C_{\\mathrm{cluster}}(x_i))$, with adaptive threshold $T=\\frac{1}{3}(\\mathrm{mean}(C)+\\mathrm{median}(C)+P_{75}(C))$ and retention of samples with $C_{\\mathrm{agg}}(x_i)\\ge T$. In the paper's stated design, lower entropy, larger margin, and higher silhouette mark trustworthy samples, so this filter is supposed to discard mislabeled or ambiguous points before relabeling survivors with $\\arg\\max p$; the second mechanism is the federated conditional GAN, whose generator and discriminator weights are averaged with FedAvg each round and then used to create class-conditioned images for missing classes.","core_discovery":"The authors contend that data-quality corruption in federated settings should be handled before and during aggregation rather than by changing the classifier alone. They propose a modular pipeline: each client trains a small CNN with stratified K-fold cross-validation, computes per-sample confidence scores from prediction entropy, prediction margin, and silhouette-based cluster consistency, and removes samples below an adaptive threshold; a federated conditional GAN is then trained over the cleaned sets to synthesize samples for absent classes; finally, the completed datasets are used to train a global CNN with FedAvg or FedProx. The reported experiments show that the cleaning-only variants (CleanAvg, CleanProx) consistently beat the noisy FedAvg and FedProx baselines across 10–70% asymmetric label noise and 2–7 missing classes, with CleanProx reaching 0.98 macro-F1 on MNIST, and that GAN augmentation adds a smaller but visible gain on Fashion-MNIST, especially under severe imbalance.","pith_inferences":["Editorial: if Equation (1) is not a typo, the printed scoring formula adds raw entropy (which rises with uncertainty) to margin and silhouette, so the filter would preferentially retain high-entropy samples; the reported cleaning gains would then need another explanation, such as relabeling by argmax or shifts in class balance.","Editorial: the adaptive threshold based on mean, median, and 75th percentile depends on the score distribution, and under heavy noise a filter that keeps high-scoring samples may simply shrink the dataset; tracking retention ratios and whether retained samples are more often correctly labeled would separate cleaning from subsampling effects.","Editorial: because synthetic data quality is measured only through downstream F1, a direct quality check on generated images would clarify whether GAN completion adds genuine class coverage or acts mainly as a regularizer.","Editorial: the privacy guarantee asserted in the abstract is not measured in the experiments—the paper states that differential privacy mechanisms were not explicitly applied—so 'privacy compliant' should be read as architectural intent rather than a demonstrated property."],"forward_implications":["Under the reported results, FedAvg and FedProx can recover most of their macro-F1 on MNIST by adding only the local cleaning stage, even with 70% label noise and seven missing classes.","On Fashion-MNIST, conditional-GAN augmentation gives a performance lift mainly when classes are severely missing, so generative completion is a targeted remedy for class sparsity rather than a universal add-on.","Because cleaning-only models match or beat GAN-augmented models on MNIST, the practical implication is that low-resource deployments can skip the GAN stage without losing much accuracy.","The framework keeps the communication pattern of standard FedAvg and FedProx, so its reported gains come from data-quality preprocessing plus the same round-based weight exchange."],"supporting_citations":[{"why":"Supplies the federated averaging protocol used for both the collaborative GAN stage and the final classifier training.","marker":"[1]"},{"why":"Supplies the FedProx proximal regularization term that stabilizes local training under non-IID data in the final stage.","marker":"[25]"},{"why":"Supplies the conditional GAN formulation that the synthetic-data generation stage builds on.","marker":"[31]"},{"why":"Provides the non-IID weight-divergence analysis that motivates the missing-class and class-imbalance problems targeted by the pipeline.","marker":"[11]"},{"why":"Supplies the LeNet-style CNN architecture used as the local and global classifier in the experiments.","marker":"[32]"},{"why":"Supplies the stratified K-fold cross-validation procedure used to obtain out-of-fold predictions for the cleaning confidence scores.","marker":"[27]"},{"why":"Supplies the K-means clustering used to compute the silhouette-based cluster confidence component of the cleaning score.","marker":"[29]"}],"fun_headline_variants":["Confidence filtering lifts noisy federated F1 to 0.98","CleanProx beats noisy FedAvg, hits 0.98 macro-F1","Robust FL: filter noise, synthesize absent classes","Federated learning: confidence filter wins on messy data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a locally trained CNN's prediction probabilities and silhouette scores separate mislabeled from correctly labeled samples—and, as written, the scoring formula adds raw entropy, so high-entropy samples score higher and may be retained instead of removed.","fun_headline_variants_meta":{"raw":{"variants":["Confidence filtering lifts noisy federated F1 to 0.98","CleanProx beats noisy FedAvg, hits 0.98 macro-F1","Robust FL: filter noise, synthesize absent classes","Federated learning: confidence filter wins on messy data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000937,"raw_usage":{"total_tokens":3995,"prompt_tokens":919,"completion_tokens":3076,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":3002}},"tokens_in":535,"tokens_out":3076,"duration_ms":22477,"temperature":1.0,"reasoning_tokens":3002,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:26:04.033746+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the cleaning stage on MNIST with 50% known label noise and compare the label-correctness rates of retained versus discarded samples; if retained samples are not clearly more often correct, or if the printed score formula preferentially keeps high-entropy samples, the reported gains cannot be attributed to the stated cleaning mechanism.","supporting_citations":[{"cited_title":"Communication-efficient learning of deep networks from decentralized data,","cited_arxiv_id":null,"evidence_quote":"Supplies the federated averaging protocol used for both the collaborative GAN stage and the final classifier training."},{"cited_title":"A study of cross-validation and bootstrap for accuracy estimation and model selection,","cited_arxiv_id":null,"evidence_quote":"Supplies the stratified K-fold cross-validation procedure used to obtain out-of-fold predictions for the cleaning confidence scores."},{"cited_title":"Some methods for classification and analysis of multivariate observations,","cited_arxiv_id":null,"evidence_quote":"Supplies the K-means clustering used to compute the silhouette-based cluster confidence component of the cleaning score."}],"review_version":1}