{"id":"991c2985-e645-4b8b-8eaf-8fbe6dd98c87","arxiv_id":"2504.14694","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FedSSD applies class-level and sample-level credibility weights to the global model's logits when distilling them into local models, reducing client drift and speeding up convergence in non-IID federated learning.","lead":"This paper proposes FedSSD, a federated learning algorithm that selectively distills the global model's knowledge into each client's local model, weighting that knowledge by how trustworthy the global model is for each class and each sample. It reports faster convergence and higher accuracy than several baselines on non-IID image tasks, especially on CIFAR10 and TinyImageNet.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 2, on which Theorem 1 rests, is not implied by the FedSSD local objective and appears algebraically inconsistent with Lemma 1, so the convergence guarantee is not established.","rationale":"The most load-bearing condition for the paper's central claim is the theoretical convergence guarantee. If Theorem 1 fails, the 'convergence guarantee' advertised in the abstract and Section 4.3 is unsupported; the algorithm may still work empirically, but the claim is materially weaker. The reader already flagged Assumption 2 as unverified, and my pass strengthens that: Assumption 2's linear term does not match the gradient of the surrogate built in Lemma 1, and the local objective's true gradient contains sample-dependent Jacobian terms, not a constant proximal term. This is an internal inconsistency, not merely a disagreement with a stronger outside assumption. The empirical experiments provide some support for the practical algorithm, and the paper's motivation and visualizations are reasonable, so this is not a rejection of the method; it is a request to repair the proof or to explicitly label the convergence result as conditional on an unverified assumption. Because fixing the theorem is feasible and the empirical core remains plausible, the reader's CONDITIONAL verdict remains appropriate.","tokens_in":16087,"tokens_out":8532,"duration_ms":77571,"concrete_test":"Independently re-derive §4.3: write out the gradient of L_SSD from Eq. (3) and the gradient of the upper bound \\tilde L in Lemma 1. If neither equals ∇L_CE(w) + M_max L_f (w − w_t), then Assumption 2 cannot be justified by Lemma 1 and Theorem 1 needs a corrected proof. Then re-run Theorem 1's derivation with the corrected expression to check whether the ρ formula in Theorem 1 still yields a positive lower bound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3's Theorem 1 is the paper's theoretical support for its advertised convergence guarantee. Its only FedSSD-specific premise is Assumption 2: ||∇L_CE(w_{t+1}^i) + M_max L_f (w_{t+1}^i − w_t)|| ≤ η||∇L_i(w_t)|| with η ∈ [0,1). However, the actual local update in Algorithm 1 minimizes L_i = L_CE + L_SSD, where L_SSD is a sum of sample-dependent MSE logit losses. Nothing in that objective yields a proximal term of the form M_max L_f (w − w_t). Lemma 1 constructs a surrogate \\tilde L(w,w_t) = L_CE(w) + M_max^2 L_f^2 ||w − w_t||^2; its gradient is ∇L_CE(w) + 2 M_max^2 L_f^2 (w − w_t), not ∇L_CE(w) + M_max L_f (w − w_t). Even granting the also-suspect L_f^2 in Lemma 1, Assumption 2's linear term is not the gradient of the bound it supposedly comes from. Moreover, Lemma 1 only compares function values for an exact minimizer of \\tilde L; the actual SGD trajectory after E local epochs is never shown to satisfy the stationarity-style bound. The proof then cites FedProx Theorem 4, which relies on an analogous inexactness assumption for a genuine proximal objective; here the selective self-distillation loss is sample-dependent and is not reduced to a single quadratic regularizer. Thus Theorem 1 does not follow from the stated assumptions, and the 'convergence guarantee' is an unverified premise rather than a derived result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedSSD, a federated learning algorithm that adds an adaptive self-distillation regularizer to each client's local training. The regularizer matches the local model's logits to the frozen global model's logits, with the matching weighted by a class-level credibility matrix computed on a server-side auxiliary dataset and by a sample-level confidence score for each local example. The authors motivate the method by showing that local models forget global knowledge and that the global model's reliability varies across classes and samples. The manuscript presents Algorithm 1, a convergence theorem (Theorem 1) based on Assumptions 1 and 2, and experiments on CIFAR-10, CIFAR-100, and TinyImageNet comparing FedSSD with FedAvg, FedProx, FedCurv, MOON, FedNTD, and the authors' prior FedCAD. The central claims are that FedSSD achieves better generalization and robustness in fewer communication rounds than other state-of-the-art FL methods and that its convergence is theoretically guaranteed.","tokens_in":16524,"tokens_out":9865,"duration_ms":80851,"significance":"The FedSSD idea is practically attractive: it requires no extra uplink communication beyond the model update, since the server sends only the model and a K-by-K credibility matrix, and the reported results show consistent improvements on CIFAR-10 and TinyImageNet across several heterogeneity settings. The ablation study and sensitivity analyses are useful and support the claim that the selective weighting mechanism contributes beyond fixed KL or MSE distillation. The paper is also honest in disclosing that it extends the authors' earlier FedCAD paper. However, the theoretical convergence guarantee is not established, because the key assumption used in Theorem 1 is not derived from the actual FedSSD local objective and the proof is delegated to an external theorem in a way that does not match the algorithm. In addition, the CIFAR-100 result in Table 1 contradicts the abstract's broad 'better generalization' claim. The empirical core is promising, but the theoretical claim and the headline wording need substantive revision before the paper can be accepted.","major_comments":[{"comment":"The bridge from the FedSSD objective to the convergence assumption is not established. The actual local loss in Eq. (6) is L_i = L_CE,i + L_SSD,i, where L_SSD,i (Eq. (3)) is a sum of per-sample masked MSE logit losses whose mask M(x) depends on the sample and on the round-dependent credibility matrix. Lemma 1 replaces this with the surrogate L_tilde(w,w_t) = L_CE(w) + M_max^2 L_f^2 ||w - w_t||^2. Three problems arise. First, the coefficient L_f^2 in Lemma 1 is inconsistent with Assumption 1.3 as printed, which yields L_f after squaring the displayed bound, so the lemma's algebra is incorrect. Second, even if the bound were correct, the gradient of L_tilde is grad L_CE(w) + 2 M_max^2 L_f^2 (w - w_t), not grad L_CE(w) + M_max L_f (w - w_t) as Assumption 2 states. Third, Lemma 1 is a function-value bound for an exact minimizer of L_tilde, whereas Algorithm 1 runs E epochs of SGD on L_i, and no argument shows that the resulting w_{t+1}^i satisfies either the lemma's condition or Assumption 2. Assumption 2 is therefore an unverified postulate, and Theorem 1's descent guarantee does not follow from the stated premises.","section":"Section 4.3, Lemma 1 and Assumption 2"},{"comment":"No proof of Theorem 1 appears in the manuscript; the sentence 'The theorem derives from the Theorem 4 in [8]' is not a proof. FedProx's Theorem 4 relies on an inexactness condition for a proximal objective with a fixed coefficient, whereas FedSSD's regularizer is sample-dependent and is not a single proximal term, so the reduction to FedProx is not immediate. The authors should either provide a self-contained proof that verifies the required assumptions for Algorithm 1 or clearly state that the theoretical analysis is conditional on an unverified Assumption 2; in the latter case the advertised 'convergence guarantee' in the abstract should be removed.","section":"Section 4.3, Theorem 1"},{"comment":"FedSSD's final global test accuracy on CIFAR-100 (66.19 ± 0.28) is lower than MOON (67.32 ± 0.74), yet the abstract claims that FedSSD 'achieves better generalization ... compared with other state-of-the-art FL methods' and Section 5.2 states a 'similar conclusion' for deeper networks. The paper's caveat that average local accuracy and convergence speed are higher does not rescue the global-accuracy claim. Please qualify the claim to the datasets and settings where it holds, or provide a statistical comparison showing that the 1.13-point deficit is not significant; as written, the central claim is contradicted by the paper's own table.","section":"Table 1, CIFAR-100 row"}],"minor_comments":[{"comment":"The quantifier in Assumption 1.1 is garbled; it should read 'For each client i in [N] and any parameter w in R^d, E_i[||grad L_i(w)||^2] <= ||grad L(w)||^2 B^2.'","section":"Section 4.3, Assumption 1.1"},{"comment":"The inequality in Assumption 1.3 as printed uses sqrt(2 L_f), but Lemma 1's derivation requires a factor of L_f^2 after squaring; the assumption and the lemma should be made dimensionally consistent.","section":"Section 4.3, Assumption 1.3 and Lemma 1"},{"comment":"'Global locit' is a typo for 'global logit'.","section":"Section 4.2, text near Eq. (4)"},{"comment":"The T entries for FedCurv, FedNTD, and FedCAD on CIFAR-100 appear to be missing from the table.","section":"Table 1"},{"comment":"The method relies on a server-side auxiliary dataset D_V with 64 samples per class, but the paper does not discuss the privacy and availability assumptions for this dataset or the sensitivity of the results to its composition; a discussion would help readers judge the method's practical scope.","section":"Section 5.1 and Section 5.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: FedSSD is a genuinely new combination that works. Class- and sample-level credibility weighting applied to logits-wise MSE self-distillation in federated learning is not something I've seen in the cited prior work, including the authors' own FedCAD. The motivation figures (global vs local accuracy, class-wise reliability) are convincing, and the empirical gains in convergence speed are consistent on CIFAR10 and TinyImageNet. The ablation against constant KL and MSE losses shows the selective weighting is doing real work. The paper is clearly written and honestly differentiates itself from FedCAD.\n\nThe soft spots are real, and one of them is load-bearing. The convergence guarantee in Section 4.3 does not follow from the stated assumptions. Assumption 2 is not implied by the local objective: the surrogate bound in Lemma 1 has gradient ∇L_CE + 2 M_max^2 L_f^2 (w − w_t), not ∇L_CE + M_max L_f (w − w_t), so the assumption is algebraically inconsistent with the lemma. Nothing in the actual SGD update—which minimizes a sample-dependent MSE distillation loss plus cross-entropy—is shown to satisfy that bound. The proof then leans on FedProx's Theorem 4, whose proximal term has a different structure. So Theorem 1 is an unverified premise, not a derived result. A reviewer should ask the authors to either prove Assumption 2 for their actual loss or remove the theoretical claim.\n\nSecond, the abstract's 'better generalization' claim is too strong. On CIFAR100, FedSSD's final accuracy (66.19) trails MOON (67.32). The paper admits this but still frames the comparison as superior; that is an overstatement. Minor: no code release, Mmax is tuned per dataset, and the server needs an auxiliary dataset, which is an extra assumption that gets little discussion.\n\nOverall: the method is plausible and the experiments mostly support it. The theory needs a major revision or removal, and the CIFAR100 result should be reported without spin. I'd accept it for peer review with the expectation of heavy revision. The empirical contribution deserves a serious referee.","headline":"A useful new FL method with solid, mostly positive experiments, but the convergence theorem is algebraically inconsistent and should be fixed or dropped.","tokens_in":16994,"tokens_out":5841,"would_cite":true,"duration_ms":49146,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedSSD selectively weights the global model's distilled knowledge by class and sample credibility, improving accuracy and convergence speed in non-IID federated learning.","keywords":["federated learning","non-IID data","knowledge distillation","self-distillation","client drift","catastrophic forgetting","data heterogeneity","convergence analysis"],"falsifier":"Run FedSSD on CIFAR10 with the paper's default settings and, at each round, measure for each client the quantity $\\|\\nabla L_{CE,i}(w_{t+1}^i) + M_{max}L_f(w_{t+1}^i - w_t)\\|$ and compare it with $\\eta \\|\\nabla L_i(w_t)\\|$ for $\\eta < 1$. If a substantial fraction of client updates violate the inequality, the theorem's descent guarantee does not cover the actual optimization trajectory.","tokens_in":1690,"feed_emoji":"🧠","tokens_out":2900,"duration_ms":80362,"temperature":0.7,"pith_summary":"The paper claims that catastrophic forgetting of global knowledge during local training is a main cause of slow convergence and poor accuracy in federated learning on non-IID data. It proposes FedSSD, a method that makes each client distill the global model's output logits into its local model, but weights that distillation by how credible the global model is for each class and for each sample. Class credibility is measured by recall and confusion on a small server-held auxiliary set; sample credibility is the global model's confidence in the true label. The paper proves convergence under a descent condition and reports that FedSSD beats FedAvg, FedProx, MOON, FedNTD and others in accuracy and speed on CIFAR10, CIFAR100 and TinyImageNet. The central idea is that adaptive credibility weighting is what makes self-distillation work under heterogeneity, while a constant-weight distillation is too blunt.","feed_headline":"Selective self-distillation cuts federated learning rounds","feed_subtitle":"Weighing the global model's advice by class and sample credibility preserves knowledge and lifts accuracy.","key_machinery":"The central object is the selective distillation weight vector $M(x)\\in\\mathbb{R}^K$ (Eq. 4), a per-class, per-sample credibility mask applied element-wise to both teacher and student logits before an MSE. Class credibility is $M_{\\text{class}}[k_1] = A_{k_1,k_1}(1-\\max_{k\\neq k_1} A_{k,k_1})$, computed from the global model's confusion matrix $A_t$ on a small server-side auxiliary set, so it is high when the global model both recalls class $k_1$ and rarely confuses other classes with it. Sample credibility is $M_{\\text{sample}}(x) = 1 - (1-p_g(x)[k_2])^{0.5}$, which grows with the global model's predicted probability of the true label $k_2$. The product, clipped and scaled by $M_{\\text{max}}$, controls how strongly each logit channel is distilled into the local model.","core_discovery":"FedSSD's central discovery is that the global model in federated learning, while better overall than local models, is not uniformly trustworthy as a distillation teacher: its logits vary in reliability across classes and across individual samples, especially in early rounds and on skewed data. The method therefore adds a selective self-distillation loss $L_{SSD} = E(\\|M\\odot z_g - M\\odot z\\|_2^2)$ to the local cross-entropy objective, where the channel-wise weight $M(x)[k_1]$ combines a class-level credibility score (recall of class $k_1$ minus the maximum rate at which other classes are mistaken for $k_1$) and a sample-level score (the global model's predicted probability of the true label, mapped to $[0,1]$). Because the weights suppress unreliable logits, local models preserve global knowledge without being misled, and the aggregate model converges in fewer rounds. The paper also proves a non-convex convergence result, adapted from FedProx's theorem, under bounded-dissimilarity and Lipschitz-smoothness assumptions plus a technical condition on the local optimizer.","pith_inferences":["The credibility-weighting scheme is teacher-agnostic: the same class-level and sample-level masks could be applied to any distillation setting (e.g., partially trained teachers in data-free or ensemble distillation) where teacher reliability varies along those two axes.","A natural testable extension is to replace the sample-level confidence $p_g(x)[k_2]$ with the teacher's predictive entropy or margin, which would also penalize confidently wrong samples; the paper's square-root mapping only distinguishes confident-correct from unconfident cases.","Because the class credibility matrix is estimated on a small auxiliary set, the method inherits sensitivity to that set's composition; one could instead estimate per-class credibility from the uploaded local models or from the server's own predictions on generated data, removing the public-data assumption."],"forward_implications":["On non-IID image benchmarks, FedSSD should require fewer communication rounds to reach FedAvg's final accuracy: 33 rounds vs 100 on CIFAR10, 48 vs 100 on CIFAR100, and 10 vs 30 on TinyImageNet in the paper's runs.","FedSSD's gains widen as heterogeneity increases, e.g., on CIFAR10 with #K=2 it reports 53.37% versus the best baseline's 50.75%, whereas on near-IID it is within roughly two points, so the method is most useful where client drift is worst.","Using MSE logit matching with selective weights should dominate both plain MSE and KL distillation at the same coefficient, as the paper's ablation in Table 3 shows SSD highest across $\\alpha$ values.","Beyond FedAvg's model transmission, the only extra communication is one $K\\times K$ credibility matrix per round, keeping FedSSD communication-cheap relative to methods that transmit Fisher diagonals or control variates."],"supporting_citations":[{"why":"Defines FedAvg, the foundational baseline and the local-update loop that FedSSD extends.","marker":"[2]"},{"why":"Supplies the non-IID convergence analysis and assumptions the paper adapts for its theorem.","marker":"[4]"},{"why":"Proposes the proximal regularization baseline and the Theorem 4 framework the FedSSD convergence proof derives from.","marker":"[8]"},{"why":"Establishes that the aggregated global model extracts better representations than local models, motivating distillation.","marker":"[9]"},{"why":"Is the authors' prior class-wise adaptive self-distillation method (FedCAD) that FedSSD improves on.","marker":"[11]"},{"why":"Argues MSE logit matching beats KL divergence in distillation, grounding FedSSD's choice of loss.","marker":"[35]"},{"why":"Uses the global model as a teacher in self-distillation, the baseline FedSSD selectively weights.","marker":"[41]"},{"why":"Also self-distills global knowledge into local models, a key comparison point for selective weighting.","marker":"[42]"}],"fun_headline_variants":["Selective self-distillation speeds non-IID federated learning","Weigh global-model wisdom by class and sample credibility","FedSSD: trust the global model only when it's credible","Credibility-gated distillation cuts federated learning rounds"],"cache_read_input_tokens":19072,"weakest_assumption_plain":"The convergence theorem relies on Assumption 2, which asserts that after each round of local SGD, the optimized local parameter satisfies a particular norm bound involving the gradient and the distance from the global model; the paper does not verify that ordinary local training actually lands inside that bound.","fun_headline_variants_meta":{"raw":{"variants":["Selective self-distillation speeds non-IID federated learning","Weigh global-model wisdom by class and sample credibility","FedSSD: trust the global model only when it's credible","Credibility-gated distillation cuts federated learning rounds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1613,"prompt_tokens":963,"completion_tokens":650,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":582}},"tokens_in":579,"tokens_out":650,"duration_ms":6680,"temperature":1.0,"reasoning_tokens":582,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:42:19.611219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FedSSD on CIFAR10 with the paper's default settings and, at each round, measure for each client the quantity $\\|\\nabla L_{CE,i}(w_{t+1}^i) + M_{max}L_f(w_{t+1}^i - w_t)\\|$ and compare it with $\\eta \\|\\nabla L_i(w_t)\\|$ for $\\eta < 1$. If a substantial fraction of client updates violate the inequality, the theorem's descent guarantee does not cover the actual optimization trajectory.","supporting_citations":[{"cited_title":"On the conver- gence of fedavg on non-iid data,","cited_arxiv_id":null,"evidence_quote":"Supplies the non-IID convergence analysis and assumptions the paper adapts for its theorem."},{"cited_title":"Model-contrastive federated learning,","cited_arxiv_id":null,"evidence_quote":"Establishes that the aggregated global model extracts better representations than local models, motivating distillation."},{"cited_title":"Class-wise adaptive self distillation for heterogeneous federated learning,","cited_arxiv_id":null,"evidence_quote":"Is the authors' prior class-wise adaptive self-distillation method (FedCAD) that FedSSD improves on."}],"review_version":1}