{"id":"9e9722d6-eff7-4486-afc9-0dd0ec23c49b","arxiv_id":"2412.04521","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"FedDW regularizes each client's classifier-weight similarity matrix toward a globally aggregated soft-label matrix, improving non-IID federated accuracy on four benchmarks by about 3% over ten baselines.","lead":"FedDW is a federated learning method that uses a shared class-relationship table to keep client models aligned when different clients have very different local data. On four test datasets it improved accuracy by about three points over ten existing methods while adding almost no training overhead.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (11) relies on an unverified SL-CR equality; the paper's IID evidence (t-SNE, heatmaps) does not establish it, so the central mechanism is currently unsupported.","rationale":"I read the paper as proposing a regularizer whose validity rests on the claimed IID consistency between soft-label and classifier-gram matrices. The empirical tables are internally consistent and the timing table supports low overhead, so the central empirical claim is not fraudulent. But the mechanism is load-bearing because the method is named and motivated by this consistency; if the equality is false, FedDW is still a plausible regularizer, but the paper's explanation and theoretical framing are wrong. The paper itself flags that the convergence proof is omitted, and the Section IV-B analysis is for a different objective, further weakening the theory; however, the empirical claim could survive those fixes. The reader's weakest assumption matches mine, and a direct numerical test of the equality under IID would settle it. Until then, conditional acceptance is appropriate: the authors should either prove or verify the equality, or reframe FedDW as a heuristic and remove the unsupported theory.","tokens_in":17124,"tokens_out":6360,"duration_ms":69916,"concrete_test":"Use the released code to train FedAvg on IID CIFAR-10 (or MNIST) to convergence; compute the per-class average soft-label matrix Omega (Eq. 4) and the row-softmaxed Gram matrix Sigma = rowsoftmax(omega omega^T) from the same aggregated classifier; report the mean per-row KL divergence between Omega[i,:] and Sigma[i,:] and the normalized Frobenius distance. If the mean KL exceeds 0.05 or the normalized distance exceeds 10%, the equality asserted in Section III-B fails and the regularization target in Eq. (11) is biased.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism requires that under IID data, Omega_global in Eq. (7)/(4) is numerically equal to sigma(omega omega^T) in Eq. (11). Section III-B only argues that weight moduli are roughly uniform under IID, so inner products can replace cosine similarity; it never derives the much stronger statement that average soft-label rows equal row-softmaxed Gram rows. That equality would require, roughly, that feature vectors h(x) for each class are concentrated at (or share a common scalar multiple of) the class's classifier weight vector, so logits equal the corresponding Gram row. The t-SNE plot (Figure 5) shows only that weight vectors lie inside class clusters, not that h is approximately omega; the heatmaps (Figure 8) are visual and provide no distance or error measure. If the equality is only approximate, the regularizer in Eq. (11) is a biased target and the reported gains may come from generic classifier smoothing rather than the claimed soft-label/class-relation consistency. Additionally, Section IV-B's derivative and convexity analysis is written for A^T A with A a generic row-stochastic matrix, not for the actual sigma(omega omega^T) in Eq. (11), so it does not supply theoretical support for the actual objective.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedDW, a federated learning method that regularizes each client's last-layer classifier weights by matching the row-softmaxed Gram matrix σ(ωω^T) to a globally aggregated soft-label matrix Ω. The central claim is that under IID data these two matrices coincide and encode inter-class relationships, so the global soft-label matrix can serve as a target to mitigate performance loss under non-IID data. The method requires uploading only a |C|×|C| matrix per client and adds a regularization term that depends only on the classifier layer. The paper reports experiments on MNIST, CIFAR-10/100, and IMDB against ten baselines, scalability tests, efficiency measurements, compatibility experiments, and a theoretical convergence analysis.","tokens_in":17395,"tokens_out":12193,"duration_ms":109880,"significance":"If the key consistency assumption were rigorously established, FedDW would be a simple and communication-efficient regularizer for non-IID federated classification. The experimental work is broad: four datasets, ten baselines, five-run means with standard deviations, and scalability experiments in client count, training rounds, and network architectures, with code publicly released. The empirical results are internally consistent and the method is competitive or state-of-the-art in most settings. However, the theoretical sections currently do not support the paper's claims: the proof of Theorem 3 is omitted, the derivative analysis in Section IV-B is performed on a different objective, and the central IID equality is only demonstrated visually. These issues prevent acceptance in the current form.","major_comments":[{"comment":"The central assumption that under IID data the global soft-label matrix Ω_global equals σ(ωω^⊤) is not quantitatively established. The supporting evidence (Figures 4, 5, and 8) is visual: weight-modulus trends, a t-SNE plot, and heatmaps, with no distance, correlation, or error measure between Ω and σ(ωω^⊤). Because Eq. (11) directly penalizes the difference between these two matrices under non-IID conditions, an approximate or dataset-specific equality would bias the regularization target, and the reported gains could stem from generic classifier smoothing rather than the claimed consistency. Please provide a quantitative IID test, such as the average Frobenius distance or cosine similarity between the two matrices across datasets, compared against a baseline such as a randomly permuted matrix.","section":"Section III-B, Eq. (11)"},{"comment":"In the pathological non-IID setting (β=0.1), a client may have no data from a particular class, making |D_i^n|=0 and Eq. (4) undefined for that row. The paper does not specify how to construct the local average soft-label matrix for missing classes, even though Figure 4 explicitly shows clients with zero data in some classes. The implementation must define a convention for these rows (for example, omitting them from aggregation, using a uniform distribution, or assigning a default row), and the aggregation in Eq. (7) should be stated in terms of that convention. Please clarify how missing classes are handled in the experiments and in Algorithm 1.","section":"Section III-C1, Eq. (4)"},{"comment":"The derivative analysis is performed on a surrogate objective rather than the actual regularizer. In Eq. (23), σ(ωω^⊤) is replaced by a generic row-stochastic matrix A, and the regularizer is rewritten as L_reg(A)=||Ω−A^⊤A||_F^2, which is not the same function as Eq. (11), whose argument is σ(ωω^⊤) and whose variable is ω. Even for the surrogate, Eq. (24) is incorrect: the gradient of ||Ω−A^⊤A||_F^2 with respect to A is −4A(Ω−A^⊤A) (for symmetric Ω), not −2(Ω−A^⊤A)A. Consequently, Eq. (25) does not give the Hessian of the true objective, and the claims about computational cost and convexity of the actual L_reg are not supported by this section.","section":"Section IV-B, Eqs. (23)-(25)"},{"comment":"Theorem 3 is stated without proof, with the text noting that the proof is omitted due to space limitations. The statement also appears to have technical problems: Eq. (21) gives an upper bound on T for a given ε, whereas a convergence guarantee of the form (1/T)Σ E||∇L||² ≤ ε requires T to be sufficiently large, not bounded above; Eq. (22) can involve a negative denominator or an imaginary square root for allowed parameters; and Eq. (20) appears to contain a typographical error ('Eη²' instead of the Rη²σ² term that appears in Eq. (19)). Moreover, Assumption 3 assumes convexity of L_cla, but the theorem claims a non-convex convergence rate, and the paper does not explain how the convex assumption is reconciled with that claim. A complete, correct proof is needed before the convergence analysis can support the paper's claims.","section":"Section IV-A, Theorem 3, Eqs. (20)-(22)"},{"comment":"The abstract claims that FedDW 'outperforms 10 state-of-the-art FL methods, improving accuracy by an average of 3% in highly heterogeneous settings.' The numbers in Table I do not clearly support this figure: averaged over the four pathological-heterogeneity tasks, FedDW improves over the best baseline by about 1.1 percentage points (and it is 0.12 points below FedBABU on MNIST), while averaged over all eight practical and pathological tasks the improvement over the best baseline is below 1 point. Please state explicitly which baselines and metric (absolute percentage points, relative improvement, comparison with FedAvg or the best baseline) produce the 3% average, or adjust the claim accordingly.","section":"Abstract and Table I"}],"minor_comments":[{"comment":"The summation index is written as '∑_{j=1}^n A_ij = 1'; the upper limit should be |C|, and similar typographical inconsistencies with 'n' instead of '|C|' appear in the surrounding text. Please correct these.","section":"Eq. (23)"},{"comment":"The caption says 'The font deepening represents the best result'; this should read 'boldface indicates the best result'. Additionally, the term 'counterexample' in the Figure 4 caption is not defined; please clarify what constitutes a counterexample in the 100 experiments.","section":"Table I caption"},{"comment":"The sentence 'The other two datasets have not changed much' is vague; please report the µ values selected for MNIST and IMDB and describe how the selection was made.","section":"Section V-A"},{"comment":"The text cites 'FedDyn [14]', but reference [14] is the FedProto paper; FedDyn appears not to be cited elsewhere. Please correct the citation and ensure all references are used consistently.","section":"Related Work"},{"comment":"The statement 'before visualization, we need to perform Vector Unitization' is unclear; please specify what normalization was applied to the weight vectors and features before the t-SNE embedding.","section":"Section III-B, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The experimental study is solid and the code availability is a plus, but the theoretical sections appear to be assembled from other papers' results without thorough verification. The omitted proof of Theorem 3, the surrogate-objective analysis in Section IV-B, and the unsupported 3% claim are serious gaps. I would require a full rewrite of Section IV and a quantitative validation of the IID consistency assumption before considering the paper for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: FedDW introduces a genuinely new regularizer for non-IID federated learning and backs it with an unusually broad set of experiments. The theoretical sections, however, are not reliable as written, and the paper's core assumption—that under IID data the row-softmaxed classifier Gram matrix equals the global soft-label matrix—is supported only by visual evidence.\n\nWhat's new: the method aligns each client's CR matrix (sigma(omega omega^T)) with the globally averaged soft-label matrix from all clients. That specific mechanism is not in the cited soft-label or classifier-regularization papers. It's a clean, simple idea with negligible additional compute, since the regularizer only touches the last layer. The experimental work is a real strength: four datasets spanning CV and NLP, multiple backbones (CNN, ResNet, LSTM), two heterogeneity regimes, client-count and round-scalability tests, and a compatibility table showing the regularizer boosts several existing FL methods. Five runs with standard deviations are reported, which is more than many FL papers bother with. The roughly 3-point gains over strong baselines in pathological settings are plausible from the tables.\n\nSoft spots. The theory is in poor shape. Theorem 3's proof is explicitly omitted, and the displayed convergence bound looks wrong: Eq. (21) gives an upper bound on T, which would imply convergence only for small T, the reverse of what you'd need. More seriously, Section IV-B defines A as sigma(omega omega^T) and then analyzes Lreg(A) = ||Omega - A^T A||_F^2, but the actual regularizer in Eq. (11) is ||Omega - A||_F^2. The derivatives and non-convexity discussion are about a different objective. That's a load-bearing error in the theory.\n\nThe key consistency hypothesis also has thin evidence. The paper shows a t-SNE plot and heatmaps under IID, but never measures the distance between SL and CR quantitatively, or compares it to a chance baseline. And because the regularizer forces CR toward SL, the agreement shown in the non-IID case is partly by construction. A referee should ask for an independent test: train a model on IID data, freeze it, compute both matrices, report Frobenius distance relative to a random permutation.\n\nMinor issues: mu is tuned on the same test sets, which can inflate results; the 'DLE data' terminology is a new name for known ideas but that's harmless; the privacy discussion correctly notes that uploading soft labels may leak information but doesn't analyze it.\n\nBottom line: the empirical contribution is worth taking seriously, but the theory needs to be corrected or removed, and the consistency claim needs real evidence. I'd send this to peer review with major-revision expectations, not desk-reject it.","headline":"FedDW has a genuinely new regularizer and a broad empirical study, but the theory is broken and the central consistency assumption is under-supported.","tokens_in":17889,"tokens_out":5039,"would_cite":false,"duration_ms":46477,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedDW claims that a single consistency regularizer, aligning each client's row-softmaxed classifier Gram matrix with the global soft-label matrix, outperforms 10 state-of-the-art federated learning methods by an average of about 3% in…","keywords":["federated learning","non-IID data","soft labels","knowledge distillation","classifier weight regularization","Gram matrix","heterogeneous data","consistency optimization"],"falsifier":"Train a FedAvg model on an IID partition of CIFAR-10, compute the global soft-label matrix $\\Omega$ and the row-softmaxed CR matrix $\\sigma(\\omega\\omega^\\top)$ at the end of training, and measure $\\|\\Omega - \\sigma(\\omega\\omega^\\top)\\|_F$. If this distance is not close to zero, or does not shrink with more IID data or more classes, the consistency principle at the core of FedDW is violated and the regularization target is biased.","tokens_in":16894,"feed_emoji":"🔄","tokens_out":5939,"duration_ms":54744,"temperature":0.7,"pith_summary":"FedDW claims that federated learning with non-IID data can be repaired by a single regularization term: pull each client's classifier-head weight Gram matrix toward a globally aggregated matrix of soft labels. The paper argues that in IID training these two matrices carry the same inter-class similarity information, so the global soft-label matrix is a legitimate target for local classifier parameters. Across MNIST, CIFAR-10/100, and IMDB, the method reports top or near-top accuracy among 10 baselines, with the largest gains in highly heterogeneous settings. A convergence proof gives an O(1/T) rate, and the extra backpropagation cost is limited to the last layer.","feed_headline":"FedDW regularizer lifts federated accuracy 3% over 10 baselines","feed_subtitle":"When client data is skewed, pulling each classifier's Gram matrix toward global soft labels recovers IID-like performance.","key_machinery":"The Class Relation (CR) matrix $\\omega\\omega^\\top$ is the last-layer weight matrix multiplied by its own transpose, whose $(i,j)$ entry records the similarity between class $i$ and class $j$ through the inner product of their classifier weight vectors. The paper pairs this with the global soft-label (SL) matrix $\\Omega$, a weighted average of per-class softmax outputs across clients, and regularizes via Eq. (11): $L_{\\text{reg}} = \\frac{1}{|C|^2}\\|\\Omega - \\sigma(\\omega\\omega^\\top)\\|_F^2$. Because the gradient of this term depends only on the classification layer, backpropagation cost does not grow with model size.","core_discovery":"Under IID data the per-class average soft-label matrix and the row-softmaxed product of the last-layer weight matrix with its transpose coincide. FedDW turns this equality into a training objective: each client minimizes the Frobenius distance between the server-aggregated soft-label matrix and its local $\\sigma(\\omega\\omega^\\top)$, thereby steering the classifier head back toward IID-like parameter geometry even when local data are skewed. The paper reports that this simple regularizer outperforms 10 existing FL methods, raising accuracy on average by about 3% in pathological non-IID settings, and that the added computation is negligible because the regularization gradient touches only the classifier parameters.","pith_inferences":["A reader could test the core IID consistency directly: train any classifier on an IID split and compare the average soft-label matrix with the row-softmaxed weight Gram matrix; if the Frobenius distance is not small, the regularization target is approximate and the reported gains may be due to implicit averaging rather than exact consistency.","The method's global target is the average of per-class soft labels, so classes that are rare across all clients get less reliable entries; reweighting the SL matrix by class frequency might make FedDW more robust on long-tailed federated data.","Uploading per-class average softmax vectors reveals class proportions on each client; combining FedDW with differential privacy or a public dataset, as the paper suggests, would be needed to close that privacy gap.","For large models, the flat back-propagation cost is the most distinctive property; whether that advantage survives when the classifier head itself is huge is an open extension."],"forward_implications":["Any federated classifier with a final linear layer can add FedDW by uploading per-class average softmax vectors, so adoption requires no change to the backbone architecture.","The extra communication is a $|C|\\times|C|$ matrix per client per round, which stays small relative to model parameters even for large networks.","Because the regularization gradient involves only the classification layer, the per-epoch client runtime stays near FedAvg; the paper measures a 0.06 ms/epoch difference on ResNet-18.","The DW regularizer can be stacked on other FL methods, improving accuracy when combined with FedAvg, FedDistill, MOON, and FedProx in the reported experiments.","The convergence bound of $O(1/T)$ matches standard federated rates under convex loss, so the added term does not worsen the asymptotic optimization guarantee."],"supporting_citations":[{"why":"Supplies the base aggregation algorithm and the accuracy baseline FedDW builds on.","marker":"[4]"},{"why":"Provides the proximal-term regularization family and the convergence assumptions FedDW adapts.","marker":"[6]"},{"why":"Gives the proof template for FedDW's convergence rate and the shared-feature baseline.","marker":"[11]"},{"why":"Supplies the prototype-regularization setup and the proof for the one-round deviation bound.","marker":"[14]"},{"why":"Defines federated distillation with soft labels, the closest family FedDW extends.","marker":"[17]"},{"why":"Introduces knowledge distillation and soft labels, the source of the SL matrix.","marker":"[18]"},{"why":"Motivates the similarity-based interpretation of weight vectors via contrastive learning.","marker":"[25]"}],"fun_headline_variants":["FedDW: 3% accuracy boost in heterogeneous federated learning","New regularizer tames data skew in federated learning","Soft labels and weight geometry: FedDW's simple fix for non-IID FL","FedDW leverages classifier consistency to outdo 10 FL baselines","Distilling weights: FedDW's low-cost path to IID-like federated performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"FedDW presupposes that under evenly spread (IID) data, the matrix of average per-class predictions equals the classifier's weight-similarity matrix, which makes the global prediction matrix a valid target for local classifier weights.","fun_headline_variants_meta":{"raw":{"variants":["FedDW: 3% accuracy boost in heterogeneous federated learning","New regularizer tames data skew in federated learning","Soft labels and weight geometry: FedDW's simple fix for non-IID FL","FedDW leverages classifier consistency to outdo 10 FL baselines","Distilling weights: FedDW's low-cost path to IID-like federated performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000408,"raw_usage":{"total_tokens":2094,"prompt_tokens":897,"completion_tokens":1197,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":1101}},"tokens_in":513,"tokens_out":1197,"duration_ms":9307,"temperature":1.0,"reasoning_tokens":1101,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:45:07.449631+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a FedAvg model on an IID partition of CIFAR-10, compute the global soft-label matrix $\\Omega$ and the row-softmaxed CR matrix $\\sigma(\\omega\\omega^\\top)$ at the end of training, and measure $\\|\\Omega - \\sigma(\\omega\\omega^\\top)\\|_F$. If this distance is not close to zero, or does not shrink with more IID data or more classes, the consistency principle at the core of FedDW is violated and the regularization target is biased.","supporting_citations":[{"cited_title":"FedGH: Heterogeneous federated learning with generalized global header,","cited_arxiv_id":null,"evidence_quote":"Gives the proof template for FedDW's convergence rate and the shared-feature baseline."},{"cited_title":"Communication-efficient learning of deep networks from decentralized data,","cited_arxiv_id":null,"evidence_quote":"Supplies the base aggregation algorithm and the accuracy baseline FedDW builds on."},{"cited_title":"Federated optimization in heterogeneous networks,","cited_arxiv_id":null,"evidence_quote":"Provides the proximal-term regularization family and the convergence assumptions FedDW adapts."},{"cited_title":"Fedproto: Federated prototype learning across heterogeneous clients,","cited_arxiv_id":null,"evidence_quote":"Supplies the prototype-regularization setup and the proof for the one-round deviation bound."},{"cited_title":"A simple framework for contrastive learning of visual representations,","cited_arxiv_id":null,"evidence_quote":"Motivates the similarity-based interpretation of weight vectors via contrastive learning."}],"review_version":1}