{"id":"3137ba16-4c60-4b5d-836e-9baa44452dcc","arxiv_id":"2502.03231","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In federated learning, model averaging progressively degrades feature quality and feature-classifier alignment with network depth, a pattern the authors call Cumulative Feature Degradation.","lead":"This paper shows that averaging different clients' models in federated learning degrades the learned features layer by layer, with the damage growing as information moves through deeper layers. The finding offers a concrete feature-level explanation for why common fixes such as personalization, pretrained initialization, and classifier fine-tuning work.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal claim that CFD drives the accuracy drop is not isolated; the depth-increasing relative-change metric in Eq. (14) may be partly a normalization artifact, and no intervention ties the metric to performance.","rationale":"The reader's conditional verdict is appropriate. The paper documents a plausible and reproducible empirical regularity: after FedAvg aggregation, normalized within-class variance rises, between-class variance falls, and feature-parameter alignment drops, with trends that strengthen with depth. However, the strongest claim is causal: CFD significantly impairs penultimate-layer features and coupling, ultimately degrading performance. The manuscript does not isolate this mechanism from alternative explanations. My specific technical concern adds to the reader's weakest assumption: the primary CFD-V metric is computed as a relative change of a normalized quantity whose magnitude decreases with depth, so the monotonic depth trend can arise from the metric's denominator alone. This is not an accusation of error but a concrete, testable confound. The proposed intervention directly manipulates localized components of the post-aggregated model and measures whether CFD reductions predict accuracy recovery, which would settle the causal question. The label-skew extension tests whether the cross-domain-only evidence supports the general FL claim. Given that the empirical observations remain valuable and the framework is coherent, the correct response is to keep the conditional verdict and request these checks rather than accept the causal framing or reject the paper.","tokens_in":71110,"tokens_out":3497,"duration_ms":53570,"concrete_test":"Run an intervention study on DomainNet with ResNet34: after FedAvg aggregation at a fixed global round, for each client separately replace only the classifier head with that client's pre-aggregation head, then measure test accuracy and the change in CFD-A at the penultimate layer. Repeat this replacement for the first stage, middle stages, and penultimate stage, recording CFD-V/CFD-A before and after each replacement. If accuracy recovery does not track the reduction in CFD metrics, the causal claim fails. Additionally, vary local epochs E in {1,5,10,20} and use a label-skew Dirichlet(0.5) partition, then compute the Spearman correlation across runs between the post-aggregation accuracy drop and the average CFD-V/CFD-A; if the correlation is weak or non-monotonic, the metrics are not the driver of performance degradation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Sections 4.2-4.3) is that cumulative feature degradation (CFD-V/CFD-A) is a fundamental cause of the post-aggregation accuracy drop. The evidence is correlational and confounded by the metric definition itself. The relative-change metric in Eq. (14), Delta^l(V) = |V_post - V_pre| / (|V_pre| + |V_post|) * 100%, is applied to normalized variances (Eqs. 8-9) that systematically shrink with depth (Figs. 2-3). For a fixed absolute perturbation in the covariance traces, a smaller denominator at deeper layers automatically yields a larger relative change, so the observed monotonic depth trend in CFD-V may be a scaling artifact rather than evidence of accumulating degradation. The feature-distance result in Fig. 5, where feature distance grows with depth while parameter distance shrinks, is more robust, but it is not quantitatively linked to the accuracy drop. No experiment manipulates the CFD metrics while holding all other properties of the model fixed. The solution analyses in Section 5 alter multiple components simultaneously: personalizing shallow layers changes local training trajectories, pretraining changes the initialization distribution, and classifier fine-tuning directly changes the classifier that defines accuracy. These interventions therefore do not isolate the hypothesized mechanism. The claim is also only tested under cross-domain heterogeneity (Appendix B), leaving label-skew heterogeneity, the most common FL setting, unexamined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a layer-peeled feature analysis framework to study how model aggregation affects feature representations in federated learning (FL). The framework measures normalized within-class/between-class feature variance, linear probing accuracy, and alignment between features and subsequent parameters. Across three cross-domain datasets and several architectures (ConvNet, ResNet variants, VGG, ViT), the authors report that aggregation degrades feature discrimination and feature-parameter alignment, that this degradation accumulates with network depth (termed Cumulative Feature Degradation, CFD), and that it most severely affects penultimate-layer features and their coupling with the classifier. They then reinterpret three common FL remedies—parameter personalization, pretrained initialization, and classifier fine-tuning—as mitigating CFD. The paper also shows that the post-aggregated model achieves better out-of-distribution generalization.","tokens_in":71360,"tokens_out":3308,"duration_ms":32551,"significance":"If the main claims hold, this is a useful diagnostic perspective: it provides single-model, layer-wise metrics for understanding why FedAvg exhibits temporary performance drops under heterogeneity, and it offers an interpretable explanation for why several existing FL solutions work. The empirical coverage is broad (three datasets, six architectures), and the proposed metrics are defined externally with no fitted parameters, so the framework itself is not circular. The paper's value is primarily observational and explanatory rather than prescriptive; it does not introduce a new FL algorithm but may inspire mechanism-aware designs.","major_comments":[{"comment":"The depth-increasing relative change in normalized variance (CFD-V) is confounded by the normalization in Eqs. (8)-(9). Since normalized within-class variance systematically decreases with depth (Figs. 2a and 3a), and the relative change in Eq. (14) divides by |V_pre| + |V_post|, a constant absolute perturbation produces a larger relative change at deeper layers even with no true accumulation. The authors should analyze the absolute changes of the unnormalized variances (which they report in Appendix G) or directly regress the relative change against depth while controlling for the denominator magnitude. Without this, the central 'cumulative' claim for CFD-V is not established.","section":"§4.2"},{"comment":"The paper states that all experiments are repeated three times with different random seeds, but no figure or table reports error bars, standard deviations, or confidence intervals. This matters because many claims are about monotonic trends (e.g., relative change increasing with depth in Figs. 2-3, the penultimate-layer spike in Fig. 6). I ask the authors to add error bars or shaded regions to at least the main figures, and to report seed-level variance for the key quantitative claims.","section":"§3.1"},{"comment":"The paper repeatedly states that CFD 'impairs', 'degrades', and 'ultimately degrades model performance', but the evidence is correlational. No experiment manipulates the CFD metrics while holding all other model properties fixed. The solution analyses in Section 5 alter multiple components simultaneously: personalizing shallow layers changes local training trajectories, pretraining changes the initialization distribution, and classifier fine-tuning directly changes the classifier that defines accuracy. These interventions therefore do not isolate the hypothesized mechanism. At minimum, the authors should either (a) soften the causal language to 'is associated with' and 'is consistent with', or (b) add a controlled experiment (e.g., perturbing penultimate features in a trained model and measuring accuracy) that directly links the metric to performance.","section":"§4.2"},{"comment":"All experiments use cross-domain heterogeneity, where each client has a different input domain. Label-skew heterogeneity (different class distributions across clients) is the most common FL heterogeneity in the literature and is absent from this study. The claim that CFD is a fundamental cause of the aggregation-induced accuracy drop cannot be generalized without at least one label-skew setting (e.g., Dirichlet-partitioned CIFAR-10 or FEMNIST). Without this, the scope of the paper's main conclusion is limited to cross-domain FL.","section":"Appendix B"},{"comment":"The generalization result in Section 4.4 appears to conflict with the paper's overall 'degradation' framing: the post-aggregated model has worse ID accuracy but better OOD accuracy. The paper does not quantify this trade-off or reconcile it with the claim that CFD 'degrades' feature quality. If aggregation improves generalization, then the observed variance and alignment changes are not uniformly negative. I recommend adding a quantitative comparison of the ID accuracy drop versus OOD accuracy gain, and discussing how the benefit of aggregation interacts with the CFD phenomenon.","section":"§4.4"}],"minor_comments":[{"comment":"The manuscript contains numerous typos and inconsistent notation. Examples include 'intermidiate' (Eq. 4), 'featre' (Section 4.1), 'Reset50' (Section 4.3), 'VGG1_BN' (Appendix C, should be VGG13_BN), 'presenst' (Section 5.2), 'aligment' (Figure 9 caption), and inconsistent use of 'pre-aggregated' vs 'pre-aggregated' (hyphenation). A thorough language and proofreading pass is needed.","section":"Throughout"},{"comment":"The axis labels in Figures 2, 3, and 4 are difficult to read, and the figure captions do not explain the color scheme or what 'L1', 'L2', etc. refer to. I recommend larger fonts and a legend that maps layer indices to the architecture.","section":"Figures 2-4"},{"comment":"The alignment metric is described as using principal angles between the range space of class-wise feature means and the top-C subspace of the subsequent layer parameters. The description of the SVD-based computation for convolutional layers is dense and should include an explicit dimension check or a small example, especially given that the metric is central to CFD-A.","section":"§3.2"},{"comment":"The 'Discussion and Limitations' section is only in the appendix. Given the strong causal language used in the main text, I recommend moving a condensed limitations discussion into the main body (or at least explicitly referencing it in Section 4.2 where the causal claims are made).","section":"Appendix Q"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study with a new diagnostic framework, but its central causal claim (CFD drives the accuracy drop) is not isolated and the depth-trend metric has a normalization confound. The absence of error bars and the limited heterogeneity setting further weaken the quantitative support. I believe the issues are fixable within the scope of the manuscript, so major revision is appropriate rather than rejection. The authors should be encouraged to add controlled experiments or substantially soften the causal language."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before reading. First, the paper actually delivers a new diagnostic observation: Cumulative Feature Degradation (CFD), the finding that aggregation degrades feature quality and feature-parameter alignment progressively with depth, is documented across three datasets and several architectures and is genuinely worth taking seriously. Second, the paper's central causal claim—that CFD drives the post-aggregation accuracy drop—is not established by the experiments, and the authors' language (\"impairs\", \"degrades\", \"fundamental factor\") goes beyond what the evidence supports.\n\nWhat is actually new: the layer-peeled framework applies single-model metrics—normalized within/between-class variance and principal-angle alignment—to pre- and post-aggregation models. Earlier layer-wise FL analyses rely on pairwise model comparisons or parameter-space loss metrics, so the single-model diagnostic angle is a real step forward. The CFD trend itself, with the worst disruption at the penultimate layer and classifier interface, is a new observation. Section 4.4 also earns its keep: aggregation hurts local discrimination but improves out-of-distribution transfer, which is a useful complement to the degradation story. The reinterpretation of FedPer, FedBN, pretraining, and classifier fine-tuning as CFD mitigations is plausible and worth having, even if it is not a proof.\n\nThe soft spots are in proportion. The load-bearing issue is that no experiment manipulates the CFD metrics while holding everything else fixed; the Section 5 solutions change multiple components at once, so they do not isolate the mechanism. The stress-test concern about Eq. (14) is partly right: normalized within-class variance shrinks with depth, so some of the monotone CFD-V trend could come from the denominator. That said, the between-class and alignment trends and the feature-distance result in Figure 5 are less vulnerable, so the phenomenon is probably real rather than purely an artifact. The paper should show unnormalized or standardized changes and downrank the causal phrasing. Also, three seeds are averaged without error bars, and only cross-domain heterogeneity is tested; label-skew, the more common FL setting, is unexamined.\n\nBottom line: this is a diagnostic paper, not a proof. For readers working on FL feature analysis or aggregation-aware algorithms, the CFD framework and observations are useful and cite-worthy. A careful referee should engage with it rather than desk-reject; the revisions I'd demand are softer causal language, error bars, a label-skew experiment, and either an intervention or a partial-correlation analysis connecting the metrics to performance.","headline":"The CFD phenomenon is a real and useful diagnostic observation; the causal story overreaches, and part of the depth trend in CFD-V may be a normalization artifact.","tokens_in":71879,"tokens_out":2582,"would_cite":true,"duration_ms":26798,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In federated learning, averaging client models degrades learned feature representations and their coupling to later layers; the damage accumulates with depth (Cumulative Feature Degradation) and explains the recurring post-aggregation…","keywords":["federated learning","model aggregation","layer-peeled analysis","feature representations","cumulative feature degradation","feature-parameter alignment","feature variance","personalized federated learning"],"falsifier":"In a fixed federated round, record the pre-aggregation penultimate features and classifier, aggregate normally, then surgically restore only the pre-aggregation alignment (or feature variance) in the post-aggregation model—for instance by locally re-fitting the classifier or projecting features—while leaving the other metric untouched. If local accuracy does not recover, the claimed causal role of Cumulative Feature Degradation is not supported.","tokens_in":70923,"feed_emoji":"📉","tokens_out":7265,"duration_ms":61582,"temperature":0.7,"pith_summary":"This paper tries to explain a familiar but poorly understood event in federated learning: right after the server averages client models, the global model performs noticeably worse on local data than the models that were averaged did. Its proposed explanation is that aggregation damages the geometry of the features the network has learned—same-class features become less compact, different-class features become less separated—and also weakens how well each layer's features line up with the parameters of the layer that consumes them. The damage is not confined to one layer; the paper's measurements show it compounding through the network, a phenomenon it names Cumulative Feature Degradation, so the penultimate layer and its classifier take the hardest hit. If this diagnosis is right, common remedies (personalizing selected layers, starting from pretrained weights, fine-tuning the classifier locally) succeed for a shared reason: they interrupt or shorten that accumulation path. The paper also shows aggregation improves out-of-distribution generalization, so its picture is double-sided rather than purely negative.","feed_headline":"Layer by layer, federated averaging degrades features","feed_subtitle":"New metrics trace the post-aggregation accuracy drop to cumulatively damaged features and a mismatched classifier.","key_machinery":"The core object is a layer-peeled diagnostic framework that evaluates, for every layer $\\ell$, two quantities: the normalized within-class variance $\\bar\\sigma^\\ell_W = \\operatorname{Tr}(\\Sigma^\\ell_W)/\\operatorname{Tr}(\\Sigma^\\ell_T)$ and normalized between-class variance $\\bar\\sigma^\\ell_B = \\operatorname{Tr}(\\Sigma^\\ell_B)/\\operatorname{Tr}(\\Sigma^\\ell_T)$, together with the alignment between the range space of class-wise feature means $\\bar Z^\\ell$ and the top-$C$ subspace of the next layer's parameter matrix $W^{\\ell+1}$, measured by the mean singular value of $(V^{\\ell+1}_W)^\\top U^\\ell_{\\bar Z}$ (principal angles between subspaces). The relative change $\\Delta^\\ell(V) = |V^\\ell_{\\mathrm{post}} - V^\\ell_{\\mathrm{pre}}| / (|V^\\ell_{\\mathrm{pre}}| + |V^\\ell_{\\mathrm{post}}|) \\times 100\\%$ before versus after aggregation is the quantity that reveals Cumulative Feature Degradation, and the same machinery is then applied to FedPer, FedBN, progressive personalization, pretrained initialization, and classifier fine-tuning to locate which part of the degradation each method repairs.","core_discovery":"The paper's central claim is that the post-aggregation accuracy drop in federated learning is caused mainly by damage to the feature extraction process itself, not just by parameter divergence or client drift. Using metrics computed directly on a single model—normalized within-class and between-class feature variance, and the alignment between the subspace of class-mean features and the input subspace of subsequent parameters (computed via principal angles)—the authors observe that aggregation raises within-class variance, lowers between-class variance, and weakens feature-parameter alignment. These disruptions grow with network depth: degraded features from early layers propagate into later ones, so the relative change of the metrics increases layer by layer. The paper names this accumulation Cumulative Feature Degradation (CFD) and identifies two concrete manifestations, CFD-V in feature variance and CFD-A in alignment, with the sharpest alignment spike at the penultimate-layer/classifier interface. It further argues that personalizing parameters, pretrained initialization, and classifier fine-tuning all work because they mitigate CFD, while the same aggregation that hurts local discrimination improves generalization to other clients' distributions.","pith_inferences":["Editorial extension: the metrics are observational, so a direct intervention test—such as re-projecting post-aggregation penultimate features back to the pre-aggregation variance and alignment values and checking whether local accuracy recovers—would convert the correlation into causation.","Editorial extension: if CFD accumulates with depth, then architectures with more stages (deeper ResNets, long ViTs) should show larger relative changes in late-layer metrics, a prediction that could be tested by sweeping depth while holding data and aggregation fixed.","Editorial extension: the framework suggests a tunable design—keep shallow layers global (they carry transferable, generalizable features) and personalize or locally fine-tune deeper layers (where task-specific compression happens)—which would sit between FedAvg and full personalization on the generalization-localization trade-off."],"forward_implications":["If CFD is the true cause of the post-aggregation drop, then treating the drop as an unavoidable cost of knowledge sharing is no longer necessary; convergence could be accelerated by protecting feature structure across the aggregation step.","Deeper layers—especially the penultimate layer and its classifier—are the fragile point, so aggregation-aware methods should pay special attention to re-aligning the final classifier with locally produced features after each round.","The three studied solution families (parameter personalization, pretrained initialization, classifier fine-tuning) succeed by the same mechanism of mitigating feature degradation, which gives a principled design criterion rather than separate empirical stories.","Because aggregation simultaneously hurts local feature discrimination and improves cross-client generalization, methods that simply reduce aggregation strength may trade away the generalization benefit; the paper implies the goal is to repair local degradation while keeping the fusion advantage."],"supporting_citations":[{"why":"Defines the FedAvg protocol whose aggregation step and performance drop are the object of study.","marker":"[2]"},{"why":"Attributes the drop to client drift; the paper contrasts this parameter-space explanation with its feature-level account.","marker":"[4]"},{"why":"Supplies the layerwise feature compression and discrimination perspective that motivates the within/between-class variance metrics.","marker":"[25]"},{"why":"Provides intermediate neural collapse results and the principal-angle alignment methodology reused for feature-parameter coupling.","marker":"[27]"},{"why":"Define principal angles between subspaces, the computation behind the alignment metric.","marker":"[39, 40]"},{"why":"FedPer, the classifier-personalization method revisited in Section 5.1 as one of the effective mitigations.","marker":"[13]"},{"why":"FedBN, the batch-normalization personalization method revisited as mitigating CFD by preserving shallow feature extraction.","marker":"[10]"},{"why":"Pretrained-initialization studies revisited in Section 5.2 for the claim that they shorten the effective CFD path.","marker":"[16, 17]"},{"why":"Classifier fine-tuning methods revisited in Section 5.3 for the claim that they restore penultimate-feature/classifier alignment.","marker":"[18, 19]"}],"fun_headline_variants":["Cumulative feature damage explains federated averaging's accuracy dip","Layer by layer, aggregation degrades features, harming FL performance","Why federated models underperform: aggregation's hidden feature cost","Feature degradation, not divergence, drives federated learning drops","Aggregation's real downside: feature quality suffers with depth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the two measured quantities—feature variance and feature-parameter alignment—capture what actually determines downstream performance, and that the post-aggregation changes in them cause the accuracy drop rather than merely accompany it.","fun_headline_variants_meta":{"raw":{"variants":["Cumulative feature damage explains federated averaging's accuracy dip","Layer by layer, aggregation degrades features, harming FL performance","Why federated models underperform: aggregation's hidden feature cost","Feature degradation, not divergence, drives federated learning drops","Aggregation's real downside: feature quality suffers with depth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000743,"raw_usage":{"total_tokens":3361,"prompt_tokens":1042,"completion_tokens":2319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":2235}},"tokens_in":658,"tokens_out":2319,"duration_ms":15744,"temperature":1.0,"reasoning_tokens":2235,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T05:27:07.507351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a fixed federated round, record the pre-aggregation penultimate features and classifier, aggregate normally, then surgically restore only the pre-aggregation alignment (or feature variance) in the post-aggregation model—for instance by locally re-fitting the classifier or projecting features—while leaving the other metric untouched. If local accuracy does not recover, the claimed causal role of Cumulative Feature Degradation is not supported.","supporting_citations":[{"cited_title":"Feature learning in deep classifiers through intermediate neural collapse","cited_arxiv_id":null,"evidence_quote":"Provides intermediate neural collapse results and the principal-angle alignment methodology reused for feature-parameter coupling."},{"cited_title":"Fedbn: Federated learning on non-iid features via local batch normalization","cited_arxiv_id":null,"evidence_quote":"FedBN, the batch-normalization personalization method revisited as mitigating CFD by preserving shallow feature extraction."}],"review_version":1}