Pith. sign in

REVIEW 5 major objections 5 minor 28 references

FedAH: Aggregated Head for Personalized Federated Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FedAH claims that blending each client's previous local head with the server's current global head, using per-element learned weights, improves personalized federated learning accuracy across vision and text benchmarks.

desk verdict Honest incremental empirical paper on personalized-head FL; solid experiments but the theory promise is empty, the novelty claim is oversold, and the abstract needs qualification. read the letter →

arxiv 2412.01295 v1 pith:YGI7NEI4 submitted 2024-12-02 cs.LG cs.AIcs.DC

Pengzhan Zhou , Yuepeng He , Yijun Zhai , Kaixin Gao , Chao Chen , Zhida Qin , Chong Zhang , Songtao Guo This is my paper · ORCID
classification cs.LGcs.AIcs.DC
keywords PersonalizedfederatedlearningStatisticalheterogeneityAggregatedheadElement-wiseaggregationFeatureextractorModelTestaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes FedAH, a personalized federated learning method that modifies how each client's classification head is initialized. Instead of keeping the locally trained head from the previous round or ignoring the head split entirely, FedAH computes an Aggregated Head as an element-wise blend of the client's previous local head and the server's current global head, with per-element weights learned by gradient descent. The claim is that this one change lets the personalized head retain global knowledge while still fitting local data, and the experiments report that it beats ten existing federated learning methods on five image and text benchmarks, with the largest gain being 2.87% over the second-best method on Cifar100. If true, it means a client-specific head does not have to be trained in isolation to achieve personalization; a small learned interpolation step can capture both global and local information.

What carries the argument

The central object is the Aggregated Head, defined by the element-wise interpolation formula $\hat{h}_i^t = h_i^{t-1} + (h^t - h_i^{t-1}) \odot W_i^t$, with $W_i^t \in [0,1]$ learned via a gradient step per round (Eq. 4). This single mechanism does three jobs: it injects the global head's information into each client's personalized head, it preserves the client-specific part through the $1 - W_i^t$ weighting, and it subsumes FedAvg ($W=1$) and FedPer/FedRep ($W=0$) as extreme cases. The rest of the method—freezing the extractor while training the head, then freezing the head while training the extractor—follows the FedRep training loop, so the only new machinery is the learned blend and its gradient-based weight update.

What would settle it

Run FedAH and FedRep on a synthetic federated task with a known linear classifier and fully disjoint label partitions across clients; if the global head's parameters for a client's unseen classes are random, the learned aggregation weights should drive those entries of $W_i^t$ to zero, and the accuracy gain over FedRep should vanish.

Watch

Extended reading notes

Core claim

FedAH's central claim is that introducing global information at the level of the model head, not just the feature extractor, improves personalized accuracy. At each round, client $i$ computes $\hat{h}_i^t := h_i^{t-1} + (h^t - h_i^{t-1}) \odot W_i^t$, where $h_i^{t-1}$ is the client's head from the previous round, $h^t$ is the server-aggregated global head, and $W_i^t$ is a learned per-element weight vector clipped to $[0,1]$. The weights are trained by gradient descent with all other parameters frozen. The client then trains the blended head while freezing the shared feature extractor, then trains the feature extractor with the head frozen, following the alternating protocol of FedRep. The paper reports that this Aggregated Head initialization outperforms ten state-of-the-art federated learning baselines across MNIST, Cifar10, Cifar100, Tiny-ImageNet, and AG News, and remains ahead under stronger heterogeneity, larger client counts, and client dropout.

Load-bearing premise

The load-bearing premise is that the global head—the weighted average of the selected clients' locally trained heads—is a meaningful source of transferable knowledge for every client, so that blending it into each local head with learned weights improves accuracy rather than injecting noise.

Editorial extensions

If this is right

  • Personalized-head federated learning does not have to forfeit global knowledge: blending the global head into the local head before local training yields higher test accuracy than keeping heads fully local.
  • The learned weight vector $W_i^t$ acts as a per-client, per-parameter gate; when a client's local head is informative it can suppress global influence, and when local data are scarce it can lean on the global head.
  • Because the Aggregated Head is computed from the previous local head and the current global head, the method requires no extra communication beyond the standard model upload/download, and it converges in fewer iterations than baselines in the reported settings.
  • Under client dropout, re-joining clients re-learn their aggregation weights at the start of each round, which the experiments show keeps accuracy stable relative to fixed-ratio baselines.
  • The method extends to text classification (AG News) and larger models (ResNet-18 on Tiny-ImageNet), indicating the benefit is not limited to small CNN image tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the blend's benefit comes from the global head carrying transferable class-boundary information, then the gain should shrink when client label distributions are completely disjoint: there the global head's parameters for a client's unseen classes are arbitrary, and the learned $W_i^t$ entries would need to be driven to zero for those classes to avoid noise. This is a testable stress case the pa
  • The per-element weight learning is a form of learned interpolation; one could extend it to per-layer or per-block weights in deeper heads, or add a momentum term that smooths $W_i^t$ across rounds, with no change to the communication pattern.
  • Because $W_i^t$ is trained with all other parameters frozen, the method is compatible with robust or privacy-preserving aggregation on the server: the global head could be replaced by any robust aggregate, and the local weight learning would adapt.
  • The reported accuracy gain over FedALA is largest on Cifar100; a natural follow-up is to measure whether the head blend helps most when the label space is large relative to the feature dimension, since that is where the head carries the most class-specific information.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper proposes FedAH, a personalized federated learning method that combines the head-only personalization of FedRep with an element-wise interpolation between each client's previous head and the server-aggregated global head. The interpolation weights W_i^t are learned by one gradient step per round under a [0,1] clipping constraint (Eqs. 3-4). The server aggregates both feature extractor and head; clients fine-tune the aggregated head and then the shared extractor (Eqs. 5-6). Experiments on MNIST, Cifar10, Cifar100, Tiny-ImageNet, and AG News compare FedAH with ten baselines under pathological and Dirichlet-heterogeneous settings, plus scalability, dropout, and local-epoch ablations. FedAH reports the best average test accuracy in most settings, with the largest margin on Cifar100 (2.87% over FedALA), but is slightly below FedALA on MNIST under the practical setting.

Significance. If the mechanism is as claimed, FedAH would be a practically useful, low-complexity extension of FedRep and FedALA, and the paper provides extensive, careful evaluation with standard deviations and open code. The contribution is, however, incremental relative to FedALA, and the central explanatory claim—that the learned element-wise head aggregation (Eq. 3) is responsible for the gains—is not isolated by ablations. The paper also promises but does not deliver a theoretical derivation. The experimental evidence is strong enough to justify a major revision rather than rejection.

major comments (5)
  1. [Section III] Section III, first paragraph, states that the authors will 'finally perform a theoretical derivation of FedAH,' but no theoretical derivation appears anywhere in the section or the rest of the paper. The method description in Section III-C is algorithmic only. The paper should either supply a convergence or noise-robustness argument for the learned aggregation weights W (e.g., under heterogeneous label distributions), or explicitly retract the promise. This is not merely a wording issue: Section IV-F attributes FedAH's dropout stability to the learned W without any supporting analysis.
  2. [Section III-C, Eqs. (3)-(4)] The central attribution claim is untested. FedAH differs from FedRep by (i) sharing and aggregating the head, (ii) transmitting the full model rather than only the feature extractor, and (iii) the learned interpolation W in Eq. (3). The experiments compare against FedRep, FedPer, FedALA, etc., but never fix W=1 or W=0 within FedAH. Without these ablations (e.g., on Cifar100 and TINY*), the observed improvement cannot be attributed specifically to the learned element-level aggregation weights. Add results for FedAH with W identically 1, with W identically 0, and (optionally) with a random fixed W.
  3. [Algorithm 1, line 12] Algorithm 1 returns {r_i, \hat h_i} for each client, but \hat h_i is the aggregated head before local training (Eq. 3, step 7), while the trained head h_i^t is produced in step 8. This makes the output specification inconsistent with the evaluation, which uses the average test accuracy of the best local models. Clarify what the final personalized model is and correct the pseudocode accordingly.
  4. [Equation (4)] Equation (4) is circular as written: the loss is evaluated at \hat h_i^t, which by Eq. (3) depends on W_i^t, the same quantity being updated. Specify whether \hat h_i^t in Eq. (4) is computed using W_i^{t-1} or the current W_i^t, and state the exact number of gradient steps used to train W_i^t per round. This is necessary for reproducibility of the method.
  5. [Abstract and Table I] The abstract and conclusion overstate the results. Table I shows FedAH is not the best on MNIST under the practical heterogeneous setting (99.66 vs. FedALA's 99.71), and the 2.87% margin is specific to Cifar100, not a general average across all tasks. The claim that FedAH 'outperforms ten SOTA FL methods in terms of test accuracy by 2.87%' should be qualified to match the reported results.
minor comments (5)
  1. [Algorithm 1 and Eq. (4)] The notation for learning rates is inconsistent: Algorithm 1 lists 'weight learning rate α' while Eq. (4) uses η for the weight update. Section IV-B says the aggregation-weight learning rate is set to the local learning rate, but the symbols should be made consistent in the algorithm and equations.
  2. [Table I caption] The text introduces the notation 'TINY' and 'TINY*' but Table I's caption does not define them; the caption should state that TINY uses a 4-layer CNN and TINY* uses ResNet-18.
  3. [Section IV-E] The phrase 'we can theoretically compare the communication overhead' is vague; the comparison is a parameter-count calculation, not a theoretical analysis, and the wording should be adjusted.
  4. [Section IV-A] The statement that all tasks are run for 2000 iterations until all methods empirically converge conflicts with Table III's early-stopping iteration counts (e.g., FedAH 40 iterations, FedAvg 165); clarify the stopping rule and how the 2000-iteration budget is used.
  5. [Section IV-B] The sentence 'the learning rate of the aggregation weights in FedAH is set to be the same as the local learning rate' should give the actual numerical value rather than only a relationship, since Table I does not report hyperparameters for the methods.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FedAH's aggregation weights are learned on local data and are not fitted to the reported test accuracies.

full rationale

The derivation chain in FedAH does not reduce to its own inputs. Equation (3) defines the aggregated head as h_hat_i^t := h_i^{t-1} + (h^t - h_i^{t-1}) ⊙ W_i^t, but W_i^t is not a free parameter tuned to the reported empirical results. It is trained on client i's local data via Equation (4), with the global head and feature extractor frozen, using gradient descent and element-wise clipping. The resulting head and extractor are then trained on local data through Equations (5) and (6), and test accuracy is measured on a held-out 25% portion of each client's local dataset. No equation is identical by construction to the claimed accuracy improvement, and no fitted quantity is renamed as a prediction. The paper contains no load-bearing self-citation: the methods it builds on, such as FedRep and FedALA, are external prior work by other authors, and the weight-clipping regularization is explicitly attributed to FedALA, not to the present authors. The claimed 2.87% advantage over ten baselines is an empirical comparison against independent methods, not a definitional consequence of the FedAH update rules. One non-circular concern: Section III states that the authors will 'perform a theoretical derivation of FedAH,' but no such derivation appears in the paper; this is an omitted-proof or completeness gap, not a circularity step. The resemblance of Equation (3) to FedALA-style adaptive aggregation is a novelty and attribution question, not evidence that the result is forced by the input definition. Overall circularity score: 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method rests on two modeling axioms about the head and its global average, and one unproven assumption that a single gradient step on the aggregation weights suffices. No new physical or architectural entities are postulated.

free parameters (2)
  • Initial aggregation weight value for W_i^t = 1 (all elements)
    Chosen by hand in Section III-C; every element starts at 1 so the first Aggregated Head equals the global head. The value is a design choice, not fitted to test accuracy.
  • Weight clipping range = [0, 1]
    Imposed after Eq. (4) via sigma(w)=max(0,min(1,w)), following FedALA; keeps the Aggregated Head inside the convex hull of local and global heads.
assumptions (3)
  • domain assumption The model can be split into a shared feature extractor and a client-specific head (the last FC layer), and this split preserves enough shared representation to make personalization effective.
    Invoked in Section III-B, following FedPer and FedRep; if the split is wrong for a task, the method's benefit disappears.
  • domain assumption The global head h^t, formed by weighted averaging of local heads over sampled clients, is meaningful for all clients even under non-IID data.
    Server aggregation in Algorithm 1, line 11; this is the key premise behind introducing global head information.
  • ad hoc to paper Element-wise aggregation weights in [0,1], updated by a single gradient step per iteration, can capture useful global information without destroying personalization.
    Eq. (3)-(4); no convergence or optimality proof is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FedAH: Aggregated Head for Personalized Federated Learning." pith.science (2026). https://pith.science/paper/YGI7NEI4

@misc{pith2026241201295,
  author       = {Pith},
  title        = {Pith review of: FedAH: Aggregated Head for Personalized Federated Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGI7NEI4}},
  note         = {Machine review of arXiv:2412.01295}
}
read the original abstract

Recently, Federated Learning (FL) has gained popularity for its privacy-preserving and collaborative learning capabilities. Personalized Federated Learning (PFL), building upon FL, aims to address the issue of statistical heterogeneity and achieve personalization. Personalized-head-based PFL is a common and effective PFL method that splits the model into a feature extractor and a head, where the feature extractor is collaboratively trained and shared, while the head is locally trained and not shared. However, retaining the head locally, although achieving personalization, prevents the model from learning global knowledge in the head, thus affecting the performance of the personalized model. To solve this problem, we propose a novel PFL method called Federated Learning with Aggregated Head (FedAH), which initializes the head with an Aggregated Head at each iteration. The key feature of FedAH is to perform element-level aggregation between the local model head and the global model head to introduce global information from the global model head. To evaluate the effectiveness of FedAH, we conduct extensive experiments on five benchmark datasets in the fields of computer vision and natural language processing. FedAH outperforms ten state-of-the-art FL methods in terms of test accuracy by 2.87%. Additionally, FedAH maintains its advantage even in scenarios where some clients drop out unexpectedly. Our code is open-accessed at https://github.com/heyuepeng/FedAH.

Figures

Figures reproduced from arXiv: 2412.01295 by the authors.

Figure 1
Figure 1. An example for FedAH. r t : global feature extractor; h t : global heads; h t−1 1 , . . . , ht−1 n : local head of clients 1, . . . , n in the previous iteration; hˆt 1 , . . . , hˆt n : Aggregated Heads of clients 1, . . . , n in the current iteration. The remained paper is organized as following. Section II summarizes the related work. Section III demonstrates the methods used. Section IV explains the experiments.… view at source ↗
Figure 2
Figure 2. Local learning process of FedAH on client [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The inputs and outputs of the feature extractor and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Test accuracy (%) curves of different methods on Tiny [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 5 canonical work pages

  1. [1]

    Federated recommendation systems,

    L. Yang, B. Tan, V . W. Zheng, K. Chen, and Q. Yang, “Federated recommendation systems,” Federated Learning: Privacy and Incentive , pp. 225–239, 2020

    2020

  2. [2]

    Federated learning for internet of things: A comprehensive survey,

    D. C. Nguyen, M. Ding, P. N. Pathirana, A. Seneviratne, J. Li, and H. V . Poor, “Federated learning for internet of things: A comprehensive survey,” IEEE Communications Surveys & Tutorials , vol. 23, no. 3, pp. 1622–1658, 2021

    2021

  3. [3]

    Fedcp: Separating feature information for personalized federated learn- ing via conditional policy,

    J. Zhang, Y . Hua, H. Wang, T. Song, Z. Xue, R. Ma, and H. Guan, “Fedcp: Separating feature information for personalized federated learn- ing via conditional policy,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 3249– 3261

    work page 2023

  4. [4]

    Advances and open problems in federated learning,

    P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al. , “Advances and open problems in federated learning,” Foundations and trends® in machine learning , vol. 14, no. 1–2, pp. 1–210, 2021

    2021

  5. [5]

    Federated optimization in heterogeneous networks,

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,” Proceedings of Machine learning and systems , vol. 2, pp. 429–450, 2020

    2020

  6. [6]

    Towards personalized federated learning,

    A. Z. Tan, H. Yu, L. Cui, and Q. Yang, “Towards personalized federated learning,” IEEE transactions on neural networks and learning systems , vol. 34, no. 12, pp. 9587–9603, 2022

    2022

  7. [7]

    Eliminating domain bias for federated learning in representation space,

    J. Zhang, Y . Hua, J. Cao, H. Wang, T. Song, Z. Xue, R. Ma, and H. Guan, “Eliminating domain bias for federated learning in representation space,” Advances in Neural Information Processing Systems , vol. 36, 2024

    work page 2024

  8. [8]

    Communication-efficient learning of deep networks from decentralized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics . PMLR, 2017, pp. 1273– 1282

    2017

Show all 28 references
  1. [9]

    Federated machine learning: Concept and applications,

    Q. Yang, Y . Liu, T. Chen, and Y . Tong, “Federated machine learning: Concept and applications,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1–19, 2019

    2019

  2. [10]

    Per- sonalized cross-silo federated learning on non-iid data,

    Y . Huang, L. Chu, Z. Zhou, L. Wang, J. Liu, J. Pei, and Y . Zhang, “Per- sonalized cross-silo federated learning on non-iid data,” in Proceedings of the AAAI conference on artificial intelligence , vol. 35, no. 9, 2021, pp. 7865–7873

    2021

  3. [11]

    Personalized federated learning with moreau envelopes,

    C. T Dinh, N. Tran, and J. Nguyen, “Personalized federated learning with moreau envelopes,” Advances in Neural Information Processing Systems, vol. 33, pp. 21 394–21 405, 2020

    2020

  4. [12]

    Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach,

    A. Fallah, A. Mokhtari, and A. Ozdaglar, “Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach,” Advances in Neural Information Processing Systems, vol. 33, pp. 3557–3568, 2020

    2020

  5. [13]

    Federated meta-learning with fast convergence and efficient communication,

    F. Chen, M. Luo, Z. Dong, Z. Li, and X. He, “Federated meta-learning with fast convergence and efficient communication,” arXiv preprint arXiv:1802.07876, 2018

    2018 arXiv

  6. [14]

    Ditto: Fair and robust federated learning through personalization,

    T. Li, S. Hu, A. Beirami, and V . Smith, “Ditto: Fair and robust federated learning through personalization,” in International conference on machine learning . PMLR, 2021, pp. 6357–6368

    2021

  7. [15]

    Person- alized federated learning with first order model optimization,

    M. Zhang, K. Sapra, S. Fidler, S. Yeung, and J. M. Alvarez, “Person- alized federated learning with first order model optimization,” arXiv preprint arXiv:2012.08565, 2020

    2012 arXiv

  8. [16]

    Adapt to adaptation: Learning personalization for cross-silo federated learning,

    J. Luo and S. Wu, “Adapt to adaptation: Learning personalization for cross-silo federated learning,” in IJCAI: proceedings of the conference , vol. 2022. NIH Public Access, 2022, p. 2166

    2022

  9. [17]

    Fedala: Adaptive local aggregation for personalized federated learning,

    J. Zhang, Y . Hua, H. Wang, T. Song, Z. Xue, R. Ma, and H. Guan, “Fedala: Adaptive local aggregation for personalized federated learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 9, 2023, pp. 11 237–11 244

    2023

  10. [18]

    Federated learning with personalization layers,

    M. G. Arivazhagan, V . Aggarwal, A. K. Singh, and S. Choud- hary, “Federated learning with personalization layers,” arXiv preprint arXiv:1912.00818, 2019

    1912 arXiv

  11. [19]

    Exploiting shared representations for personalized federated learning,

    L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai, “Exploiting shared representations for personalized federated learning,” in Interna- tional conference on machine learning . PMLR, 2021, pp. 2089–2099

    2021

  12. [20]

    Model-contrastive federated learning,

    Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 10 713–10 722

    2021

  13. [21]

    Gradient-based learning applied to document recognition,

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998

    1998

  14. [22]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al. , “Learning multiple layers of features from tiny images,” 2009

    2009

  15. [23]

    A downsampled variant of imagenet as an alternative to the cifar datasets,

    P. Chrabaszcz, I. Loshchilov, and F. Hutter, “A downsampled variant of imagenet as an alternative to the cifar datasets,” arXiv preprint arXiv:1707.08819, 2017

    2017 arXiv

  16. [24]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

    2016

  17. [25]

    Character-level convolutional net- works for text classification,

    X. Zhang, J. Zhao, and Y . LeCun, “Character-level convolutional net- works for text classification,” Advances in neural information processing systems, vol. 28, 2015

    2015

  18. [26]

    Bag of tricks for efficient text classification,

    A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of tricks for efficient text classification,” arXiv preprint arXiv:1607.01759 , 2016

    2016 arXiv

  19. [27]

    Personalized federated learning using hypernetworks,

    A. Shamsian, A. Navon, E. Fetaya, and G. Chechik, “Personalized federated learning using hypernetworks,” in International Conference on Machine Learning . PMLR, 2021, pp. 9489–9502

    2021

  20. [28]

    Ensemble distillation for robust model fusion in federated learning,

    T. Lin, L. Kong, S. U. Stich, and M. Jaggi, “Ensemble distillation for robust model fusion in federated learning,” Advances in neural information processing systems , vol. 33, pp. 2351–2363, 2020

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.