REVIEW 3 major objections 5 minor 32 references
Optimizing Personalized Federated Learning through Adaptive Layer-Wise Learning
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read FLAYER treats each neural network layer differently to lift personalized federated learning accuracy by 5.4%.
desk verdict Good engineering recipe for personalized FL, but the server aggregation rule as written would zero out the global model; the paper needs a major revision before its central accuracy claim can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a three-stage layer-wise controller applied inside each client. First, head layers (the last $s$ layers) are initialized as $\tilde{\theta}_k^{t} = [\theta_g^{(1:L-s,t-1)},\ A_{k,l}^{t-1}\odot\theta_k^{(L-s+1:L,t-1)} + A_{k,g}^{t-1}\odot\theta_g^{(L-s+1:L,t-1)}]$, with $A_{k,l}+A_{k,g}=1$ and $A_{k,l}$ set by the local model's recent accuracy (Eq. 2). Second, each layer $i$ trains with learning rate $\eta^{(i,t)} = \eta\left(1+\log\left(1+1/\|g^{(i,t)}\|_2\right)\cdot i/L\right)$ (Eq. 4), so deeper layers and smaller gradients take larger steps. Third, each client uploads $\hat{\theta}_k^t \odot M_k^t$, where the binary mask keeps the top $UP_i = \min(\max(i/L,0.1),1)$ fraction of weights by absolute change per layer (Eqs. 5-9). This three-step design is what carries the argument: it decides, at every stage, how much global versus local information enters each layer.
What would settle it
Run the published Algorithm 1 verbatim on CIFAR-100 with ResNet-18 and 20 clients; if masked-out coordinates enter the server average as zeros, the global model loses those parameters entirely, so the accuracy claim collapses. The alternative—keeping previous global values for masked coordinates—is a rule the paper does not state, and the difference between the two variants is measurable in one training run.
Extended reading notes
Core claim
The central claim is that the conflict between personalization and shared global knowledge in federated learning can be managed layer by layer, cheaply, at all three points where local and global information meet. FLAYER's performance-guided aggregation sets the head-layer initialization from the client's recent local accuracy; its adaptive layer-wise learning rate lets deeper, more client-specific layers take larger steps while shallow layers stay close to the global representation; and its layer-wise masking uploads a depth-dependent fraction of each layer's most-changed parameters, so server averaging retains essential base features. On CIFAR-10, CIFAR-100, Tiny-ImageNet, and AG News under Dirichlet(0.1) heterogeneity, the paper reports FLAYER as the most accurate of seven compared methods, averaging a 5.40% inference-accuracy improvement over the six pFL baselines and up to 14.29% on individual tasks, while reducing total training time to convergence by 58.9% on average relative to FedAvg.
Load-bearing premise
The reported accuracy depends on an unstated server-side rule for parameters that no sampled client uploads; Algorithm 1 as written would set those coordinates to zero, and the paper never says how the global model keeps them intact.
Editorial extensions
If this is right
- If the central claim holds, personalized federated learning need not choose between global generalization and local specialization: the same layer-wise rules can be applied to any client model.
- Communication per round falls for early layers because each layer uploads only the top $UP_i$ fraction of parameters by absolute change, with $UP_i$ as low as 0.1 for shallow layers.
- Because FLAYER is a wrapper rather than a new network architecture, the paper reports that applying it to other pFL methods improves their accuracy by 4.90 to 11.97 percentage points, meaning the layer-wise controls transfer across methods.
- The reported convergence speedup (e.g., 78 versus 152 iterations on CNN/CIFAR-10 against FedALA) implies that adaptive layer-wise learning rates are the main driver of faster training, with masking mattering more in deeper residual networks.
Reading between the lines
- If the layer-position monotonicity in Eq. (5) is what drives the gain, then models with skip connections or attention heads may need non-monotone upload fractions, because residual paths make some shallow parameters load-bearing in later layers; this is a testable variant.
- The accuracy-based aggregation weight in Eq. (2) creates a feedback loop: a client that reports high local accuracy receives more of its own weights back, so in adversarial deployment the rule would need a trustworthy accuracy estimate.
- A natural cheap extension is replacing the gradient-norm term in Eq. (4) with a depth-only schedule, since gradient norms and layer depth are correlated; if accuracy holds, the per-layer norm computation can be skipped on resource-limited devices.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FLAYER, a personalized federated learning method with three layer-wise components: (i) performance-guided local aggregation that initializes head layers from a weighted mix of local and global parameters, (ii) an adaptive per-layer learning rate based on layer position and gradient norm, and (iii) a layer-wise sparse binary masking scheme that uploads only selected parameters to the server. The authors claim that FLAYER improves inference accuracy by 5.40% on average (up to 14.29%) over six pFL baselines while reducing computation cost, based on experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet, and AG News.
Significance. If the method and results are reproducible, FLAYER would be a practically useful pFL technique: it combines a clear motivation (different layers play different roles) with three concrete mechanisms, and the evaluation is broad, covering four datasets, seven baselines, heterogeneity, scalability, applicability, and an ablation study. The reported computation-cost reductions are also practically valuable. However, the paper currently does not specify a load-bearing part of the server aggregation rule, so the experimental claims cannot be verified from the text as written. The paper also ships no code, which raises the bar for precision in the algorithmic description.
major comments (3)
- [Layer-wise Sparse Binary Masking / Algorithm 1] Algorithm 1 line 12 sets the next global model to a weighted average of the uploaded masked parameters Theta_t_k, where Theta_t_k = hat_Theta_t_k ⊙ M_t_k by Eq. (9). For any coordinate that no sampled client selects, every uploaded Theta_t_k is zero, so that coordinate of the global model becomes exactly zero. For a coordinate selected by only a subset of clients, the average is shrunk by the zero contributions of the non-selecting clients. The paper never states a mechanism that prevents this behavior, such as carrying over the previous global values for unselected coordinates, averaging only selected coordinates, or uploading masked deltas and adding them to the global model. This is the central aggregation rule of the method, and the reported accuracies in Table 2 cannot be produced by the algorithm as written unless an additional rule is assumed. The authors must specify the exact aggregation rule, analyze its effect on the global model, and confirm that the reported experiments use that rule.
- [Computation cost / Table 3] The computation-cost comparison is based on the number of iterations and total time "until convergence," but the paper does not define the convergence criterion. Without a stopping rule (for example, a threshold on accuracy change or validation loss, or a fixed number of rounds), the iteration counts in Table 3 are not reproducible and the claimed average reduction of 58.9% in training time is not well grounded. Please state the stopping criterion used for each method and dataset.
- [Performance-guided Layer-wise Local Aggregation / Eq. (2)] Eq. (2) uses aggregation weights A_{k,l} and A_{k,g} with A_{k,l}+A_{k,g}=1, but the paper only says that "the local model inference accuracy ... sets the local weight A_{k,l}." This is not a precise definition: it is unclear whether A_{k,l} equals raw accuracy, a normalized or clipped accuracy, or some other mapping, and how accuracy is evaluated (on the local training set or a validation split). Since this performance-guided aggregation is the first contribution of the paper, the mapping must be specified explicitly so that the method is reproducible.
minor comments (5)
- [Abstract / Overall Performance] The abstract's "5.40% average improvement" is supported by Table 2 only if it is interpreted as an average absolute percentage-point difference across all 42 baseline-task pairs; it is not a relative improvement. The text should state this explicitly, since the current wording is ambiguous and could mislead readers.
- [Table 5] Table 5 includes GPFL and FedCP as compared methods, but these are not described in Section 2 or in the related-work discussion, and FedCP has no citation. Please add the missing references and briefly describe their experimental settings.
- [Eq. (4)] The adaptive learning rate formula grows without bound as the gradient norm approaches zero. The paper does not discuss clipping or a lower bound on the gradient norm; a brief statement on numerical stability would be helpful.
- [Evaluation on Personalization Layers / Table 4] The head size s is tuned per architecture (s=1 for CNN, s=2 for ResNet-18), but the sensitivity analysis is only reported for CIFAR-100. Please state how s was selected for the other datasets and whether the results are stable with respect to this choice.
- [Table 3] In the CIFAR-10 CNN row for FedAMP, the table reports "#517" with a stray marker; this appears to be a typesetting error that should be corrected.
Circularity Check
No circularity found: FLAYER's components are heuristic design choices and empirical comparisons, not predictions derived from fitted inputs.
full rationale
FLAYER's three components are each specified by explicit formulas or measured quantities that are independent of the reported accuracy results. The aggregation weights in Eq. (2) are set by the client's local inference accuracy, which is a control signal from local data, not a parameter fitted to the target test accuracy and then renamed as a prediction. The adaptive learning rate in Eq. (4) is a fixed function of the layer index and gradient norm, with no free constants fitted to the reported results. The masking proportion in Eq. (5) is a simple positional heuristic (min(max(i/L, 0.1), 1)), and the mask itself is selected from weight fluctuations (Eqs. 6-9); none of these steps uses the final accuracy numbers as an input. The ablation study and CKA similarity analysis are empirical observations used to motivate design choices, not circular derivations. Citations to Yosinski et al. and Luo et al. are external domain knowledge about layer transferability and classifier calibration, and no load-bearing self-citation chain appears. The most serious concern is a reproducibility gap in Algorithm 1: Eq. (9) zeroes masked-out parameters and Eq. (12) averages those zeroed vectors, so coordinates not selected by any sampled client would become zero unless an unstated carry-over rule is used. That issue affects the correctness of the reported implementation, but it is not circularity because the claimed improvement is not derived from the method's own outputs. Hence no circular step can be quoted and exhibited, and the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- head layer count s =
1 for CNN, 2 for ResNet-18, 1 for fastText
- base learning rate eta =
0.1 for ResNet-18 and fastText, 0.005 for CNN
- upload proportion floor 0.1 =
0.1
- layer-wise LR constants =
none
assumptions (4)
- domain assumption Base layers capture generalized features and head layers capture task-specific features
- domain assumption Local inference accuracy on D_k is a reliable indicator of local model quality for setting aggregation weights
- ad hoc to paper Global model can be reconstructed by weighted averaging of masked local parameters, with zeros for unselected coordinates
- domain assumption Layer position and gradient norm determine appropriate learning rate in federated settings
Cite this review
Pith. "Pith review of Optimizing Personalized Federated Learning through Adaptive Layer-Wise Learning." pith.science (2026). https://pith.science/paper/JMWPXTGX
@misc{pith2026241207062,
author = {Pith},
title = {Pith review of: Optimizing Personalized Federated Learning through Adaptive Layer-Wise Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/JMWPXTGX}},
note = {Machine review of arXiv:2412.07062}
}
read the original abstract
Real-life deployment of federated Learning (FL) often faces non-IID data, which leads to poor accuracy and slow convergence. Personalized FL (pFL) tackles these issues by tailoring local models to individual data sources and using weighted aggregation methods for client-specific learning. However, existing pFL methods often fail to provide each local model with global knowledge on demand while maintaining low computational overhead. Additionally, local models tend to over-personalize their data during the training process, potentially dropping previously acquired global information. We propose FLAYER, a novel layer-wise learning method for pFL that optimizes local model personalization performance. FLAYER considers the different roles and learning abilities of neural network layers of individual local models. It incorporates global information for each local model as needed to initialize the local model cost-effectively. It then dynamically adjusts learning rates for each layer during local training, optimizing the personalized learning process for each local model while preserving global knowledge. Additionally, to enhance global representation in pFL, FLAYER selectively uploads parameters for global aggregation in a layer-wise manner. We evaluate FLAYER on four representative datasets in computer vision and natural language processing domains. Compared to six state-of-the-art pFL methods, FLAYER improves the inference accuracy, on average, by 5.40\% (up to 14.29\%).
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Arivazhagan, M. G.; Aggarwal, V.; Singh, A. K.; and Choudhary, S. 2019. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818
arXiv 2019
-
[4]
Chrabaszcz, P.; Loshchilov, I.; and Hutter, F. 2017. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819
arXiv 2017
-
[5]
Collins, L.; Hassani, H.; Mokhtari, A.; and Shakkottai, S. 2021. Exploiting shared representations for personalized federated learning. In International conference on machine learning, 2089--2099. PMLR
2021
-
[6]
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778
2016
-
[7]
Huang, Y.; Chu, L.; Zhou, Z.; Wang, L.; Liu, J.; Pei, J.; and Zhang, Y. 2021. Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI conference on artificial intelligence, volume 35, 7865--7873
2021
-
[8]
Joulin, A.; Grave, \'E .; Bojanowski, P.; and Mikolov, T. 2017. Bag of Tricks for Efficient Text Classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, 427--431
work page 2017
Show all 32 references
-
[9]
Kornblith, S.; Norouzi, M.; Lee, H.; and Hinton, G. 2019. Similarity of neural network representations revisited. In International conference on machine learning, 3519--3529. PMLR
2019
-
[10]
Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images
2009
-
[11]
Li, A.; Sun, J.; Zeng, X.; Zhang, M.; Li, H.; and Chen, Y. 2021 a . Fedmask: Joint computation and communication-efficient personalized federated learning via heterogeneous masking. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, 42--55
2021
-
[12]
Li, T.; Hu, S.; Beirami, A.; and Smith, V. 2021 b . Ditto: Fair and robust federated learning through personalization. In International conference on machine learning, 6357--6368. PMLR
2021
-
[13]
U.; and Jaggi, M
Lin, T.; Kong, L.; Stich, S. U.; and Jaggi, M. 2020. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33: 2351--2363
2020
-
[14]
Luo, J.; and Wu, S. 2022. Adapt to adaptation: Learning personalization for cross-silo federated learning. In IJCAI: proceedings of the conference, volume 2022, 2166. NIH Public Access
2022
-
[15]
Luo, M.; Chen, F.; Hu, D.; Zhang, Y.; Liang, J.; and Feng, J. 2021. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34: 5972--5984
2021
-
[16]
Ma, X.; Zhang, J.; Guo, S.; and Xu, W. 2022. Layer-wised model aggregation for personalized federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10092--10101
2022
-
[17]
McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; and y Arcas, B. A. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, 1273--1282. PMLR
2017
-
[18]
Niu, Y.; and Deng, W. 2022. Federated learning for face recognition with gradient correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 1999--2007
2022
-
[19]
Pillutla, K.; Malik, K.; Mohamed, A.-R.; Rabbat, M.; Sanjabi, M.; and Xiao, L. 2022. Federated learning with partial model personalization. In International Conference on Machine Learning, 17716--17758. PMLR
2022
-
[20]
Shamsian, A.; Navon, A.; Fetaya, E.; and Chechik, G. 2021. Personalized federated learning using hypernetworks. In International Conference on Machine Learning, 9489--9502. PMLR
2021
-
[21]
Singh, B.; De, S.; Zhang, Y.; Goldstein, T.; and Taylor, G. 2015. Layer-specific adaptive learning rates for deep networks. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), 364--368. IEEE
2015
-
[22]
Song, C.; Granqvist, F.; and Talwar, K. 2022. Flair: Federated learning annotated image repository. Advances in Neural Information Processing Systems, 35: 37792--37805
2022
-
[23]
Z.; Yu, H.; Cui, L.; and Yang, Q
Tan, A. Z.; Yu, H.; Cui, L.; and Yang, Q. 2022. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems
2022
-
[24]
Wang, J.; Liu, Q.; Liang, H.; Joshi, G.; and Poor, H. V. 2020. Tackling the objective inconsistency problem in heterogeneous federated optimization. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 7611--7623
2020
-
[25]
Yosinski, J.; Clune, J.; Bengio, Y.; and Lipson, H. 2014. How transferable are features in deep neural networks? Advances in neural information processing systems, 27
2014
-
[26]
Zhang, J.; Hua, Y.; Wang, H.; Song, T.; Xue, Z.; Ma, R.; Cao, J.; and Guan, H. 2023 a . Gpfl: Simultaneously learning global and personalized feature information for personalized federated learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 5041--5051
2023
-
[27]
Zhang, J.; Hua, Y.; Wang, H.; Song, T.; Xue, Z.; Ma, R.; and Guan, H. 2023 b . FedALA: Adaptive local aggregation for personalized federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 11237--11244
2023
-
[28]
Zhang, M.; Sapra, K.; Fidler, S.; Yeung, S.; and Alvarez, J. M. 2020. Personalized federated learning with first order model optimization. arXiv preprint arXiv:2012.08565
2020 arXiv
-
[29]
Zhang, X.; Zhao, J.; and LeCun, Y. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28
2015
-
[30]
Zhao, Y.; Li, M.; Lai, L.; Suda, N.; Civin, D.; and Chandra, V. 2018. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582
2018 arXiv
-
[31]
Zhu, H.; Xu, J.; Liu, S.; and Jin, Y. 2021. Federated learning on non-IID data: A survey. Neurocomputing, 465: 371--390
2021
-
[32]
Zhu, Z.; Hong, J.; and Zhou, J. 2021. Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning, 12878--12889. PMLR
2021
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.