Pith. sign in

REVIEW 3 major objections 7 minor 32 references

FLITE shows federated fine-tuning can transmit a 5 KB latent per client per round and match full-weight FedAvg within 0.5 pp.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 17:29 UTC pith:ZR7XT6RC

load-bearing objection The averaging identity is exact and the bandwidth reduction is real, but the reported experiments show the latent channel barely moves the needle — it sits at the pretrained base, not at FedAvg parity. the 3 major comments →

arxiv 2607.18343 v1 pith:ZR7XT6RC submitted 2026-07-20 cs.LG cs.AI

Federated Lightweight Fine-Tuning

Radhakrishna Achanta , Will Reed This is my paper
classification cs.LG cs.AI
keywords federated learningcommunication efficiencymapping networkslow-rank adaptationlatent space averagingFedAvgfine-tuningquantization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Federated fine-tuning is normally bottlenecked by per-round communication that scales with model size. This paper shows that when the fine-tuning update is generated from a small latent vector through a shared frozen affine map, averaging latents on the server is exactly averaging the generated weights — so the per-round payload collapses to a tiny latent while the FedAvg update rule is unchanged. The method, FLITE, wraps this identity in a low-rank, seed-regenerable projection and a delta formulation around a shared pretrained base, and reports 74.67% on CIFAR-100 with a ~5 KB per-client message versus 75.16% for full-weight FedAvg. The paper's premise, supported by its experiments, is that the correction a well-pretrained model needs is itself low-dimensional, which is what makes a 1,280-float latent sufficient.

Core claim

The central claim is that a low-dimensional latent can carry the full fine-tuning signal in federated learning without changing the aggregation semantics. Concretely, with θ_l = θ_pre_l + U_l V_l^T z_l, where θ_pre and the factors U_l, V_l are frozen and shared across clients, averaging the per-client latents produces exactly the same result as averaging the generated weights, so the server can average 1,280 latent coordinates instead of 11.2M weights. Empirically, on CIFAR-100 with ResNet-18+GroupNorm, FLITE reaches 74.67±0.01% accuracy with a per-round payload of ~5 KB (8718× reduction), within ≈0.5 pp of full-weight FedAvg (75.16±0.15%), and the averaging identity error is 5.96×10^-8. The

What carries the argument

The key object is the affine mapping-network generator θ = θ_pre + U V^T z, in which the pretrained base θ_pre and the low-rank factors U, V are frozen and shared, and only the per-layer latent z is trained. Because the map is affine and shared, the FedAvg averaging rule in weight space is exactly mirrored by averaging latents; the paper verifies this to floating-point precision. The practical machinery that makes the tiny latent sufficient is the delta formulation (an additive correction around a strong pretrained base rather than a whole network), the low-rank seed-regenerable factorization of the projection (cutting generator memory from ~80 GB to ~10 MB), and a frozen orthogonal classifi

Load-bearing premise

The load-bearing premise is that the fine-tuning correction a pretrained model needs is low-dimensional and lies inside the frozen random subspace U V^T; if a significant part of the useful weight update falls outside that 64-dimensional per-layer subspace, the latent cannot express it and accuracy parity with full-weight FedAvg collapses.

What would settle it

Train a model on a task with a known high-rank or out-of-subspace fine-tuning delta (for example, adapting a pretrained vision model to a radically different input distribution or a new set of classes), then compare FLITE's accuracy against full-weight FedAvg while also measuring the relative projection error of the full delta onto U V^T; a large projection error accompanied by a large accuracy gap would falsify the low-dimensionality assumption.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Federated fine-tuning can be performed with per-round messages that are model-independent in size: 1,280 floats for ResNet-18, so communication cost no longer scales with model parameter count.
  • Because the aggregation is exactly FedAvg on the generated weights, existing server-side optimization and aggregation logic can be reused without modification.
  • At a fixed byte budget, the method can afford many more synchronization rounds than full-weight FedAvg, converting communication savings into drift reduction under non-IID data.
  • The latent channel composes with integer quantization: int4 latents keep accuracy unchanged at 648 bytes per round, while int4 full-weight FedAvg collapses to chance.
  • The method extends to larger backbones (ResNet-34 on TinyImageNet) and to autoregressive language models (TinyGPT on WikiText-2), with payload reductions of 313–577×.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A formal convergence analysis for FedAvg with latent averaging under non-convex objectives is a natural next step; the paper does not provide one, but the exact averaging identity suggests existing drift bounds should transfer directly.
  • The identity is not specific to the random projection: any shared affine map from a low-dimensional latent to weights with frozen random factors would give the same exact-averaging property, potentially turning other parameter-efficient fine-tuning schemes into federated channels.
  • A direct stress test — measuring the projection error of the full fine-tuning delta onto the frozen subspace — would tell practitioners how far the 1,280-float bound can be pushed before accuracy parity breaks.
  • Because the shared seed regenerates identical orthogonal factors on every client, the subspace is known to all participants; whether this opens new attack surfaces for malicious clients is not examined in the paper and deserves attention.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes FLITE, a communication-efficient federated fine-tuning method. It uses a mapping-network-style generator: each client trains a small latent vector z that is mapped to a weight delta through a frozen affine projection, theta_l = theta_pre_l + U_l V_l^T z_l, with U and V shared and frozen across clients. Because the generator is affine, averaging latents is exactly equivalent to averaging the generated weights (Eq. 4), and the paper verifies this identity numerically to ~6e-8. The headline result is that on CIFAR-100 with ResNet-18+GroupNorm, FLITE transmits only 1,280 floats (~5 KB) per client per round, an ~8700x reduction, and reaches 74.67%, within ~0.5 pp of full-weight FedAvg (75.16%). Additional experiments address non-IID robustness, a larger ResNet-34/TinyImageNet setting, a TinyGPT/WikiText-2 language task, int4 quantization of the latent, and a negative result showing that mapping networks fail at ResNet-18 scale when used for from-scratch training.

Significance. The exact averaging identity (Eq. 4) is a conceptually clean and potentially useful observation: it shows that FedAvg aggregation can be carried out in a low-dimensional latent space with zero approximation error when the generator is affine and shared. The low-rank, seed-regenerable factorization is a practical fix to the memory bottleneck of dense mapping networks, and the numerical verification of the identity to floating-point precision is a strength. However, the empirical support for the central claim that the latent carries the fine-tuning signal is weak. In the headline CIFAR-100 IID result, FLITE improves over the pretrained base by only +0.07 pp, whereas full-weight FedAvg improves by +0.56 pp; on TinyImageNet and WikiText-2 the latent is at or below the base. Thus the method is better described as a low-bandwidth channel that preserves pretrained accuracy and avoids federated drift, rather than a method that captures federated fine-tuning gains. With a substantial reframing and additional experiments, the contribution could still be valuable.

major comments (3)
  1. [§5.1, Table 1] The claim that 'the latent carries essentially all of the federated signal' is contradicted by the paper's own numbers. The centralized base is 74.6%; full-weight FedAvg reaches 75.16±0.15%, an improvement of +0.56 pp; FLITE reaches 74.67±0.01%, an improvement of only +0.07 pp, roughly 12% of the FedAvg gain. Similarly, Table 4 shows the latent at 62.14% versus a 62.5% base, and Table 6 shows final perplexity 66.10 versus a 66.1 base. In none of the IID settings does the latent produce a nontrivial correction. This is load-bearing because the paper's contribution is framed as federated fine-tuning at extreme compression; the evidence supports only 'a low-bandwidth channel that remains near the pretrained base.' The authors should either report a task where the latent captures a substantial share of the FedAvg improvement, or substantially reframe the claims.
  2. [§3.3, Table 3] The latent-dimension sweep in Table 3 does not test whether the fine-tuning correction is low-dimensional. Since the generated delta is Δθ = U V^T z and rank(U V^T) ≤ r, the expressible deltas are confined to the r-dimensional column space of U. With r=32 fixed, varying d from 64 to 1024 only re-samples V and leaves the set of reachable deltas unchanged; the d−r nullspace directions in z are dead. The flatness across d is therefore a mathematical necessity, not empirical evidence for a genuinely small correction. The rank sweep (§5.6) is the relevant experiment, but the paper reports only that accuracy moves by <0.2 pp without giving the actual accuracies. Please report the accuracy values for r∈{8,32,128,256} and discuss whether any rank achieves an improvement over the 74.6% base; without this, the low-dimensional-delta premise is unsupported.
  3. [§4, §5] The experimental setup is not a strong test of the method's core premise. The centralized base is pretrained on the same task used for federated fine-tuning (CIFAR-100, CIFAR-10, TinyImageNet, WikiText-2). Consequently, the federated fine-tuning signal is small (e.g., +0.56 pp on CIFAR-100). A more informative experiment would start from a general-purpose pretrained model and fine-tune on a downstream federated distribution (e.g., a backbone pretrained on a different domain or dataset), where the delta to be learned is larger and the low-dimensional subspace hypothesis is actually testable. Without such an experiment, the claim that a 1,280-float latent can express the useful fine-tuning correction remains untested.
minor comments (7)
  1. [Table 1] The centralized base is reported as a single number (74.6%) with no variance; the statement in §5.1 that FLITE is 'statistically indistinguishable' from this ceiling is not supportable. Please report the base over multiple seeds or soften the claim.
  2. [Table 3] The d=256 and d=1024 rows lack error bars and seed counts; please include them.
  3. [§5.6] The head ablation reports 72.99% (trainable head) and 73.53% (frozen orthogonal head) on CIFAR-100, whereas Table 1 reports 74.67% for the frozen-orthogonal-head configuration in the same federated setting. Please clarify the configuration (latent dimension, rank, rounds, hyperparameters) used in the ablation; as written, the discrepancy suggests an inconsistency.
  4. [§4] The statement that PowerSGD and top-k reduce full-weight cost by 40–140× is not derived. Add a sentence explaining the calculation (rank/density plus index bits, etc.) or point to the relevant appendix figure.
  5. [§G, Code] The paper says code will be released upon acceptance but provides no artifact or detailed per-seed results. Given that the central identity is verified numerically and the method is conceptually simple, a minimal code release or exact seeds/configurations in the appendix would substantially strengthen reproducibility.
  6. [Algorithm 1] It is not specified how the global latent is initialized for the first round. The text says 'initialise ¯z←0', but this step is not explicitly placed in the algorithm loop. Please clarify.
  7. [§3.3, §3.4] The memory numbers (≈2 GB for stored factors, ≈10 MB peak for seed regeneration) would benefit from a precise parameter count (sum_l r(P_l+d_l)) and a statement of whether the 10 MB figure excludes the regenerated weight tensors.

Circularity Check

1 steps flagged

Core FedAvg identity is non-circular, but the d-sweep flatness used to 'confirm' low-dimensional deltas is forced by the rank-r parameterization.

specific steps
  1. self definitional [§5.3 / Table 3, with Eq. (3) in §3.3]
    "θ_l = θ_pre_l + U_l V_l^T z_l, U_l ∈ R^{P_l×r}, V_l ∈ R^{d_l×r} ... The latent dimension d traces a controllable frontier (Table 3): from d=64 (8718×, 74.67%) up to d=1024 (545×, 74.66%), accuracy is essentially flat, confirming that the correction a pretrained model needs is genuinely small."

    In Eq. (3) the generated delta is U V^T z. For frozen U, V and fixed rank r, this map depends on z only through V^T z ∈ R^r, and z is initialized at zero, so gradient components orthogonal to col(V) stay zero. Thus increasing d while r is fixed never increases the expressive capacity of the parameterization; the d-sweep flatness in Table 3 follows from the construction itself. The paper nevertheless cites this flatness as 'confirming that the correction a pretrained model needs is genuinely small.' The evidence is an artifact of the model definition, not an independent test of the low-dimensional-delta premise.

full rationale

The main derivation is not circular: Eq. (4) is pure linear algebra with shared frozen θ_pre, U, V, verified numerically, and the reported accuracy and bandwidth results are measured against external baselines (full-weight FedAvg, PowerSGD, top-k), so they are independent evidence. There is no load-bearing self-citation chain or imported uniqueness theorem: the mapping-network and intrinsic-dimension references are by other authors and are used only as external context. The one genuine circular step is the d-sweep in §5.3/Table 3. Because the low-rank factorization in Eq. (3) fixes the effective rank to r regardless of d, the latent coordinates beyond V^T z cannot influence training; flatness in d is therefore a necessary consequence of the parameterization. Presenting that flatness as confirmation that the needed correction is low-dimensional is a self-definitional argument. This affects a supporting ablation rather than the headline FedAvg-parity result, so the paper is partially circular but not wholly so.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The derivation of the averaging identity is self-contained linear algebra. The empirical performance rests on the low-dimensional-delta assumption, frozen random projection factors, a shared pretrained base, GroupNorm, and the frozen-head design. d and r are hand-chosen but shown to be insensitive.

free parameters (2)
  • Latent dimension per layer d = 64 (default); swept 64/256/1024
    Determines the 1,280-float payload claim. Accuracy is flat across the sweep (Table 3), so d is a hand-chosen design knob rather than a fitted constant.
  • Projection rank r = 32 (default); swept 8/32/128/256
    Sets generator memory and regeneration compute. Accuracy varies by <0.2 pp across the sweep (§5.6), so it trades resources, not accuracy.
axioms (6)
  • domain assumption The delta a well-pretrained network needs is low-dimensional and reachable in a random frozen subspace.
    Central premise of §3.3 and §5.6, supported by flatness of d and r sweeps and intrinsic-dimension literature [14], but unproven for new tasks or models.
  • domain assumption Seed-generated random orthonormal factors U_l, V_l provide sufficient spanning coverage for the fine-tuning delta.
    §3.3–3.4. The factors are never learned or adapted; performance depends on random projections being good enough.
  • domain assumption A shared centrally-pretrained base is available to all clients and broadcast once; the task is fine-tuning, not from-scratch training.
    Stated in §2 and limitation (ii) in §6. The method does not work from a random base (§5.7).
  • domain assumption GroupNorm is used because BatchNorm running statistics are not naturally averageable across clients.
    §4. The paper accepts absolute accuracy below BatchNorm SOTA.
  • standard math Unweighted averaging of latents equals weight-space FedAvg by Eq. (4).
    Eq. (2)/(4). Exact by linearity of the shared affine map.
  • domain assumption A frozen orthogonal classifier head is accuracy-positive or at least harmless.
    §3.5 and §5.6 ablation; supported by prior work [6,24], but the reported ablation numbers do not directly match the main federated runs.

pith-pipeline@v1.3.0-alltime-deepseek · 14277 in / 14672 out tokens · 132857 ms · 2026-08-01T17:29:00.561210+00:00 · methodology

0 comments
read the original abstract

Federated fine-tuning is bottlenecked by communication: FedAvg and pseudo-gradient schemes transmit a payload that scales with the model, and gradient compression shrinks it by only a constant factor. We take a different lever. Mapping networks generate a network's weights from a small trainable latent through a frozen affine projection; because the map is shared and affine, averaging latents is exactly averaging the generated weights. We turn this into a practical low-bandwidth federated channel with two changes: a low-rank, seed-regenerable factorisation of the projection (cutting generator memory from ~80 GB to ~10 MB), and a delta formulation $\theta = \theta^{\mathrm{pre}} + U V^{\top} z$ that learns an additive correction around a shared centrally-pretrained base -- federated fine-tuning, which is what makes the method work at scale. A frozen orthogonal classifier head further removes the head from the payload while improving accuracy. On CIFAR-100 with ResNet-18+GroupNorm, our method (FLITE, Federated Low-rank Iterative Training Engine) communicates 1,280 floats (~5 KB) per client per round -- an 8718x reduction -- and reaches 74.67%, within ~0.5 pp of full-weight FedAvg. The averaging identity holds to floating-point precision ($6 \times 10^{-8}$); the method sits one to two orders of magnitude below PowerSGD and top-k on the bandwidth-accuracy Pareto; it matches or exceeds full-weight FedAvg under strong non-IID skew. int4 latents reach 648 bytes per round at unchanged accuracy, whereas int4 full-weight FedAvg collapses to chance.

Figures

Figures reproduced from arXiv: 2607.18343 by Radhakrishna Achanta, Will Reed.

Figure 1
Figure 1. Figure 1: Our scheme (frozen-orthogonal head + latent) vs textbook FedAvg (trainable head, full-weight). Parity under IID, a clear advantage under non-IID (α=0.1), at a fraction of the bandwidth. 5.2 Head-to-head: ours vs textbook FedAvg [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Bandwidth–accuracy Pareto. Left: CIFAR-100, latent (ours) vs full-weight Fe￾dAvg. Right: CIFAR-10, vs full-weight, PowerSGD (r ∈ {2, 8}) and top-k ({1%, 10%}). Our latent occupies the low-bandwidth frontier [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Non-IID robustness on CIFAR-10. Our latent matches full-weight FedAvg at α = 0.5 and exceeds it under strong heterogeneity (α = 0.1), at ≈ 545× lower per￾round bandwidth. 5.5 Fixed-byte budget and client-count sensitivity Two further axes support the same picture (details in Appendix A): under a fixed 200 MB per-client budget on CIFAR-10 (α=0.5), our method completes ∼150 latent rounds and reaches 94.21% w… view at source ↗
Figure 4
Figure 4. Figure 4: Fixed 200 MB/client budget (CIFAR-10, α=0.5): many cheap latent rounds vs one expensive full-weight round. A.2 Client-count sweep [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Scaling in client count K (CIFAR-100). Latent accuracy is stable in K; full￾weight is more sensitive at small K. Trainable FC Frozen (Kaiming) Frozen (ortho, ours) 72.0 72.5 73.0 73.5 74.0 74.5 Best test accuracy (%) 72.99 72.95 73.53 CIFAR-100: frozen orthogonal FC ablation [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Classifier-head ablation on CIFAR-100 (frozen orthogonal vs. trainable vs. frozen Kaiming). A.5 Quantization–bandwidth Pareto Combining the latent channel with integer quantization pushes the per-round payload below 1 KB at parity accuracy ( [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Latent quantization (CIFAR-100, d=64): accuracy vs communicated precision. 10 3 10 4 10 5 10 6 10 7 Payload per client per round (bytes) 0 10 20 30 40 50 60 70 80 Best test accuracy (%) int4 int8 fp32 int4 int8 fp32 latent int4 = 648 B/client, no acc loss; full-weight int4 collapses CIFAR-100 (d = 64): matched-bit quantization Pareto Ours: frozen-ortho FC + latent Textbook FedAvg: full-weight [PITH_FULL_I… view at source ↗
Figure 8
Figure 8. Figure 8: Latent-quantization bandwidth–accuracy points: int4 latent reaches 648 B/round at parity accuracy. ≈ 313× reduction — while holding validation perplexity essentially at the shared pretrained base (66.09 throughout federation). By contrast, full-weight FedAvg can briefly improve perplexity in early rounds (61.71 ± 0.03 best) but then diverges as heterogeneous local updates are averaged (81.06 ± 0.09 final).… view at source ↗
Figure 9
Figure 9. Figure 9: WikiText-2 (TinyGPT): validation perplexity vs. communication round. Shaded bands are ±1 std over 3 seeds. Latent federation is stable at the pretrained base; full-weight FedAvg improves briefly then drifts. 32 groups. The classifier head is orthogonally initialised and frozen from the first step. Per-epoch checkpointing enables resume. Latent (mapping) training. Adam, learning rate 10−2 , weight decay 10−… view at source ↗
Figure 10
Figure 10. Figure 10: Generator storage for ResNet-18: dense (≈ 80 GB) vs low-rank (≈ 2 GB) vs seed-regenerated (≈ 10 MB working memory). E From-scratch failure at scale Training the latent from a randomly (Kaiming) initialised base, with no pre￾trained offset, collapses at ResNet-18 scale: on CIFAR-100 it reaches only ≈ 2.5% test accuracy, against 74.7% in delta mode under otherwise identical settings. The latent has the capa… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 7 linked inside Pith

  1. [1]

    In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    Aji, A.F., Heafield, K.: Sparse communication for distributed gradient descent. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 440–445 (2017)

    2017

  2. [2]

    In: Advances in Neural In- formation Processing Systems 30 (NeurIPS) (2017)

    Alistarh, D., Grubic, D., Li, J., Tomioka, R., Vojnovic, M.: QSGD: Communication- efficient SGD via gradient quantization and encoding. In: Advances in Neural In- formation Processing Systems 30 (NeurIPS) (2017)

    2017

  3. [3]

    In: Proceedings of the 35th Inter- national Conference on Machine Learning (ICML) (2018)

    Bernstein, J., Wang, Y.X., Azizzadenesheli, K., Anandkumar, A.: signSGD: Com- pressed optimisation for non-convex problems. In: Proceedings of the 35th Inter- national Conference on Machine Learning (ICML) (2018)

    2018

  4. [4]

    Transactions on Machine Learning Research (2026), accepted by TMLR

    Choudhary, S., Aketi, S.A., Roy, K.: Achieving global flatness in decentralized learning with heterogeneous data. Transactions on Machine Learning Research (2026), accepted by TMLR

    2026

  5. [5]

    arXiv preprint arXiv:2311.08105 (2023)

    Douillard, A., Feng, Q., Rusu, A.A., Chhaparia, R., Donchev, Y., Kuncoro, A., Ranzato, M., Szlam, A., Shen, J.: DiLoCo: Distributed low-communication training of language models. arXiv preprint arXiv:2311.08105 (2023)

    Pith/arXiv arXiv 2023

  6. [6]

    In: International Conference on Learning Representations (2018)

    Hoffer, E., Hubara, I., Soudry, D.: Fix your classifier: The marginal value of training the last weight layer. In: International Conference on Learning Representations (2018)

    2018

  7. [7]

    In: Proceedings of the 37th International Conference on Machine Learning (ICML) (2020)

    Hsieh, K., Phanishayee, A., Mutlu, O., Gibbons, P.B.: The non-IID data quag- mire of decentralized machine learning. In: Proceedings of the 37th International Conference on Machine Learning (ICML) (2020)

    2020

  8. [8]

    In: International Conference on Learning Representations (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: International Conference on Learning Representations (2022)

    2022

  9. [9]

    arXiv preprint arXiv:1912.04977 (2019)

    Kairouz, P., McMahan, H.B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A.N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al.: Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 (2019)

    Pith/arXiv arXiv 1912

  10. [10]

    arXiv preprint arXiv:1910.06378 (2019)

    Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S.J., Stich, S.U., Suresh, A.T.: Scaffold: Stochastic controlled averaging for on-device federated learning. arXiv preprint arXiv:1910.06378 (2019)

    Pith/arXiv arXiv 1910

  11. [11]

    In: Proceedings of the 36th International Conference on Machine Learning (ICML) (2019)

    Karimireddy, S.P., Rebjock, Q., Stich, S.U., Jaggi, M.: Error feedback fixes SignSGD and other gradient compression schemes. In: Proceedings of the 36th International Conference on Machine Learning (ICML) (2019)

    2019

  12. [12]

    In: Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS)

    Khaled, A., Mishchenko, K., Richt´ arik, P.: Tighter theory for local SGD on identical and heterogeneous data. In: Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS). Proceedings of Machine Learning Research, vol. 108, pp. 4519–4529. PMLR (2020)

    2020

  13. [13]

    In: Proceedings of The 28th International Conference on Artificial In- telligence and Statistics

    Kharrat, S., Canini, M., Horv´ ath, S.: Dpfl: Decentralized personalized federated learning. In: Proceedings of The 28th International Conference on Artificial In- telligence and Statistics. Proceedings of Machine Learning Research, vol. 258, pp. 5086–5094. PMLR (2025)

    2025

  14. [14]

    In: International Conference on Learning Representations (2018)

    Li, C., Farkhoor, H., Liu, R., Yosinski, J.: Measuring the intrinsic dimension of objective landscapes. In: International Conference on Learning Representations (2018)

    2018

  15. [15]

    arXiv preprint arXiv:1910.03581 (2019), neurIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality 16 R

    Li, D., Wang, J.: Fedmd: Heterogeneous federated learning via model distillation. arXiv preprint arXiv:1910.03581 (2019), neurIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality 16 R. Achanta and W. Reed

    Pith/arXiv arXiv 1910

  16. [16]

    arXiv preprint arXiv:1812.06127 (2018)

    Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogeneous networks. arXiv preprint arXiv:1812.06127 (2018)

    Pith/arXiv arXiv 2018

  17. [17]

    In: International Conference on Learning Representations (2021)

    Li, X., Jiang, M., Zhang, X., Kamp, M., Dou, Q.: FedBN: Federated learning on non-IID features via local batch normalization. In: International Conference on Learning Representations (2021)

    2021

  18. [18]

    In: Proceedings of the Forty-first Con- ference on Uncertainty in Artificial Intelligence

    Lin, I.C., Yagan, O., Joe-Wong, C.: Fedspd: A soft-clustering approach for per- sonalized decentralized federated learning. In: Proceedings of the Forty-first Con- ference on Uncertainty in Artificial Intelligence. Proceedings of Machine Learning Research, vol. 286, pp. 2618–2641. PMLR (2025)

    2025

  19. [19]

    arXiv preprint arXiv:2006.07242 (2020)

    Lin, T., Kong, L., Stich, S.U., Jaggi, M.: Ensemble distillation for robust model fusion in federated learning. arXiv preprint arXiv:2006.07242 (2020)

    Pith/arXiv arXiv 2006

  20. [20]

    In: International Conference on Learning Representations (2018)

    Lin, Y., Han, S., Mao, H., Wang, Y., Dally, W.J.: Deep gradient compression: Reducing the communication bandwidth for distributed training. In: International Conference on Learning Representations (2018)

    2018

  21. [21]

    In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS)

    McMahan, H.B., Moore, E., Ramage, D., Hampson, S., Ag¨ uera y Arcas, B.: Communication-efficient learning of deep networks from decentralized data. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS). Proceedings of Machine Learning Research, vol. 54, pp. 1273–1282. PMLR (2017)

    2017

  22. [22]

    In: International Conference on Learning Representations (2022)

    Oh, J., Kim, S., Yun, S.Y.: FedBABU: Toward enhanced representation for feder- ated image classification. In: International Conference on Learning Representations (2022)

    2022

  23. [23]

    Proceedings of the National Academy of Sciences117(40), 24652–24663 (2020)

    Papyan, V., Han, X.Y., Donoho, D.L.: Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences117(40), 24652–24663 (2020)

    2020

  24. [24]

    IEEE Transactions on Neural Networks and Learning Systems (2021)

    Pernici, F., Bruni, M., Baecchi, C., Del Bimbo, A.: Regular polytope networks. IEEE Transactions on Neural Networks and Learning Systems (2021)

    2021

  25. [25]

    arXiv preprint arXiv:2003.00295 (2020)

    Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Koneˇ cn´ y, J., Ku- mar, S., McMahan, H.B.: Adaptive federated optimization. arXiv preprint arXiv:2003.00295 (2020)

    Pith/arXiv arXiv 2003

  26. [26]

    arXiv preprint arXiv:2602.19134 (2026)

    Sen, L., Mukherjee, S.: Mapping networks: Generating network weights from low- dimensional latents. arXiv preprint arXiv:2602.19134 (2026)

    arXiv 2026

  27. [27]

    In: International Conference on Learning Representations (2019)

    Stich, S.U.: Local SGD converges fast and communicates little. In: International Conference on Learning Representations (2019)

    2019

  28. [28]

    In: Pro- ceedings of the 42nd International Conference on Machine Learning

    Thompson, G., Yue, K., Wong, C.W., Dai, H.: Ntk-dfl: Enhancing decentralized federated learning in heterogeneous settings via neural tangent kernel. In: Pro- ceedings of the 42nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 267, pp. 59470–59491. PMLR (2025)

    2025

  29. [29]

    In: Advances in Neural Information Pro- cessing Systems 32 (NeurIPS) (2019)

    Vogels, T., Karimireddy, S.P., Jaggi, M.: PowerSGD: Practical low-rank gradient compression for distributed optimization. In: Advances in Neural Information Pro- cessing Systems 32 (NeurIPS) (2019)

    2019

  30. [30]

    In: International Conference on Learning Repre- sentations (2020)

    Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., Khazaeni, Y.: Federated learning with matched averaging. In: International Conference on Learning Repre- sentations (2020)

    2020

  31. [31]

    In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)

    Wu, Y., He, K.: Group normalization. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)

    2018

  32. [32]

    Zehtabi, S., Han, D.J., Parasnis, R., Hosseinalipour, S., Brinton, C.: Decentralized sporadic federated learning: A unified algorithmic framework with convergence guarantees. In: International Conference on Learning Representations (2025), spot- light Federated Lightweight Fine-Tuning 17 A Additional experimental figures This appendix collects extra figur...

    2025