FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimation

Amirreza Talebi; Elizabeth Serena Bentley; Jia Liu; Michael Lin; Peizhong Ju; Srijith Nair

REVIEW 2 major objections 4 minor 51 references

FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimation

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FSL-SAGE claims that federated split learning can match FedAvg's O(1/√T) convergence rate while cutting communication and client memory.

desk verdict A genuinely useful method with an overclaimed O(1/sqrt(T)) rate; the dropped auxiliary approximation term and an incomplete server-side proof are fixable but require a revision. read the letter →

arxiv 2505.23182 v2 pith:3GXBF3FM submitted 2025-05-29 cs.LG

Srijith Nair , Michael Lin , Peizhong Ju , Amirreza Talebi , Elizabeth Serena Bentley , Jia Liu This is my paper

classification cs.LG

keywords federatedsplitlearningauxiliarymodelsgradientestimationconvergencerateanalysiscommunicationefficiencynonconvexoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FSL-SAGE is a federated split learning method in which each client trains a small client-side model plus an auxiliary model that mimics the server's cut-layer gradient response, so clients update in parallel without waiting for the server's backpropagation. The paper's central claim is that this scheme converges in stationarity at rate $O(1/\sqrt{T})$ over $T$ communication rounds under a PAC-learnability assumption on the auxiliary models (essentially, that they fit the server's gradient mapping better as alignment data grows), matching FedAvg. If true, large models could be trained across memory-constrained clients at a fraction of the communication cost of vanilla split learning while preserving FedAvg's asymptotic guarantee. The paper also analyzes a lazy version that stops aligning the auxiliary models after $T'$ rounds and states an explicit communication–accuracy trade-off.

What carries the argument

The central object is the auxiliary model $x_{a,i}$ at each client, a small network that outputs an estimated backward-pass gradient $\hat{z}_{i,b}(x_{a,i}; z_{i,f}, y_i)$ for the cut-layer features $z_{i,f}$ (the smashed activations). Every $l$ rounds the S-server refits each client's auxiliary model on stored smashed activations by minimizing the squared $\ell^2$ error between the auxiliary estimate and the true server-side gradient (the alignment objective, Eq. (2)), then sends the updated auxiliary model back to the client, where it stays frozen until the next alignment. In the analysis, the quality of this fit enters through the estimation error $\varepsilon_t$, and the proof controls four coupled time-scales—client local steps, $Q$ server updates per round, federated averaging, and the slower alignment step—using a client-drift lemma and step-size choices that make variance and drift terms decay as $O(1/\sqrt{T})$.

What would settle it

Train FSL-SAGE with a fixed auxiliary architecture while measuring the minimum gradient-estimation error that architecture can achieve on held-out smashed activations (cut-layer features) after abundant alignment data; if that floor stays bounded away from zero and the stationarity gap plateaus at a level proportional to it instead of continuing to decay as $1/\sqrt{T}$, then the assumption that $\varepsilon_t^\star$ should shrink fast enough is what failed.

Watch

Extended reading notes

Core claim

On the paper's own terms, the core claim is that replacing the server's returned cut-layer gradients with estimates from a periodically aligned auxiliary model does not change the asymptotic convergence rate of federated training. For step sizes $\eta_L = O(1/\sqrt{T})$ and $\eta = O(1/(m\sqrt{T}))$, non-lazy FSL-SAGE achieves a stationarity gap of $\min_t \mathbb{E}[\|\nabla f(x_t)\|^2] = O(1/\sqrt{T})$ (Corollary 4.9), the same rate as FedAvg. The general bound (Theorem 4.3) separates the usual variance and client-drift terms from the method-specific term $\varepsilon_t$, the clients' averaged squared error in estimating the server-side gradient; an in-expectation PAC-learnability assumption (Assumption 4.6) converts that term into an $O(1/\sqrt{T})$ learnability term plus an irreducible best-approximation error $\varepsilon_t^\star$ of the auxiliary hypothesis class, which the paper argues can be made arbitrarily small by universal approximation. Empirically, on ResNet-18/CIFAR-10, ResNet-18/CIFAR-100, and LoRA fine-tuning of GPT2-medium on E2E, FSL-SAGE reports accuracies on par with or above SplitFed and CSE-FSL at lower communication load.

Load-bearing premise

The load-bearing premise is that the deliberately small auxiliary models can approximate the server-side cut-layer gradient mapping with negligible error; the $O(1/\sqrt{T})$ conclusion requires the average best-approximation error $\varepsilon_t^\star$ to be zero or to shrink at rate $O(1/\sqrt{T})$, which the paper justifies through universal approximation for sufficiently expressive networks.

Editorial extensions

If this is right

Clients can train models much larger than their local memory: each device stores only the client-side model plus the auxiliary model, while the server keeps the rest of the network.
A federated split system can keep FL-style parallelism and periodic aggregation without giving up the asymptotic rate, since $O(1/\sqrt{T})$ stationarity matches FedAvg even when local data is heterogeneous.
Communication load drops because the server receives cut-layer features only $Q$ times per round and transmits updated auxiliary models once every $l$ rounds, instead of exchanging cut-layer gradients every iteration or full models every round.
The lazy variant gives a tunable cost–accuracy knob: stopping alignment after $T'$ rounds replaces the $O(1/\sqrt{T})$ learnability term with roughly $(1+T'/T)/\sqrt{T'}$, quantifying the communication saved by freezing the auxiliary models.
On the reported workloads, the accuracy at fixed communication budgets is on par with or above SplitFed and CSE-FSL, e.g., roughly 2.2x less communication than CSE-FSL to reach 81% test accuracy on CIFAR-10.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The ablation results suggest an auxiliary-size and alignment-interval frontier that the paper does not optimize: CIFAR-100 accuracy degrades as the alignment interval $l$ grows, so a schedule that aligns more often early in training and less often later could yield further communication savings.
If $\varepsilon_t^\star$ is the dominant error floor, then the alignment loss on held-out smashed activations could serve as an online stopping signal for the lazy variant, replacing the fixed freeze point $T'$ with the round at which the auxiliary fit stops improving.
The same cut-layer gradient estimation mechanism could be applied to split fine-tuning of larger generative models, where repeated transmission of smashed activations rather than model weights is the dominant communication cost; the paper's LoRA GPT2-medium experiment is a first step in that direction.
The proof's lack of linear speedup in the number of clients is a structural consequence of one sequentially updated server-side model; a clustered or multi-head server design might recover part of that speedup while keeping the auxiliary-alignment machinery.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the method is a real contribution, but the headline rate claim is not proved as stated. The periodic alignment of client auxiliary models to the server's cut-layer gradient mapping is a sensible extension of CSE-FSL, and the experiments show consistent communication savings and accuracy gains on CIFAR-10/100 and E2E/GPT-2 LoRA, with code released. The theoretical analysis also does something genuinely new: it isolates the auxiliary-model estimation error epsilon_t as its own term in the bound rather than folding it into constants.

The soft spots are both in the theory. First, Corollary 4.9 drops the (1/T) sum epsilon_t* term from Theorem 4.8 without showing it decays. With the stated step sizes, that term has an O(1) coefficient times the average best-approximation error of the auxiliary class. The universal-approximation discussion gives no rate, no uniformity over t, and is in tension with the small auxiliary models used in the experiments. So the O(1/sqrt(T)) claim 'matching FedAvg' is an overstatement; the proven bound has a constant floor set by the auxiliary's representational power. That's fixable—state the bound with the floor, or impose an explicit uniformly decaying epsilon_t*—but it's a genuine gap.

Second, Theorem 4.3's server-side half is asserted by 'following the same steps' as the client side, but the server model is updated Q times per round across clients and the proof provides no analogous drift bound for server gradients evaluated at intermediate points. That looks like another fixable but real incompleteness.

None of this kills the paper. The mechanism works empirically, the auxiliary-size ablations are informative, and the epsilon_t decomposition should be useful to the FSL community. But the published claim is stronger than the proof supports, and a referee should ask for a corrected statement.

Who's this for: researchers building on federated split learning, and anyone who wants a clean way to talk about the cost of auxiliary approximations. I'd send it to review rather than desk-reject, and I'd expect the revision to tone down the rate claim or prove the decay. Reading group: worth a session. I'd cite it for the alignment mechanism, not for the matching-FedAvg statement.

Referee Report

2 major / 4 minor

Summary. The paper proposes FSL-SAGE, a federated split learning algorithm in which each client uses a small auxiliary model to estimate the server-side cut-layer gradient, with the auxiliary models periodically aligned to the true server-side gradient mapping. The main theoretical claim is a finite-time stationarity convergence rate of O(1/sqrt(T)) for the non-lazy version, which would match FedAvg, together with a 'lazy' variant that stops alignment after T' rounds. The paper also reports experiments with ResNet-18 on CIFAR-10/100 and LoRA fine-tuning of GPT2-medium on E2E, comparing communication load and accuracy against FedAvg, SplitFed, and CSE-FSL, and includes ablations on the alignment interval and auxiliary model size.

Significance. The algorithmic idea has clear practical value: using periodically aligned auxiliary models to provide server feedback while retaining client-side parallelism is a natural and promising middle ground between FL and SL, and the paper supports it with extensive experiments, ablations, and released source code. The general decomposition in Theorem 4.3, which separates optimization error, stochastic variance, data heterogeneity, and auxiliary estimation error, is also a useful conceptual contribution that could inform later FSL analyses. However, the headline O(1/sqrt(T)) claim is not established by the displayed proofs: the bound in Theorem 4.8 contains a non-vanishing best-approximation term that Corollary 4.9 discards without proof, and the server-side half of the main theorem is asserted rather than proved. These issues are load-bearing for the central claim, so the paper needs major revision even though the overall approach and experiments are promising.

major comments (2)

[§4.2, Theorem 4.8 (Eq. (14)) and Corollary 4.9] Corollary 4.9 does not follow from Theorem 4.8. The last term in Eq. (14) is (3K eta_L L_f^2)/(2cQ min{eta_L, m eta}) * (1/T) sum_{t=1}^T epsilon_t^*, and with the stated step sizes eta_L = O(1/sqrt(T)) and eta = O(1/(m sqrt(T))) the prefactor is O(1). The proven bound is therefore O(1/sqrt(T)) + O((1/T) sum epsilon_t^*), not O(1/sqrt(T)). Assumption 4.6 does not control epsilon_t^*: it only bounds the ERM excess relative to the best auxiliary hypothesis, whereas epsilon_t^* is the best-approximation error of the auxiliary model class for the current server-side mapping. The universal-approximation discussion after Eq. (14) gives no rate, no uniformity in t, and is in tension with the deliberately small auxiliary models used in the experiments (e.g., a 2.1M-parameter auxiliary for a 10.5M-parameter server model). If the auxiliary class has positive approximation error, the bound has a constant floor and does not vanish as T grows. The abstract, Corollary 4.9, and the conclusion should either add an explicit condition such as (1/T) sum epsilon_t^* = O(1/sqrt(T)) or state the rate as O(1/sqrt(T)) plus an auxiliary-approximation floor rather than claiming it matches FedAvg.
[Appendix B.1, server-side bound (Eqs. (B.16)-(B.18))] The server-side half of Theorem 4.3 is asserted by saying that Eq. (B.17) is 'identical' to Eq. (B.5) under substitutions and that the client-side steps can be followed. This is not a proof: the server model is updated Qm times per round, so the gradients g_{s,i}^{t,q} in Eq. (B.16) are evaluated at intermediate server parameters x_s^{t,q} that drift within the round. On the client side, drift is controlled by Lemma B.1, but no analogous server-drift lemma is stated or proved. Without a bound on E ||x_s^{t,q} - x_s^t||^2, the inequality (B.18) is unsupported. The authors need to add the server analogue of Lemma B.1 or restructure the server update so that the existing argument applies.

minor comments (4)

[Theorem 4.3, Eq. (3)] The right-hand side of Eq. (3) contains the random quantity epsilon_t defined in Eq. (5), while the left-hand side is the expectation min_t E ||grad f(x^t)||^2. The proof's final step says 'taking full expectation on both sides,' suggesting the intended term is E[epsilon_t], as later written in Eq. (B.21). Please restate the theorem with E[epsilon_t] in the bound or define epsilon_t in Eq. (5) as an expectation.
[Appendix B.2, derivation after Eq. (B.27)] The transition from r_i(epsilon) = O(1/epsilon^2) to an excess term O(1/sqrt(t)) should account for the fact that alignment occurs every l rounds, so the number of alignment samples at round t is proportional to t/l. The resulting constant sqrt(l) factor does not change the rate, but the substitution should be explicit.
[Figures 11 and 12 captions] The captions of Figures 11 and 12 say 'Effect of auxiliary model size on final test performance', but the x-axes are labeled 'Alignment interval (l)'. Either the figures or the captions are mislabeled, and this should be corrected.
[Appendix D.2, latency analysis] There is a typo in the sentence 'which can is computationally expensive for the server'; it should read 'which can be computationally expensive for the server.'

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the auxiliary-model estimation error is explicitly separated in Theorems 4.3 and 4.8, neither fitted to data nor defined in terms of the claimed O(1/√T) rate; the only overlapping-author citation (Yang et al. 2021) is non-load-bearing. The unstated εt* decay condition in Corollary 4.9 is a correctness gap, not a circular reduction.

full rationale

The paper's headline claim (O(1/√T) stationarity convergence 'matching FedAvg') is the output of a derivation chain that is self-contained given standard assumptions. Theorem 4.3 bounds the stationarity gap under smoothness and bounded-variance assumptions (4.1, 4.2), leaving an explicit residual (3Kη_L/2cQ min{η_L,mη})·(1/T)Σε_t for the auxiliary-model gradient estimation error (eq. 5). Corollary 4.4 honestly displays this as O(1/√T) + O(1)·(1/T)Σε_t. Theorem 4.8 then bounds ε_t by the best-in-class approximation error ε_t* plus a PAC-learnability term C₁/√t, using in-expectation learnability (Assumption 4.6) from external results (Grunwald et al. 2021; Mey 2022; Steinke & Zakynthinou 2020). The in-manuscript text itself states that the coefficient of (1/T)Σε_t* is O(1) and that ε_t* is 'entirely determined by the architecture of the auxiliary model'; the term is a genuine residual, neither fitted to the training data nor defined so as to force the rate. The transition to Corollary 4.9's clean O(1/√T) tacitly requires ε_t* = O(1/√T) or zero, which is not implied by the displayed bound (14); this is an overstatement/correctness gap (the skeptical reading) rather than circularity, because no equation in the paper makes the rate equal to the auxiliary error by construction, and the universal-approximation citations (all external) supply no rate. No load-bearing self-citation appears: Yang et al. 2021 (Jia Liu is a co-author of both papers) is cited only for standard assumptions and the linear-speedup comparison, and the same assumptions are cited to Karimireddy et al. 2019 and Reddi et al. 2021; the client-drift lemma follows Reddi et al. 2021, which has no author overlap. Per the reviewing rule, in-manuscript limitations were weighed: the post-Theorem 4.8 admission about ε_t*, the stated lack of linear speedup, and the server-side half of Theorem 4.3 asserted by 'following the same steps' as the client side without a displayed server-drift lemma (B.17 to B.18). These are flagged correctness risks and omitted-proof issues, not circular steps. Verdict: no significant circularity; score 1 reflects only the single non-load-bearing overlapping-author citation.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The convergence guarantee rests on standard smoothness and variance assumptions plus a new PAC-learnability assumption specific to this method. The algorithm also depends on hand-chosen alignment interval, cut layer, and auxiliary architecture, which affect the approximation floor and communication cost.

free parameters (3)

alignment interval l = 10
Server aligns auxiliary models once every l rounds; chosen by hand, ablated in D.3, trades server compute/communication against accuracy.
auxiliary model architecture = 2.1M params (CIFAR), 92.4M params (GPT-2)
Arbitrarily chosen subset of server model; controls approximation error εt* and therefore the non-vanishing floor in Theorem 4.8; ablation D.4 shows sensitivity.
cut layer location = after block 2 of ResNet-18; after first attention block of GPT-2
Determines client memory and smashed-data communication; hand-picked and affects communication advantage.

assumptions (6)

domain assumption Assumption 4.1: client and server loss gradients are Lc- and Lhatc-Lipschitz smooth
Standard smoothness assumption for SGD/FedAvg analysis; invoked throughout the proof.
domain assumption Assumption 4.2: unbiased local gradients with bounded variance (σL², σhatL²) and bounded heterogeneity (σG²)
Standard bounded-variance assumptions; needed for client drift and server-side variance bounds.
ad hoc to paper Assumption 4.6: auxiliary models are in-expectation PAC-learnable with sample complexity O(1/ε²)
This assumption is specific to FSL-SAGE; it converts the auxiliary fitting error εt into the O(1/√T) term. It is not verified for the neural aux models used in experiments.
domain assumption Assumption 4.7: client cut-layer activation zi,f is Lzf-Lipschitz in client parameters
Used in Theorem 4.8 to bound εt through the Jacobian spectral norm; not assumed in Theorem 4.3, creating a gap.
ad hoc to paper Exact ERM solution for auxiliary alignment in Eq. (2) and sufficiently large server storage
The analysis assumes the auxiliary model reaches the ERM optimum; in practice only approximate optimization is done. Also the alignment dataset grows unbounded, requiring unbounded server storage.
ad hoc to paper Universal approximation: εt* can be made arbitrarily small with sufficiently large auxiliary models
Invoked after Theorem 4.8 to justify the O(1/√T) conclusion; not a formal assumption of the theorem, and it conflicts with the small-auxiliary-model motivation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimation." pith.science (2026). https://pith.science/paper/3GXBF3FM

@misc{pith2026250523182,
  author       = {Pith},
  title        = {Pith review of: FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3GXBF3FM}},
  note         = {Machine review of arXiv:2505.23182}
}

abstract

Collaborative training methods like Federated Learning (FL) and Split Learning (SL) enable distributed machine learning without sharing raw data. However, FL assumes clients can train entire models, which is infeasible for large-scale models. In contrast, while SL alleviates the client memory constraint in FL by offloading most training to the server, it increases network latency due to its sequential nature. Other methods address the conundrum by using local loss functions for parallel client-side training to improve efficiency, but they lack server feedback and potentially suffer poor accuracy. We propose FSL-SAGE (Federated Split Learning via Smashed Activation Gradient Estimation), a new federated split learning algorithm that estimates server-side gradient feedback via auxiliary models. These auxiliary models periodically adapt to emulate server behavior on local datasets. We show that FSL-SAGE achieves a convergence rate of $\mathcal{O}(1/\sqrt{T})$, where $T$ is the number of communication rounds. This result matches FedAvg, while significantly reducing communication costs and client memory requirements. Our empirical results also verify that it outperforms existing state-of-the-art FSL methods, offering both communication efficiency and accuracy.

Figures

Figures reproduced from arXiv: 2505.23182 by the authors.

**Figure 1.** Schematic diagram of the FSL-SAGE algorithm. Text on the arrows indicates (sender → receiver, message, rate), e.g., the red arrows transmit xc between Ci and the F-server at the rate of 1 per round (p.r.), the updated xa,i is sent to Ci once in l rounds. Arrows are color coded: brown arrows represent local operations, blue arrows represent the smashed data and label transmission to the server, red arrows indicate fe… view at source ↗

**Figure 2.** Test accuracy vs. communication load for ResNet-18 on CIFAR-10 distributed homogeneously across 10 clients. Curves are labeled from best (leftmost) to worst (rightmost) in final accuracy. 0 50 100 150 200 Communication Load (GB) 0.0 0.1 0.2 0.3 0.4 0.5 Test Accuracy 13.45GB 26.18GB 48.02GB FSL-SAGE SplitFed-MS CSE-FSL FedAvg SplitFed-SS [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 5.** Test loss vs. communication load for LoRA finetuning of GPT2-m on i.i.d. E2E distributed across 3 clients. say 81%, FSL-SAGE achieves almost 2.2× lesser communication cost than CSE-FSL, and 10× lesser than SplitFedSS [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

Figures from the paper (6 more)

**Figure 6.** Figure 6: Outline and notation of gradient flows in FSL: Blue arrows denote gradients w.r.t. model parameters; green arrows represent gradients w.r.t. the cut-layer; and brown arrow indicates the Jacobian of the cut-layer w.r.t. the client-side parameters. We use the following a…

**Figure 7.** Figure 7: Illustration of the inequality in (B.36) Lemma B.2 (Series bound). Given p ∈ N, the series: S(p) , 1 p Xp r=1 1 √ r (B.34) satisfies the bounds 2 √ p + 1 + 1 ≤ S(p) ≤ 2 √ p . (B.35) In other words S(p) = O √ 1 p . Proof. The series in (B.34) can be bounded using in…

**Figure 8.** Figure 8: Class label distribution for Dirichlet distributed data with α = 1 simulating a heterogeneous data distribution across clients. 65 70 75 80 85 90 95 100 Communication Load (GB) 0.5 0.6 0.7 0.8 Test Accuracy FedAvg SplitFed-MS SplitFed-SS CSE-FSL FSL-SAGE α 1.0e+00 1.0e…

**Figure 10.** Figure 10: Computation and communication latency in various FL methods for 200 rounds. (Note that the y-axis is in logarithmic scale.) latency of FSL-SAGE is about half that of CSE-FSL, since CSE-FSL requires the server to send the auxiliary models back to the clients after aggr…

**Figure 11.** Figure 11: Effect of auxiliary model size on final test performance of CIFAR-10 after 200 rounds. 101 Alignment interval (l) 0.515 0.520 0.525 0.530 0.535 0.540 0.545 Test Accuracy (200 rounds) [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]

**Figure 13.** Figure 13: Effect of ResNet-18 auxiliary model size on CIFAR10. 10−2 10−1 100 101 Auxiliary size (in MB) 0.48 0.50 0.52 0.54 0.56 Test Accuracy (@ 200 GB) CSE-FSL FSL-SAGE [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 23 canonical work pages

[1]

PAC-Bayesian Statistical Learning Theory

Audibert, J.-Y. PAC-Bayesian Statistical Learning Theory. PhD thesis, Paris 6, Jul 2004

work page 2004
[2]

Bartlett, P. L. and Mendelson, S. Empirical minimization. Probability theory and related fields, 135 0 (3): 0 311--334, 2006

work page 2006
[3]

Bhatti, H. I. and Moon, J. Locally supervised learning with periodic global guidance. arXiv preprint arXiv:2208.00821, 2022

work page Pith review arXiv 2022
[4]

Cybenko, G. V. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2: 0 303--314, 1989. URL https://api.semanticscholar.org/CorpusID:3958369

work page 1989
[5]

Bert: Pre-training of deep bidirectional transformers for language understanding

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp.\ 4171--4186, 2019

2019
[6]

and Lan, G

Ghadimi, S. and Lan, G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23 0 (4): 0 2341--2368, 2013

2013
[7]

Pac-bayes, mac-bayes and conditional mutual information: Fast rate bounds that handle general vc classes

Grunwald, P., Steinke, T., and Zakynthinou, L. Pac-bayes, mac-bayes and conditional mutual information: Fast rate bounds that handle general vc classes. In Conference on Learning Theory, pp.\ 2217--2247. PMLR, 2021

work page 2021
[8]

and Raskar, R

Gupta, O. and Raskar, R. Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications, 116: 0 1--8, 2018

work page 2018

Show all 51 references

[9]

I., Lee, J., and Moon, J

Han, D.-J., Bhatti, H. I., Lee, J., and Moon, J. Accelerating federated learning with split learning on locally generated losses. In ICML 2021 workshop on federated learning for user privacy and data confidentiality. ICML Board, 2021

2021
[10]

Convergence analysis of split federated learning on heterogeneous data

Han, P., Huang, C., Tian, G., Tang, M., and Liu, X. Convergence analysis of split federated learning on heterogeneous data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=ud0RBkdBfE

2024
[11]

Universal function approximation by deep neural nets with bounded width and relu activations

Hanin, B. Universal function approximation by deep neural nets with bounded width and relu activations. Mathematics, 7 0 (10): 0 992, 2019

2019
[12]

and Sellke, M

Hanin, B. and Sellke, M. Approximating continuous functions by relu nets of minimal width, 2018. URL https://arxiv.org/abs/1710.11278

2018 arXiv
[13]

Group knowledge transfer: Federated learning of large cnns at the edge

He, C., Annavaram, M., and Avestimehr, S. Group knowledge transfer: Federated learning of large cnns at the edge. Advances in Neural Information Processing Systems, 33: 0 14068--14080, 2020

2020
[14]

Deep residual learning for image recognition

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

2016
[15]

Distilling the knowledge in a neural network

Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

2015 arXiv
[16]

A., Welbl, J., Clark, A., et al

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. An empirical analysis of compute-optimal large language model training. Advances in neural information processing systems, 35: 0 3...

2022
[17]

Multilayer feedforward networks are universal approximators

Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neural networks, 2 0 (5): 0 359--366, 1989

1989
[18]

H., Qi, H., and Brown, M

Hsu, T.-M. H., Qi, H., and Brown, M. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019

1909 arXiv
[19]

J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

2021 arXiv
[20]

B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

2001 arXiv
[21]

P., Kale, S., Mohri, M., Reddi, S

Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., and Suresh, A. T. SCAFFOLD: stochastic controlled averaging for on-device federated learning. CoRR, abs/1910.06378, 2019. URL http://arxiv.org/abs/1910.06378

1910 arXiv
[22]

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6980

2015 arXiv
[23]

Learning multiple layers of features from tiny images

Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

2009
[24]

and Lyu, X

Li, Y. and Lyu, X. Convergence analysis of sequential federated learning on heterogeneous data. Advances in Neural Information Processing Systems, 36: 0 56700--56755, 2023

2023
[25]

Lin, Z., Qu, G., Wei, W., Chen, X., and Leung, K. K. Adaptsfl: Adaptive split federated learning in resource-constrained edge networks. arXiv preprint arXiv:2403.13101, 2024

2024 arXiv
[26]

The expressive power of neural networks: A view from the width

Lu, Z., Pu, H., Wang, F., Hu, Z., and Wang, L. The expressive power of neural networks: A view from the width. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/32cbf68...

2017
[27]

S., Hare, J

Marquez, E. S., Hare, J. S., and Niranjan, M. Deep cascade learning. IEEE transactions on neural networks and learning systems, 29 0 (11): 0 5475--5485, 2018

2018
[28]

B., Moore, E., Ramage, D., and y Arcas, B

McMahan, H. B., Moore, E., Ramage, D., and y Arcas, B. A. Federated learning of deep networks using model averaging. CoRR, abs/1602.05629, 2016. URL http://arxiv.org/abs/1602.05629

2016 arXiv
[29]

A note on high-probability versus in-expectation guarantees of generalization bounds in machine learning, 2022

Mey, A. A note on high-probability versus in-expectation guarantees of generalization bounds in machine learning, 2022. URL https://arxiv.org/abs/2010.02576

2022 arXiv
[30]

and Shen, C

Mu, Y. and Shen, C. Communication and storage efficient federated split learning. In ICC 2023 - IEEE International Conference on Communications, pp.\ 2976--2981, 2023. doi:10.1109/ICC45041.2023.10278891

2023
[31]

The E 2 E dataset: New challenges for end-to-end generation

Novikova, J., Du s ek, O., and Rieser, V. The E 2 E dataset: New challenges for end-to-end generation. In Jokinen, K., Stede, M., DeVault, D., and Louis, A. (eds.), Proceedings of the 18th Annual SIG dial Meeting on Discourse and Dialogue , pp.\ 201--206, Saarbr \"u cken, Germ...

2017 doi
[32]

Locfedmix-sl: Localize, federate, and mix for improved scalability, convergence, and latency in split learning

Oh, S., Park, J., Vepakomma, P., Baek, S., Raskar, R., Bennis, M., and Kim, S.-L. Locfedmix-sl: Localize, federate, and mix for improved scalability, convergence, and latency in split learning. In Proceedings of the ACM Web Conference 2022, WWW '22, pp.\ 3347–3357, New York, N...

2022
[33]

Gpt-4 technical report, 2024

OpenAI, Achiam, J., et al. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

2024 arXiv
[34]

Language models are unsupervised multitask learners

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

2019
[35]

W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org/abs/2103.00020

2021 arXiv
[36]

Hierarchical text-conditional image generation with clip latents, 2022

Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents, 2022. URL https://arxiv.org/abs/2204.06125

2022 arXiv
[37]

J., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Kone c n \'y , J., Kumar, S., and McMahan, H

Reddi, S. J., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Kone c n \'y , J., Kumar, S., and McMahan, H. B. Adaptive federated optimization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=LkFG3lB13U5

2021
[38]

K., Li, T., Sanjabi, M., Zaheer, M., Talwalkar, A., and Smith, V

Sahu, A. K., Li, T., Sanjabi, M., Zaheer, M., Talwalkar, A., and Smith, V. On the convergence of federated optimization in heterogeneous networks. CoRR, abs/1812.06127, 2018. URL http://arxiv.org/abs/1812.06127

2018 arXiv
[39]

Photon: Federated llm pre-training

Sani, L., Iacob, A., Cao, Z., Lee, R., Marino, B., Gao, Y., Cai, D., Li, Z., Zhao, W., Qiu, X., et al. Photon: Federated llm pre-training. arXiv preprint arXiv:2411.02908, 2024 a

2024
[40]

F., Aleksandrov, P., Qiu, X., and Lane, N

Sani, L., Iacob, A., Cao, Z., Marino, B., Gao, Y., Paulik, T., Zhao, W., Shen, W. F., Aleksandrov, P., Qiu, X., and Lane, N. D. The future of large language model pre-training is federated. In International Workshop on Federated Foundation Models in Conjunction with NeurIPS 20...

2024
[41]

and Zakynthinou, L

Steinke, T. and Zakynthinou, L. Reasoning about generalization via conditional mutual information. In Conference on Learning Theory, pp.\ 3437--3452. PMLR, 2020

2020
[42]

Improving lo RA in privacy-preserving federated learning

Sun, Y., Li, Z., Li, Y., and Ding, B. Improving lo RA in privacy-preserving federated learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=NLPzL6HWNl

2024
[43]

Thapa, C., Arachchige, P. C. M., Camtepe, S., and Sun, L. Splitfed: When federated learning meets split learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 8485--8493, 2022

2022
[44]

Llama: Open and efficient foundation language models, 2023

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023. URL https://arxiv.org/abs/2302.13971

2023 arXiv
[45]

Split learning for health: Distributed deep learning without sharing raw patient data

Vepakomma, P., Gupta, O., Swedish, T., and Raskar, R. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00564, 2018

2018 arXiv
[46]

FL o RA : Federated fine-tuning large language models with heterogeneous low-rank adaptations

Wang, Z., Shen, Z., He, Y., Sun, G., Wang, H., Lyu, L., and Li, A. FL o RA : Federated fine-tuning large language models with heterogeneous low-rank adaptations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/f...

2024
[47]

Split learning over wireless networks: Parallel design and resource management

Wu, W., Li, M., Qu, K., Zhou, C., Shen, X., Zhuang, W., Li, X., and Shi, W. Split learning over wireless networks: Parallel design and resource management. IEEE Journal on Selected Areas in Communications, 41 0 (4): 0 1051--1066, 2023

2023
[48]

Achieving linear speedup with partial worker participation in non- IID federated learning

Yang, H., Fang, M., and Liu, J. Achieving linear speedup with partial worker participation in non- IID federated learning. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=jDdzh5ul-d

2021
[49]

Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning

Yu, H., Yang, S., and Zhu, S. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp.\ 5693--5700, 2019

2019
[50]

From -entropy to kl-entropy: Analysis of minimum information complexity density estimation

Zhang, T. From -entropy to kl-entropy: Analysis of minimum information complexity density estimation. The Annals of Statistics, pp.\ 2180--2210, 2006

2006
[51]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

PAC-Bayesian Statistical Learning Theory

Audibert, J.-Y. PAC-Bayesian Statistical Learning Theory. PhD thesis, Paris 6, Jul 2004

work page 2004

[2] [2]

Bartlett, P. L. and Mendelson, S. Empirical minimization. Probability theory and related fields, 135 0 (3): 0 311--334, 2006

work page 2006

[3] [3]

Bhatti, H. I. and Moon, J. Locally supervised learning with periodic global guidance. arXiv preprint arXiv:2208.00821, 2022

work page Pith review arXiv 2022

[4] [4]

Cybenko, G. V. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2: 0 303--314, 1989. URL https://api.semanticscholar.org/CorpusID:3958369

work page 1989

[5] [5]

Bert: Pre-training of deep bidirectional transformers for language understanding

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp.\ 4171--4186, 2019

2019

[6] [6]

and Lan, G

Ghadimi, S. and Lan, G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23 0 (4): 0 2341--2368, 2013

2013

[7] [7]

Pac-bayes, mac-bayes and conditional mutual information: Fast rate bounds that handle general vc classes

Grunwald, P., Steinke, T., and Zakynthinou, L. Pac-bayes, mac-bayes and conditional mutual information: Fast rate bounds that handle general vc classes. In Conference on Learning Theory, pp.\ 2217--2247. PMLR, 2021

work page 2021

[8] [8]

and Raskar, R

Gupta, O. and Raskar, R. Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications, 116: 0 1--8, 2018

work page 2018