Trustformer: A Trusted Federated Transformer

Ali Abbasi Tadi; Dima Alhadidi; Luis Rueda

REVIEW 4 major objections 5 minor 62 references

Trustformer: A Trusted Federated Transformer

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a federated Transformer can be trained from scratch without ever sharing full model weights: each client clusters each layer's weights with k-means, sends only the centroids, and reconstructs a close approximation…

desk verdict A clean compression trick with reproducible experiments, but the convergence proof is trivial in the general case and the privacy claim is contradicted at the tested operating point. read the letter →

arxiv 2501.11706 v1 pith:7QCGTZRL submitted 2025-01-20 cs.LG cs.CR

Ali Abbasi Tadi , Dima Alhadidi , Luis Rueda This is my paper · ORCID

classification cs.LGcs.CR

keywords TrustformerfederatedlearningTransformerk-meansclusteringprivacy-preservingmachinecommunicationefficiencyIntelSGXtranslation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to show that privacy-preserving federated learning for Transformers does not have to choose between leaking information and huge communication bills. It proposes Trustformer, in which each client trains a Transformer locally, runs k-means on each layer's weight matrix, and sends only the centroids to the server. The server averages the centroids and returns them, and each client uses the difference between global and local centroids, plus the cluster labels it kept, to reconstruct a close approximation of the global FedAvg model. The paper claims this provides no privacy leakage, cuts per-round data transfer to as little as about 10% of FedAvg's volume at its lowest tested clustering ratio, and on a Russian-to-English translation benchmark maintains translation quality comparable to privacy-preserving baselines. If true, this would make from-scratch federated training of large Transformers more practical in settings where privacy and bandwidth are both constrained.

What carries the argument

The load-bearing mechanism is centroid-mediated model reconstruction: after local training, each client runs k-means on every layer's weight matrix, keeps the cluster labels private, and transmits only the N_oc centroids; the server averages the centroids into global centroids; each client then moves every weight by the difference between the global and its local centroid for that weight's cluster. This difference-vector update is what lets clients approximate the FedAvg global model without the server ever seeing the weight matrices, with Theorem 1 bounding the approximation error by the cluster count.

What would settle it

Run Trustformer and FedAvg on the same deliberately heterogeneous client data (for example, each client gets a different topic or genre), set β to 0.5, and compare each client's reconstructed weights with the FedAvg average after each round; if the per-parameter difference systematically exceeds the bound in the paper's Lemma 5 or the translation scores fall below the DP-FedSAM baseline, then the claimed convergence to FedAvg does not hold for non-IID data.

Watch

Extended reading notes

Core claim

The central claim is that Transformer training can be federated by clustering weights instead of sending them. Each client trains locally, clusters each layer's weight matrix, and transmits only the N_oc centroids; the server averages the centroids and sends back global centroids; each client then shifts its own weights by the centroid differences and obtains a model that, by Theorem 1, converges to the FedAvg global model as N_oc approaches the number of parameters. The authors further claim that this scheme provides no privacy leakage under their threat model, with centroids exchanged inside Intel SGX enclaves, and that it preserves translation quality while substantially reducing communication overhead.

Load-bearing premise

The proof in the appendix that clients arrive at exactly the FedAvg model assumes the same weight positions fall into the same clusters on every client, which is only guaranteed when the number of clusters equals the number of parameters; with different local data the clusters can differ, so the equality is approximate.

Editorial extensions

If this is right

Per-round communication scales with the clustering ratio β; at β = 0.1 the reported data per round is about 10% of FedAvg's volume, and even at β = 0.9 the traffic is still about 10% lower than FedAvg.
At the limit N_oc = r, the update rule reproduces FedAvg exactly, making Trustformer a tunable generalization of FedAvg rather than a fundamentally different optimizer.
Because the server only averages centroids and never receives a complete weight matrix, it does not hold the final global model, reducing what an attacker gains from a server breach.
In the reported Russian-to-English experiments, the reconstructed model's loss overlaps FedAvg's at β = 0.9, and its translation quality is comparable to DP-FedSAM on BLEU and METEOR while achieving the highest BERT F1 among the compared methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the authors leave implicit is that the privacy guarantee is not uniform in β: as β approaches 1 the transmitted centroids approach raw weights, so "no privacy leakage" is only meaningful for small clustering ratios unless a formal privacy bound is added.
The authors restrict their claim to training from scratch; the same centroid-exchange mechanism could plausibly extend to federated fine-tuning of pre-trained language models, but that extension is untested.
Because k-means runs independently on each client, non-IID client data can yield mismatched cluster assignments across clients; a natural next experiment is measuring how much reconstruction error grows on skewed data splits.
The paper's security discussion notes two caveats: SGX was simulated with a fixed key and AES rather than real enclave hardware, and page-fault attacks on SGX are acknowledged as out of scope.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The concrete idea—cluster each layer's weights locally with k-means, send only centroids, average centroids on the server, then reconstruct the global model by shifting local weights according to the centroid differences—is clean and easy to implement. It's not revolutionary; it sits naturally in the line of quantization/compression work in FL, but the combination is new as far as I know. The authors ship code and data, and the WMT19 experiments with BLEU/METEOR/BERT F1 show that at beta=0.9 the utility is close to FedAvg. The communication table is honest and useful.

The soft spots are the load-bearing parts. Theorem 1's proof relies on Lemma 2, which claims the average of updated parameters equals FedAvg for every parameter position. That cancellation only works if every client assigns the same parameter index to the same cluster label. With non-IID data and separate k-means runs, the assignments differ, so the equality breaks. What remains is the trivial N_oc=r case, which just recovers FedAvg. So the formal convergence claim does not cover the regime the method actually uses.

The privacy claim is worse. At the only utility-tested setting, beta=0.9, N_oc=0.9r, so most clusters are singletons and the centroids are essentially raw weight rows. The server receives about 90% of the model in near-raw form. That directly contradicts the abstract's 'no privacy leakage' and Section 8's 'server cannot infer any useful information.' There is no formal privacy definition, SGX is simulated with fixed-key AES, and the authors concede side-channel attacks. So the privacy argument is not just weak; it's misleading at the operating point they emphasize.

Methodologically, only 3 clients, one translation dataset, and no error bars or variance across runs. The claim of 'significantly reducing' communication at beta=0.9 is 10%, which is modest.

Who's this for: people working on weight compression in FL might find the reconstruction trick worth a look. But as a privacy-preserving framework, the current write-up overclaims.

My recommendation: send it to peer review. The compression scheme is concrete and reproducible, and a serious referee could push the authors to either prove convergence properly, adopt a real privacy guarantee like DP, or substantially reduce their claims. It needs major revision, but it's not a desk-reject.

Referee Report

4 major / 5 minor

Summary. The paper proposes Trustformer, a federated learning scheme for training Transformers from scratch. After local training, each client applies k-means clustering to the rows of each layer's weight matrix, sends the resulting centroids to a central server, the server averages the centroids across clients, and each client reconstructs an approximation of the FedAvg global model by adding the difference between global and local centroids to every point in each cluster. The paper claims that this approach provides no privacy leakage, reduces communication overhead, and maintains translation quality comparable to DP-FedAvg, DP-FedSAM, and DP-BLUR-LUS baselines. The central theoretical result, Theorem 1, states that the reconstructed models converge to the FedAvg global model as the number of clusters N_oc approaches the number of parameters r. Experiments on Russian-to-English translation with three clients report loss curves, BLEU/METEOR/BERT F1 scores, communication sizes, and training times.

Significance. The method is straightforward and the experimental setup is clearly described, with code made available, multiple evaluation metrics, and explicit reporting of communication volume and timing. If the central claims were valid, the scheme would be an interesting lightweight baseline for compressed federated learning. However, the theoretical result is not established for the practical regime N_oc < r, and the privacy claim is contradicted at the operating point used for the utility comparisons (β = 0.9). The claimed significance therefore rests on unsupported assertions rather than on the evidence presented.

major comments (4)

[§6 and Appendix, Theorem 1 and Lemma 2] The proof of Theorem 1 depends on Lemma 2, which asserts that the average of the updated parameters across clients equals the FedAvg global model for every parameter position l. The cancellation in the proof, cavg,j - (1/n)Σ_i ci,j = 0, is valid only if all clients assign a given parameter index l to the same cluster index j. The assumptions in §4.3 (same clustering algorithm, same β, same seed) do not guarantee that k-means, run independently on each client's non-IID local weights, produces aligned cluster assignments. Without aligned assignments, the average of the per-client centroid corrections does not cancel, and Lemma 2 is false. The only part of the theorem that is rigorously true is the exact equality at N_oc = r (Corollary 2), which is tautological because singleton clusters make the centroids equal to the raw parameters. Corollary 1 also asserts that the bounds B and C decrease with N_oc, but B and C are assumed constants in Lemmas 3 and 4, with no formal dependence on N_oc established. Consequently, Theorem 1 does not support the paper's claims of convergence to FedAvg for finite N_oc.
[§5.3, Eq. (3), §7.3.2 Table 3, §8] The privacy guarantee is contradicted at the configuration that provides utility close to the baseline. With β = 0.9, Eq. (3) gives N_oc = floor(0.9 r), so the average cluster contains r/N_oc ≈ 1.11 parameter rows; most clusters are singletons whose centroids are exactly the raw weight rows. The server thus receives roughly 90% of the model parameters essentially unchanged. This directly contradicts §5.1's claim that centroids, 'due to generalization, do not reveal much information about the weights,' and §8's claim that an untrusted server 'cannot infer any useful information.' No formal privacy definition (differential privacy, semantic security, or an explicit adversary model) is given anywhere in the manuscript. The only concrete mechanism, described in §7.1.4, is a simulation of Intel SGX using a fixed key and AES cryptography, and §8 itself acknowledges side-channel attacks such as page-fault attacks. The central privacy motivation is therefore unsupported at the tested operating point.
[§6 and §7.3.1, Figure 4] The statement that Figure 4 'experimentally underscores the correctness of Theorem 1' is not justified. The figure shows that training loss moves toward the FedAvg baseline as β increases; this is expected for any compression scheme that smoothly decreases the amount of compression. It does not test the specific claim of Lemma 2, namely that the average of the reconstructed models across clients equals the FedAvg model for intermediate N_oc, nor does it provide any measure of the distance between the reconstructed and FedAvg models. With only three clients, ten training epochs, and no repeated runs or confidence intervals, the experiment cannot substantiate a convergence theorem.
[Abstract and §7.3.2, Table 3] The abstract states that the method 'significantly reduces the model size transmitted.' At the operating point where utility is close to the baseline (β = 0.9, Figures 4 and 6), Table 3 shows a reduction from 782,003 KB to 702,608 KB, i.e., only about 10%. The much larger reductions (e.g., β = 0.1 gives 78,019 KB) correspond to configurations with degraded utility, as Figure 4 indicates. While Section 6 explicitly describes the trade-off between N_oc and approximation error, the framing in the abstract and conclusion overstates the communication benefit at the configuration that supports the accuracy claim.

minor comments (5)

[§4.3 and throughout] The notation 'N oc' should be typeset consistently as N_oc with a proper subscript; the subscript is inconsistently rendered across the manuscript.
[Table 4] In the DP-FedAvg row, the total time (664 s) does not match the sum of training time (563 s) and aggregation time (11 s); either an additional component is missing or this is a typo.
[§3 and §7] DP-BLUR-LUS is cited as [11] in Section 3 but as [10] in Section 7.3.1 and Figure 5; the reference numbering should be consistent.
[Abstract and §7.1.4] The abstract says SGX is 'leveraged,' but Section 7.1.4 states that Intel SGX was simulated with fixed-key AES and not run on actual SGX hardware; this discrepancy should be stated clearly in the abstract and introduction.
[§7.3.1] The text says that at β = 0.9 all clients have the same loss value, but the figures do not report numeric values or confidence intervals; presenting the actual values and standard deviations would improve clarity.

Circularity Check

3 steps flagged · score 6.0 of 10

Theorem 1's convergence-to-FedAvg result is a definitional restatement of the update rule: at N_oc=r the centroids are the weights, so the claimed prediction is built into Equation (3) and Algorithm 5 rather than independently derived.

self definitional [Section 6 (Theorem 1) and Appendix (Corollary 2)]
"At N_oc = r: Each cluster contains exactly one parameter (|S_i,j| = 1). Local centroids: c_i,j = w_i,l. Global centroids: c_avg,j = 1/n Σ_i w_i,l = w_FedAvg,l. Parameter adjustment: w_global_i,l = w_i,l + (w_FedAvg,l − w_i,l) = w_FedAvg,l."

Corollary 2 obtains Trustformer = FedAvg by substituting the paper's own definitions: when each parameter is its own k-means cluster, the local centroid is the parameter value and the global centroid is the FedAvg average. The update rule in Algorithm 5 (w_global = w_local + (global_centroid - local_centroid)) then forces exact equality. No property of clustering is used; the result is a restatement of the update rule at the extreme setting, not an independent derivation of convergence.
self definitional [Appendix, Lemma 2]
"w_avg,global_l = 1/n Σ_i (w_i,l + (c_avg,j − c_i,j)) = (1/n Σ_i w_i,l) + c_avg,j − (1/n Σ_i c_i,j) = 1/n Σ_i w_i,l (since c_avg,j = 1/n Σ_i c_i,j)."

The cancellation is an algebraic identity: c_avg,j is defined in Equation (2) as the average of the c_i,j, so the proof simply removes the term it introduced. It also assumes the same cluster index j applies to parameter l for every client; with per-client k-means on non-IID data, cluster labels are not aligned, so the equality holds only in the definitional N_oc=r regime. Thus the lemma does not provide an independent first-principles derivation of FedAvg convergence.

1 more flagged steps

self definitional [Section 7.3.1, with Equation (3) in Section 5.3]
"This figure also experimentally underscores the correctness of Theorem 1, in which we proved that when we increase the number of clusters to the number of records, we will have the convergence equal to FedAvg."

The 'prediction' is forced by the input: Equation (3) sets N_oc = floor(r·β), and Corollary 2 shows β=1 gives exact FedAvg by construction. Varying β therefore traces a curve that is defined to end at FedAvg at β=1; observing that larger β gives loss closer to FedAvg is a sanity check of the implementation, not an experimental confirmation of an independent prediction.

full rationale

The circularity in Trustformer sits in its theoretical justification for utility. Theorem 1 and its supporting Lemma 2 reduce to the definitions of k-means centroids and the Algorithm 5 update rule: at N_oc=r the global centroid is by construction the FedAvg weight, so the claimed convergence is a tautology; the proof's cancellation in Lemma 2 is an identity in the definition of c_avg, and it fails for finite N_oc when per-client cluster labels are unaligned. The experimental 'validation' of Theorem 1 by sweeping β is likewise a plot of the construction's endpoint, not a test of an external prediction. The paper's empirical comparisons on BLEU, METEOR, and BERT-F1 against DP-FedAvg, DP-FedSAM, and DP-BLUR-LUS are genuine external benchmarks and are not circular. Separately, Section 8's security conclusion that 'the untrusted server cannot infer any useful information' is not a circular step but is unsupported: it relies on an informal 'generalization' argument (Section 5.1), a simulated SGX layer (Section 7.1.4), and no formal privacy definition, while the tested β=0.9 operating point transmits 90% of the weight rows as centroids, a substantial fraction of which are singleton or near-singleton clusters equal or nearly equal to the original weights. The paper does not rest on load-bearing self-citations, so the score is moderate rather than extreme.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. Its load-bearing assumptions are the alignment of cluster structures across clients, the privacy of centroids, the behavior of the parameter-difference bounds B and C, and the TEE-based trust model.

free parameters (1)

beta (clustering ratio) = 0.9 in quality experiments; swept 0.1 to 0.9
N_oc = floor(r*beta) determines how many centroids per layer are transmitted. The utility/communication tradeoff is entirely controlled by this hand-chosen hyperparameter; at beta=0.9 the communication saving over FedAvg is only about 10%, while the large savings advertised in Table 3 correspond to small beta values whose translation quality is not reported.

assumptions (4)

domain assumption All clients agree on the same clustering scheme, clustering ratio beta, and random seed for k-means initialization
Section 4.3. This is needed for the cluster structure to be comparable across clients; the paper does not test non-IID cluster misalignment.
domain assumption Cluster centroids transmitted to the server do not reveal sensitive information about client data
Section 5.1 and Section 8. The privacy claim rests on this; no formal privacy bound is provided.
domain assumption The server can be trusted to aggregate inside an SGX enclave while the host server cannot access plaintext centroids
Section 4.2 and Section 8. However Algorithm 3 unseals centroids on the server; the paper does not address the honest-but-curious server's access to unsealed centroids inside the enclave.
ad hoc to paper Parameter differences across clients are bounded by B and centroid magnitudes by C, and both decrease as N_oc grows
Appendix Lemmas 3 to 5 and Corollary 1. These bounds are introduced to prove error decay but are not derived from the training process.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Trustformer: A Trusted Federated Transformer." pith.science (2026). https://pith.science/paper/7QCGTZRL

@misc{pith2026250111706,
  author       = {Pith},
  title        = {Pith review of: Trustformer: A Trusted Federated Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7QCGTZRL}},
  note         = {Machine review of arXiv:2501.11706}
}

read the original abstract

Transformers, a cornerstone of deep-learning architectures for sequential data, have achieved state-of-the-art results in tasks like Natural Language Processing (NLP). Models such as BERT and GPT-3 exemplify their success and have driven the rise of large language models (LLMs). However, a critical challenge persists: safeguarding the privacy of data used in LLM training. Privacy-preserving techniques like Federated Learning (FL) offer potential solutions, but practical limitations hinder their effectiveness for Transformer training. Two primary issues are (I) the risk of sensitive information leakage due to aggregation methods like FedAvg or FedSGD, and (II) the high communication overhead caused by the large size of Transformer models. This paper introduces a novel FL method that reduces communication overhead while maintaining competitive utility. Our approach avoids sharing full model weights by simulating a global model locally. We apply k-means clustering to each Transformer layer, compute centroids locally, and transmit only these centroids to the server instead of full weights or gradients. To enhance security, we leverage Intel SGX for secure transmission of centroids. Evaluated on a translation task, our method achieves utility comparable to state-of-the-art baselines while significantly reducing communication costs. This provides a more efficient and privacy-preserving FL solution for Transformer models.

Figures

Figures reproduced from arXiv: 2501.11706 by the authors.

**Figure 2.** Training and aggregation weights. Because of this reduction, we ensure the data transmitted between the client and server are way less than the whole model weights Wlocal. According to Theorem 1, clients eventually converge to the global model generated by FedAvg once the server averages the centroids instead of actual weights, provided that the number of clusters is sufficiently high. Other than mathematical proof … view at source ↗

**Figure 3.** Example of Simulating the global model by using the difference between global centroids and local centroids. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: Loss value in three clients with the varying [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 4.** Figure 4: Loss value in three clients with the varying number [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

**Figure 6.** Figure 6: Global model quality on each client on 1000 held [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 57 canonical work pages

[1]

Comparative analysis of membership inference attacks in federated and centralized learning

A BBASI TADI , A., D AYAL, S., A LHADIDI , D., AND MOHAMMED , N. Comparative analysis of membership inference attacks in federated and centralized learning. Information 14, 11 (2023), 620

work page 2023
[2]

A CAR , D. A. E., Z HAO, Y., N AVARRO, R. M., M ATTINA , M., WHATMOUGH , P., AND SALIGRAMA , V. Federated learning based on dynamic regularization. In International Conference on Learning Representations (2021)

work page 2021
[3]

K., Y U, F

A GARWAL , N., S URESH , A. K., Y U, F. X., K UMAR , S., AND MCMAHAN , H. B. cpsgd: Communication-efficient and differentially-private distributed sgd. In Advances in Neural Infor- mation Processing Systems (2018), pp. 7564–7575

work page 2018
[4]

V., I SLAM , M

A HUJA , K., N GUYEN , M. V., I SLAM , M. A. A., B ADIE -MODIRI , A., AND RABINOVICH , M. Federated learning with adaptive commu- nication compression. In Advances in Neural Information Processing Systems (2020), vol. 33, pp. 464–476

work page 2020
[5]

Trustformer, 2024

ANONYMOUS . Trustformer, 2024. https://github.com/anonymoustrustformer/trustformer

work page 2024
[6]

R., F EDERMANN , C., F ISHEL , M., G RAHAM , Y., H ADDOW , B., H UCK , M., K OEHN , P., M ALMASI , S., ET AL

B ARRAULT , L., B OJAR , O., C OSTA-JUSSA , M. R., F EDERMANN , C., F ISHEL , M., G RAHAM , Y., H ADDOW , B., H UCK , M., K OEHN , P., M ALMASI , S., ET AL . Findings of the 2019 conference on machine translation (wmt19). ACL

work page 2019
[7]

Scalable and efficient training of large convolutional neural networks with differential privacy

B U, Z., M AO, J., AND XU, S. Scalable and efficient training of large convolutional neural networks with differential privacy. Advances in Neural Information Processing Systems 35 (2022), 38305–38318

work page 2022
[8]

How to evaluate machine translation: A review of automated and human metrics

C HATZIKOUMI , E. How to evaluate machine translation: A review of automated and human metrics. Natural Language Engineering 26, 2 (2020), 137–161

work page 2020

Show all 62 references

[9]

W., W U, S., S AHU , A

C HEN , H., D ING , J., T RAMEL , E. W., W U, S., S AHU , A. K., AVESTIMEHR , S., AND ZHANG , T. Self-aware personalized federated learning. Advances in Neural Information Processing Systems 35 (2022), 20675–20688

2022
[10]

Fedtune: A deep dive into efficient federated fine-tuning with pre-trained transformers

C HEN , J., X U, W., G UO, S., W ANG , J., Z HANG , J., AND WANG , H. Fedtune: A deep dive into efficient federated fine-tuning with pre-trained transformers. arXiv preprint arXiv:2211.08025 (2022)

2022 arXiv
[11]

S., AND CHENG , J

C HENG , A., W ANG , P., Z HANG , X. S., AND CHENG , J. Differen- tially private federated learning with local regularization and sparsi- fication. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2022), pp. 10122–10131

2022
[12]

Comparative analysis of membership inference attacks in fed- erated learning

D AYAL, S., A LHADIDI , D., A BBASI TADI , A., AND MOHAMMED , N. Comparative analysis of membership inference attacks in fed- erated learning. In Proceedings of the 27th International Database Engineered Applications Symposium (2023), pp. 185–192

2023
[13]

Meteor universal: Language spe- cific translation evaluation for any target language

D ENKOWSKI , M., AND LAVIE, A. Meteor universal: Language spe- cific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation(2014), pp. 376– 380

2014
[14]

Digital research al- liance of canada, 2024

D IGITAL RESEARCH ALLIANCE OF CANADA . Digital research al- liance of canada, 2024. Accessed: 2024-06-10

2024
[15]

An image is worth 16x16 words: Transformers for image recognition at scale

D OSOVITSKIY , A., B EYER , L., K OLESNIKOV , A., W EISSENBORN , D., Z HAI , X., U NTERTHINER , T., D EHGHANI , M., M INDERER , M., H EIGOLD , G., G ELLY, S., ET AL . An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Lea...

2021
[16]

Taming transformers for high-resolution image synthesis

E SSER , P., ROMBACH , R., AND OMMER , B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 12873–12883

2021
[17]

Decepticons: Corrupted transform- ers breach privacy in federated learning for language models

F OWL, L., G EIPING , J., R EICH , S., W EN, Y., C ZAJA , W., G OLD - BLUM , M., AND GOLDSTEIN , T. Decepticons: Corrupted transform- ers breach privacy in federated learning for language models. arXiv preprint arXiv:2201.12675 (2022)

2022 arXiv
[18]

Gradient leakage attacks in federated learning

G ONG , H., J IANG , L., L IU, X., W ANG , Y., G ASTRO , O., W ANG , L., Z HANG , K., AND GUO, Z. Gradient leakage attacks in federated learning. Artificial Intelligence Review 56 , Suppl 1 (2023), 1337– 1374

2023
[19]

One-shot federated learning

G UHA , N., T ALWALKAR , A., AND SMITH , V. One-shot federated learning. In Proceedings of the 36th International Conference on Machine Learning (2019)

2019
[20]

J I, S., L UO, T., C HEN , H., L I, J., AND JORDAN , M. I. Learning private neural language modeling with attentive aggregation. arXiv preprint arXiv:2106.07821 (2021)

2021 arXiv
[21]

Highly accurate protein structure prediction with alphafold

J UMPER , J., E VANS, R., P RITZEL , A., G REEN , T., F IGURNOV , M., RONNEBERGER , O., T UNYASUVUNAKOOL , K., B ATES, R., ˇZ´IDEK , A., P OTAPENKO , A., ET AL . Highly accurate protein structure prediction with alphafold. Nature 596, 7873 (2021), 583–589

2021
[22]

Incentive design for efficient federated learning in mobile networks: A contract theory approach

K ANG , J., X IONG , Z., N IYATO, D., Y U, P., AND LIANG , Y.-C. Incentive design for efficient federated learning in mobile networks: A contract theory approach. IEEE Journal on Selected Areas in Communications 39, 1 (2020), 152–165

2020
[23]

W., K HAN , F

K HAN , S., N ASEER , M., H AYAT, M., Z AMIR , S. W., K HAN , F. S., AND SHAH , M. Transformers in vision: A survey. ACM Computing Surveys (CSUR) 54 , 10s (2022), 1–41

2022
[24]

Unifiedqa: Crossing format boundaries with a single qa system

K HASHABI , D., M IN, S., K HOT, T., S ABHARWAL , A., T AFJORD , O., C LARK , P., AND HAJISHIRZI , H. Unifiedqa: Crossing format boundaries with a single qa system. In Findings of the Association for Computational Linguistics: EMNLP 2020 (2020), pp. 1896–1907

2020
[25]

P., AND BA, J

K INGMA , D. P., AND BA, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

2014 arXiv
[26]

Crypten: Secure multi- party computation meets machine learning

K NOTT, B., V ENKATARAMAN , S., H ANNUN , A., S ENGUPTA , S., IBRAHIM , M., AND VAN DER MAATEN , L. Crypten: Secure multi- party computation meets machine learning. Advances in Neural Information Processing Systems 34 (2021), 4961–4973

2021
[27]

K., G HOSH , S., B ASU , S., S URESH , A

L ALITHA , A. K., G HOSH , S., B ASU , S., S URESH , A. K., AND KAN- NAN, R. Fully decentralized federated learning. In Third Workshop on Bayesian Deep Learning (NeurIPS) (2019)

2019
[28]

Cross-lingual language model pretraining

L AMPLE , G., AND CONNEAU , A. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems (2019), pp. 7059–7069

2019
[29]

Mole: Mitigation of side-channel attacks against sgx via dynamic data location escape

L ANG , F., W ANG , W., M ENG , L., L IN, J., W ANG , Q., AND LU, L. Mole: Mitigation of side-channel attacks against sgx via dynamic data location escape. In Proceedings of the 38th Annual Computer Security Applications Conference (2022), pp. 978–988

2022
[30]

Privacy- preserving machine learning with fully homomorphic encryption for deep neural network

L EE, J.-W., K ANG , H., L EE, Y., C HOI , W., E OM, J., D ERYABIN , M., L EE, E., L EE, J., Y OO, D., K IM, Y.-S., ET AL . Privacy- preserving machine learning with fully homomorphic encryption for deep neural network. iEEE Access 10 (2022), 30039–30054

2022
[31]

On the convergence of fedavg on non-iid data

L I, X., H UANG , K., Y ANG , W., WANG , S., AND ZHANG , Z. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations (2020)

2020
[32]

L IN, Y., H AN, S., M AO, H., W ANG , Y., AND DALLY, W. J. Deep gradient compression: Reducing the communication bandwidth for distributed training. In International Conference on Learning Repre- sentations (2020)

2020
[33]

B., M OORE , E., R AMAGE , D., H AMPSON , S., AND Y ARCAS , B

M CMAHAN , H. B., M OORE , E., R AMAGE , D., H AMPSON , S., AND Y ARCAS , B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics (2017), PMLR, pp. 1273–1282

2017
[34]

B., R AMAGE , D., T ALWAR, K., AND ZHANG , L

M CMAHAN , H. B., R AMAGE , D., T ALWAR, K., AND ZHANG , L. Learning differentially private recurrent language models. In Inter- national Conference on Learning Representations (ICLR) (2018)

2018
[35]

K-means clustering based feature consistency alignment for label-free model evaluation

M IAO, S., Z HENG , L., L IU, J., AND JIN, H. K-means clustering based feature consistency alignment for label-free model evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023), pp. 3299–3307

2023
[36]

T., A ONO , Y., H AYASHI, T., W ANG , L., AND MORIAI , S

P HONG , L. T., A ONO , Y., H AYASHI, T., W ANG , L., AND MORIAI , S. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security 13, 5 (2018), 1333–1345

2018
[37]

A call for clarity in reporting bleu scores

P OST, M. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771 (2018)

2018 arXiv
[38]

R AFFEL , C., S HAZEER , N., R OBERTS , A., L EE, K., N ARANG , S., MATENA , M., Z HOU , Y., LI, W., AND LIU, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. In Journal of Machine Learning Research (2020), p. 140

2020
[39]

J., C HARLES , Z., Z AHEER , M., G ARRETT , Z., R USH , K., K ONECN ´Y, J., K UMAR , S., M CMAHAN , B., AND HSIEH , C.- J

R EDDI , S. J., C HARLES , Z., Z AHEER , M., G ARRETT , Z., R USH , K., K ONECN ´Y, J., K UMAR , S., M CMAHAN , B., AND HSIEH , C.- J. Adaptive federated optimization. In International Conference on Learning Representations (2021)

2021
[40]

Make landscape flatter in differentially private federated learning

S HI, Y., L IU, Y., W EI, K., S HEN , L., W ANG , X., AND TAO, D. Make landscape flatter in differentially private federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023), pp. 24552–24562

2023
[41]

L., N ARAYANAN , V., AND SAXENA , P

S HINDE , S., C HUA, Z. L., N ARAYANAN , V., AND SAXENA , P. Preventing page faults from telling your secrets. In Proceedings of the 11th ACM on Asia Conference on Computer and Communications Security (2016), pp. 317–328

2016
[42]

A., A LHADIDI , D., AND RUEDA , L

T ADI , A. A., A LHADIDI , D., AND RUEDA , L. Pppct: Privacy- preserving framework for parallel clustering transcriptomics data. Computers in Biology and Medicine 173 (2024), 108351

2024
[43]

A., R UEDA , L., AND ALHADIDI , D

T ADI , A. A., R UEDA , L., AND ALHADIDI , D. Nicasn: Non-negative matrix factorization and independent component analysis for cluster- ing social networks. In Canadian AI (2022)

2022
[44]

A hybrid approach to privacy- preserving federated learning

T RUEX , S., B ARACALDO , N., A NWAR, A., S TEINKE , T., L UDWIG , H., W EBER , B., AND ZHANG , R. A hybrid approach to privacy- preserving federated learning. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security (2019), ACM, pp. 1– 11

2019
[45]

N., K AISER , Ł., AND POLOSUKHIN , I

V ASWANI , A., S HAZEER , N., P ARMAR , N., U SZKOREIT , J., J ONES , L., G OMEZ , A. N., K AISER , Ł., AND POLOSUKHIN , I. Attention is all you need. In Advances in neural information processing systems (2017), pp. 5998–6008

2017
[46]

Federated learning with matched averaging

W ANG , H., Y UROCHKIN , M., S UN, Y., PAPAILIOPOULOS , D., AND KHAZAENI , Y. Federated learning with matched averaging. In International Conference on Learning Representations (2020)

2020
[47]

Dynamic personalized fed- erated learning with adaptive differential privacy

Y ANG , X., H UANG , W., AND YE, M. Dynamic personalized fed- erated learning with adaptive differential privacy. In Advances in Neural Information Processing Systems (2023), A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36, Curran Associates,...

2023
[48]

Z HANG , J., Z HAO, Y., S ALEH , M., AND LIU, P. J. Pegasus: Pre- training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning (2020), pp. 11328– 11339

2020
[49]

Vinvl: Revisiting visual representations in vision-language models

Z HANG , P., L I, X., H U, X., Y ANG , J., Z HANG , L., W ANG , Y., AND GAO, J. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 5579–5588

2021
[50]

Q., AND ARTZI , Y

Z HANG , T., K ISHORE , V., W U, F., W EINBERGER , K. Q., AND ARTZI , Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019). Appendix Definition and notations

2019 arXiv
[51]

Total number of clients: n
[52]

For simplicity, we assume there is only 1 hidden layer, meaning that W i local = {wi 1} in the local models

Each client i has a local model parameter vector wi ∈ Rr. For simplicity, we assume there is only 1 hidden layer, meaning that W i local = {wi 1} in the local models. We show wi 1 as wi. Therefore, wi = [wi,1, wi,2, ..., wi,r]
[53]

Each client clusters their parameter vector wi into N oc clusters using k-means clustering
[54]

For client i, clusters are denoted by Si,j for j = 1, 2, .., N oc
[55]

Centroids for client i: ci,j = 1 |Si,j | Σl∈Si,j wi,l
[56]

,ˆci,N oc ] to the server

Client i sends centroids ci = [ˆci,1, ˆci,2, . . . ,ˆci,N oc ] to the server
[57]

We showcavg,1 as cavg for sim- plicity

Since we have just one layer in the clients, the Cglobal = {cavg,1}. We showcavg,1 as cavg for sim- plicity. The server aggregates centroids: cavg,j = 1 n Σn i=1ci,j for j = 1 , 2, . . . , N oc and sends it to clients
[58]

For simplicity, we call the global weights on client i as wglobal i

Clients approximate the global model and since we have only 1 layer, Wglobal = {w1}. For simplicity, we call the global weights on client i as wglobal i . Clients approximate wglobal i using ci,j, wi, and cavg,j . Theorem and Proofs Theorem 1. In a federated learning setting w...
[59]

The number of parameters per cluster |Si,j| de- creases
[60]

Centroids represent fewer parameters
[61]

Centroid differences decrease: |ci,j − cavg,j| de- creases
[62]

Since both B and C decrease with increasing N oc, the total error bound |εi,l| decreases

Approximation error decreases: Overall, |εi,l| de- creases with increasing N oc. Since both B and C decrease with increasing N oc, the total error bound |εi,l| decreases. Proof of Theorem Proof of Theorem 1. From Lemma 5, the total error per parameter is: |εi,l| ≤B + 2C n − 1 ...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Comparative analysis of membership inference attacks in federated and centralized learning

A BBASI TADI , A., D AYAL, S., A LHADIDI , D., AND MOHAMMED , N. Comparative analysis of membership inference attacks in federated and centralized learning. Information 14, 11 (2023), 620

work page 2023

[2] [2]

A CAR , D. A. E., Z HAO, Y., N AVARRO, R. M., M ATTINA , M., WHATMOUGH , P., AND SALIGRAMA , V. Federated learning based on dynamic regularization. In International Conference on Learning Representations (2021)

work page 2021

[3] [3]

K., Y U, F

A GARWAL , N., S URESH , A. K., Y U, F. X., K UMAR , S., AND MCMAHAN , H. B. cpsgd: Communication-efficient and differentially-private distributed sgd. In Advances in Neural Infor- mation Processing Systems (2018), pp. 7564–7575

work page 2018

[4] [4]

V., I SLAM , M

A HUJA , K., N GUYEN , M. V., I SLAM , M. A. A., B ADIE -MODIRI , A., AND RABINOVICH , M. Federated learning with adaptive commu- nication compression. In Advances in Neural Information Processing Systems (2020), vol. 33, pp. 464–476

work page 2020

[5] [5]

Trustformer, 2024

ANONYMOUS . Trustformer, 2024. https://github.com/anonymoustrustformer/trustformer

work page 2024

[6] [6]

R., F EDERMANN , C., F ISHEL , M., G RAHAM , Y., H ADDOW , B., H UCK , M., K OEHN , P., M ALMASI , S., ET AL

B ARRAULT , L., B OJAR , O., C OSTA-JUSSA , M. R., F EDERMANN , C., F ISHEL , M., G RAHAM , Y., H ADDOW , B., H UCK , M., K OEHN , P., M ALMASI , S., ET AL . Findings of the 2019 conference on machine translation (wmt19). ACL

work page 2019

[7] [7]

Scalable and efficient training of large convolutional neural networks with differential privacy

B U, Z., M AO, J., AND XU, S. Scalable and efficient training of large convolutional neural networks with differential privacy. Advances in Neural Information Processing Systems 35 (2022), 38305–38318

work page 2022

[8] [8]

How to evaluate machine translation: A review of automated and human metrics

C HATZIKOUMI , E. How to evaluate machine translation: A review of automated and human metrics. Natural Language Engineering 26, 2 (2020), 137–161

work page 2020