Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers

Hariharan Ramesh; Jyotikrishna Dass

REVIEW 4 major objections 5 minor 26 references

Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SpecTraL shows that a single small core matrix, built from stacked federated LoRA adapters, carries the exact singular values of the aggregated update and yields automatic per-layer ranks via a random-matrix threshold.

desk verdict SpecTraL has a clean, checkable spectral-core identity and a plausible automatic rank-selection story, but the paper's key statistical premise and headline efficiency claims need better evidence before I'd trust the margins. read the letter →

arxiv 2607.21074 v1 pith:HVAEF3KZ submitted 2026-07-23 cs.LG cs.DC

Hariharan Ramesh , Jyotikrishna Dass This is my paper

classification cs.LGcs.DC

keywords federatedlearningLoRAvisiontransformerssingularvaluethresholdingrandommatrixtheoryspikedcovariancemodelHouseholderQRrankdiscovery

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SpecTraL is trying to establish that federated LoRA aggregation can be both exact and automatically compact. The paper stacks client adapters, applies Householder QR to extract a small r-by-r core matrix C, and proves that the singular values of C are exactly the singular values of the true weighted sum of client updates. It then uses the ScreeNOT estimator to place a hard threshold that separates consensus signal from non-IID noise, discovering layer-wise ranks with no manual tuning. The authors argue this resolves the tension between exact aggregation, communication efficiency, and support for heterogeneous client ranks, and they show empirical gains on vision transformers under non-IID federated settings.

What carries the argument

The central object is the core matrix C = RB RA^T, where RB and RA come from thin Householder QR decompositions of the stacked LoRA factors. This r-by-r matrix contains the complete spectral information of ΔW* without ever forming the m-by-n update. The second load-bearing object is ScreeNOT, a random-matrix-theory estimator that takes the observed singular values and matrix dimensions and returns an MSE-optimal hard threshold between signal and noise. Together they convert federated LoRA aggregation into a per-layer spectral denoising problem whose rank is discovered automatically, not tuned.

What would settle it

Construct a synthetic federation with a known low-rank global signal and a set of client-specific signals that are low-rank and task-relevant (not white noise). Run SpecTraL's thresholding on the stacked adapters and compare the recovered rank and reconstruction error against the oracle rank. If the automatic threshold drops any of those task-relevant directions, the spiked covariance assumption is violated in a concrete, reproducible way.

Watch

Extended reading notes

Core claim

The paper claims that spectral transformation of stacked LoRA factors makes federated aggregation simultaneously exact and rank-adaptive. Given client adapters Bk and Ak, stacking yields Bstack and Astack whose product is exactly ΔW*. Instead of forming ΔW* or doing expensive SVDs on the stacked matrices, SpecTraL computes thin Householder QR decompositions Bstack = QB RB and Astack^T = QA RA, then forms C = RB RA^T. Because QB and QA have orthonormal columns, the singular values of C equal those of ΔW*. Applying ScreeNOT to those singular values returns an MSE-optimal hard threshold θ̂, so the retained rank r* = #{i : σi > θ̂} separates directions of inter-client consensus from incoherent c

Load-bearing premise

The whole rank-discovery engine assumes that the aggregated update's singular value spectrum follows a spiked covariance model, where trailing singular values are pure noise with a compactly supported distribution; the paper concedes the noise is not white and has unknown covariance, so if client-specific directions carry task-relevant signal, the automatic threshold could discard them.

Editorial extensions

If this is right

If the spiked covariance model holds, federated LoRA aggregation needs no manually tuned energy threshold: the signal–noise boundary is computed from the observed spectrum alone.
The exact singular values of the true aggregated update are recoverable at O((m+n)r² + r³) server cost per layer, avoiding dense reconstruction and full stacked SVD.
Per-layer ranks that vary across layers and across rounds become natural outputs of the algorithm, replacing the fixed-rank assumption used by most existing methods.
Broadcasting compact, denoised adapters reduces download cost as a direct consequence of principled thresholding rather than forcing clients to adopt smaller ranks arbitrarily.
The padding-aware initialization preserves the denoised global signal at the first step of local training while giving extra adapter dimensions exploratory gradient signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's optimism about non-IID noise rests on an analogy between federated client drift and additive random-matrix noise; a skeptical reader could construct a synthetic federation with structured, low-rank client-specific signals and check whether ScreeNOT's threshold keeps directions that later prove useful for a rare domain.
A natural extension is to apply the same spectral pipeline to federated fine-tuning of decoder-only language models, where LoRA adapters are often placed on attention and MLP projections; the per-layer rank pattern may differ and the thresholding behavior under extreme label skew is untested.
The paper implies that the intrinsic dimensionality of a layer's update is a meaningful quantity that changes over training; that claim is testable by tracking r*(t) across rounds and correlating it with client drift or task difficulty.
Because ScreeNOT's guarantee is asymptotic in matrix dimensions, small stacked ranks (small K and small rk) may be a regime where the threshold is unreliable; the paper does not report failure cases for very small client counts.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Short version: SpecTraL is a solid incremental step in federated LoRA aggregation. The math at the center is right, and surprisingly clean: after stacking client adapters and doing two thin Householder QRs, the r×r core C = RB·RA^T has exactly the singular values of the true aggregated update ΔW*, at server cost O((m+n)r² + r³). The padding trick in Eq. 10 also preserves BgAg exactly. I checked that logic independently and did not find a hole. The actual innovation is modest—QR instead of SVD is a standard substitution, and the spectral-thresholding idea comes from their own FLoRIST paper—but they do not hide the debt, and applying ScreeNOT to this setting is new.

Where the paper is softer: the empirical claims are thinner than the tone suggests. Table 1 has no error bars or seeds; the wins over FlexLoRA and FLoRIST are often 0.1–0.5 points. The headline benefits—communication savings, server wall-clock, actual bytes over the wire—are never measured. The heterogeneous-rank experiment, where the r* > rk path matters, is only a figure with no final numbers. And Algorithm 1 needs an upper rank bound k for ScreeNOT, but the paper never reports what k was used, which undercuts the “no manual tuning” claim. Those are fixable, but they need to be fixed.

The statistical premise deserves scrutiny, but I disagree with the harshest version of the stress-test note. ScreeNOT is explicitly designed for correlated noise, not white noise, so the fact that federated noise is non-white is not itself a disqualifier. The real question is whether the trailing singular values of ΔW* are in fact incoherent drift rather than rare-domain signal. The paper asserts this through the spiked-covariance model but does not validate it for federated LoRA; on DomainNet/NICO++ clients are grouped by domain, so a direction shared by one or two clients is at risk of being thresholded away. That is a genuine open risk, not a demonstrated failure. The ablation showing ScreeNOT beats energy thresholding is suggestive but not a direct test.

Bottom line: this deserves a serious referee. Send it to review, but the authors should be asked for seeds/error bars, communication and wall-clock measurements, the ScreeNOT configuration, and an empirical check of the signal/noise split—ideally a small experiment poisoning the trailing spectrum with a held-out-domain direction to see if ScreeNOT discards it. If that comes back clean, I’d cite it.

Referee Report

4 major / 5 minor

Summary. The paper proposes SpecTraL, a server-side aggregation method for federated LoRA fine-tuning of Vision Transformers. The server stacks client adapters, performs thin Householder QR on the stacked factors, forms an r×r core matrix C = R_B R_A^T, and observes that the singular values of C equal those of the true aggregated update ΔW* = Σ p_k B_k A_k. It then applies the ScreeNOT estimator to the spectrum of C to select per-layer ranks automatically, reconstructs compact global adapters by applying the stored Householder reflectors, and returns them to clients. Clients whose local rank exceeds the global rank pad the received adapters with zeros in B and Gaussian rows in A. Experiments on DomainNet and NICO++ with ViT-B/16 and ViT-L/16 report small average-accuracy improvements over six baselines, and an ablation compares five initialization strategies.

Significance. If the central claims hold, SpecTraL is a useful contribution: the algebraic core (Eqs. 3–5) is clean and machine-checkable, the server cost O((m+n)r^2 + r^3) avoids dense reconstruction, and the padding rule in Eq. (10) preserves the global update exactly. The ScreeNOT-based rank selection is a plausible replacement for manually tuned energy thresholds, and the code release is a strength. However, the load-bearing statistical premise — that federated LoRA updates obey the spiked-covariance/noise-bulk model — is asserted rather than validated, and the experimental evidence has no repeated-seed statistics. The value of the paper therefore hinges on whether those two gaps can be closed.

major comments (4)

[§3.2, Step 2; Eq. (8)] The central claim that ScreeNOT 'analytically separates' consensus from noise is not established. The text acknowledges in Step 2 that the noise 'is not white' with 'unknown' covariance structure, yet Eq. (8)'s MSE-optimality guarantee is invoked as if the spiked-covariance model provably holds for ΔW*. In this federated setting, a direction shared by only a few clients (e.g., a rare visual domain) is exactly the kind of sub-bulk component ScreeNOT would classify as noise and discard. No experiment shows that discarded directions are uninformative. Please validate the noise-bulk assumption on the actual aggregated updates, e.g., by comparing downstream accuracy after retaining versus discarding components below θ̂, or by showing that the retained subspace is stable under client subsampling.
[Algorithm 1; §3.2 Step 2] ScreeNOT is presented as automatic, but Algorithm 1 requires an upper rank bound k and passes r as its only dimension. Section 4.1 states ScreeNOT 'requires no hyperparameter tuning; it takes only the observed singular values and matrix dimensions as input.' No k value, no sensitivity analysis, and no statement of whether the relevant dimensions are (m,n) or (r,r) are given. Since r is the stacked rank (≈320 in the experiments) while the noise bulk of a random m×n matrix scales with m and n, the threshold's behavior is not well defined as written. Reporting k and a sensitivity sweep is necessary to substantiate the 'no manual tuning' claim.
[Table 1; §4.2] All main accuracy numbers appear to come from single runs. Several headline improvements are small — DomainNet ViT-B/16 76.21 vs 76.04 (FlexLoRA) and 75.25 (FLoRIST), NICO++ ViT-B/16 90.86 vs 90.75 (FlexLoRA), and ViT-L/16 DomainNet 79.68 vs 79.48 (FlexLoRA). Without multiple seeds, error bars, or significance tests, the claim that SpecTraL 'consistently achieves the best average accuracy' is not supported. Please also clarify how FLoRIST's energy threshold was selected: §4.1 fixes τ=0.95, while §2 notes the optimal τ varies; a threshold tuned per setting could change the comparison.
[§3.2 Step 2; §4.3] The motivating evidence for ScreeNOT (Fig. 2) is a visual 'elbow' on two layers of one model, and the ablation in Table 2 compares initialization strategies but not ScreeNOT against alternative rank rules with repeated runs. The claim that ScreeNOT's rank discovery is the primary driver of performance needs an apples-to-apples comparison with FLoRIST using identical reconstruction and initialization except for the thresholding rule, as well as a fixed-rank oracle. Eq. (8)'s asymptotic optimality is not evidence for finite r≈320 or for per-layer cores with much smaller effective rank.

minor comments (5)

[Figure 1] Figure 1 is too small; many labels are illegible (e.g., 'No padding', 'Normal-pad', the per-client rank annotations). Please enlarge or split into subfigures.
[References] Reference [20] lists the title as 'Screenot...' while the text consistently uses 'ScreeNOT'; please correct and use journal formatting consistently.
[Algorithm 1] Algorithm 1, line 11, uses σ_A without restating its definition from Eq. (10); add a one-line explanation so the algorithm is self-contained.
[§4.1] The setup says 'All clients train LoRA adapters at rank r=32,' while §4.4 uses heterogeneous ranks. State explicitly that Tables 1–2 and Figure 3 are homogeneous-rank settings and that §4.4 is the only heterogeneous-rank experiment.
[Title page] The header simultaneously reads 'Preprint. This is the submitted version, prior to peer review' and 'Accepted at ECML-PKDD 2026 (Research Track).' This is contradictory and should be removed or cleaned up before publication.

Circularity Check

0 steps flagged · score 2.0 of 10

Minor self-citation in motivation; central spectral derivation is self-contained and not circular.

full rationale

The claimed derivation chain is not circular. Step 1 derives the r x r core matrix C = R_B R_A^T and shows via Eq. (5) that its singular values exactly equal those of Delta W*, a mathematical identity from thin QR factorizations rather than an input-output equivalence. Step 2 imports ScreeNOT from external work [20] (Donoho, Gavish, Romanov), not from the authors' own prior results. The threshold theta-hat is computed from observed singular values and matrix dimensions without using test accuracy, so the recovered rank is an unsupervised estimate rather than a prediction fitted to the reported metric. The only self-citation is the motivational appeal to the authors' prior FLoRIST [18] for the empirical premise that aggressive rank reduction helps and that its energy threshold tau varies; this premise is not used to derive the spectral identity or the ScreeNOT cutoff, and it is independently corroborated by the paper's own comparisons against FLoRA and FLoRIST (Table 1). Concerns that the ScreeNOT guarantee may not transfer because the federated noise is non-white with unknown covariance, and the unspecified upper bound k in Algorithm 1, are correctness/validity risks rather than circularity and are therefore noted but not scored here.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on (a) standard linear algebra that is correct and parameter-free (the QR/core-matrix identity), (b) an imported estimator (ScreeNOT) with real guarantees in its own model class, and (c) an asserted statistical model — spiked covariance for non-IID client drift — which is the fragile part. The free parameters are mostly hidden implementation details of that import (rank bound k, noise-level estimate, padding scale). No invented entities.

free parameters (3)

ScreeNOT upper rank bound k = unspecified
Algorithm 1 requires a 'ScreeNOT upper bound k' on the signal rank; the main text claims no tuning is needed and that ScreeNOT uses only observed singular values and dimensions, so k's setting is a hidden choice.
ScreeNOT noise-level estimate sigma-hat = unspecified
The threshold theta-hat depends on the estimated noise scale; how sigma-hat is obtained for the core matrix C (e.g., a median-based bulk estimate) is not stated in §3.2.
Gaussian padding scale sigma_A = matched to std of Ag rows (Eq. 10)
Heuristic scale for the Anew rows; data-derived, not accuracy-tuned, but still a choice affecting the client's local exploration capacity.

assumptions (5)

ad hoc to paper Aggregated federated update follows a spiked covariance model: ΔW* = low-rank consensus signal + noise with compactly supported singular-value bulk.
§3.2 Step 2 asserts this framing for non-IID client drift; the paper itself notes the noise is non-white with unknown covariance structure, so the model is an assumption, not a consequence.
domain assumption ScreeNOT's MSE-optimality guarantee (Eq. 8) transfers to the core matrix C of federated LoRA updates.
ScreeNOT [20] proves optimality for correlated-Gaussian-noise models; the paper imports the guarantee without verifying that model applies to client-drift structure.
domain assumption Trailing singular values of ΔW* are harmful noise; discarding them improves accuracy.
Supports Step 2 truncation; motivated empirically by the authors' FLoRIST (energy-threshold gains), not proven; Figure 2's 'noise tail' label is interpretive.
standard math Thin Householder QR with orthonormal Q preserves the singular values of ΔW* in C = RB·RA^T.
Eqs. 3–5; correct linear algebra and independently checkable — this part of the paper is solid.
domain assumption No client-side transform alters the meaning of uploaded BkAk (FedAvg-style independence).
Eq. 1 defines ΔW* as a weighted sum of local updates; if clients apply local regularization or asymmetric scaling, the 'consensus signal' interpretation of stacking changes.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers." pith.science (2026). https://pith.science/paper/HVAEF3KZ

@misc{pith2026260721074,
  author       = {Pith},
  title        = {Pith review of: Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HVAEF3KZ}},
  note         = {Machine review of arXiv:2607.21074}
}

read the original abstract

Fine-tuning Vision Transformers (ViTs) with low-rank adapters (LoRA) promises better communication efficiency under federated setup, yet existing aggregation strategies face fundamental limitations. Independently averaging these LoRA factors is mathematically inconsistent, introducing cross-term aggregation error. In contrast, approaches that preserve heterogeneous client ranks by concatenating local adapters on the server substantially increase download cost and often require merging global LoRA updates into pretrained weights on the clients, causing reinitialization lag and unstable convergence. Other approaches further increase server-side overhead by reconstructing dense weight updates or training auxiliary models to refine aggregation error. In this work, we propose SpecTraL, spectral transformation for layer-wise global rank discovery, that resolves these challenges within a unified design. SpecTraL stacks local LoRA modules from clients and performs orthonormal Householder Transformation of the stacked adapters directly in the low-rank latent space, eliminating dense reconstruction of the global update and any auxiliary refinement on the server. By leveraging the Spiked Covariance Model from Random Matrix Theory, SpecTraL analytically separates the global consensus signal from non-IID noise, discovering optimal layer-wise global ranks without manual hyperparameter tuning. To match local ranks in subsequent rounds, we introduce a padding-aware initialization framework that lets clients incorporate residual LoRA dimensions without re-merging them into the pre-trained base model. Experiments on federated fine-tuning of ViT-B/16 and ViT-L/16 over DomainNet and NICO++ demonstrate improved accuracy-communication trade-offs, reduced server computation, and elimination of hyperparameter search for rank selection. Our code is available at https://github.com/DASS-Lab-Group/SpecTraL

Figures

Figures reproduced from arXiv: 2607.21074 by the authors.

**Figure 2.** Singular value spectrum of the core matrix [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Training convergence on DomainNet with ViT-B/16 under homogeneous rank settings. SpecTraL consistently achieves higher accuracy throughout training and converges faster than existing federated LoRA aggregation methods. Benefits increase with model scale. The performance gap between SpecTraL and FLoRIST widens on ViT-L/16 compared to ViT-B/16. Larger models produce richer singular value spectra across more adapted… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Convergence on DomainNet (ViTL/16, heterogeneous ranks, 100 clients). SpecTraL matches or exceeds all baselines throughout training, while FLoRA lags due to reinitialization. To evaluate SpecTraL under computational heterogeneity, we adopt the heavy-tail–light rank d…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 2 linked inside Pith

[1]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InICLR, 2021

2021
[2]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, 2021

2021
[3]

Dinov2: Learning robust visual features without supervision.TMLR, 2024

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.TMLR, 2024

2024
[4]

Parameter-efficient transfer learning for nlp

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InICML, 2019

2019
[5]

Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

arXiv 2021
[6]

Communication-efficient learning of deep networks from decentralized data

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. InArtificial Intelligence and Statistics, pages 1273–1282. PMLR, 2017

2017
[7]

Federated optimization in heterogeneous networks.MLSys, 2:429–450, 2020

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks.MLSys, 2:429–450, 2020

2020
[8]

Scaffold: Stochastic controlled averaging for federated learning

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICML, pages 5132–5143, 2020

2020

Show all 26 references

[9]

Model-contrastive federated learning

Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. InCVPR, pages 10713–10722, 2021

2021
[10]

Federated learning based on dynamic regularization

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. InICLR, 2021

2021
[11]

Heterogeneous lora for federated fine-tuning of on-device foundation models

Yae Jee Cho, Luyang Liu, Zheng Xu, Akhil Fahrezi, and Gauri Joshi. Heterogeneous lora for federated fine-tuning of on-device foundation models. InEMNLP, 2024

2024
[12]

Federated learning with buffered asynchronous aggregation

John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, and Dzmitry Huba. Federated learning with buffered asynchronous aggregation. InAISTATS, pages 3581–3607, 2022

2022
[13]

Heterofl: Computation and communication efficient federated learning for heterogeneous clients

Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. InICLR, 2021

2021
[14]

Towards building the federatedgpt: Federated instruction tuning

Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In FL@NeurIPS Workshop, 2023

2023
[15]

Improving lora in privacy-preserving federated learning

Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-preserving federated learning. InICLR, 2024. 12

2024
[16]

Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations

Yue He, Aofan Li, Lingjuan Lyu, Zhaopeng Shen, Guangyu Sun, Hao Wang, and Zhenguo Wang. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. 2024

2024
[17]

Federated fine-tuning of large language models under heterogeneous tasks and client resources

Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. InNeurIPS, 2024

2024
[18]

Florist: Singular value thresholding for efficient and accurate federated fine-tuning of large language models.arXiv preprint arXiv:2506.09199, 2025

Hariharan Ramesh and Jyotikrishna Dass. Florist: Singular value thresholding for efficient and accurate federated fine-tuning of large language models.arXiv preprint arXiv:2506.09199, 2025

2025 arXiv
[19]

Lora-fair: Federated lora fine-tuning with aggregation and initialization refinement

Jieming Bian, Lei Wang, Letian Zhang, and Jie Xu. Lora-fair: Federated lora fine-tuning with aggregation and initialization refinement. InICCV, 2025

2025
[20]

Screenot: Exact mse-optimal singular value thresholding in correlated noise.Annals of Statistics, 2023

David Donoho, Matan Gavish, and Elad Romanov. Screenot: Exact mse-optimal singular value thresholding in correlated noise.Annals of Statistics, 2023

2023
[21]

Golub and Charles F

Gene H. Golub and Charles F. Van Loan.Matrix Computations - 4th Edition. Johns Hopkins University Press, Philadelphia, PA, 2013

2013
[22]

On the distribution of the largest eigenvalue in principal components analysis

Iain M Johnstone. On the distribution of the largest eigenvalue in principal components analysis. Annals of Statistics, 29(2):295–327, 2001

2001
[23]

Pytorch image models

Ross Wightman. Pytorch image models. https://github.com/rwightman/ pytorch-image-models, 2019

2019
[24]

Moment matching for multi-source domain adaptation

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. InICCV, pages 1406–1415, 2019

2019
[25]

Towards non-iid image classification: A dataset and baselines.Pattern Recognition, 110:107383, 2021

Yue He, Zheyan Shen, and Peng Cui. Towards non-iid image classification: A dataset and baselines.Pattern Recognition, 110:107383, 2021

2021
[26]

Federated learning on non-iid data silos: An experimental study

Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. InICDE, pages 965–978, 2022. 13

2022

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InICLR, 2021

2021

[2] [2]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, 2021

2021

[3] [3]

Dinov2: Learning robust visual features without supervision.TMLR, 2024

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.TMLR, 2024

2024

[4] [4]

Parameter-efficient transfer learning for nlp

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InICML, 2019

2019

[5] [5]

Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

arXiv 2021

[6] [6]

Communication-efficient learning of deep networks from decentralized data

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. InArtificial Intelligence and Statistics, pages 1273–1282. PMLR, 2017

2017

[7] [7]

Federated optimization in heterogeneous networks.MLSys, 2:429–450, 2020

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks.MLSys, 2:429–450, 2020

2020

[8] [8]

Scaffold: Stochastic controlled averaging for federated learning

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICML, pages 5132–5143, 2020

2020