REVIEW 3 major objections 7 minor 35 references
Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning
T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Federated PEFT under non-IID data fails when client updates are spectrally misaligned, and TRISHUL's three mechanisms—exact shared-basis aggregation, nuclear-norm shrinkage, and water-filled head budgets—fix it without extra communication.
desk verdict A coherent three-part recipe for spectral control in federated PEFT, with a plausible mechanism and good diagnostics, but the headline gains are vulnerable to circular hyperparameter selection and missing error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the multi-head low-rank parameterization with frozen shared bases: each layer $l$ has $h_l$ heads, each head defined by orthonormal frozen matrices $B_i \in \mathbb{R}^{d\times r}$, $A_i \in \mathbb{R}^{r\times d}$ with mutually orthogonal subspaces across heads, a trainable $r\times r$ core $H_i$, and a trainable scalar $s_i$. Three mechanisms act on this substrate: exact aggregation, since shared bases let the server average $\{s_{c,i}H_{c,i}\}$ and reconstruct the exact mean ambient update; nuclear-norm proximal shrinkage, where each local gradient step is followed by singular value thresholding $\operatorname{prox}_{\lambda\eta\|\cdot\|_*}(H) = U\operatorname{diag}(\max(\sigma_j - \lambda\eta, 0))V^\top$ on the core, exploiting $\|B_i H_i A_i\|_* = \|H_i\|_*$; and concave water-filling, where head counts $h_l$ maximize $\sum_l a_l \log(1+h_l)$ subject to $\sum_l h_l = h$ with $a_l = \|W_l\|_F^2 + \epsilon$, solved in closed form as $h_l^* = (a_l/\mu - 1)_+$. The argument's force is that all spectral operations happen on $r\times r$ cores, so the claims of exactness, variance reduction, and allocation optimality come with negligible cost.
What would settle it
Set the scalar cap $s_{\max}$ very large, or remove the cap entirely, in the TRISHUL implementation on CIFAR-100 non-IID with $\alpha=0.3$, and compare final accuracy and per-head aggregation variance against the reported setting: if accuracy and variance do not degrade, the rank-controlled shrinkage of the core is not what delivers the gains, while if they degrade sharply, the variance bound depends on the unreported cap.
Extended reading notes
Core claim
TRISHUL's central claim is that controlling the singular spectrum of client updates before aggregation, rather than only correcting gradient drift, is what makes federated PEFT robust to heterogeneity. Concretely: because frozen orthonormal bases $B_i$, $A_i$ are shared, averaging the uploaded core products $s_{c,i}H_{c,i}$ recovers exactly the mean ambient update, eliminating the factor-wise averaging bias $\mathbb{E}[B_c A_c] \neq \mathbb{E}[B_c]\mathbb{E}[A_c]$ of standard LoRA. Because the bases are isometric, penalizing and thresholding the nuclear norm of $H_{c,i}$ is equivalent to spectral shrinkage of the full update, so proximal singular value thresholding zeroes client-specific high-rank modes before upload while preserving dominant shared directions. A concave water-filling allocation then assigns more of the fixed head budget to layers with larger pretrained Frobenius norms. Across CIFAR-100, SVHN, 20 Newsgroups, MRQA, and GLUE with LLaMA3.2-1B, the paper reports that TRISHUL beats federated LoRA baselines, with margins widening from about 3.2% to 4.7% over the strongest baseline as Dirichlet heterogeneity goes from $\alpha=0.3$ to $\alpha=0.05$ on CIFAR-100.
Load-bearing premise
The load-bearing premise is that nuclear-norm shrinkage of the core $H_i$ acts with full strength on the uploaded update, but the uploaded object is $s_i B_i H_i A_i$, and the effective shrinkage strength for the actual penalty is $\lambda\|H_i\|_*/|s_i|$, so if the trainable scalar $s_i$ grows large the variance-reduction claim in Eq. (15) can fail; the paper never reports the scalar bound $s_{\max}$ that Algorithm 1 line 15 refers to.
Editorial extensions
If this is right
- If TRISHUL is right, federated PEFT under non-IID data should be formulated as a spectral-control problem, and methods that only correct gradient drift will remain below methods that shape the update spectrum before aggregation.
- The gains should grow monotonically with heterogeneity: on CIFAR-100 the reported margin over the strongest baseline widens from 3.2% at $\alpha=0.3$ to 4.7% at $\alpha=0.05$.
- Because shrinkage is applied to $r\times r$ cores, the method preserves the communication budget of the underlying multi-head PEFT protocol and adds less than about 1% per-round compute at 85M to 1B parameter scale.
- The spectral diagnostics, including principal-angle similarity, dominant singular-vector similarity, spectral entropy, effective rank, and inter-client aggregation variance, should show improved alignment and lower variance for TRISHUL relative to the multi-head baseline across rounds and heterogeneity levels.
- Layer-wise allocation from pretrained Frobenius norms should dominate uniform allocation and heuristic gradient- or curvature-based allocations at zero runtime cost.
Reading between the lines
- Editorial inference: the same spectral-shrinkage mechanism may transfer to centralized PEFT, where applying singular value thresholding to core matrices during fine-tuning could act as a regularizer that improves robustness to distribution shift, though the paper only claims federated benefits.
- Editorial inference: the variance bound in Eq. (15) suggests a testable scaling law, namely that halving the effective rank $\rho_i$ via shrinkage should halve the aggregation-variance bound; measuring per-head variance across $\lambda$ values would directly confirm whether the mechanism itself, rather than the scalar cap, drives the gains.
- Editorial inference: if spectrally compact updates admit tighter sensitivity analysis, TRISHUL-style shrinkage could make differentially private federated PEFT cheaper for a given privacy budget; the paper lists DP as future work and does not claim this.
- Editorial inference: dynamic allocation that re-solves the water-filling problem during training using running estimates of layer importance may outperform the one-shot pretrained-norm allocation, since the paper itself notes fixed allocation as a limitation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. TRISHUL is a federated parameter-efficient fine-tuning (PEFT) framework built on three mechanisms: (i) frozen, shared, orthonormal multi-head low-rank bases that make server-side aggregation of r×r core updates algebraically exact in the ambient parameter space; (ii) client-side nuclear-norm proximal shrinkage (singular value thresholding) applied to the small core matrices before upload; and (iii) a concave water-filling rule that allocates a fixed total head budget across layers using pretrained layer Frobenius norms as capacity scores. The paper proves the aggregation identity, derives a variance bound based on post-threshold rank, solves the water-filling program, and states a convergence theorem for the composite objective. Empirically, it compares TRISHUL against FedIT, FedEx-LoRA, FFA-LoRA, Fed-SB, SCAFFOLD+LoRA, and RAVAN on CIFAR-100, SVHN, 20 Newsgroups, MRQA, and GLUE with LLaMA3.2-1B, reporting consistent gains that grow with non-IID heterogeneity, together with ablations of each prong and direct spectral diagnostics.
Significance. If the empirical claims hold, TRISHUL is a useful contribution to federated PEFT: it offers a communication-neutral way to stabilize heterogeneous client updates, makes a specific spectral-control mechanism operational, and provides diagnostics that go beyond accuracy alone. The controlled experimental design—parameter-matched budgets, all baselines implemented in the same codebase, ablations of each individual prong, a 1B-parameter experiment, and a careful complexity analysis—is a genuine strength. The significance is currently tempered by the evaluation protocol, in particular the test-set-based selection of the regularization coefficient and the absence of error bars, and by several loose ends in the theoretical claims. With those issues fixed, the paper would be a solid contribution to the federated PEFT literature.
major comments (3)
- [Section IV-A3 and Section IV-C2 (Tables II–IV, VI, VII)] The regularization coefficient λ is selected by grid search over {0.001, 0.01, 0.1, 1.0} on the same benchmark test sets that are later used to report results, and the sensitivity study in Table VI and Figure 10 uses the same test numbers to justify the choice λ=0.01. No validation split is described anywhere in the evaluation. Because every main table uses this test-selected λ, the reported gains (e.g., 0.7–0.8% IID and 3.2–3.5% non-IID on CIFAR-100, up to 4.7% at α=0.05 in Table VII) are at risk of being a selection artifact. Additionally, with only three seeds and three participating clients per round, the absence of standard deviations or confidence intervals makes it impossible to tell whether the smaller IID differences are above run-to-run noise. I request a validation-based (or nested) selection procedure and reported error bars for the headline tables; without this, the claim in Section IV-B that TRISHUL is best across all datasets and budgets is not yet supported.
- [Section III-C, Eq. (8)–(9); Algorithm 1, line 15] The claim that nuclear-norm shrinkage of the core matrix is exactly equivalent to shrinkage of the full update omits the trainable scalar s_i. The actual update is s_i B_i H_i A_i, whose nuclear norm is |s_i| ||H_i||_*, so the penalty in Eq. (9) corresponds to an effective shrinkage coefficient λ/|s_i| on the full update, which varies across clients and across rounds unless |s_i| is constrained to be close to 1. The variance bound in Eq. (15) requires |s_i| ≤ s_max, but s_max is never reported and the scalars are reset to 1 on the server after each round (Algorithm 1, line 22), so the shrinkage strength of the uploaded product s_i H_i is not controlled as stated. The paper should either penalize λ |s_i| ||H_i||_*, remove the trainable scalar from the penalty analysis, or report s_max and justify that |s_i| remains near 1 under the training dynamics.
- [Appendix E, Theorem A.5 and Eq. (40)] The convergence theorem does not cover the algorithm as written. After aggregation, the server resets all scalars to 1 (Algorithm 1, line 22); this reset is not a proximal step with respect to Ψ in Eq. (35), and the descent inequality in Eq. (40) does not account for the change in θ it induces. In addition, Assumption A.8 requires client sampling probabilities consistent with aggregation weights, but Section IV-A2 states that the server uniformly samples three clients while Eq. (21) aggregates by data-size weights; for unequal local sample sizes these are incompatible. The theorem should be revised to model the scalar reset explicitly (or the reset should be removed from the algorithm) and to state the exact sampling scheme used in the experiments.
minor comments (7)
- [Section III-D, Eq. (17)–(20)] The optimality statement for the water-filling allocation is with respect to the paper's own logarithmic utility; the real supporting evidence is the empirical comparison in Table VIII. I recommend framing this as a design choice validated empirically rather than as an independent optimality result.
- [Tables II–III and Section IV-A3] The meaning of the 'Budget' column is confusing: the text says the lower budget matches LoRA rank 32, but TRISHUL and RAVAN are listed as budget 110. Please define the parameter-equivalent rank explicitly in terms of trainable parameters and clarify the core rank r used in the sensitivity and ablation plots.
- [Algorithm 1, line 15] The scalar clipping bound s_max is a free parameter of the method but is never specified in the experimental setup; please report its value and, ideally, its sensitivity.
- [Figures 8–9] Figures 8 and 9 each contain multiple panels but are referenced by a single figure number; please add (a)–(c) sub-labels and refer to them explicitly in the text.
- [Appendix E, Eq. (39)] The proof of Theorem A.5 relies on a 'standard proximal descent inequality' in Eq. (39), but no reference or derivation is given; please provide a citation or a self-contained proof, since the constants are not immediate.
- [Table X and Section IV-C6] The text says random-normal initialization performs best on 20 Newsgroups, which is consistent with Table X, but the caption should state the dataset-specific best in one sentence for readability.
- [Footnote 1] The repository reference 'See Trishul Github' provides no URL or identifier; please include a full link or a reproducibility statement.
Circularity Check
Headline empirical claim is partly in-sample because the regularization coefficient λ is selected on the same test sets later used to establish superiority; the theoretical derivation chain is otherwise self-contained.
-
fitted input called prediction
[Section IV-A3 (Parameter Budget and Hyperparameter) and Section IV-C2, Table VI / Figure 10]
"For TRISHUL, the spectral regularization coefficient is selected by grid search overλ∈{0.001,0.01,0.1,1.0}, withλ= 0.01providing the best and most stable performance across tasks. The full sensitivity study is reported in Section IV-C2."
The paper chooses λ=0.01 by maximizing test accuracy on the same benchmark test sets it later uses for evaluation, and then reports those same test numbers in Table VI and Figure 10 as sensitivity evidence and in Tables II–III as the main comparison. No held-out validation split is described anywhere in the experimental setup; all results are three-seed averages on the same test sets. Therefore the headline claim in Section IV-B that 'Across all datasets and parameter budgets, TRISHUL achieves the best performance' is not an independent out-of-sample prediction: the key regularization strength was fitted to the very test labels used to compare against baselines, so the reported gains are in-sample and can be inflated by selection. The theoretical chain (Eqs.
full rationale
TRISHUL's formal derivation is largely self-contained. Exact aggregation follows from linearity of the shared frozen bases (Eqs. 7 and 22); the variance bound follows from SVT rank reduction and the rank-norm inequality (Eqs. 13–15); and the water-filling allocation is the direct KKT solution of the explicitly stated concave program (Eqs. 17–18). No load-bearing conclusion depends on a chain of self-citations: the multi-head substrate is attributed to an external prior work [4], and the aggregation exactness is proved in the text rather than imported by citation. The water-filling optimality theorem is definitional of the paper's chosen utility, but the paper supplements it with an independent empirical ablation (Table VIII), so I do not treat that theorem as a circular prediction of accuracy. The main circularity is experimental: λ is selected on the same test sets later used to demonstrate TRISHUL's superiority, with no validation split, so the empirical claim is partially in-sample. The omission of the trainable scalar s_i in Eq. (8)'s equivalence claim is a technical correctness concern, not a circularity, and the unreported s_max further weakens the applied variance bound but does not reduce a prediction to an input. Overall score 4 reflects one significant fitted-input/evaluation circularity while the derivation core retains independent content and is benchmarked against external baselines.
Assumptions & free parameters
free parameters (3)
- lambda (nuclear-norm shrinkage coefficient) =
0.01 (from grid search over 0.001, 0.01, 0.1, 1.0)
- s_max (scalar clipping bound) =
not reported
- epsilon (capacity score offset) =
not reported (any epsilon > 0)
assumptions (5)
- standard math The frozen shared bases satisfy B_i^T B_i = I_r, A_i A_i^T = I_r, and inter-head orthogonality, so singular values of H_i are preserved by the embedding (Eq. 8).
- standard math Nuclear norm is the tightest convex surrogate for rank and its proximal operator is singular value thresholding.
- domain assumption Each local objective is L-smooth with bounded stochastic variance, bounded heterogeneity, and bounded second moment (Assumptions A.4-A.7).
- domain assumption Participating clients are sampled independently according to the aggregation weights (Assumptions A.3 and A.8).
- ad hoc to paper The pretrained layer Frobenius norm ||W_l||_F^2 is a positive, stable proxy for layer adaptation capacity.
Cite this review
Pith. "Pith review of Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning." pith.science (2026). https://pith.science/paper/2EOXNYQN
@misc{pith2026260720914,
author = {Pith},
title = {Pith review of: Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning},
year = {2026},
howpublished = {\url{https://pith.science/paper/2EOXNYQN}},
note = {Machine review of arXiv:2607.20914}
}
read the original abstract
Federated parameter-efficient fine-tuning (PEFT) enables communication-efficient adaptation of large pretrained models on decentralized edge data, but it remains fragile under non-IID client heterogeneity. In low-rank adaptation (LoRA), different clients may learn locally useful but spectrally misaligned update subspaces, causing high-variance aggregation and poor global transfer. We propose TRISHUL, a spectral-control framework for robust federated PEFT. TRISHUL follows the FL no-raw-data-sharing setting but does not itself provide formal privacy guarantees. TRISHUL uses shared frozen multi-head low-rank bases to obtain algebraically exact aggregation of compact core updates, applies nuclear norm proximal shrinkage to suppress client-specific high-rank spectral components before upload, and allocates adaptation heads non-uniformly across layers using a concave water filling budget rule derived from pretrained layer capacity. Because shrinkage is performed only on small core matrices, TRISHUL adds negligible computation and no extra per-round communication over the underlying multi-head PEFT protocol. Across vision and language benchmarks, including CIFAR-100, SVHN, 20 Newsgroups, MRQA, and GLUE with LLaMA3.2-1B, TRISHUL improves convergence, stability, and final performance over federated LoRA baselines, with greater gains under stronger heterogeneity.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
Communication-efficient learning of deep net- works from decentralized data,
H. B. McMahanet al., “Communication-efficient learning of deep net- works from decentralized data,” inProceedings of the 20th International Conference on Artificial Intelligence and Statistics. PMLR, 2017
work page 2017
-
[2]
Lora: Low-rank adaptation of large language models,
E. J. Huet al., “Lora: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations (ICLR), 2022
work page 2022
-
[3]
Efficient pretraining and finetuning of quantized llms with low-rank structure,
X.-Y . Liu, J. Zhang, G. Wang, W. Tong, and A. Walid, “Efficient pretraining and finetuning of quantized llms with low-rank structure,” in2024 IEEE 44th international conference on distributed computing systems (ICDCS). IEEE, 2024, pp. 300–311
work page 2024
-
[4]
Ravan: Multi-head low-rank adaptation for federated fine-tuning,
A. Rajeet al., “Ravan: Multi-head low-rank adaptation for federated fine-tuning,”arXiv preprint arXiv:2506.05568, 2025
arXiv 2025
-
[5]
Federated optimization in heterogeneous networks,
T. Li, A. K. Sahu, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,” inProceedings of Machine Learning and Systems (MLSys), 2020
work page 2020
-
[6]
Scaffold: Stochastic controlled averaging for federated learning,
S. P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” inProceedings of the 37th International Conference on Machine Learning (ICML), 2020
work page 2020
-
[7]
Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization,
B. Recht, M. Fazel, and P. A. Parrilo, “Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization,” SIAM Review, vol. 52, no. 3, pp. 471–501, 2010
2010
-
[8]
A singular value thresholding al- gorithm for matrix completion,
J.-F. Cai, E. J. Cand `es, and Z. Shen, “A singular value thresholding al- gorithm for matrix completion,”SIAM Journal on Optimization, vol. 20, no. 4, pp. 1956–1982, 2010
1956
Show all 35 references
-
[9]
Boyd and L
S. Boyd and L. Vandenberghe,Convex Optimization. Cambridge University Press, 2004
2004
-
[10]
Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations,
Z. Wanget al., “Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations,” inAdvances in Neural Information Processing Systems (NeurIPS), 2024
2024
-
[11]
Fedex-lora: Exact ag- gregation for federated parameter-efficient fine-tuning of foundation models,
R. Singhal, K. Ponkshe, and P. Vepakomma, “Fedex-lora: Exact ag- gregation for federated parameter-efficient fine-tuning of foundation models,” inNeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024
2024
-
[12]
Fed-sb: A silver bullet for extreme communication efficiency and performance in (private) federated lora fine-tuning,
R. Singhal, K. Ponkshe, R. Vartak, L. R. Varshney, and P. Vepakomma, “Fed-sb: A silver bullet for extreme communication efficiency and performance in (private) federated lora fine-tuning,”arXiv preprint arXiv:2502.15436, 2025
2025
-
[13]
Improving lora in privacy-preserving federated learning,
Y . Sun, Z. Li, Y . Li, and B. Ding, “Improving lora in privacy-preserving federated learning,” inInternational Conference on Learning Represen- tations (ICLR), 2024
2024
-
[14]
Fdlora: Personalized federated learning of large language models via dual lora tuning,
J. Qiet al., “Fdlora: Personalized federated learning of large language models via dual lora tuning,”arXiv preprint arXiv:2406.07925, 2024
2024 arXiv
-
[15]
Selective aggregation for low-rank adaptation in fed- erated learning (fedsa-lora),
P. Guoet al., “Selective aggregation for low-rank adaptation in fed- erated learning (fedsa-lora),” inInternational Conference on Learning Representations (ICLR), 2025
2025
-
[16]
Heterogeneous lora for federated fine-tuning of on-device foundation models,
Y . J. Cho, L. Liu, Z. Xu, A. Fahrezi, and G. Joshi, “Heterogeneous lora for federated fine-tuning of on-device foundation models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024, pp. 12 903–12 913
2024
-
[17]
Federated fine-tuning of large language models under heterogeneous tasks and client resources,
J. Bai, D. Chen, B. Qian, L. Yao, and Y . Li, “Federated fine-tuning of large language models under heterogeneous tasks and client resources,” inAdvances in Neural Information Processing Systems (NeurIPS), 2024
2024
-
[18]
Adalora: Adaptive budget allocation for parameter-efficient fine-tuning,
Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y . Cheng, W. Chen, and T. Zhao, “Adalora: Adaptive budget allocation for parameter-efficient fine-tuning,” inInternational Conference on Learn- ing Representations (ICLR), 2023
2023
-
[19]
Dora: Weight-decomposed low-rank adapta- tion,
S.-Y . Liu, C.-Y . Wang, H. Yin, P. Molchanov, Y .-C. F. Wang, K.-T. Cheng, and M.-H. Chen, “Dora: Weight-decomposed low-rank adapta- tion,” inInternational Conference on Machine Learning (ICML), 2024
2024
-
[20]
Vera: Vector-based random matrix adaptation,
D. J. Kopiczko, T. Blankevoort, and Y . M. Asano, “Vera: Vector-based random matrix adaptation,” inInternational Conference on Learning Representations (ICLR), 2024
2024
-
[21]
Pissa: Principal singular values and singular vectors adaptation of large language models,
F. Meng, Z. Wang, and M. Zhang, “Pissa: Principal singular values and singular vectors adaptation of large language models,” inAdvances in Neural Information Processing Systems (NeurIPS), 2024
2024
-
[22]
Olora: Orthonormal low-rank adaptation of large language models,
K. B ¨uy¨ukaky¨uz, “Olora: Orthonormal low-rank adaptation of large language models,”arXiv preprint arXiv:2406.01775, 2024
2024 arXiv
-
[23]
Exact matrix completion via convex optimization,
E. J. Cand `es and B. Recht, “Exact matrix completion via convex optimization,”Foundations of Computational Mathematics, vol. 9, no. 6, pp. 717–772, 2009
2009
-
[24]
A fast iterative shrinkage-thresholding algo- rithm for linear inverse problems,
A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algo- rithm for linear inverse problems,”SIAM Journal on Imaging Sciences, vol. 2, no. 1, pp. 183–202, 2009
2009
-
[25]
Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning,
C. H. Martin and M. W. Mahoney, “Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning,”Journal of Machine Learning Research, vol. 22, no. 165, pp. 1–73, 2021
2021
-
[26]
An image is worth 16x16 words: Trans- formers for image recognition at scale,
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Trans- formers for image recognition at scale,” inInternational Conference on Learnin...
2021
-
[27]
Exploring the limits of transfer learning with a unified text-to-text transformer,
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020
2020
-
[28]
Towards building the federatedgpt: Federated instruction tuning,
J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, G. Wang, and Y . Chen, “Towards building the federatedgpt: Federated instruction tuning,” inICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 6915– 6919
2024
-
[29]
Learning multiple layers of features from tiny images,
A. Krizhevsky, “Learning multiple layers of features from tiny images,” University of Toronto, Tech. Rep., 2009
2009
-
[30]
Reading digits in natural images with unsupervised feature learning,
Y . Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y . Ng, “Reading digits in natural images with unsupervised feature learning,” in NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011
2011
-
[31]
Twenty newsgroups,
T. Mitchell, “Twenty newsgroups,” UCI Machine Learning Repository, 1997
1997
-
[32]
Mrqa 2019 shared task: Evaluating generalization in reading comprehension,
A. Fisch, A. Talmor, R. Jia, M. Seo, E. Choi, and D. Chen, “Mrqa 2019 shared task: Evaluating generalization in reading comprehension,” inProceedings of the 2nd Workshop on Machine Reading for Question Answering, 2019, pp. 1–13
2019
-
[33]
Glue: A multi-task benchmark and analysis platform for natural lan- guage understanding,
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural lan- guage understanding,” inProceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018, pp. 353–355
2018
-
[34]
The llama 3 herd of models,
A. Grattafioriet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024. APPENDIX This appendix formalizes the three mechanisms used by TRISHUL: exact shared-basis aggregation, rank-controlled spectral shrinkage, and concave water-filling allocation. The resul...
2024 arXiv
-
[35]
LX l=1 hl =h(31) is h∗ l = al µ−1 + ,(32) whereµ>0is chosen such that P lh∗ l =h
The solution of max {hl≥0} LX l=1 al log(1 +hl) s.t. LX l=1 hl =h(31) is h∗ l = al µ−1 + ,(32) whereµ>0is chosen such that P lh∗ l =h. Proof.The Lagrangian is L= X l al log(1 +hl)−µ X l hl−h ! + X l νlhl, withν l≥0. The KKT stationarity condition gives al 1 +hl −µ+ν l = 0. For...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.