Collaborative and Efficient Fine-tuning: Leveraging Task Similarity

Amirhossein Reisizadeh; Asuman Ozdaglar; Chanwoo Park; Gagik Magakyan; Pablo A. Parrilo

REVIEW 2 major objections 4 minor 21 references

The paper proposes CoLoRA, a collaborative fine-tuning scheme that trains one shared low-rank adapter plus tiny per-task adapters, and gives recovery guarantees under a formal task-similarity condition.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:39 UTC pith:5M5Z3YFO

load-bearing objection CoLoRA is a useful federated PEFT idea with an honest empirical story, but Theorem 1's similarity assumption is r times weaker than what the supplied proof actually requires — an addressable, load-bearing gap. the 2 major comments →

arxiv 2602.07218 v2 pith:5M5Z3YFO submitted 2026-02-06 cs.LG cs.AIstat.ML

Collaborative and Efficient Fine-tuning: Leveraging Task Similarity

Gagik Magakyan , Amirhossein Reisizadeh , Chanwoo Park , Pablo A. Parrilo , Asuman Ozdaglar This is my paper

classification cs.LG cs.AIstat.ML

keywords parameter-efficient fine-tuninglow-rank adaptationtask similaritycollaborative fine-tuningalternating minimizationmatrix sensingsubspace distancepersonalization

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Fine-tuning large foundation models on scarce labeled data is a bottleneck; the paper tries to relieve it by letting users with similar tasks share data implicitly. It proposes CoLoRA, which replaces k separate low-rank adapters with one shared adapter (two small matrices) used by all tasks plus a small r-by-r personalized matrix per task. This cuts trainable parameters from O(kdr) to O(dr+kr^2). The paper's theoretical claim is that, for heterogeneous linear regression whose tasks have sufficiently overlapping column and row subspaces, an alternating-minimization version provably recovers the shared subspaces up to an error that shrinks as task similarity grows; experiments on instruction-style language tasks show that a fixed task improves when trained alongside similar tasks.

Core claim

On its own terms, CoLoRA claims that similar downstream tasks are not independent: their LoRA updates nearly factor as B_i A_i ≈ B Λ_i A, with B and A shared and Λ_i personal. In the linear-regression model, the ground truths M_i* are assumed close to U* Λ_i* V*^T for one common U*, V*; the paper's CoAltMin alternates local least-squares updates of Λ_i with global least-squares updates of V and U, using fresh small batches for the local steps. Theorem 1 states that under a generalized restricted-isometry property and a condition that pairwise subspace similarity is at least ξ, the recovered subspaces satisfy dist ≤ ε + O(κ^2 r sqrt(1−ξ^2)) after O(log(1/ε)) iterations, with sample complexity

What carries the argument

The load-bearing object is the shared factorization M_i* ≈ U* Λ_i* V*^T (equivalently B Λ_i A in LoRA coordinates): it is what lets one global pair of adapters serve all tasks while Λ_i preserves personalization. The proof machinery is CoAltMin — alternating minimization over Λ_i (local), then V, then U (global) — analyzed through a generalized restricted isometry property (GRIP, a RIP-type concentration condition adapted to the U,V,Λ structure) plus a sub-isometric bound on the measurement ensemble; these control the noise and approximation terms in each update.

Load-bearing premise

The theorem's guarantee rests on the unproved assertion, made just after Definition 2, that pairwise subspace similarity at least ξ implies a common reference pair U*, V* within distance sqrt(r(1−ξ²)) of every task's subspaces; if that implication fails, the error bound in Theorem 1 does not follow as stated.

What would settle it

Search for three r-dimensional subspaces in sufficiently high dimension with all pairwise subspace similarities ≥ξ but no common r-dimensional subspace within distance sqrt(r(1−ξ²)) of all three, by random sampling on the Grassmannian and solving for the minimax center; a concrete instance would disprove equation (13), the step on which Theorem 1 depends.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

With k users, trainable parameters drop from O(kdr) to O(dr+kr^2); for small rank r this makes collaborative fine-tuning scalable to many users.
Similar tasks effectively pool their fine-tuning data through the shared adapter, mitigating data scarcity, while each user keeps a personalized Λ_i.
When all tasks share identical row and column subspaces (ξ=1), the recovery error can be driven to any ε, matching the classical single-task low-rank matrix sensing behavior.
When task similarity is low, the shared structure helps less and the method can underperform purely local fine-tuning, as the experiments show.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The pairwise-similarity notion is a proxy for actual transferability; selecting collaborators by measured adapter-subspace overlap (especially in later layers) is a testable way to decide which users should train together.
If the unproved step from pairwise similarity to a common reference subspace fails, grouping tasks into clusters that each admit a common reference would preserve the CoLoRA benefit; the paper's own experiments hint that cluster-level structure is what matters.
The linear-regression analysis is Gaussian and factorization-based; the same guarantee should not be expected for nonlinear losses, but the empirical success on instruction tasks suggests the shared-subspace inductive bias, not the exact theorem, is doing the work.
The Kronecker structure V*⊗U* of the shared representation suggests specialized sample-complexity gains over generic multitask representation learning; one could test this by comparing CoLoRA against a generic shared-head method on synthetic tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

CoLoRA is a useful federated PEFT idea with an honest empirical story, but Theorem 1's similarity assumption is r times weaker than what the supplied proof actually requires — an addressable, load-bearing gap.

read the letter

Two things to know. First, the theory has a real gap that the reader's report missed, and the reader's own main objection does not survive contact with the paper. The CoLoRA idea and its experiments are in decent shape; the theorem needs an r-factor fix.

What's actually new: the structure BΛiA with trained shared A,B and per-task Λi, giving O(dr+kr²) trainable parameters, and a collaborative alternating-minimization analysis for two-sided shared representations. That goes beyond Fed-SB's frozen A,B and the single-representation theory of Collins et al. The empirical core is a clean dose-response: for a fixed target task, Rouge-L goes up monotonically as the average subspace similarity of the collaborating tasks increases (Figure 3, Tables 3-4), and CoLoRA beats Local LoRA, FedDPA, RoLoRA, and ALoRA in most regimes while losing to Local LoRA at very low similarity — exactly what you'd expect. The related work section is honest, including the Brüel-Gabrielsson et al. caveat about single shared decompositions.

Soft spots. The reader claims Eq. (13) — pairwise similarity implying a common reference subspace — is asserted without proof. It's not a problem: take U* = U_1*, and identity (16) gives dist(U_1*,U_i*) ≤ √(r(1−ξ²)) for every i. The objection doesn't land. The actual gap is between Theorem 1 and the proof. Theorem 1 requires ξ² ≥ 1 − Θ(1)/(κ²ϲ²r(1+rd/N)), but Theorem 2's condition (19) needs β = √r√(1−ξ²) ≤ 1/(4·10⁴√r κϲ √A_{N,d,r}), which after squaring gives 1−ξ² ≲ 1/(r²κ²ϲ²(1+rd/N)) — a factor of r stricter. So the main recovery guarantee, as stated, does not follow from the supplied proof. Either the theorem's assumption must be strengthened by that r factor or the analysis sharpened. That's load-bearing for the theory section, but plausibly fixable.

Also: experiments are single-seed, no error bars, no code, and the closest baseline, Fed-SB, is discussed but never run. The similarity metric is measured from the very LoRA objects whose shareability the paper is trying to justify — a motivation in a circle, though the theorem's constants don't depend on it.

Who this is for: people working on federated or multi-task PEFT. A reading group would get a good hour out of the proof-gap diagnosis alone. It deserves a serious referee — send to review, and ask the authors to fix the r-factor, add Fed-SB, and report multiple seeds.

Referee Report

2 major / 4 minor

Summary. The paper proposes CoLoRA, a collaborative parameter-efficient fine-tuning method. Instead of learning a separate LoRA adapter per task, it parameterizes each adapter as B Λ_i A, with shared A, B ∈ R^{d×r} and task-specific Λ_i ∈ R^{r×r}. The paper introduces a subspace-similarity measure between adapters, motivates CoLoRA empirically, and analyzes a linear-regression analogue via a collaborative alternating-minimization algorithm (CoAltMin). The main theoretical claim (Theorem 1) is that, under a task-similarity condition ξ, CoAltMin recovers shared subspaces up to error ε + O(κ² r √(1−ξ²)) with the stated sample complexity. Experiments on NaturalInstructions with Qwen2.5-1.5B compare CoLoRA with local LoRA, RoLoRA, FedDPA, and ALoRA.

Significance. If correct, the central idea is attractive: the shared A, B plus per-task Λ_i decomposition is a natural parameter-efficient personalization scheme, and the matrix-sensing analysis with a generalized RIP is a nontrivial extension of the single-task AltMin literature. The appendix contains a substantial proof skeleton with GRIP, (U,V)-RIP, sub-isometry, and concentration lemmas, which is a strength and makes the claimed mechanisms checkable. However, I find a load-bearing mismatch between Theorem 1 and the proof of Theorem 2, and the empirical section does not provide the statistical support needed for the claimed gains. I also concur with the stress-test note that the Eq. (13) concern is not valid: pairwise subspace similarity plus Eq. (16) directly gives a common reference by choosing any task's basis.

major comments (2)

[§4.2, Theorem 1; §B.2, Theorem 2, Eq. (19)] The theorem as stated is not supported by the proof. Theorem 1 assumes ξ² ≥ 1 − Θ(1)/(κ²ϱ² r(1 + rd/N)). The proof of Theorem 2 requires condition (19), β ≤ 1/(4·10⁴ √r max(κ², κγ) √A_{N,d,r}). Since β = √r √(1−ξ²) (Eq. (17)) and max(κ², κγ) = κϱ, condition (19) is 1−ξ² ≤ Θ(1)/(r² κ² ϱ² A_{N,d,r}). With A_{N,d,r} ≈ 1 + rd/N, this is a factor of r stricter than the assumption in Theorem 1. Thus the stated guarantee does not follow from the supplied proof. Please strengthen the theorem statement (e.g., replace r by r² in the denominator) or provide a sharper argument; Corollary 1 inherits this issue.
[§5, Tables 3–5, Figures 2–4; §A.2] All experimental results appear to be single runs. Tables 3–5 and Figures 2–4 report one Rouge-L value per condition, and §A.2 says only that the same seed was used in initialization, without giving multiple seeds or variability. Given the 50-point-per-client data regime and the paper's claims of 'significant' improvement and 'consistently outperforms', the empirical conclusions are not statistically supported. Please report means and standard errors (or confidence intervals) over multiple seeds and state the number of runs.

minor comments (4)

[§4, Eq. (13)] The transition to reference matrices U*, V* is asserted without proof. The assertion is true—choose U* = U_1^*, V* = V_1^* and use Eq. (16)—but a one-line derivation would remove ambiguity.
[§3 and Algorithm 2] The O(dr + kr²) parameter count applies to the centralized parameterization. In Algorithm 2 each client still stores local B_i, A_i ∈ R^{d×r} during training, so the per-client training state is O(dr), not O(r²). Please clarify what the counting includes (global model vs. per-client local state).
[§5.1, §A.3] The text says performance is 'predicted in our theory', but the theory measures ξ on ground-truth matrices M_i^*, whereas the experiments measure similarity of trained adapters. This link is heuristic and should be stated as such.
[Throughout] There are numerous typos and OCR artifacts: 'coresponding', 'Rougle-L', 'gaurantee', 'tax similarity' in §5.1, and inconsistent notation such as Λ_i^* vs. (Λ^i)^*. These should be cleaned up.

Circularity Check

1 steps flagged

No load-bearing circularity; only a definitional motivation tautology, while the main AltMin guarantee is self-contained.

specific steps

self definitional [Section 2.3, Definition 1 and the 'Takeaway' paragraph]
"simc(Ti,Tj) := 1/√r ∥U_i^⊤ U_j∥_F ... larger values of subspace similarity ... hinting of a common matrix B that aims at satisfying (4) approximately. ... This experiment further corroborates our initial hypothesis that similar tasks tend to have common underlying structures in the sense of (4) when fine-tuned with LoRA."

Task similarity is defined as the overlap of LoRA adapter column/row subspaces. The conclusion that 'similar tasks have common underlying structures in the sense of (4)' restates that definition: high sim_c and sim_r means a common B (and A) with B_iA_i ≈ BΛ_iA exists by construction. So the motivational 'corroboration' is a tautology, not an independent empirical derivation. It is not used in the proof of Theorem 1, so it does not make the recovery guarantee circular.

full rationale

The theoretical derivation is self-contained once the structural assumption (8)/(13) is granted. The reader's objection to (13) is invalid: for pairwise similarity ≥ ξ, choosing U^*=U_1^*, V^*=V_1^* gives dist(U^*,U_i^*), dist(V^*,V_i^*) ≤ sqrt(r(1−ξ^2)) by the principal-angle identity (16), so no missing common-reference theorem is needed. The substantive issue is the skeptic's factor-r mismatch: Theorem 1 states ξ^2 ≥ 1 − Θ(1)/(κ^2ϱ^2 r (1+rd/N)), while the proof's condition (19) with β=√r√(1−ξ^2) requires (1−ξ^2) ≲ 1/(r^2 κ^2 ϱ^2 (1+rd/N)), i.e. an extra factor of r; this is a proof gap affecting the stated theorem, not a circular reduction. The empirical similarity metric is defined in terms of the same adapter subspaces CoLoRA exploits, which gives the motivation a mildly definitional flavor, but the central claim (CoAltMin recovery and parameter efficiency) is established by AltMin/GRIP arguments independent of fitted constants, so circularity is low.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 1 invented entities

The core theoretical result relies on a structural data model and on an unproved inference from pairwise similarity to a common reference subspace. The empirical method relies on a heuristic similarity measure. No fitted numerical free parameters are used in the theory; the experimental hyperparameters are stated but not swept.

axioms (5)

domain assumption Ground-truth tasks share row/column subspaces: M_i* ≈ U* Lambda_i* V*^T (Eq. 8)
The linear theory presumes this structure; it is the phenomenon CoLoRA is designed to exploit, not derived from LoRA or from data.
ad hoc to paper Pairwise subspace similarity (10) implies existence of common U*, V* within beta of every task (Eq. 13)
Stated as a consequence of Definition 2 but not proved; pairwise closeness to a common center is stronger than pairwise closeness.
domain assumption Gaussian/sub-Gaussian measurement ensemble G_j^i in Eq. (7)
Standard in matrix sensing; needed for GRIP concentration, but not justified as a model of language fine-tuning data.
domain assumption LoRA subspace similarity measured from independently trained adapters predicts collaborative gains of CoLoRA
Used to construct collaborator groups (Figs. 2-3) and to motivate Eq. (4); heuristic, with no formal link to the language objective.
standard math RIP/GRIP concentration and subspace perturbation inequalities (Lemmas 11-15)
Background results invoked in proofs; accepted as standard in the high-dimensional probability literature.

invented entities (1)

Column-subspace task similarity sim_c (Definition 1) no independent evidence
purpose: Quantify task similarity from LoRA adapters; used to rank tasks and to state the theoretical task-similarity condition (10).
The metric is constructed from the same trained adapters whose shared structure CoLoRA assumes; no external falsifiable handle is provided outside the paper's own framework.

pith-pipeline@v1.3.0-alltime-deepseek · 45453 in / 16775 out tokens · 173700 ms · 2026-08-03T03:39:27.456973+00:00 · methodology

0 comments

read the original abstract

Adaptability has been regarded as a central feature in the foundation models, enabling them to effectively acclimate to unseen downstream tasks. Parameter-efficient fine-tuning methods such as celebrated LoRA facilitate efficient adaptation of large foundation models using labeled, high-quality and generally scarce task data. To mitigate data scarcity in fine-tuning of foundation models, we propose to leverage task similarity across multiple downstream users. Intuitively, users with similar tasks must be able to assist each other in boosting the effective fine-tuning data size. We propose Collaborative Low-Rank Adaptation, or CoLoRA, which exploits task similarity to collaboratively and efficiently fine-tune personalized foundation models. The main idea in CoLoRA is to train one shared adapter capturing underlying task similarities across all tasks, and personalized adapters tailored to user-specific tasks. We theoretically study CoLoRA on heterogeneous linear regression and provide provable guarantees for ground truth recovery. We also conduct several natural language experiments with varying task similarity, which further demonstrate that when trained together with similar tasks, individual performances are significantly boosted.

Figures

Figures reproduced from arXiv: 2602.07218 by Amirhossein Reisizadeh, Asuman Ozdaglar, Chanwoo Park, Gagik Magakyan, Pablo A. Parrilo.

**Figure 2.** Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Performance of CoLoRA for a fixed task w.r.t different levels of similarity. The dashed line is the performance when only using local data. 5.2. Comparison with baselines Next, we compare CoLoRA against the following baselines: • Local LoRA. Each client independently fine-tunes LoRA adapters on its private data; no parameters are communicated. • RoLoRA. Chen et al. (2025) alternate between local training o… view at source ↗

**Figure 5.** Figure 5: Similarity across network layers. Smaller matrix indices correspond to parameters from earlier layers in the network. Red points denote the similarity between tasks T1 and T2, while blue points correspond to T1 and T3. A.3. Experiment tables [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

21 extracted references · 10 linked inside Pith

[1]

For all the comparisons experiments, we used the same seed in the initialization to ensure consistency

We used rouge-L score for all evaluation metrics. For all the comparisons experiments, we used the same seed in the initialization to ensure consistency. We used one NVIDIA-A100/L40 for our experiments. Similarity calculation protocol.Following Bai et al. (2024) and Mishra et al. (2022), we do not use the full data for training adapters for reducing compu...

2024
[2]

Hence, it satisfies the distance properties such as triangle inequality

= Q1(Q1)⊤ −Q 2(Q2)⊤ 2 =∥P 1 −P 2∥2 . Hence, it satisfies the distance properties such as triangle inequality. C.2. Classical Gaussian Concentration We are using the following classical concentration result several times in the manuscript. Lemma 11.Let n be a positive integer. Suppose G1, . . . ,Gn are i.i.d and for every i, the entroes of Ji are i.i.d fro...

2019
[3]

Experiment tables Table 3.Similarity and performance values forT 1 1

A.3. Experiment tables Table 3.Similarity and performance values forT 1 1 . Subspace similarity 0.129 0.133 0.138 0.163 0.180 0.192 0.207 0.230 0.234 0.242 0.263 Rougle-L score 0.772 0.786 0.760 0.823 0.805 0.847 0.910 0.899 0.890 0.891 0.924 Table 4.Similarity and performance values forT 2 1 . Task 2 Subspace similarity 0.114 0.136 0.151 0.196 0.238 0.28...

arXiv 2013
[4]

and Richt ´arik, P

Hanzely, F. and Richt ´arik, P. Federated learning of a mixture of global and local models.arXiv preprint arXiv:2002.05516,

Pith/arXiv arXiv 2002
[5]

Mansour, Y ., Mohri, M., Ro, J., and Suresh, A. T. Three approaches for personalization with applications to feder- ated learning.arXiv preprint arXiv:2002.10619,

Pith/arXiv arXiv 2002
[6]

Rlhf from heterogeneous feedback via personal- ization and preference aggregation.arXiv preprint arXiv:2405.00254,

Park, C., Liu, M., Kong, D., Zhang, K., and Ozdaglar, A. Rlhf from heterogeneous feedback via personal- ization and preference aggregation.arXiv preprint arXiv:2405.00254,

Pith/arXiv arXiv
[7]

Qwen2.5 technical report.arXiv preprint at arxiv:2412.15115,

Qwen Team. Qwen2.5 technical report.arXiv preprint at arxiv:2412.15115,

Pith/arXiv arXiv
[9]

R., Chen, D., Guibas, L., Malik, J., and Savarese, S

Standley, T., Zamir, A. R., Chen, D., Guibas, L., Malik, J., and Savarese, S. Identifying beneficial task relations for multi-task learning in deep neural networks.arXiv preprint arXiv:1702.08303,

Pith/arXiv arXiv
[10]

A practical guide to fine-tuning language mod- els with limited data.arXiv preprint arXiv:2411.09539,

Szep, M., Rueckert, D., von Eisenhart-Rothe, R., and Hinter- wimmer, F. A practical guide to fine-tuning language mod- els with limited data.arXiv preprint arXiv:2411.09539,

arXiv
[12]

L., and Pal, C

Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., Hamilton, W. L., and Pal, C. Ex- ploring and predicting transferability across nlp tasks. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP),

2020
[13]

However, their method requires computing a full singular value decomposition (SVD) at each iteration, which limits its computational efficiency

proposed a projected gradient descent algorithm that achieves similar recovery guarantees as (Recht et al., 2010). However, their method requires computing a full singular value decomposition (SVD) at each iteration, which limits its computational efficiency. (Jain et al.,

2010
[15]

Fallah et al

propose frameworks that combine global and local parameters, along with alternating optimization framework to jointly learn them. Fallah et al. (2020) adopts a meta-learning perspective, aiming to learn a global model that performs well on each local task after a single gradient update. While these methods are related to our setting, their analyses focus ...

2020
[18]

The next lemma shows that across all iterations the ”local” concentrations happen with high probability

Define eU0,t =U t, eV0,t =V t, eU1,t = Ut,U tUt ⊤U i∗ , eV1,t = Vt,V tVt ⊤V i∗ −V i∗ , eU2,t = Ut,U tUt ⊤U i∗ −U i∗ , eV2,t = Vt,V i∗ , where[A,B]is concatenation across column axis. The next lemma shows that across all iterations the ”local” concentrations happen with high probability. Lemma 2.With probability at least: 1−3C·T k·exp − cn2(δ′ 2r)2 162 (29...

2010
[21]

40 Collaborative and Efficient Fine-tuning D.11

Combining these bounds, we conclude: x⊤F2 = 1 + 2 ≤36m· k max i=1 Λi∗ 2 Λi∗ F · δ3rdist(U,U ∗) +β p An,d,r Sincexwas arbitrary, this completes the proof. 40 Collaborative and Efficient Fine-tuning D.11. Proof of Lemma 10 We have: σmin(Rt+1) = min ∥z∥=1 ∥Rt+1z∥2 = min ∥z∥=1 ∥Vt+1Rt+1z∥2 , = min ∥z∥=1 bVt+1z 2 = min ∥z∥=1 V ∗((Dt)−1)⊤z−mat(H t)z 2 ≥min ∥z∥=...

2019
[32]

Consequently, we can update the factor Λ directly on the large subset without needing the chunks

However, when there is only a single client, the GRIP condition 30 already guarantees that the sample set{Gj}N j=1 satisfies the (U,V) -RIP for all U,V∈R d×r. Consequently, we can update the factor Λ directly on the large subset without needing the chunks. In this case, settingβ= 0, the resulting sample complexity reduces to O(κ4dr2), which improves upon ...

2013
[2010]

Fedex-lora: Exact aggregation for federated and efficient fine-tuning of foundation models.arXiv preprint arXiv:2410.09432,

Singhal, R., Ponkshe, K., and Vepakomma, P. Fedex-lora: Exact aggregation for federated and efficient fine-tuning of foundation models.arXiv preprint arXiv:2410.09432,

Pith/arXiv arXiv
[2016]

demonstrated that a simple gradient descent algorithm can achieve recovery under the milder condition δ6r ≤c , reducing the sample complexity back to Ω(dr) while remaining computationally efficient. Linear Representation Learning.Our framework is related to the literature on multitask linear representation learning (Collins et al., 2021; Du et al., 2021; ...

2021
[2022]

M., and Mahdavi, M

Deng, Y ., Kamani, M. M., and Mahdavi, M. Adap- tive personalized federated learning.arXiv preprint arXiv:2003.13461,

Pith/arXiv arXiv 2003
[2023]

K., Jain, P., Netrapalli, P., and Oh, S

Thekumparampil, K. K., Jain, P., Netrapalli, P., and Oh, S. Sample efficient linear meta-learning by alternating minimization.arXiv preprint arXiv:2105.08306,

Pith/arXiv arXiv
[2024]

and Ji, K

Ban, H. and Ji, K. Rethinking parameter sharing for llm fine-tuning with multiple loras.arXiv preprint at arxiv:2509.25414,

Pith/arXiv arXiv
[2025]

A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258,

Pith/arXiv arXiv

[1] [1]

For all the comparisons experiments, we used the same seed in the initialization to ensure consistency

We used rouge-L score for all evaluation metrics. For all the comparisons experiments, we used the same seed in the initialization to ensure consistency. We used one NVIDIA-A100/L40 for our experiments. Similarity calculation protocol.Following Bai et al. (2024) and Mishra et al. (2022), we do not use the full data for training adapters for reducing compu...

2024

[2] [2]

Hence, it satisfies the distance properties such as triangle inequality

= Q1(Q1)⊤ −Q 2(Q2)⊤ 2 =∥P 1 −P 2∥2 . Hence, it satisfies the distance properties such as triangle inequality. C.2. Classical Gaussian Concentration We are using the following classical concentration result several times in the manuscript. Lemma 11.Let n be a positive integer. Suppose G1, . . . ,Gn are i.i.d and for every i, the entroes of Ji are i.i.d fro...

2019

[3] [3]

Experiment tables Table 3.Similarity and performance values forT 1 1

A.3. Experiment tables Table 3.Similarity and performance values forT 1 1 . Subspace similarity 0.129 0.133 0.138 0.163 0.180 0.192 0.207 0.230 0.234 0.242 0.263 Rougle-L score 0.772 0.786 0.760 0.823 0.805 0.847 0.910 0.899 0.890 0.891 0.924 Table 4.Similarity and performance values forT 2 1 . Task 2 Subspace similarity 0.114 0.136 0.151 0.196 0.238 0.28...

arXiv 2013

[4] [4]

and Richt ´arik, P

Hanzely, F. and Richt ´arik, P. Federated learning of a mixture of global and local models.arXiv preprint arXiv:2002.05516,

Pith/arXiv arXiv 2002

[5] [5]

Mansour, Y ., Mohri, M., Ro, J., and Suresh, A. T. Three approaches for personalization with applications to feder- ated learning.arXiv preprint arXiv:2002.10619,

Pith/arXiv arXiv 2002

[6] [6]

Rlhf from heterogeneous feedback via personal- ization and preference aggregation.arXiv preprint arXiv:2405.00254,

Park, C., Liu, M., Kong, D., Zhang, K., and Ozdaglar, A. Rlhf from heterogeneous feedback via personal- ization and preference aggregation.arXiv preprint arXiv:2405.00254,

Pith/arXiv arXiv

[7] [7]

Qwen2.5 technical report.arXiv preprint at arxiv:2412.15115,

Qwen Team. Qwen2.5 technical report.arXiv preprint at arxiv:2412.15115,

Pith/arXiv arXiv

[8] [9]

R., Chen, D., Guibas, L., Malik, J., and Savarese, S

Standley, T., Zamir, A. R., Chen, D., Guibas, L., Malik, J., and Savarese, S. Identifying beneficial task relations for multi-task learning in deep neural networks.arXiv preprint arXiv:1702.08303,

Pith/arXiv arXiv

[9] [10]

A practical guide to fine-tuning language mod- els with limited data.arXiv preprint arXiv:2411.09539,

Szep, M., Rueckert, D., von Eisenhart-Rothe, R., and Hinter- wimmer, F. A practical guide to fine-tuning language mod- els with limited data.arXiv preprint arXiv:2411.09539,

arXiv

[10] [12]

L., and Pal, C

Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., Hamilton, W. L., and Pal, C. Ex- ploring and predicting transferability across nlp tasks. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP),

2020

[11] [13]

However, their method requires computing a full singular value decomposition (SVD) at each iteration, which limits its computational efficiency

proposed a projected gradient descent algorithm that achieves similar recovery guarantees as (Recht et al., 2010). However, their method requires computing a full singular value decomposition (SVD) at each iteration, which limits its computational efficiency. (Jain et al.,

2010

[12] [15]

Fallah et al

propose frameworks that combine global and local parameters, along with alternating optimization framework to jointly learn them. Fallah et al. (2020) adopts a meta-learning perspective, aiming to learn a global model that performs well on each local task after a single gradient update. While these methods are related to our setting, their analyses focus ...

2020

[13] [18]

The next lemma shows that across all iterations the ”local” concentrations happen with high probability

Define eU0,t =U t, eV0,t =V t, eU1,t = Ut,U tUt ⊤U i∗ , eV1,t = Vt,V tVt ⊤V i∗ −V i∗ , eU2,t = Ut,U tUt ⊤U i∗ −U i∗ , eV2,t = Vt,V i∗ , where[A,B]is concatenation across column axis. The next lemma shows that across all iterations the ”local” concentrations happen with high probability. Lemma 2.With probability at least: 1−3C·T k·exp − cn2(δ′ 2r)2 162 (29...

2010

[14] [21]

40 Collaborative and Efficient Fine-tuning D.11

Combining these bounds, we conclude: x⊤F2 = 1 + 2 ≤36m· k max i=1 Λi∗ 2 Λi∗ F · δ3rdist(U,U ∗) +β p An,d,r Sincexwas arbitrary, this completes the proof. 40 Collaborative and Efficient Fine-tuning D.11. Proof of Lemma 10 We have: σmin(Rt+1) = min ∥z∥=1 ∥Rt+1z∥2 = min ∥z∥=1 ∥Vt+1Rt+1z∥2 , = min ∥z∥=1 bVt+1z 2 = min ∥z∥=1 V ∗((Dt)−1)⊤z−mat(H t)z 2 ≥min ∥z∥=...

2019

[15] [32]

Consequently, we can update the factor Λ directly on the large subset without needing the chunks

However, when there is only a single client, the GRIP condition 30 already guarantees that the sample set{Gj}N j=1 satisfies the (U,V) -RIP for all U,V∈R d×r. Consequently, we can update the factor Λ directly on the large subset without needing the chunks. In this case, settingβ= 0, the resulting sample complexity reduces to O(κ4dr2), which improves upon ...

2013

[16] [2010]

Fedex-lora: Exact aggregation for federated and efficient fine-tuning of foundation models.arXiv preprint arXiv:2410.09432,

Singhal, R., Ponkshe, K., and Vepakomma, P. Fedex-lora: Exact aggregation for federated and efficient fine-tuning of foundation models.arXiv preprint arXiv:2410.09432,

Pith/arXiv arXiv

[17] [2016]

demonstrated that a simple gradient descent algorithm can achieve recovery under the milder condition δ6r ≤c , reducing the sample complexity back to Ω(dr) while remaining computationally efficient. Linear Representation Learning.Our framework is related to the literature on multitask linear representation learning (Collins et al., 2021; Du et al., 2021; ...

2021

[18] [2022]

M., and Mahdavi, M

Deng, Y ., Kamani, M. M., and Mahdavi, M. Adap- tive personalized federated learning.arXiv preprint arXiv:2003.13461,

Pith/arXiv arXiv 2003

[19] [2023]

K., Jain, P., Netrapalli, P., and Oh, S

Thekumparampil, K. K., Jain, P., Netrapalli, P., and Oh, S. Sample efficient linear meta-learning by alternating minimization.arXiv preprint arXiv:2105.08306,

Pith/arXiv arXiv

[20] [2024]

and Ji, K

Ban, H. and Ji, K. Rethinking parameter sharing for llm fine-tuning with multiple loras.arXiv preprint at arxiv:2509.25414,

Pith/arXiv arXiv

[21] [2025]

A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258,

Pith/arXiv arXiv