{"id":"ed1b19d2-74d4-4dd7-b952-20026e67705f","arxiv_id":"2501.15361","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Decentralized LoRA fine-tuning with gossip averaging converges to a stationary point at O(1/sqrt(T)) and matches centralized LoRA accuracy on several NLP tasks.","lead":"This paper introduces Dec-LoRA, a way to fine-tune large language models across a network of clients that talk directly to each other, with no central server. It reports that the method matches centralized LoRA accuracy in several benchmarks while also providing a convergence proof.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 4.3 (uniform boundedness of A_i and B_i) is not implied by the other assumptions; Remark 4.4's finiteness argument is invalid, so Theorem 4.8's O(1/sqrt(T)) rate is conditional on an unverified trajectory property.","rationale":"The reader identified Assumption 4.3 as the weakest assumption, and my stress-test agrees: it is the single load-bearing point where the proof of the main theorem could fail. The issue is not that the assumption is unusual; bounded iterates are common in optimization proofs. The problem is that the paper treats it as automatically satisfied by finiteness in Remark 4.4, and that justification is mathematically incorrect. Finiteness for each finite T gives T-dependent bounds, but the theorem's constants are required to be T-independent. The proof gives no independent argument that the LoRA iterates stay in a fixed ball, and the chain-rule structure makes Assumption 4.2 insufficient to ensure this. Despite this gap, the paper has real value: the algorithm is clearly specified, the empirical comparisons are extensive, and the convergence proof is otherwise detailed and largely self-contained. The gap is repairable in principle, for example by stating the theorem with an explicit projected variant or by proving a uniform bound under additional assumptions. For this reason the reader's CONDITIONAL verdict remains appropriate; I do not see grounds to move to REJECT or to upgrade to full acceptance without addressing Assumption 4.3.","tokens_in":30436,"tokens_out":11694,"duration_ms":111062,"concrete_test":"Run the minimal instance n=2, rank r=1, K=1 with the 1-D bounded smooth objective f_i(W) = arctan(W - W0), which satisfies Assumptions 4.1 and 4.2 with a doubly stochastic 2-client mixing matrix, using the exact Dec-LoRA updates (4)-(5) with eta = 1/(2 sqrt(T)) for T = 10^2, 10^3, 10^4, 10^5, 10^6. Record M(T) = max_{t<=T, i} max(|A_i^t|, |B_i^t|). If M(T) grows with T, Assumption 4.3 is not implied and the theorem's premise is not guaranteed. Independently, inspect the proof of Lemma E.8: replace the asserted global bound C_B by sup_{t<=T} ||B_i^t|| and verify whether the final bound can be closed without a separate argument showing this supremum is O(1); if no such argument exists, Theorem 4.8 is incomplete as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 4.8's rate is not established because Assumption 4.3 is an invariant that the algorithm does not enforce. It is used exactly where the proof needs the LoRA chain-rule gradients to be controlled: Lemma E.6 uses C_B^2 in the smoothness constants, Lemma E.7/E.8 bound gradient differences by C_A^2 and C_B^2, and the final bound contains C_A^4 and C_B^4. Assumption 4.2 only bounds the full gradient in W, while grad_A f_i = B_i^T grad_W f_i and grad_B f_i = grad_W f_i A_i^T; without a uniform bound on A_i and B_i these gradients are not bounded. Remark 4.4 claims Assumption 4.3 holds if all elements remain finite. For each fixed T this is true, but the constants C_A and C_B in Theorem 4.8 must be independent of T. Pointwise finiteness only gives constants C_A(T), C_B(T) that may grow with T, and if they grow at all the explicit term L(C_A^4 + C_B^4)G^2/T^{1/2} destroys the claimed rate. The proof supplies no mechanism, such as projection, regularization, or a stepsize-dependent invariant, that would enforce the required uniform boundedness. Thus the central convergence guarantee rests on an unproven property of the trajectory, not on Assumptions 4.1, 4.2, and 4.5 alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dec-LoRA, a decentralized gossip-style algorithm for fine-tuning large language models with low-rank adapters. Clients perform K local SGD steps on their LoRA factors A and B, exchange the updated factors with neighbors, and aggregate them with a doubly stochastic mixing matrix. The authors introduce a factorized stationarity metric evaluated at the averages \\bar{B} \\bar{A}, prove an O(1/T^{1/2}) convergence rate to a stationary point under smoothness, bounded stochastic gradients, uniform boundedness of the LoRA iterates, and a spectral gap condition on the mixing matrix, and report experiments on RoBERTa-base and LLaMA-2-7B/13B and OPT-2.7B comparing Dec-LoRA with centralized LoRA and other PEFT baselines under different topologies, ranks, quantization, and data heterogeneity.","tokens_in":30817,"tokens_out":2510,"duration_ms":23979,"significance":"If the convergence claim and the empirical comparisons hold, Dec-LoRA would be a useful contribution: it is, to my knowledge, the first decentralized LoRA fine-tuning method with a non-convex convergence guarantee, and the paper includes a substantial experimental study across two model families, several GLUE/SuperGLUE tasks, ring and Erdős-Rényi topologies, quantization, and non-i.i.d. data. The explicit factorized stationarity metric is a reasonable response to the known difficulty of analyzing averages of products BA, and the appendix contains a detailed proof with clearly stated lemmas. The authors also acknowledge limitations of the method under data heterogeneity in Section F, which is helpful. The main weakness is that the central convergence theorem rests on an unverified uniform-boundedness assumption that the algorithm does not enforce; this must be fixed before the theoretical contribution can be accepted as stated.","major_comments":[{"comment":"Assumption 4.3 is load-bearing but is not guaranteed by the algorithm. It is used exactly where the proof needs control of the LoRA chain-rule gradients: Lemma E.6 obtains LC_B^2- and LC_A^2-smoothness, Lemma E.8 bounds stacked gradients by C_B^2G^2 and C_A^2G^2, and the final bound in Theorem 4.8 contains terms L(C_A^4+C_B^4)G^2/T^{1/2}. Remark 4.4 argues that the assumption holds if all elements remain finite, but for each fixed T this only yields constants C_A(T), C_B(T) that may depend on T. The theorem requires T-independent constants; if the iterates grow with T, the term L(C_A^4+C_B^4)G^2/T^{1/2} need not vanish at the claimed rate. The proof provides no projection, regularization, or stepsize-dependent invariant that would enforce uniform boundedness. Thus the O(1/T^{1/2}) guarantee is conditional on an unproven trajectory property, not on Assumptions 4.1, 4.2, and 4.5 alone.","section":"§4, Assumption 4.3 and Remark 4.4; Lemma E.6; Theorem 4.8"},{"comment":"The empirical comparisons are reported as single-run best validation accuracies with no error bars, standard deviations, or number of seeds. The central empirical claim that Dec-LoRA achieves performance comparable to centralized LoRA is therefore not quantitatively supported; for example, Table 3 shows DROP accuracy differences of roughly 3–4 points (e.g., 48.32 vs. 44.35 at rank 2) with no indication of run-to-run variability. Given that the convergence theorem is also conditional, the experiments should include repeated runs or confidence intervals before the abstract's 'performance comparable to centralized LoRA' claim can be evaluated.","section":"§5.2–5.3, Tables 1–3 and Figures 2–3"},{"comment":"The validation of the local-update effect is internally inconsistent with the stated theorem. Appendix D.5 says 'the final convergence bound is proportional to 1/T^{1/3}', but Theorem 4.8 states an O(1/T^{1/2}) bound after setting eta = 1/(K T^{1/2}). The qualitative conclusion in Fig. 2(c) and (f) may still be correct, but the passage as written does not follow from the theory and needs correction.","section":"Appendix D.5 vs. Theorem 4.8"},{"comment":"The stationarity statement is with respect to the factorized metric at \\bar{B}^{(t)}\\bar{A}^{(t)}, not with respect to the original objective in W. This is disclosed and is a reasonable choice given separate aggregation of A and B, but the abstract and introduction should state more prominently that 'converges to a stationary point' means a stationary point of the factorized metric, not necessarily of f(W0+BA) over the low-rank manifold. As written, the phrasing in the abstract and Section 1 could mislead readers into expecting a stronger guarantee.","section":"Eq. (7) and Theorem 4.8"}],"minor_comments":[{"comment":"The second displayed inequality in Lemma E.7 is labeled with E||∇A fi(BA) - ∇A fi(B'A')||^2 again, but from the proof and the following text it should be E||∇B fi(BA) - ∇B fi(B'A')||^2; please correct the statement.","section":"Lemma E.7, Eq. (24)"},{"comment":"The dimension notation in Remark 4.4 is inconsistent with the setup: it says A_i is r × k and B_i is d × r, while the paper defines A ∈ R^{r×d2} and B ∈ R^{d1×r}. Please harmonize the notation.","section":"Remark 4.4"},{"comment":"The statement in Section 5.2.2 that 'We validate these empirical results with our theoretical findings in Appendix D.5' is not fully accurate because the appendix discussion uses a different rate (1/T^{1/3}) than the theorem and only addresses monotonicity in β, not the precise finite-T behavior of the bound.","section":"Section 5.2.2 / Appendix D.5"},{"comment":"There is a duplicated citation marker in the text: '... converges to zero as T → ∞[30, 31]:[30, 31]:' appears twice; please remove the duplicate.","section":"Appendix B.4"},{"comment":"The table entry for FedSA-LoRA says 'Week metric'; this should be 'Weak metric'.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the unverified Assumption 4.3. If the authors can either add a projection step and prove the projected algorithm satisfies the theorem, or prove uniform boundedness from the existing update rule under reasonable extra conditions, the theoretical contribution would be solid and the paper would be a good fit for the journal. The empirical section also needs basic statistical reporting before the 'comparable to centralized LoRA' claim can be assessed. I would not recommend rejection, because the algorithmic idea and the proof structure are promising and the identified gap is potentially fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What should you know? The paper combines LoRA local updates with gossip averaging, and the experiments are the most solid part. The theory is structurally complete but has a real hole: Assumption 4.3 assumes uniform bounds on ||A_i|| and ||B_i|| across all rounds and clients, and Remark 4.4's justification (finiteness implies the bound) is wrong for the constants needed in Theorem 4.8, which must be independent of T. Without projection or regularization, pointwise finiteness only gives bounds that can grow with T, and the final bound contains L(C_A^4 + C_B^4)G^2 / T^{1/2}, so the rate collapses if the iterates grow. This is load-bearing, not cosmetic.\n\nWhat is genuinely new: this is the first convergence analysis for decentralized LoRA, and the authors introduce a new stationarity metric evaluated at the product of averages. That metric is weaker than the standard product-averaging metric, but they are explicit about the substitution and about why they make it. The empirical work is substantial: RoBERTa and LLaMA-2-7B/13B, multiple topologies, quantization, non-i.i.d. splits, ablations on rank and client count, and a fair comparison against Dec-Adapter, Dec-BitFit, and Dec-IA3.\n\nThe soft spots beyond the theory: LLaMA results are parity at best, with genuine drops on DROP (about 4 points) and WIC at rank 8. The abstract's phrase \"consistently superior accuracy\" oversells the small margins in Table 1. No code or data, no error bars, and reporting best validation accuracy means the numbers are selected post hoc. None of these are fatal, but they matter. The weaker metric also means the theory does not show what a reader might hope: convergence to a stationary point of the original objective at the averaged product.\n\nIf the boundedness issue is fixed—say, by adding a projection step or explicitly assuming bounded iterates as a standard condition—the theorem would go through as written. As it stands, the main theoretical claim is conditional on an unverified trajectory property.\n\nWho is this for? People working on decentralized PEFT or federated LoRA theory. It deserves a serious referee because the combination is timely and the empirical base is real, but the referee should require the boundedness issue be addressed and the empirical claims toned down. I would not cite it for the rate, but I would read it.","headline":"Decentralized LoRA with gossip is a sensible method and the experiments are the strongest part, but the central O(1/sqrt(T)) guarantee rests on a boundedness assumption that the algorithm does not enforce.","tokens_in":31296,"tokens_out":2889,"would_cite":false,"duration_ms":27515,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dec-LoRA replaces the parameter server with peer-to-peer averaging of LoRA factors and proves $O(T^{-1/2})$ convergence, with experiments matching centralized LoRA accuracy.","keywords":["decentralized federated learning","low-rank adaptation (LoRA)","large language models","parameter-efficient fine-tuning","non-convex optimization","gossip averaging","convergence analysis","quantization"],"falsifier":"On a smooth nonconvex problem with bounded data, run Dec-LoRA for growing $T$ with $\\eta=1/(K\\sqrt{T})$ and record $\\frac{1}{T}\\sum_t (\\|\\nabla_A f(\\bar B^{(t)}\\bar A^{(t)})\\|_F^2 + \\|\\nabla_B f(\\bar B^{(t)}\\bar A^{(t)})\\|_F^2)$; if the metric does not approach zero at the predicted $O(T^{-1/2})$ rate while the iterates stay bounded, the theorem is contradicted. Alternatively, if $\\max_{i,t,k}(\\|A_i^{(t)+k}\\|_F + \\|B_i^{(t)+k}\\|_F)$ diverges as $T$ grows, then Assumption 4.3 is violated and the theorem's conditions do not hold for the actual trajectory.","tokens_in":30250,"feed_emoji":"🌐","tokens_out":11975,"duration_ms":95549,"temperature":0.7,"pith_summary":"Dec-LoRA is a fully decentralized version of LoRA: a network of clients fine-tunes the low-rank factors $A$ and $B$ locally on private data, exchanges only those small matrices with neighbors, and averages them with a doubly stochastic mixing matrix $Q$ instead of sending anything to a server. The paper's central claim is twofold. Empirically, Dec-LoRA matches the accuracy of centralized LoRA on RoBERTa and LLaMA-2 across GLUE and SuperGLUE-style tasks, including with 4-bit quantized base models and non-i.i.d. data, and it outperforms decentralized Adapter, BitFit, and IA3 baselines on average. Theoretically, under $L$-smoothness, bounded stochastic gradients, and a uniform bound on the LoRA factors, the algorithm converges to a stationary point of the global loss in the averaged variables at rate $O(T^{-1/2})$ in the number of communication rounds $T$. If true, this establishes that serverless peer-to-peer fine-tuning is a practical and provably convergent alternative to centralized federated LoRA.","feed_headline":"Matches centralized LoRA accuracy without a central server","feed_subtitle":"Peer-to-peer gossip replaces the parameter server, with a proof of O(1/sqrt(T)) convergence to a stationary point.","key_machinery":"The engine is a consensus-averaged LoRA step. Each client runs $K$ local SGD steps on $A_i$ and $B_i$ with learning rate $\\eta=1/(K\\sqrt{T})$, then sets $A_i^{(t+1)}=\\sum_j q_{ij} A_j^{(t)+K}$ and similarly for $B_i$, where $Q$ is symmetric doubly stochastic with second-largest-magnitude eigenvalue $\\beta<1$. The proof separates two effects: geometric contraction of the deviation from the network mean (Lemma 4.7) and smoothness of the loss as a function of each factor when the other is fixed (Lemma 4.10, with constants $LC_B^2$ and $LC_A^2$). Summing the descent inequalities for $A$ and $B$ cancels the cross terms and yields the bound in Theorem 4.8; the tractable stationarity metric is the squared norm of $\\nabla_{(A,B)} f(\\bar B \\bar A)$, which is what one can actually control when $A$ and $B$ are aggregated separately.","core_discovery":"The discovery is that LoRA's separate $A$ and $B$ updates can be averaged peer-to-peer without destroying convergence, provided stationarity is measured at the averaged point $(\\bar B^{(t)}, \\bar A^{(t)})$ rather than at the average of the products $B_i A_i$. The paper proves (Theorem 4.8) that when each client performs $K$ local SGD steps and neighbors mix via $Q$, the averaged squared gradient norms, $\\frac{1}{T}\\sum_{t} (\\mathbb{E}\\|\\nabla_A f(\\bar B^{(t)}\\bar A^{(t)})\\|_F^2 + \\mathbb{E}\\|\\nabla_B f(\\bar B^{(t)}\\bar A^{(t)})\\|_F^2)$, decay as $O(T^{-1/2})$ with step size $\\eta = 1/(K\\sqrt{T})$. The experiments support the same story: on RoBERTa-base, classification accuracy is within a point or two of centralized LoRA, and on LLaMA-2-7B the gap is small on most benchmarks; 4-bit quantization barely changes accuracy; non-i.i.d. splits cause only a modest drop.","pith_inferences":["The convergence guarantee is for stationarity at the averaged factors $(\\bar B,\\bar A)$, not at the average product $\\frac{1}{n}\\sum_i B_i A_i$; those two points need not coincide, so the theory leaves the product-average gap unquantified.","Assumption 4.3's uniform bounds on $\\|A_i\\|_F$ and $\\|B_i\\|_F$ are not enforced by the update rules, so a natural testable extension is to add norm clipping or projection after each aggregation and see whether the same $O(T^{-1/2})$ rate holds with explicit constants.","Because the deviation bounds in Lemma 4.7 grow like $(1-\\beta^2)^{-2}$ and a ring's $\\beta$ approaches 1 as $1-O(1/n^2)$, the theory implies that very large ring networks will need many more rounds; the paper's Table 5 shows exactly this qualitative pattern."],"forward_implications":["A language model can be fine-tuned on private data spread over many devices with no parameter server and no single point of failure; per-client communication stays at $O((d_1+d_2)r)$ LoRA parameters per round, independent of the number of clients.","The convergence rate $O(T^{-1/2})$ means that, up to constants, decentralized fine-tuning needs the same order of communication rounds as standard nonconvex stochastic optimization, so removing the server does not change the asymptotic convergence order.","The bound degrades as the mixing matrix's second eigenvalue approaches 1, so the theory predicts—and the paper's figures confirm—that sparser topologies converge slower than well-connected ones.","The paper reports that 4-bit quantized base models preserve almost the same Dec-LoRA accuracy as full precision, while non-i.i.d. data causes only a small drop when client counts are small."],"supporting_citations":[{"why":"defines the LoRA parameterization $W_0 + BA$ that Dec-LoRA decentralizes.","marker":"[9]"},{"why":"flags that averaging $A$ and $B$ separately differs from averaging products $BA$, motivating the paper's stationarity metric.","marker":"[24]"},{"why":"supplies the consensus-analysis framework for decentralized SGD with local updates that the proof adapts.","marker":"[74]"},{"why":"gives the spectral bound for the ring mixing matrix used in the experiments.","marker":"[75]"},{"why":"provides the convergence bound for powers of the mixing matrix used in Lemma E.5.","marker":"[82]"},{"why":"source of the stationary-point metric $\\frac{1}{T}\\sum_t \\mathbb{E}\\|\\nabla f\\|^2$ that Theorem 4.8 controls.","marker":"[30]"},{"why":"supports the claim that LoRA outperforms other PEFT methods in federated settings.","marker":"[29]"},{"why":"introduces the 4-bit QLoRA quantization used in the quantization experiments.","marker":"[38]"}],"fun_headline_variants":["Gossip-based LoRA: decentralized fine-tuning with centralized accuracy","No server needed: Dec-LoRA matches centralized fine-tuning","Peer-to-peer LoRA: same accuracy, no central server","Decentralized LoRA: proof of convergence, parity with centralized","LoRA goes fully decentralized, keeps centralized accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof assumes every client's LoRA matrices stay bounded in norm by fixed constants $C_A, C_B$ for all rounds and local steps, and the algorithm never enforces this; if the factors drift as $T$ grows, the stated convergence-rate proof no longer applies.","fun_headline_variants_meta":{"raw":{"variants":["Gossip-based LoRA: decentralized fine-tuning with centralized accuracy","No server needed: Dec-LoRA matches centralized fine-tuning","Peer-to-peer LoRA: same accuracy, no central server","Decentralized LoRA: proof of convergence, parity with centralized","LoRA goes fully decentralized, keeps centralized accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000742,"raw_usage":{"total_tokens":3343,"prompt_tokens":1008,"completion_tokens":2335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":2250}},"tokens_in":624,"tokens_out":2335,"duration_ms":14009,"temperature":1.0,"reasoning_tokens":2250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:21:31.956590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a smooth nonconvex problem with bounded data, run Dec-LoRA for growing $T$ with $\\eta=1/(K\\sqrt{T})$ and record $\\frac{1}{T}\\sum_t (\\|\\nabla_A f(\\bar B^{(t)}\\bar A^{(t)})\\|_F^2 + \\|\\nabla_B f(\\bar B^{(t)}\\bar A^{(t)})\\|_F^2)$; if the metric does not approach zero at the predicted $O(T^{-1/2})$ rate while the iterates stay bounded, the theorem is contradicted. Alternatively, if $\\max_{i,t,k}(\\|A_i^{(t)+k}\\|_F + \\|B_i^{(t)+k}\\|_F)$ diverges as $T$ grows, then Assumption 4.3 is violated and the theorem's conditions do not hold for the actual trajectory.","supporting_citations":[{"cited_title":"A unified theory of decentralized sgd with changing topology and local updates,","cited_arxiv_id":null,"evidence_quote":"supplies the consensus-analysis framework for decentralized SGD with local updates that the proof adapts."},{"cited_title":"Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning,","cited_arxiv_id":null,"evidence_quote":"supports the claim that LoRA outperforms other PEFT methods in federated settings."},{"cited_title":"Qlora: Efficient finetuning of quantized llms,","cited_arxiv_id":null,"evidence_quote":"introduces the 4-bit QLoRA quantization used in the quantization experiments."}],"review_version":1}