{"id":"5b3c109e-6bf9-42cc-b5bc-4d3951693c79","arxiv_id":"2505.18494","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"FedHL aggregates heterogeneous LoRA updates against a full-rank global baseline and claims O(1/sqrt T) convergence, with small gains on three LLM fine-tuning datasets.","lead":"A federated learning method for fine-tuning large models with different per-client LoRA ranks uses the previous full-rank model as the aggregation baseline and adaptively weights clients by their truncation error. The paper claims this removes truncation bias and gives a 1/sqrt(T) convergence guarantee, but the proof does not cover the adaptive weighting actually used.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adaptive aggregation weights in Algorithm 1 are not the fixed weights used in the Theorem 2 proof, so the O(1/sqrt T) guarantee does not cover FedHL as published.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing gap: the convergence proof fixes p_i while the algorithm updates p_i(t) from current truncation errors. This is not a novelty dispute or a stylistic concern; it is a correctness risk in the central advertised result. The proof of Lemma 2 uses the fixed objective f(W) = Σ_i p_i f_i(W) and telescopes the descent inequality over rounds. With time-varying p_i(t), the iterates are not descending on any single fixed objective, and the term involving \\hat r_i(t)^2 in the drift bound has no stated decay guarantee. The paper's own remark that \\hat r_i(t) is simplified in the notation strengthens the concern: the theorem implicitly assumes constant truncation errors as well as constant weights. The empirical results are consistently positive but modest (1-3%), and they do not compensate for the missing formal guarantee. A revised paper could fix this by either proving the theorem for the actual adaptive weights under an explicit bound on the variation of p_i(t) and \\hat r_i(t), or by clearly stating Theorem 2 for a fixed-weight variant and presenting the adaptive version as heuristic. As written, the reader's REJECT verdict remains appropriate.","tokens_in":31645,"tokens_out":6346,"duration_ms":56459,"concrete_test":"Re-derive Eq. (38) with p_i replaced by p_i(t) = (\\hat r_i(t)^2 + ε)^{-1} / Σ_j(\\hat r_j(t)^2 + ε)^{-1} and with \\hat r_i(t) time-varying. Show that the telescoping sum now contains Σ_{t=0}^{T-1} η_t Σ_i p_i(t)^2 (\\hat r_i(t)^2 + ...) and that no fixed function f(W) = Σ_i p_i f_i(W) can make the descent inequality telescope; state explicitly what additional bound on Σ_t p_i(t)^2 \\hat r_i(t)^2 would be needed to recover O(1/sqrt T). As a numerical companion, run Algorithm 1 exactly on a two-client quadratic objective with ranks 1 and 2 for T=500, record min_{t≤T} ||∇f(W_t)||^2, and compare the log-log slope to -1/2; a slope shallower than -1/2 would confirm the dynamic-weight version is not covered by the theorem.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The advertised central claim is Theorem 2's O(1/sqrt T) convergence rate for FedHL. The proof, however, is built on fixed client weights p_i. Lemma 2 (Eq. (29)) bounds Ef(V_{t+1}) with p_i independent of t, and the telescoping argument in Eq. (37)-(38) requires a single fixed objective f(W) = Σ_i p_i f_i(W) across all rounds. Algorithm 1 line 17 and Eq. (15) instead set p_i(t) ∝ 1/(\\hat r_i(t)^2 + ε), where \\hat r_i(t) = ||W_t - W_t^{r_i}||^2 changes every round. Thus the algorithm minimizes a time-varying weighted objective; the descent inequality in Lemma 2 cannot be applied with a different p(t) at each t, and the coefficient (1/2 - 9L^2 η_t^2 K^2 N Σ_i p_i(t)^2) E||∇f(W_t)||^2 no longer telescopes against f(W_t) - f(W_{t+1}) for any fixed f. The paper itself notes that \\hat r_i(t) is 'simplified in the notation', so the proof also assumes the truncation errors are constant even though they are not. No argument shows that the time-average of p_i(t)^2 \\hat r_i(t)^2 decays at the required rate. Consequently, Theorem 2 proves a rate for an idealized fixed-weight, fixed-truncation-error version, not for the published adaptive algorithm.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedHL, a federated learning method for heterogeneous LoRA fine-tuning. The server aggregates client updates on top of the full-rank global model W_t rather than the truncated rank-r_i model, and assigns each client an adaptive aggregation weight p_i(t) that is inversely proportional to the client's current truncation error. The authors claim an O(1/sqrt(T)) convergence rate, derive closed-form 'optimal' aggregation weights in Theorem 3, and report 1--3% improvements over Zero-Padding, FlexLoRA, and FLoRA on three federated LLM benchmarks. The central contribution is the claimed formal convergence guarantee for heterogeneous LoRA, which is said to be lacking in prior work.","tokens_in":31964,"tokens_out":6489,"duration_ms":54536,"significance":"If the convergence guarantee were rigorously established, this would be a useful contribution: the unbiased full-rank aggregation baseline is a simple and plausible fix for truncation bias, and the paper provides a rare theoretical treatment of heterogeneous LoRA in FL. The experimental evaluation is reasonably broad, covering cross-silo and cross-device settings, varying rank heterogeneity, and per-client performance. However, the theoretical proof is the main advertised result, and as detailed below, it does not cover the algorithm as published. The paper's value is therefore substantially weakened unless the proof is repaired or the claims are scaled back.","major_comments":[{"comment":"The convergence proof treats the client weights p_i and truncation errors as constants independent of t, but the published algorithm updates both every round. Lemma 2 (Eq. (29)) is a descent inequality for a fixed objective f(W) = Σ_i p_i f_i(W), and the telescoping sum in Eq. (38) relies on a single fixed f across all rounds. In Algorithm 1, line 17, and Eq. (15), p_i(t) is recomputed from the current truncation error r-hat_i(t) = ||W_t - W_t^{r_i}||^2, which itself changes with W_t. Thus the algorithm minimizes a time-varying weighted objective, and the telescoping argument cannot be applied. The manuscript itself concedes in Section V that 'the definition of r-hat_i, which depends on t, is simplified in the notation for the convergence analysis.' As a result, Theorem 2 proves O(1/sqrt(T)) convergence only for an idealized fixed-weight, fixed-truncation-error version, not for FedHL as implemented. The same issue affects Theorem 3, since the 'optimal' weights are derived for a bound that is itself not a valid bound for the adaptive algorithm.","section":"Section V / Appendix VIII, Theorem 2 and Algorithm 1"},{"comment":"The proof models each client's local update as full-matrix gradient descent on W, writing W_{t+1} = Σ_i p_i (W_t - η_t Σ_τ ∇̃ f_i(W_{t,τ}^{r_i})). In the actual LoRA procedure, however, the client optimizes two low-rank matrices B_t^i and A_t^i with W = W_t^{r_i} + BA, and the effective update is a low-rank increment rather than the unconstrained full-matrix gradient of f_i. The proof never relates the LoRA-constrained update to the full-matrix gradient update it analyzes. This is not a cosmetic mismatch: the space of achievable local updates differs, and the claimed convergence guarantee for heterogeneous LoRA training therefore does not follow from the proof, even if the aggregation weights were fixed.","section":"Appendix VIII, Eq. (17) and local LoRA training"},{"comment":"In bounding T1 = E⟨∇f(V_{t+1}), W_{t+1} - V_{t+1}⟩, the proof drops the term (1/2) E||∇f(V_{t+1})||^2 with the assertion that in standard FedAvg this term decreases over time and asymptotically approaches zero. This is not a quantitative bound, and the term is not negligible in the preceding inequality: it can contribute constants or terms that decay more slowly than O(1/sqrt(T)). Since Lemma 1 feeds directly into Eq. (37) and the final rate, dropping this term without a bound leaves the O(1/sqrt(T)) conclusion unsupported. A rigorous treatment must either bound E||∇f(V_{t+1})||^2 in terms of the quantities already appearing in the descent inequality or subtract it with a controlled coefficient.","section":"Appendix VIII, Lemma 1, Eq. (28)"},{"comment":"The recursion for Δ_τ in Lemma 2 uses the factor 1/(K-1) and 1/(2K-1). These denominators are undefined for K=1. The paper does not state a K≥2 assumption, and its own experiments use K=1 for Fed-CodeAlpaca (Section VI.A). Thus Theorem 2, as proven, does not cover a setting the paper evaluates. The theorem needs an explicit K≥2 condition, or a separate argument for the K=1 case.","section":"Appendix VIII, Eq. (32)-(33), Lemma 2"}],"minor_comments":[{"comment":"The paper states that aggregation weights are smoothed using a softmax function with temperature 1, but Theorem 3 and the convergence analysis are for the unsmoothed formula in Eq. (15). Please clarify whether the experimental algorithm uses exactly Eq. (15) or a softmax modification, and if the latter, how the theoretical guarantee is affected.","section":"Section VI.A, hyperparameters"},{"comment":"The constant ε in Eq. (15) is defined as B/A with A and B depending on problem constants, but in Theorem 3 and Algorithm 1 it is treated as a free parameter. The paper does not specify how ε is chosen in practice, which weakens the claim that the weights in Eq. (15) are the theoretically optimal ones used in the experiments.","section":"Appendix X, Eq. (49)-(50)"},{"comment":"Eq. (5) defines W_{t+1} = Σ_i p_i W_{t+1}^i, while Eq. (6) rewrites this as Σ_i p_i (W_t^{r_i} + ΔW_t^i) and Eq. (7) as Σ_i p_i (W_t + ΔW_t^i). The relationship between the p_i in these equations and the p_i(t) used later is not explained consistently; please align the notation.","section":"Notation, Eq. (5)-(7)"},{"comment":"References [10] and [23] appear to be the same AdaLoRA paper under different titles; please consolidate or disambiguate. Also, reference [43] is listed with a different title than [41] but appears to be the same arXiv ID; please check.","section":"References"}],"recommendation":"reject","confidential_remarks":"The core issue is that Theorem 2, the paper's main theoretical contribution, does not prove convergence for the algorithm as described in Algorithm 1 and Eq. (9): the proof fixes p_i and r-hat_i while the algorithm adapts both per round. This is not a minor repair; it requires either a new analysis of time-varying aggregation weights or a substantive change to the algorithm. If the authors restrict the theorem to a fixed-weight variant, the paper's central claim of dynamic, drift-minimizing aggregation would no longer be supported. I recommend rejection, though the empirical study has value and could form the basis of a revised submission with honest theoretical scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zihao and co.'s FedHL has a genuinely useful algorithmic idea: use the full-rank global model as the aggregation baseline, and weight clients inversely to their truncation error. The experiments, while modest, consistently show 1–3% gains over three hetero-LoRA baselines across two FL settings. That part is credible.\n\nThe problem is the central convergence claim. Theorem 2 proves O(1/√T) for a version of the method with fixed aggregation weights p_i and fixed truncation errors ĥr_i. Algorithm 1 recomputes both every round (Eq. 15, line 17). The proof's descent lemma (Lemma 2) and the telescoping sum require a single time-invariant objective f(W)=Σ_i p_i f_i(W). With p_i(t) varying, the bound's coefficient no longer telescopes, and no argument shows the time-averaged weight-truncation product decays. So the guarantee does not cover the published algorithm. The stress-test note is right.\n\nThere's a second gap. The proof models local LoRA updates as full-matrix gradient descent from a truncated starting point. In reality clients update low-rank factors B_i, A_i; the local step is a projected or reparameterized update, not full gradient descent on the full matrix. The proof's Γτ recursion does not capture that.\n\nA smaller but real problem: Remark 2 calls the O(1/√T) truncation term in Theorem 1 a 'persistent' bias that 'persists without guaranteed reduction.' That term is explicitly 1/√T-scaled, so it vanishes as T grows. The remark misreads the paper's own bound. This weakens the narrative that existing methods are non-convergent; Theorem 1 as written actually gives them an O(1/√T) rate, with a larger constant.\n\nThere are also minor loose ends: the ε in Eq. (15) is defined in Theorem 3 but set as a free hyperparameter in practice, and the softmax smoothing alters the derived weights.\n\nWho is this for? People building hetero-LoRA FL systems will find the aggregation-baseline idea and the weighting scheme useful. The empirical section gives a reasonable sanity check. But readers who rely on the convergence theorem will be misled.\n\nMy recommendation: don't publish as is. The theory needs to be either corrected to cover adaptive weights (or explicitly downgraded to a heuristic), and the local-update modeling needs to match the algorithm. That's a substantial revision, but the core idea has enough merit that I'd send it to referees rather than desk-reject.","headline":"The aggregation-baseline idea is worth keeping; the advertised O(1/√T) guarantee does not cover the adaptive algorithm as written.","tokens_in":32469,"tokens_out":3915,"would_cite":false,"duration_ms":33248,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated LoRA with per-client ranks converges at the same rate as homogeneous LoRA when updates are anchored to the full-rank global model.","keywords":["federated learning","low-rank adaptation","heterogeneous LoRA","truncation bias","convergence analysis","unbiased aggregation","parameter-efficient fine-tuning","large language models"],"falsifier":"Run FedHL on a small convex problem, for example a quadratic whose optimum and gradients are known, with two clients holding different ranks, and compare three variants: the fixed-weight version the theorem analyzes, the published per-round weight recomputation, and a truncated-baseline method such as FLoRA. If the adaptive version clearly violates the $\\mathcal{O}(1/\\sqrt{T})$ decrease of the averaged squared gradient norm, or if the truncated-baseline method matches the same rate, the claim that unbiased aggregation is what restores convergence would be falsified.","tokens_in":31404,"feed_emoji":"⚖️","tokens_out":11077,"duration_ms":83324,"temperature":0.7,"pith_summary":"Federated fine-tuning of large models with LoRA is cheap, but when clients use different ranks the server must truncate the shared global model to each client's rank, and this paper argues that the bias from that truncation is what stops existing methods from having convergence guarantees. The authors propose FedHL, which aggregates each client's low-rank update as a correction on top of the full-rank global model $W_t$ instead of on top of the client's truncated copy $W_t^{r_i}$, so the truncation error never enters the update as a persistent term. They prove that this restores an $\\mathcal{O}(1/\\sqrt{T})$ convergence rate, the same rate as homogeneous LoRA, and that the optimal aggregation weights for the bound are inversely proportional to each client's squared truncation error, $\\hat{r}_i(t) = \\|W_t - W_t^{r_i}\\|_F^2$. Experiments on math, code, and instruction-following datasets with 10 and 100 clients report 1-3% gains over prior heterogeneous-LoRA baselines. If the analysis holds, it gives a simple design rule: keep the full-rank model as the reference point and weight clients by inverse truncation error, and clients are free to choose their own ranks without sacrificing convergence.","feed_headline":"A full-rank anchor makes heterogeneous LoRA provably converge","feed_subtitle":"Clients keep their own ranks while inverse truncation-error weights remove the bias that blocked convergence.","key_machinery":"The load-bearing construction is the unbiased aggregation baseline: the server keeps the full-rank model $W_t$ from the previous round and writes every client's upload as a correction on that same reference point, $W_{t+1}^i - W_t^{r_i}$, where $W_t^{r_i}$ is the rank-$r_i$ SVD truncation of $W_t$ that client $i$ actually trained on. Because all client corrections share one baseline, different ranks can be aggregated by simple weighted addition, with no zero-padding and no stacking tricks, and the truncation inconsistency between clients cancels out of the update. The second mechanism is the weight rule $p_i(t) = \\frac{1/(\\hat{r}_i(t)^2 + \\epsilon)}{\\sum_j 1/(\\hat{r}_j(t)^2 + \\epsilon)}$, derived by minimizing the slowest-decaying term of the convergence bound in Theorem 2; it down-weights exactly the clients whose truncation error is largest. The proof couples a standard FedAvg descent lemma with a per-client bound on the gap between the truncated local trajectory and the ideal full-rank trajectory, a gap that grows like $D_0^K \\hat{r}_i^2$ with $D_0 = 4(1 + L^2\\eta_t^2) \\geq 4$ and $K$ local steps, showing why removing the baseline truncation matters for the rate.","core_discovery":"The paper's central claim is that the poor convergence of earlier heterogeneous-LoRA federated methods is caused by their aggregation rule, not by the rank heterogeneity itself. Existing approaches such as FlexLoRA and FLoRA reconstruct the next global model by adding each client's local increment to that client's truncated baseline, $W_t^{r_i} = B_t^i A_t^i$; Theorem 1 shows that this injects a model-truncation-bias term into the bound that does not shrink as the number of rounds $T$ grows. FedHL replaces the per-client baselines with the stored full-rank model $W_t$, aggregating $W_{t+1} = \\sum_i p_i(t)\\,[W_t + (W_{t+1}^i - W_t^{r_i})]$, which cancels the truncation-bias term. Under standard smoothness and bounded-variance assumptions and with learning rate $\\eta_t = \\frac{1}{LK\\sqrt{T}}$, Theorem 2 bounds the average squared gradient norm by $\\mathcal{O}(1/\\sqrt{T})$, matching the homogeneous-LoRA rate. Theorem 3 shows that the weights minimizing the dominant gradient-drift term are $p_i(t) \\propto 1/(\\hat{r}_i(t)^2 + \\epsilon)$, so clients whose rank-constrained view of the global model is more accurate carry more weight. The paper positions this as the first formal convergence guarantee for heterogeneous LoRA in federated learning.","pith_inferences":["A natural next step the paper leaves implicit is an analysis of the fully adaptive procedure: the published algorithm recomputes the $p_i(t)$ every round from the current truncation errors, while the theorem treats the weights as fixed, so a time-varying-objective or two-timescale argument would close the gap between the guarantee and the implemented rule.","The inverse-truncation-error weighting treats $\\hat{r}_i$ as a proxy for update quality; one could test alternatives that use gradient norms or local validation loss, which may behave differently when a high-rank client holds noisy data.","The same construction, aggregating around the full shared object and weighting by each client's projection error, transfers to other parameter-efficient federated methods such as adapters or prompt tuning, where client-specific projections also create alignment bias.","Because the truncation error is measured in the Frobenius norm, the optimal weights implicitly assume SVD truncation; a different rank-reduction scheme, such as random projection or rank allocated by layer importance, would change both the error profile and the optimal weighting, which the paper does not explore."],"forward_implications":["Clients can pick their own LoRA ranks to match device constraints; the server no longer needs padding or stacking to reconcile them, since aggregation happens on reconstructed full matrices.","The truncation-bias term vanishes from the convergence bound, so the $\\mathcal{O}(1/\\sqrt{T})$ rate is the same as for homogeneous LoRA; the remaining drift term is controlled by the inverse-truncation-error weights.","The weight rule is directly computable on the server from the SVD it already performs, so the fix costs no extra communication and no extra storage beyond the uploaded low-rank factors.","The method carries over to cross-device settings with partial participation (15% and 30% sampling among 100 clients) and remains competitive when ranks are homogeneous, so it does not trade away standard scenarios to fix the heterogeneous one."],"supporting_citations":[{"why":"defines LoRA as the product of two low-rank matrices $B$ and $A$, the object that every client in this paper fine-tunes.","marker":"[9]"},{"why":"supplies the FedAvg update and descent framework on which the paper's convergence lemmas build.","marker":"[3]"},{"why":"is the FlexLoRA baseline that aggregates around the truncated model $W_t^{r_i}$, the rule the paper diagnoses as carrying truncation bias.","marker":"[17]"},{"why":"is the FLoRA baseline whose stacking aggregation and rank-heterogeneity protocol the experiments adopt and improve upon.","marker":"[40]"},{"why":"provides the zero-padding heterogeneous-LoRA baseline and motivates the heterogeneous rank setting.","marker":"[14]"},{"why":"supports the standard-FedAvg descent fact used to drop the $E\\|\\nabla f(V_{t+1})\\|^2$ term in the deviation bound of Lemma 1.","marker":"[53]"}],"fun_headline_variants":["Truncation-free aggregation gives heterogeneous LoRA its first proof","First convergence guarantee for federated heterogeneous LoRA","Full-rank anchor cancels LoRA truncation bias","Inverse-rank weighting removes barrier to LoRA convergence","Heterogeneous LoRA finally converges with unbiased aggregation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The convergence proof assumes the aggregation weights $p_i$ are fixed across rounds, so that the objective $\\sum_i p_i f_i(W)$ being optimized does not change; the algorithm as published recomputes the weights from the current truncation errors each round, and the stated $\\mathcal{O}(1/\\sqrt{T})$ guarantee does not formally cover that adaptive schedule.","fun_headline_variants_meta":{"raw":{"variants":["Truncation-free aggregation gives heterogeneous LoRA its first proof","First convergence guarantee for federated heterogeneous LoRA","Full-rank anchor cancels LoRA truncation bias","Inverse-rank weighting removes barrier to LoRA convergence","Heterogeneous LoRA finally converges with unbiased aggregation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000372,"raw_usage":{"total_tokens":2056,"prompt_tokens":1080,"completion_tokens":976,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":696,"completion_tokens_details":{"reasoning_tokens":899}},"tokens_in":696,"tokens_out":976,"duration_ms":6343,"temperature":1.0,"reasoning_tokens":899,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:31:21.456019+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FedHL on a small convex problem, for example a quadratic whose optimum and gradients are known, with two clients holding different ranks, and compare three variants: the fixed-weight version the theorem analyzes, the published per-round weight recomputation, and a truncated-baseline method such as FLoRA. If the adaptive version clearly violates the $\\mathcal{O}(1/\\sqrt{T})$ decrease of the averaged squared gradient norm, or if the truncated-baseline method matches the same rate, the claim that unbiased aggregation is what restores convergence would be falsified.","supporting_citations":[{"cited_title":"Federated fine-tuning of large language models under heterogeneous tasks and client resources,","cited_arxiv_id":null,"evidence_quote":"is the FlexLoRA baseline that aggregates around the truncated model $W_t^{r_i}$, the rule the paper diagnoses as carrying truncation bias."},{"cited_title":"FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations,","cited_arxiv_id":null,"evidence_quote":"is the FLoRA baseline whose stacking aggregation and rank-heterogeneity protocol the experiments adopt and improve upon."},{"cited_title":"Heterogeneous lora for federated fine-tuning of on-device foundation models,","cited_arxiv_id":null,"evidence_quote":"provides the zero-padding heterogeneous-LoRA baseline and motivates the heterogeneous rank setting."},{"cited_title":"Local sgd converges fast and communicates little,","cited_arxiv_id":null,"evidence_quote":"supports the standard-FedAvg descent fact used to drop the $E\\|\\nabla f(V_{t+1})\\|^2$ term in the deviation bound of Lemma 1."}],"review_version":1}