{"id":"4bed8c86-826f-4f01-99e9-578fd33b2799","arxiv_id":"2507.22633","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"H2Tune enables federated fine-tuning across heterogeneous foundation models by sharing sparsified rank-aligned middle matrices with learned layer mappings and alternating shared/private updates.","lead":"H2Tune is a new federated fine-tuning method that lets clients with different language models, tasks, and hardware budgets share knowledge through small aligned adapter matrices. The authors report accuracy improvements up to 15.4% over several baselines, but the experiments are small scale and the headline gain is measured against the average baseline rather than the strongest one.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Resource-heterogeneity claim fails its own accounting: Eq. 3's client-specific sparsity only affects Φ_k R_k, while dense A_k and B_k still scale with global rank r_g, so a max-rank-16 client at r_g=64 trains 4x more parameters.","rationale":"Reader's conditional verdict is well founded. My reading identifies the same weakest assumption: the only mechanism for resource adaptation in Eq. (3) is the sparsity pattern of Φ_k, and it acts only on the shared middle matrix. The private matrices A_k and B_k remain dense with dimensions tied to the uniform global rank r_g, so a client with a small resource budget is not actually spared the cost of a large rank. Table 4's support-max-rank column therefore reports a constraint that the algorithm does not enforce. This is an internal inconsistency, not a disagreement with the field. I also note a second issue flagged explicitly in Appendix B: the convergence theorem is proved for Algorithm 2, a proximal abstraction that omits Φ_k and Ω_k, so the advertised O(1/√T) guarantee is at minimum not shown for the full algorithm. Both concerns are fixable in principle, but as written they block the strongest claim that H2Tune jointly handles resource heterogeneity with a proven convergence rate. Since the empirical protocol also lacks seeds/error bars, conditional acceptance with mandatory revisions is the appropriate verdict; my read does not change the reader's verdict.","tokens_in":22981,"tokens_out":8054,"duration_ms":96646,"concrete_test":"Run the released code for the first configuration of Table 4 (Client 1 max rank 16 with global r_g=64) and log per-client trainable parameter counts and peak GPU memory for A_k, B_k, and Φ_k R_k. Compare Client 1 against a vanilla rank-16 LoRA baseline on the same model. If Client 1's dense A_k and B_k parameter counts are (a+b)·64 rather than (a+b)·16, resource heterogeneity is not actually supported. Stronger: cap each client's memory/FLOPs at the level implied by its stated max rank; if the r_g=64 run cannot fit the rank-16 budget, the headline scenario is infeasible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2/Equation (3) defines each layer update as ΔW_k = (A_k + A_k(Φ_k·R_k))B_k, with A_k∈R^{a×r_g}, B_k∈R^{r_g×b}, and R_k, Φ_k∈R^{r_g×r_g}. The stated resource-adaptation mechanism is the client-specific sparsity ratio β_k applied to Φ_k. However, A_k and B_k are dense matrices whose shapes are indexed by the global rank r_g; the sparsity of Φ_k does not reduce their parameter count or their contribution to forward/backward flops except inside the Φ_k R_k product. Consequently a client whose resource budget is described as 'support max rank 16' in Table 4 must still instantiate dense matrices of width r_g = 64, 128, or 192, i.e., 4x, 8x, or 12x the parameter and activation memory of a rank-16 LoRA client. No amount of sparsifying Φ_k changes this. The claim that H2Tune accommodates resource differences through sparsification is therefore unsupported by the method's own equations, and the resource-heterogeneity branch of the central claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes H2Tune, a federated fine-tuning framework for foundation models that targets hybrid heterogeneity: clients differ in model architecture (layers, hidden dimensions, rank), downstream task, and computational resources. The method decomposes each layer's LoRA update as ΔW = (A + A(Φ R))B, where R is a task-shared middle matrix, A and B are task-specific private matrices, and Φ is a client-specific sparsification matrix. A trainable layer-relation matrix Ω aligns local layer counts to a global layer count for server aggregation. An alternating optimization step updates (R, Φ, Ω) and (A, B) separately to disentangle task-shared from task-specific knowledge. The authors prove a convergence theorem for the alternating scheme and report experiments on MATHInstruct and GLUE across Gemma, Llama, SmolLM, and Yi models, claiming up to 15.4% average accuracy improvement over baselines.","tokens_in":23200,"tokens_out":9166,"duration_ms":98106,"significance":"If the claims were fully supported, H2Tune would be a notable contribution: it is among the first FFT frameworks to jointly address layer, dimension, architecture, task, and resource heterogeneity, and it provides a convergence theorem plus experiments across seven modern LLM families. The paper's strengths include the clearly motivated problem formulation, the breadth of the empirical evaluation, the ablation study showing each component contributes, and the availability of code. However, two load-bearing pillars need scrutiny: the resource-heterogeneity mechanism does not actually shrink the dense per-client matrices, and the convergence guarantee is proved for a simplified proximal abstraction rather than for the deployed algorithm. The empirical headline figure also needs to be read against the fact that it is an average over all baselines, including a weak no-federation lower bound.","major_comments":[{"comment":"The resource-heterogeneity claim is not supported by the method's own equations. Eq. (3) defines ΔW_l_k = (A_l_k + A_l_k(Φ_l_k R_l_k))B_l_k with A_l_k ∈ R^{a×r_g}, R_l_k, Φ_l_k ∈ R^{r_g×r_g}, and B_l_k ∈ R^{r_g×b}. The sparsification ratio β_k applies only to Φ_l_k; A_l_k and B_l_k remain dense matrices whose width is the global rank r_g. Therefore a client declared to 'support max rank 16' in Table 4 must still instantiate dense matrices of width r_g = 64, 128, or 192 when the global rank is set to those values, which is 4x, 8x, and 12x the parameter and activation memory of a rank-16 LoRA client. Sparsifying Φ_l_k does not reduce the size of A_l_k or B_l_k, so the mechanism described in §4.2 does not actually accommodate resource-heterogeneous clients within their stated budgets. The resource-heterogeneity branch of the central claim is therefore not established.","section":"§4.2, Eq. (3), and Table 4"},{"comment":"The convergence guarantee does not cover the deployed algorithm. Theorem 1 is restated in Appendix B as Theorem 9 and proved for Algorithm 2, a proximal abstraction that drops the sparsification matrix Φ_k, the layer-relation matrix Ω_k, the local sparsity constraints, and the layer-alignment mechanism. Algorithm 1, which is the algorithm actually evaluated in Section 6, includes all of these components. The theorem's variables are (R, H) only, so the O(1/√T) claim in the abstract and Section 5 does not govern the method as implemented. In addition, the proof relies on the assumption h(R_t^{τ-1}) ≤ h(R_t^0) for all t, stated in Theorem 1 and used in the derivation, but no justification or proof is provided for this trajectory condition. Finally, the abstract states a rate of O(1/√T), while Theorem 1's Eq. (11) bounds the average squared gradient norm by O(1/T); the relationship between the two rates should be clarified.","section":"Appendix A, Appendix B, and Theorem 1"},{"comment":"The empirical claim of 'consistent improvements across all scenarios' and the headline 'up to 15.4% accuracy improvement' are based on the 'avg. Imp' row, which is the average improvement over all four baselines including the LOCAL no-federation lower bound. This metric is not the same as improvement over the strongest baseline. There are concrete counterexamples: in Scenario 1, heterogeneous GLUE NLI@128, H2Tune scores 86.0 while HetLoRA scores 86.8, so H2Tune is worse than the best baseline; several other cells (e.g., Scenario 1 SPC@64 and multiple Scenario 2 GLUE cells) are ties with HetLoRA. The paper should report per-column comparisons against the best state-of-the-art baseline, and the abstract and Section 6.2 should be reworded to reflect that the 15.4% figure is an average over baselines rather than a gain over the best competing method.","section":"Table 3 and §6.2"}],"minor_comments":[{"comment":"The abstract and Section 5 claim a convergence rate of O(1/√T), while Theorem 1's displayed bound is O(1/T) for the averaged squared gradient norm. Please state which quantity is being measured and make the rate statement consistent.","section":"Abstract and Theorem 1"},{"comment":"The text in Figure 5 refers to maximum ranks 'varying from 64 to 196', but Table 4 and the surrounding discussion use global rank values 64, 128, and 192. Please harmonize these numbers.","section":"Figure 5 and Table 4"},{"comment":"The update line 'update Φ^{t,j}_k R^{t,j}_k, = Φ^{t,j}_k R^{t,j−1}_k − η′∇_R L_share(x_i, y_i)' contains malformed notation and appears to conflate the matrix product Φ R with a single variable. Please rewrite this line and separate the updates for Φ_k and R_k.","section":"Algorithm 1"},{"comment":"The definitions of y'_i and y''_i are not spelled out; in particular, Eq. (7) uses y''_i as 'the prediction based on optimized R_k' but it is unclear how this differs from y'_i and which parameters are frozen in each case.","section":"Eqs. (6) and (7)"},{"comment":"There are typographical errors in the table entries: '41,3' should be '41.3' and '42,9' should be '42.9'.","section":"Table 6"},{"comment":"The caption does not define 'avg. Imp'. The text in Section 6.2 explains that it is an average improvement, but it should be stated explicitly in the caption that this is the average improvement over all baselines, not over the best baseline.","section":"Table 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and relevant problem, and the experimental setup is broad. However, the resource-accounting issue in Eq. (3) and the gap between Algorithm 1 and Algorithm 2 in the convergence analysis are both load-bearing. If the resource mechanism cannot be redesigned so that per-client parameter counts scale with the client's declared budget, I would lean toward rejection; the empirical comparison against the strongest baseline should also be corrected. A major revision that addresses these points squarely would be the appropriate path."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"H2Tune is the first federated fine-tuning paper I've seen that takes on architecture, layer, dimension, task, and resource heterogeneity all at once, and the learned layer-alignment matrix Omega_k is a genuinely new mechanism. But the two headline claims don't survive close reading: the sparsification scheme doesn't reduce a client's real parameter load, and the O(1/sqrt(T)) theorem is proved for a simplified abstraction, not the deployed algorithm.\n\nWhat's new and good: the HHFFT scenario framing is real and under-explored, and the comparison table against HetLoRA, FlexLoRA, HeteroTune, and pFedLoRA is accurate. The trainable Omega_k in R^{L_k x L_g} that maps local layer structures to a global layer index before aggregation is a clean answer to layer-count mismatch, and I don't know of it elsewhere. Uploading only the shared middle matrix while keeping A_k, B_k private is a sensible communication/privacy design. The ablation shows the shared-matrix component carries real signal, especially on GLUE, and the anonymous code is a plus.\n\nNow the soft spots, in order of severity. First, the resource-heterogeneity claim fails its own equations. In Equation (3), A_k in R^{a x r_g} and B_k in R^{r_g x b} are dense and indexed by the global rank; sparsifying Phi_k only thins the Phi_k R_k product. A client described in Table 4 as supporting max rank 16 must still instantiate dense matrices of width r_g = 64, 128, or 192, which is 4x, 8x, or 12x its stated budget. The stress-test note is right, and this is load-bearing: adaptive sparsification based on client resources is advertised as a core component.\n\nSecond, the convergence guarantee. The proof analyzes Algorithm 2, a proximal bi-level abstraction that drops Phi, Omega, and the local sparsity constraints. The appendix is transparent about the rewrite, so this is an over-claim in the abstract rather than deception, but O(1/sqrt(T)) is not established for the method as actually run.\n\nThird, the empirical evaluation is thin: 3 clients, 5 rounds, no seeds or error bars, hyperparameters tuned on test accuracy. And the headline 15.4% improvement is against the average of all baselines, not the strongest one; on the NLI@128 cell where that number appears, HetLoRA actually beats H2Tune by 0.8 points. That's misleading presentation.\n\nOn the disentanglement framing: R is pulled toward the global average by a KL term, so 'shared' is partly true by construction. But the ablation provides independent evidence that the shared matrix transfers useful task-common knowledge, so I don't treat this as damaging circularity.\n\nBottom line: the scenario taxonomy and the Omega mechanism deserve citation, and the paper deserves referee time. The resource accounting needs to be fixed, the theory needs to either cover the deployed algorithm or be honestly scoped, and the experiments need validation splits, seeds, and baseline comparisons that don't hide behind averages.","headline":"A genuinely new hybrid-heterogeneity setup with a clever layer-alignment matrix, but the resource-adaptation claim fails the paper's own equations and the O(1/sqrt(T)) theorem covers only a simplified abstraction.","tokens_in":23866,"tokens_out":6881,"would_cite":true,"duration_ms":68618,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C26"],"pacs":[],"model":"deepseek-v4-flash","headline":"H2Tune is a federated fine-tuning framework that handles model, layer, dimension, task, and resource heterogeneity at once, proving an $O(1/\\sqrt{T})$ convergence rate and up to 15.4% accuracy gains over baselines.","keywords":["federated learning","foundation model fine-tuning","hybrid heterogeneity","LoRA","task knowledge disentanglement","parameter-efficient fine-tuning","bi-level optimization","convergence analysis"],"falsifier":"Measure a client's peak GPU memory and per-step FLOPs under the Table 4 configuration that declares a client with support max rank 16 while the global rank is set to 64. If the footprint tracks the rank-16 budget, the sparsity mechanism delivers the resource savings; if it scales with the global rank 64, the resource-heterogeneity claim fails, since the dense private matrices $A_k$ and $B_k$ are never shrunk by the sparsity mask. A second, independent check is to train with the task-shared matrix $R_k$ removed and test whether a client's accuracy on another client's task collapses, which would indicate whether task-specific knowledge had been leaking into the exchanged matrix.","tokens_in":22673,"feed_emoji":"🧩","tokens_out":21917,"duration_ms":205784,"temperature":0.7,"pith_summary":"Federated fine-tuning of foundation models usually assumes that every client runs the same model, the same layer structure, the same hidden width, and the same kind of task. This paper argues that real deployments instead mix model families, model scales, layer depths, hidden dimensions, task objectives, and hardware budgets, and it proposes H2Tune, a framework meant to make knowledge transfer work under all of those differences at once. The central claim is that three mechanisms — sparsified triple matrix decomposition, relation-guided layer alignment, and alternating task-knowledge disentanglement — let clients share a common task-shared matrix while keeping task-specific and resource-specific structure local. If the claim is right, the long-standing assumption that federated fine-tuning requires matched models is removable, with a provable $O(1/\\sqrt{T})$ convergence rate and consistent accuracy gains, up to 15.4%, across three heterogeneous-model scenarios.","feed_headline":"One shared matrix lets mixed-model clients fine-tune together","feed_subtitle":"Different-size models, differing tasks and ranks: H2Tune exchanges one shared matrix and provably converges at O(1/√T).","key_machinery":"The object that carries the argument is TriLoRA, a sparsified triple matrix decomposition of each layer's update: $\\Delta W = (A_k + A_k \\cdot (\\Phi_k \\cdot R_k)) \\cdot B_k$, with $A_k \\in \\mathbb{R}^{a \\times r_g}$, $B_k \\in \\mathbb{R}^{r_g \\times b}$, and $R_k, \\Phi_k \\in \\mathbb{R}^{r_g \\times r_g}$. $A_k$ and $B_k$ are dense private matrices that carry task-specific knowledge; $R_k$ is the public task-shared matrix that clients upload and the server averages; $\\Phi_k$ is a client-specific sparse matrix whose sparsity ratio $\\beta_k$ tunes the shared part to local resources. Two further mechanisms handle the other heterogeneity dimensions: the layer relation matrix $\\Omega_k \\in \\mathbb{R}^{L_k \\times L_g}$ linearly maps local shared matrices to a global layer count and back, and the alternating optimization (freeze $A_k$, $B_k$ while training $R_k$; then freeze $R_k$ while training $A_k$, $B_k$) is what separates task-shared from task-specific knowledge. The proof recasts the whole scheme as the bi-level problem $\\min_R L(R, H^*(R)) + h(R)$ with $H^*(R) = \\arg\\min_H G(R, H)$, where a strongly convex regularizer makes the inner problem tractable and yields the $O(1/\\sqrt{T})$ rate.","core_discovery":"The central claim, on the paper's own terms, is that hybrid heterogeneous federated fine-tuning — clients that differ at once in model architecture, layer count, hidden dimension, rank budget, and downstream task — has a workable solution called H2Tune. Each client decomposes its layer-wise low-rank update as $\\Delta W = (A_k + A_k \\cdot (\\Phi_k \\cdot R_k)) \\cdot B_k$, where $A_k$ and $B_k$ are private matrices that absorb task-specific knowledge, $R_k$ is a task-shared middle matrix held at one uniform global rank $r_g$ across all clients, and $\\Phi_k$ is a client-specific sparsity mask that adapts the shared part to local resources. A trainable relation matrix $\\Omega_k$ maps each client's shared matrices onto a common global layer count before server averaging and maps the global result back, and an alternating optimization updates $R_k$ while freezing $A_k$ and $B_k$, then updates $A_k$ and $B_k$ while freezing $R_k$, so that only task-shared knowledge enters the exchanged matrix. The paper proves an $O(1/\\sqrt{T})$ convergence rate for this scheme by treating it as a bi-level optimization problem, and reports accuracy gains over the LOCAL, FLLM, HetLoRA, and FLTLA baselines across three scenarios built from seven foundation models, with the largest gain at 15.4%.","pith_inferences":["A natural extension the paper leaves implicit is a resource-aware rule for choosing the global rank and global layer count: the paper's own sweeps (Tables 4 and 6) show accuracy plateauing after rank 128 and layer count 32, so the sweet spot could be selected per federation instead of fixed by hand.","The proof appendix skips the verification of Lemma 2, the inner-problem loss being strongly convex, calling it trivial; the claim is standard given the regularization coefficient, but the convergence theorem depends on a short verification that is not written out.","Because only the task-shared matrix $R_k$ is exchanged while $A_k$ and $B_k$ stay local, the framework's communication pattern is compatible with adding secure aggregation or differential privacy over the uploaded matrices; the paper does not analyze privacy, but nothing in the design rules it out.","The paper does not report a direct measure of task-specific knowledge leaking into the shared matrix; an obvious test is to remove the KL term from the shared-loss objective and check whether cross-client transfer changes, which would show whether that term is what prevents interference."],"forward_implications":["Clients running different foundation model families and scales — for example an 8-billion-parameter model alongside a 1-billion-parameter model — can federate fine-tuning without agreeing on architecture, layer count, hidden width, or task.","Only the task-shared middle matrix is uploaded to the server, so the channel for cross-client knowledge transfer is explicit and the task-specific matrices never leave the client.","The proved $O(1/\\sqrt{T})$ convergence rate means that handling this extra heterogeneity does not force a worse asymptotic convergence order than standard federated fine-tuning.","A client's resource budget is represented by a sparsity ratio rather than by its LoRA rank, so one global rank can serve clients of different capacities, and the hyper-parameter study indicates that the ratio can be tuned per task.","The same framework covers both the homogeneous case (identical models and tasks) and the heterogeneous case, with consistent gains across all three tested scenarios."],"supporting_citations":[{"why":"Supplies the LoRA low-rank update that H2Tune replaces with its three-matrix decomposition.","marker":"[19]"},{"why":"HetLoRA is the rank-heterogeneity baseline that pads ranks for aggregation and that H2Tune must outperform.","marker":"[11]"},{"why":"FlexLoRA is the competing method that distributes global LoRA parameters across client ranks by SVD.","marker":"[5]"},{"why":"HeteroTune is the baseline supporting hidden-dimension heterogeneity while assuming uniform model families.","marker":"[24]"},{"why":"pFedLoRA is the model-heterogeneous personalized LoRA baseline used in the comparison.","marker":"[39]"},{"why":"FedBone separates general and task-specific models, the task-knowledge separation idea H2Tune reworks for LoRA.","marker":"[9]"},{"why":"Provides the bilevel-programming convergence framework whose Lemma 2.2 the proof imports.","marker":"[15]"},{"why":"Supplies the lemma used to bound the tracking error between local and optimal inner-level parameters.","marker":"[23]"},{"why":"Gives the composite-optimization lemmas used to handle the proximal step of the convergence proof.","marker":"[16]"}],"fun_headline_variants":["Shared matrix bridges model and task gaps in federated fine-tuning","H2Tune: One shared matrix for mixed-model federated fine-tuning","Federated fine-tuning with a single shared matrix for heterogeneous clients","H2Tune aligns models and tasks via a shared matrix in federated learning","Mixed models, mixed tasks: one shared matrix to tune them all"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a per-client sparsity mask over the shared middle matrix fully reconciles one uniform global rank with smaller client budgets, even though the dense private matrices $A_k$ and $B_k$ still scale with that global rank — so a client declared to support rank 16 in Table 4 is apparently storing rank-64 matrices when the global rank is 64 — and the paper never measures that client's memory or compute.","fun_headline_variants_meta":{"raw":{"variants":["Shared matrix bridges model and task gaps in federated fine-tuning","H2Tune: One shared matrix for mixed-model federated fine-tuning","Federated fine-tuning with a single shared matrix for heterogeneous clients","H2Tune aligns models and tasks via a shared matrix in federated learning","Mixed models, mixed tasks: one shared matrix to tune them all"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000917,"raw_usage":{"total_tokens":4022,"prompt_tokens":1119,"completion_tokens":2903,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":735,"completion_tokens_details":{"reasoning_tokens":2808}},"tokens_in":735,"tokens_out":2903,"duration_ms":24481,"temperature":1.0,"reasoning_tokens":2808,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:28:04.196703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure a client's peak GPU memory and per-step FLOPs under the Table 4 configuration that declares a client with support max rank 16 while the global rank is set to 64. If the footprint tracks the rank-16 budget, the sparsity mechanism delivers the resource savings; if it scales with the global rank 64, the resource-heterogeneity claim fails, since the dense private matrices $A_k$ and $B_k$ are never shrunk by the sparsity mask. A second, independent check is to train with the task-shared matrix $R_k$ removed and test whether a client's accuracy on another client's task collapses, which would indicate whether task-specific knowledge had been leaking into the exchanged matrix.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HetLoRA is the rank-heterogeneity baseline that pads ranks for aggregation and that H2Tune must outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FlexLoRA is the competing method that distributes global LoRA parameters across client ranks by SVD."},{"cited_title":"HeteroTune: Efficient Federated Learning for Large Heterogeneous Models","cited_arxiv_id":"2411.16796","evidence_quote":"HeteroTune is the baseline supporting hidden-dimension heterogeneity while assuming uniform model families."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FedBone separates general and task-specific models, the task-knowledge separation idea H2Tune reworks for LoRA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the lemma used to bound the tracking error between local and optimal inner-level parameters."},{"cited_title":"Ghadimi, G","cited_arxiv_id":null,"evidence_quote":"Gives the composite-optimization lemmas used to handle the proximal step of the convergence proof."}],"review_version":1}