{"id":"6b717a5d-b09e-4f60-9a4d-73136fe5526a","arxiv_id":"2412.20004","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Assigning federated fine-tuning devices different numbers of LoRA layers near the output, with ranks increasing toward the output, reaches target accuracy 1.5-2.8x faster and with up to 42.3% less communication than existing LoRA federated methods.","lead":"Federated fine-tuning of large language models on phones and edge devices is slow because devices are weak and uneven. This paper assigns each device only a few trainable LoRA adapters near the model output, with ranks growing toward the output, and reports 1.5-2.8x faster convergence and about 42% less communication on 80 real Jetson devices.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 12's completion-time model treats LoRA rank as compute-free; if per-layer backward time depends on rank, Algorithm 1 solves the wrong objective and the claimed speedup is not explained. A direct per-layer timing test can settle this.","rationale":"The reader's weakest_assumption correctly identifies Eq. (12) as load-bearing. I agree with that choice and sharpen it: the model is not merely additively approximate, it omits rank-dependent compute entirely, while the paper's own resource-consumption model in Eq. (14) is linear in rank. This internal tension is important because the algorithm's depth assignments are supposed to be optimal with respect to completion time, and the claimed speedups are attributed to that optimization. However, I do not think this warrants rejection. The empirical comparison on a real 80-device WiFi testbed, the ablations showing depth and rank distribution each contribute, and the consistency with prior work on deep-layer importance all give the headline claim independent plausibility. The concern is that the quantitative speedup numbers are not yet tied to the stated model, and the algorithm as written cannot be exactly reproduced from the preprint. A focused per-layer timing experiment on the same Jetson hardware would either validate Eq. (12) or show that the optimization's objective is miscalibrated, and would therefore directly settle whether the central claims are supported. The reader's CONDITIONAL verdict is appropriate; my analysis reinforces it rather than moving it to a different category.","tokens_in":20939,"tokens_out":11013,"duration_ms":127615,"concrete_test":"On a Jetson TX2 and a Jetson NX, instrument one transformer layer of RoBERTa-base with LoRA ranks r in {1, 8, 32, 128} using the same batch size 4 and sequence length 512 as Section 6.1, and measure per-batch forward+backward time for that layer holding everything else fixed. If per-layer backprop time increases by more than about 10% from r=8 to r=128, Eq. (12)'s rank-independent compute assumption is falsified on the actual testbed hardware. Then re-solve Algorithm 1 with the measured per-layer time curves and compare the selected depths and ranks against those implied by the paper's results; a material change in configuration would show the reported speedup is not a consequence of the stated optimization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central efficiency claim rests on the joint depth/rank optimization, but the only quantitative model supporting it is Eq. (12): t_i^h = hat-t_i + k_i^h * mu_i^h + sum_{l in L_i^h} r_{i,l} * beta_i^h. In this model the LoRA-rank contribution to compute time is exactly zero: per-layer backprop time mu_i^h is constant regardless of that layer's rank. This is not a harmless approximation. Section 2.4 shows performance and resource sensitivity to ranks up to 128, and Eq. (14) counts rank-dependent compute resource consumption c per unit rank. If rank changes per-layer backward cost through extra LoRA matmuls, memory pressure from stored activations, or kernel efficiency, then the greedy depth adjustment in Algorithm 1 (line 5) is calibrating depth with a time model that is wrong in exactly the dimension that differentiates LEGEND from FedLoRA. The paper provides no per-layer timing measurements to validate the model, so the 1.5-2.8x speedup could reflect an unmodeled effect rather than the stated objective. Algorithm 1 is also under-specified in ways that make this hard to check: the arithmetic rank sequence R has no defined starting value r0, the budget psi is not connected to lambda, and the 'adjust LoRA depth' procedure in line 5 is not described.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LEGEND, a federated fine-tuning framework that assigns each device a LoRA configuration consisting of a depth (number of adapted layers near the output) and a gradually increasing rank distribution. It includes a greedy algorithm (LCD) to determine these configurations from estimated per-device compute and communication capabilities, plus adaptive layer-wise aggregation to handle heterogeneous LoRA depths. The method is evaluated on a physical testbed with 80 Jetson devices connected over WiFi, using RoBERTa, DeBERTa, and Llama2 on GLUE, MMLU, and GSM-8K tasks. The headline claims are a 1.5-2.8x speedup and about 42.3% communication savings when reaching a target accuracy, compared with FedLoRA, HetLoRA, and FedAdapter.","tokens_in":21180,"tokens_out":4739,"duration_ms":45434,"significance":"If the claims hold, LEGEND would be a practically useful contribution to federated fine-tuning on heterogeneous edge devices, where both resource constraints and system heterogeneity are real deployment obstacles. The physical testbed with 80 commercial devices is a genuine strength and goes beyond the simulation-only evaluations common in this area. The paper also provides an empirical observation that deep-position LoRA layers with larger ranks are more effective, which is consistent with results from other recent studies. The main weakness is that the analytical model underpinning the configuration algorithm is not validated and appears inconsistent with the paper's own resource-accounting equations, so the central speedup claim is not yet fully explained.","major_comments":[{"comment":"The completion-time model in Eq. (12), t_i^h = \\hat{t}_i + k_i^h * \\mu_i^h + \\sum_{l \\in L_i^h} r_{i,l} * \\beta_i^h, treats the per-layer backpropagation time \\mu_i^h as independent of LoRA rank. However, Eq. (14) charges a rank-dependent computing resource consumption \\sum r_{i,l} * c, and Section 2.4 explicitly demonstrates performance and resource sensitivity to ranks up to 128. If rank affects per-layer backward time through extra LoRA matmuls, memory pressure, or kernel efficiency, then Algorithm 1's greedy depth adjustment optimizes the wrong objective. The paper reports no per-layer timing measurements that would validate the rank-independence assumption, so the speedup claim is not yet supported by the stated model.","section":"Section 4.4, Eqs. (12) and (14)"},{"comment":"The LoRA configuration algorithm is under-specified. Line 4 defines an arithmetic rank sequence with r_l = r_{l-1} + \\lambda but does not specify the starting value r_0, and the total rank budget \\psi is never used to bound the sequence. Line 5 says 'adjust LoRA depth' but provides no procedure, making it unclear how the constraints in Eqs. (14) and (15) are enforced, especially because changing depth without changing ranks may not be sufficient to satisfy a total-rank budget. This makes the algorithm non-reproducible and the claimed optimality unverifiable.","section":"Algorithm 1, Section 4.4"},{"comment":"The evaluation sets the target accuracy as 'the minimum accuracy achieved by the four methods.' This biases the completion-time comparison in favor of a method that saturates at the minimum accuracy, and it obscures differences in final performance. Additionally, no error bars or repeated runs are reported, even though device modes are randomized every 20 rounds and WiFi bandwidth fluctuates dynamically. The headline speedup and communication-savings numbers should be accompanied by standard deviations over at least three independent runs and by results for multiple target accuracies.","section":"Section 6.1, 'Metrics'"}],"minor_comments":[{"comment":"There is a typo: 'for arbitary coupled LoRA matrices' should be 'for arbitrary coupled LoRA matrices'.","section":"Section 4.1"},{"comment":"The text says LEGEND takes 1,479s to achieve 85% accuracy on SST-2, while Figure 8(a) caption says 'Time to reach 95% accuracy'; please reconcile the target accuracy values.","section":"Section 6.2 and Figure 8"},{"comment":"The text says 'by Figure 12(c), the average waiting time of LEGEND is 352s for SST-2', but Figure 12(c) corresponds to QQP; the dataset label appears to be incorrect.","section":"Section 6.2"},{"comment":"The expression for the depth gap uses t^h and t_{i,min} without defining t_{i,min}; please clarify the notation and the intended formula.","section":"Algorithm 1, line 2"},{"comment":"The motivating experiments in Section 2 use the same tasks (SST-2, QNLI) and model family (RoBERTa) as the main evaluation in Section 6, so the inferred 'coupled relationship' between depth and rank is validated on the same task distribution from which it was derived; an independent evaluation on held-out tasks would strengthen the claim.","section":"Sections 2 and 6"},{"comment":"The paper states that this is 'the first study of federated learning based NLP in a real wireless testbed with heterogeneous devices'; please support this strong claim with a specific comparison to prior testbed studies or soften the wording.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The physical testbed is a valuable asset and the paper addresses a timely practical problem. However, the central efficiency claim rests on a completion-time model that is internally inconsistent with the paper's own resource-accounting equation, and the configuration algorithm is not specified enough to be reproduced. These issues are fixable with additional measurements and a more detailed algorithm description, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuine systems contribution with a physical 80-Jetson WiFi testbed, which is rare. The new bit is coupling LoRA depth (a contiguous suffix of layers near the output) with an increasing rank schedule, allocated per-device under capacity constraints. I don't know of prior work that assembles exactly this system, and the testbed with three device classes and dynamic bandwidth is a real strength. The paper credits FedLoRA and HetLoRA properly and extends them.\n\nWhat it does well: the motivation experiments are clear and consistent with prior work on deep-layer importance. The implementation is described concretely (FedPETuning, Docker Swarm, MPI), and the ablation shows both depth and rank distribution matter. Communication traffic and waiting-time reductions are reported, not just accuracy.\n\nSoft spots, in proportion:\n\n1. Equation (12) is load-bearing and assumes per-layer backprop time mu_i^h is independent of rank; only upload time scales linearly with rank. If rank changes backward time through extra matmuls, activation memory, or kernel efficiency, the greedy depth adjustment in Algorithm 1 is calibrating the wrong objective. The paper offers no per-layer timing measurements to validate this model. This is the weakest link. It is not fatal—the qualitative findings could survive—but the specific 1.5-2.8x numbers are not fully explained without that assumption.\n\n2. Algorithm 1 is under-specified: r0 (the starting rank) is never defined, lambda is set to 1 by default but the budget psi is not tied to lambda or r0, and the \"adjust LoRA depth\" step in line 5 is not described. A reader cannot reproduce the configuration from the text.\n\n3. The evaluation has no error bars or repeated runs, despite random mode changes every 20 rounds and fluctuating bandwidth. Target accuracy is set to the minimum across methods, which can advantage a method that saturates early. These are reporting issues, not necessarily signs that the numbers are wrong.\n\n4. Minor: the \"coupled relationship\" is an empirical observation plus a heuristic, not a derivation. The paper does not overclaim as badly as many in this space, but that phrase oversells it slightly.\n\nCitation pattern looks reasonable. Self-citations are to the authors' own prior FL work, which is relevant. No code or data release, which for a systems paper weakens reproducibility.\n\nWho it's for: researchers working on parameter-efficient federated fine-tuning and on-device LLM tuning. It deserves a serious referee. I'd send it out, with requests for a timing-model validation experiment and full algorithm details. Conditional accept territory.","headline":"Solid systems paper with a real 80-device testbed, but the headline speedups rest on an unvalidated time model that treats LoRA rank as compute-free.","tokens_in":21785,"tokens_out":2008,"would_cite":true,"duration_ms":20191,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that jointly assigning each device a LoRA depth (number of adapted layers near the output) and an increasing rank distribution, matched to device compute and upload capacity, makes federated fine-tuning of language…","keywords":["federated fine-tuning","LoRA","low-rank adaptation","system heterogeneity","parameter-efficient fine-tuning","LoRA depth","rank distribution","communication efficiency"],"falsifier":"Measure, on a single edge device, the per-batch backward-pass time and peak memory for LoRA ranks 1, 2, 4, 8, 16, 32, 64, and 128 at a fixed depth, and compare the measured completion time of several depth/rank configurations against the prediction of Eq. (12). If doubling the rank more than doubles the per-layer backward time, or if the sum of per-layer times deviates from the measured total by more than a few percent, then the linear model that the LCD algorithm optimizes is false and the claimed speedup would not transfer to that device.","tokens_in":20680,"feed_emoji":"⚡","tokens_out":6671,"duration_ms":64070,"temperature":0.7,"pith_summary":"This paper claims that federated fine-tuning of large language models on heterogeneous devices can be made much faster by letting each device adapt a different number of LoRA layers, always taken from the output side of the model, and giving deeper layers progressively higher LoRA ranks. The proposed system, LEGEND, estimates each device's compute and upload speed, then assigns a LoRA depth and an increasing rank distribution that fit within that device's memory and communication budget while keeping the slowest device's completion time in check. On a real testbed of 80 commercial edge devices, LEGEND reaches the same target accuracy 1.5–2.8 times faster and uses about 42.3% less communication traffic than existing federated fine-tuning methods such as FedLoRA, HetLoRA, and FedAdapter. The paper's key claim is that depth and rank distribution are coupled, so they must be chosen jointly rather than separately.","feed_headline":"Adapting fewer, deeper LoRA layers cuts federated tuning time 2.8x","feed_subtitle":"Giving each device its own LoRA depth and increasing rank distribution saves ~42% communication at target accuracy.","key_machinery":"The load-bearing objects are LoRA depth—the number of consecutive transformer layers counted backward from the output that get trainable low-rank bypass matrices—and rank distribution, the sequence of ranks assigned to those layers. LEGEND couples them through an additive per-round time model $t_i^h = \\hat{t}_i + k_i^h \\mu_i^h + \\sum_{l \\in L_i^h} r_{i,l} \\beta_i^h$, where $\\mu_i^h$ is the measured backward-pass time for one transformer layer and $\\beta_i^h$ the upload time per unit rank, and a greedy algorithm (LCD) assigns depth $k_i^h$ from the time gap to the slowest device while drawing ranks from a global arithmetic sequence $r_l = r_{l-1} + \\lambda$ that increases toward the output. The same rank per layer across devices keeps aggregation well-defined even though depths differ; the parameter server averages each layer only over the devices that actually trained it.","core_discovery":"The paper's central claim, on its own terms, is that existing federated LoRA methods waste both compute and communication by adapting all transformer layers with uniform ranks, and that a configuration where only the output-side layers are adapted, with ranks growing toward the output, reaches the same target accuracy much faster. LEGEND realizes this by assigning each device a LoRA depth proportional to its speed, drawing ranks from a global arithmetic sequence that increases toward the output, and aggregating each layer only over the devices that actually trained it. The paper reports that this joint configuration yields a 1.5–2.8x wall-clock speedup and about 42.3% communication-traffic reduction at the target accuracy relative to FedLoRA, HetLoRA, and FedAdapter, in experiments spanning three model sizes and three task categories on a physical WiFi testbed.","pith_inferences":["A testable extension the paper does not run is to replace the linear time model with measured per-layer cost curves: if memory bandwidth makes high-rank layers slower per unit rank, a convex cost model would change which depth/rank trade-off is optimal.","The depth-first insight may transfer to larger models with more layers, but the paper does not sweep model depth; one could check whether the optimal adapted fraction shrinks as models grow, since the same target accuracy may be reachable with a smaller output-side block.","Layer-wise aggregation over heterogeneous depths implicitly weights shallow layers by the number of capable devices that trained them; tracking per-layer participation and possibly reweighting by staleness would be a natural follow-up."],"forward_implications":["If LEGEND is right, a deployment can give weak devices a small LoRA depth and strong devices a large one, cutting per-round waiting time without changing the aggregated model's target accuracy.","Because only output-side layers are trained and uploaded, total backpropagation and communication traffic shrink roughly with the fraction of layers skipped, which is where the 1.5–2.8x speedup and about 42.3% traffic saving come from.","Consistent per-layer ranks across devices make layer-wise averaging well-defined even when depths differ, so LEGEND avoids the rank-mismatch aggregation failure that the paper attributes to HetLoRA.","The ablation results show both factors matter: removing the adaptive depth increases completion time substantially, while removing the increasing rank distribution degrades final accuracy, so the paper's claim is that depth and rank distribution must be optimized together."],"supporting_citations":[{"why":"Supplies the low-rank adaptation method that LEGEND configures and the baseline notation for LoRA layers.","marker":"[16]"},{"why":"Defines FedLoRA, the uniform-depth baseline whose efficiency LEGEND improves.","marker":"[20]"},{"why":"Defines HetLoRA, the per-device rank baseline that LEGEND compares against and whose rank-mismatch aggregation problem it avoids.","marker":"[27]"},{"why":"Defines FedAdapter, the state-of-the-art adapter-based FedFT baseline used for speedup and traffic comparisons.","marker":"[10]"},{"why":"Establishes the federated fine-tuning formulation and the non-i.i.d. data partitioning used in the experiments.","marker":"[7]"},{"why":"Provides theoretical support that partial, output-side LoRA layer fine-tuning converges, grounding the depth heuristic.","marker":"[35]"},{"why":"Supports the claim that allocating larger ranks to task-relevant deeper layers is beneficial.","marker":"[26]"}],"fun_headline_variants":["Output-side LoRA layers trim federated fine-tuning time 2.8x","Per-device LoRA depth cuts federated tuning time 2.8x, traffic 42%","Rank growth toward output slashes federated tuning time and traffic","LEGEND: fewer LoRA layers, faster federated fine-tuning","Slimmer LoRA configs yield 2.8x faster federated fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole allocation rests on the assumption that a device's per-round time is an additive linear sum of a fixed forward-pass time, a per-layer backpropagation time that does not depend on rank, and a per-rank upload time; if memory pressure or kernel efficiency makes LoRA rank change compute time nonlinearly, the greedy depth/rank assignments stop minimizing the completion time they are built on.","fun_headline_variants_meta":{"raw":{"variants":["Output-side LoRA layers trim federated fine-tuning time 2.8x","Per-device LoRA depth cuts federated tuning time 2.8x, traffic 42%","Rank growth toward output slashes federated tuning time and traffic","LEGEND: fewer LoRA layers, faster federated fine-tuning","Slimmer LoRA configs yield 2.8x faster federated fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1475,"prompt_tokens":943,"completion_tokens":532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":425}},"tokens_in":559,"tokens_out":532,"duration_ms":5058,"temperature":1.0,"reasoning_tokens":425,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:40:50.475057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, on a single edge device, the per-batch backward-pass time and peak memory for LoRA ranks 1, 2, 4, 8, 16, 32, 64, and 128 at a fixed depth, and compare the measured completion time of several depth/rank configurations against the prediction of Eq. (12). If doubling the rank more than doubles the per-layer backward time, or if the sum of per-layer times deviates from the measured total by more than a few percent, then the linear model that the LCD algorithm optimizes is false and the claimed speedup would not transfer to that device.","supporting_citations":[{"cited_title":"Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models","cited_arxiv_id":null,"evidence_quote":"Defines FedLoRA, the uniform-depth baseline whose efficiency LEGEND improves."},{"cited_title":"Heterogeneous lora for fed- erated fine-tuning of on-device foundation models","cited_arxiv_id":null,"evidence_quote":"Defines HetLoRA, the per-device rank baseline that LEGEND compares against and whose rank-mismatch aggregation problem it avoids."},{"cited_title":"Flexora: Flexible low rank adap- tation for large language models","cited_arxiv_id":null,"evidence_quote":"Provides theoretical support that partial, output-side LoRA layer fine-tuning converges, grounding the depth heuristic."},{"cited_title":"Adaptive budget allocation for parameter-efficient fine- tuning","cited_arxiv_id":null,"evidence_quote":"Supports the claim that allocating larger ranks to task-relevant deeper layers is beneficial."}],"review_version":1}