{"id":"517334a9-54b8-4521-9593-3a9c135e6bd0","arxiv_id":"2506.12213","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fed-HeLLo allocates different LoRA layers to clients of different resource levels using importance scores and geometric patterns, improving federated fine-tuning accuracy over random allocation baselines.","lead":"Fed-HeLLo lets federated learning clients with different memory limits fine-tune different LoRA layers of a shared foundation model, guided by layer-importance scores and fixed geometric allocation patterns. A smart generalist reading this will see a practical recipe for adapting large models on heterogeneous edge devices, with consistent small accuracy gains over random layer allocation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set proxy for FIM scores and post-hoc Bottleneck selection give Fed-HeLLo access to test distribution, so its gains over FedRA may not reflect the allocation strategy alone.","rationale":"The paper's central claim is that Fed-HeLLo, with its HLA strategies, outperforms existing methods under heterogeneous resources. The most load-bearing condition for this claim is that the comparison is fair and that the performance advantage comes from the allocation mechanism rather than from privileged access to the test distribution. The reader identified exactly this weakness: FIM scores on a test-set proxy may not generalize to client data, and the post-hoc selection of the Bottleneck pattern adds further test-set dependence. My reading confirms that the main results in Tables II-VI rely on test-set information in two distinct ways, while the baseline methods do not receive any such information. The paper does provide a partial safeguard in Table X, where a Wikipedia proxy still beats FedRA on DomainNet-121, but the margin shrinks and the experiment covers only one dataset and one resource setting, leaving the main multi-dataset claims exposed. I also note a secondary issue: the novelty claim of being 'first' to design a federated LoRA framework with heterogeneous LoRA allocation is contradicted by FedRA, which the paper itself cites as a baseline; however, this factual overstatement is less central to the method's effectiveness than the evaluation bias. The proposed concrete test would settle whether the test-set proxy is essential: if a non-test proxy preserves the margins, the concern is resolved; if not, the central effectiveness claim would need to be substantially weakened. Since the reader already conditioned the verdict on resolving these issues, I recommend leaving the verdict unchanged.","tokens_in":25021,"tokens_out":7937,"duration_ms":98251,"concrete_test":"Re-run the CIFAR-100 (6:3:1) and LEDGAR (6:3:1) main comparisons with FIM-HLA using a proxy dataset drawn from a held-out subset of the client training data partitioned under the same Dirichlet Non-IID settings, so the proxy matches client distributions rather than the test set, and commit to a single geometric pattern (e.g., Bottleneck) before computing test accuracy. If Fed-HeLLo's margin over FedRA drops below roughly one point or reverses, the test-set proxy is load-bearing and the reported effectiveness is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central effectiveness claim (Tables II-VI) rests on two uses of the test set that baselines do not enjoy. First, FIM-HLA computes layer importance with Eq. (4) on a proxy dataset sampled from the server's test dataset (Sec. IV-A), so the allocation probabilities embody test-distribution information even though the sampled points are excluded from evaluation. Second, the GD-HLA pattern used in the main runs appears to be Bottleneck, selected after Table VIII compares four patterns on the test set; this is post-hoc model selection on the evaluation target. Because FedRA and rank-based baselines receive no such information, the reported margins (e.g., +1.84 on CIFAR-100 and +3.15 on LEDGAR over FedRA) may shrink or vanish when the server only has a proxy unrelated to the test distribution. The paper's Wikipedia-proxy experiment (Table X) partially addresses this on DomainNet-121, but only for one setting and with a smaller margin, so the main results remain vulnerable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Fed-HeLLo proposes a federated fine-tuning framework in which clients train different subsets of LoRA layers according to their resource capabilities, coordinated by server-side heterogeneous LoRA allocation (HLA) strategies. The framework combines two allocation principles: FIM-HLA, which computes Fisher Information Matrix-based layer importance scores on a server-held proxy dataset and converts them into allocation probabilities, and GD-HLA/RGD-HLA, which imposes geometric patterns (Triangle, Inverted Triangle, Bottleneck, Uniform) on the collective distribution of trainable layers. The authors evaluate Fed-HeLLo on five datasets (CIFAR-100, LEDGAR, Natural Instruction, Dolly-15K, DomainNet-121) under IID and Non-IID data settings, reporting accuracy/F1/Rouge-L gains over FedRA and rank-based baselines, along with reduced backward computation, memory, communication, and training time. They also provide ablation studies for the GD-HLA patterns, the FIM-HLA proxy, and cross-silo settings.","tokens_in":25271,"tokens_out":6151,"duration_ms":67405,"significance":"If the reported gains are robust, Fed-HeLLo is a useful contribution to federated fine-tuning under resource heterogeneity, because it demonstrates that importance-guided, rather than purely random, allocation of trainable LoRA layers can improve global model quality while cutting per-client memory and communication cost. The paper covers a wider range of tasks, models, and non-IID settings than many prior works, and it ships code. However, the central claim depends on two practices that give the method access to test-distribution information: the FIM proxy is sampled from the test set (Section IV-A), and the GD-HLA pattern used in the main runs appears to be selected via test accuracy (Section IV-E2). The efficiency claim is also weakened by an incorrect complexity analysis (Section III-C). These issues do not invalidate the framework's potential, but they must be addressed before the reported margins over FedRA can be taken at face value.","major_comments":[{"comment":"FIM-HLA uses a proxy dataset sampled from the server's test set to compute layer importance scores, and these scores determine the allocation probabilities each round. Because the same test distribution is used for evaluation, the allocation strategy is informed by the evaluation target, whereas the baselines (FedRA, HETLoRA, etc.) receive no such information. The Wikipedia-proxy experiment (Table X) is a step in the right direction, but it covers only DomainNet-121 and one client setting, and the margin over FedRA shrinks from +1.83 to +1.37 in that setting. To support the central effectiveness claim, the paper should use proxy data that is not drawn from the test distribution for all five datasets, or explicitly restrict the claim to settings where the server holds test-distribution-like public data.","section":"Section IV-A, Eq. (4)"},{"comment":"The GD-HLA pattern used in the main experiments is not identified in Sections IV-A through IV-C, but Table VIII evaluates four patterns on test accuracy and Bottleneck is the best. If Bottleneck is the pattern used in the main runs, then Fed-HeLLo's comparison with FedRA includes a post-hoc model-selection step on the evaluation set that FedRA does not enjoy. The paper should either state the pattern-selection procedure clearly, perform pattern selection on a validation split that is not used for test evaluation, or report results for all four patterns in the main tables.","section":"Section IV-E2, Table VIII"},{"comment":"The claimed computational complexity reduction is overstated. The paper gives FedAvg complexity O(τ l(d+R)^2 N s) and Fed-HeLLo complexity O(τ c(d+R)^2 N s / l), which implies a c/l reduction in forward+backward compute. This is not correct: forward passes must still go through all l transformer layers, and backpropagation to a trainable layer still requires passing gradients through the intervening frozen layers. What is reduced is the cost of computing and storing parameter gradients and optimizer states for the frozen layers, not the total forward+backward FLOPs. The empirical TFLOPs in Tables II-VI actually show Fed-HeLLo and FedRA have identical backward cost, which is consistent with the number of trainable layers but not with a c/l factor. Please revise the complexity analysis to distinguish parameter-update cost from inference/backprop through the full network.","section":"Section III-C"},{"comment":"No error bars or multiple-seed results are reported. Several margins over FedRA are small (e.g., +0.31 accuracy on CIFAR-100 IID 6:3:1, +0.05 on CIFAR-100 IID 1:1:1, +0.59 Rouge-L on Dolly-15K 6:3:1 average). Since client sampling, local data partitioning, and allocation are stochastic, these differences could be within run-to-run noise. The paper should report mean and standard deviation over at least 3-5 seeds, and ideally a paired significance test, for the comparisons against FedRA and the other baselines.","section":"Tables II-VI"}],"minor_comments":[{"comment":"The contribution statement 'first work to design a federated LoRA-based fine-tuning framework with heterogeneous LoRA allocation' is contradicted by FedRA [45], which is a heterogeneous LoRA layer allocation method (randomly). The novelty should be framed as importance-guided or resource-aware allocation, not heterogeneous allocation per se.","section":"Abstract / Introduction"},{"comment":"The definition of trainable parameters θ is malformed: '{θ(1), · · ·, A(l), B(l)}' should be '{θ(1), ..., θ(l)}' (or include A(j), B(j) consistently).","section":"Section III-A"},{"comment":"In line 14, the aggregation uses δ^t but the local updates are indexed δ^{t+1}_i in lines 12 and the text. Please align the notation.","section":"Algorithm 1"},{"comment":"Typo: 'continures' should be 'continues'.","section":"Section IV-C, DomainNet-121 paragraph"},{"comment":"The table heading has 'CiFAR-100' with inconsistent capitalization; please use 'CIFAR-100'.","section":"Table VIII heading"},{"comment":"The descriptions of 'Proxy Data-1' and 'Proxy Data-2' are confusing: the names suggest the proxy data alone is the method, but the experiments actually use the proxy data for server-side fine-tuning. Please rename or clarify the experimental conditions.","section":"Section IV-E4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid engineering component and covers many datasets/baselines, with code made available. The main risk is that the reported gains partly stem from test-set leakage (FIM proxy) and test-based pattern selection. These are fixable by re-running with a non-test proxy and a validation-based selection rule, and by adding error bars. The complexity analysis should also be corrected. I do not think the core idea is unsound, but the current manuscript overclaims both novelty and efficiency."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Fed-HeLLo is a credible extension of FedRA, and the importance-guided allocation idea is worth engaging with. The main empirical claim is undercut by two test-set leaks, though neither looks fatal.\n\nWhat's new: the FIM-HLA allocation (gradient-norm based layer importance on a server proxy) and the GD-HLA/RGD-HLA geometric pattern priors are genuinely new relative to FedRA's random layer allocation. The co-design — RGD for warm-start, then FIM updates — is sensible. The experiments are broad: five datasets, vision and text, three non-IID splits, two RoC ratios, plus cross-silo. The gains over FedRA are consistent, around +1 to +4 points average, and the ablations are honest: Table IX shows that directly fine-tuning the server model on the proxy hurts, which argues against the crude 'proxy memorization' story; Table X shows that a Wikipedia proxy, unrelated to the test distribution, retains most of the gain on DomainNet-121. That's real evidence.\n\nSoft spots, in order. First, the main results are partly leaky. Section IV-A samples the FIM proxy from the test set — even excluding those points from evaluation, the allocation probabilities carry test-distribution information that FedRA and rank baselines don't get. Table X partially addresses this, but only for one dataset and with a smaller margin. Second, the Bottleneck pattern is chosen after comparing patterns on the test set (Table VIII); that's post-hoc model selection on the evaluation target. Third, no error bars or seeds anywhere, so I can't tell if the +1-4 point margins are noise. Fourth, the complexity analysis in III-C overstates the communication savings: the server still broadcasts the full θt_g to every client, so the download cost is O(lRs), not O(2cRs); the actual savings are on upload only. The local compute formula also has a suspicious /l. None of these are fatal, but they need fixing.\n\nWho it's for: anyone working on on-device FL fine-tuning or PEFT in heterogeneous FL will want to know this paper. It deserves a real review, but the referee should ask for a non-test proxy rerun, error bars, and a corrected complexity section. I'd accept it for review with the expectation of major revision.","headline":"Useful extension of FedRA with genuinely new importance-guided allocation strategies, but the headline gains are weakened by test-set proxy usage and post-hoc pattern selection.","tokens_in":25775,"tokens_out":3874,"would_cite":true,"duration_ms":47350,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Heterogeneous LoRA allocation, chosen by layer importance, lets memory-limited clients fine-tune foundation models efficiently.","keywords":["Federated Learning","Foundation Model Fine-Tuning","Low-Rank Adaptation","LoRA","Resource Heterogeneity","Heterogeneous LoRA Allocation","Fisher Information Matrix","Parameter-Efficient Fine-Tuning"],"falsifier":"Run Fed-HeLLo with FIM scores computed on a proxy dataset deliberately chosen to be unrepresentative of client domains, for example ImageNet images for clients holding medical text, holding everything else fixed, and compare final accuracy against FedRA; the central claim is falsified if Fed-HeLLo's advantage over random allocation disappears or reverses in that setting.","tokens_in":24829,"feed_emoji":"🧩","tokens_out":4631,"duration_ms":106647,"temperature":0.7,"pith_summary":"Federated fine-tuning of foundation models usually assumes every client can train the same LoRA adapters, but real clients have very different memory budgets. Fed-HeLLo instead assigns each client a different subset of LoRA layers, matched to its resources, and decides which layers each client should train. The paper argues that allocation should follow layer importance: Fisher Information scores capture importance that shifts during training, while geometric allocation shapes capture the stable roles of shallow and deep layers. Across five datasets and three data-distribution settings, Fed-HeLLo reports higher accuracy than random layer allocation and rank-based alternatives, with lower compute, memory, and communication costs. The practical stake is that low-memory devices can contribute data to foundation-model fine-tuning without being excluded or forcing everyone to the weakest client.","feed_headline":"Importance-based LoRA allocation wins in federated fine-tuning","feed_subtitle":"Clients train only the LoRA layers that matter, cutting memory and beating random selection on five datasets.","key_machinery":"The load-bearing mechanism is the allocation map m_i in {0,1}^l, which marks which LoRA layers a client trains, together with two importance signals that set the probabilities for sampling these maps. FIM-HLA converts per-layer gradient-norm scores into an allocation probability distribution, so layers with higher Fisher Information are trained by more clients. GD-HLA and RGD-HLA instead impose a global geometric prior, Triangle, Inverted Triangle, Bottleneck, or Uniform, on the collective allocation, stabilizing early rounds before FIM scores become informative. The co-design runs RGD-HLA for a warm-start phase and then refreshes the FIM-based probabilities every T_FIM rounds, so the final allocation blends dynamic and intrinsic layer importance.","core_discovery":"Fed-HeLLo's central claim is that heterogeneous LoRA allocation can be both memory-efficient and more accurate than uniform or random allocation, provided each client's trainable LoRA layers are chosen by layer importance rather than at random. The framework lets every client train only c_i of the l LoRA layers, and the server aggregates updates only over the layers that client actually trained. To choose subsets, the server computes Fisher Information Matrix (FIM) scores from a small proxy dataset and samples allocation maps from a probability distribution weighted by those scores, giving FIM-HLA. Because FIM scores are noisy early in training, the first rounds instead use Randomized Geometrically-Defined HLA (RGD-HLA), which biases the collective allocation toward geometric patterns, Triangle, Inverted Triangle, Bottleneck, or Uniform, reflecting whether shallow, deep, or both groups of layers carry the task. In the reported experiments, Bottleneck is the strongest fixed pattern, and the full Fed-HeLLo pipeline, a warm-start with RGD-HLA followed by periodic FIM-HLA, outperforms random allocation FedRA and rank-based HETLoRA, FlexLoRA, and FLoRA across all five datasets, with the largest margins under Non-IID data, while cutting backward computation, memory, and communication cost.","pith_inferences":["If FIM layer-importance rankings are stable across training rounds, one could refresh allocation probabilities less frequently than every T_FIM rounds or only when the client population changes, reducing server-side gradient computation further.","The reliance on a server-held labeled proxy is the main deployment constraint; a natural extension is to derive importance signals from aggregated client updates instead of server gradients, removing the proxy requirement entirely.","The Bottleneck pattern's dominance suggests a broader design principle for heterogeneous federated PEFT: preserve extreme layers, shallow and deep, while allowing the middle to be shared more sparsely; this may transfer to prompt tuning or adapter-based methods.","Within a single client, layer subsetting and rank adjustment target different memory components, so the framework could be extended to jointly allocate both the subset of LoRA layers and their ranks, rather than keeping rank fixed."],"forward_implications":["Under the tested settings, Fed-HeLLo improves average accuracy, Macro-F1, or Rouge-L over FedRA on all five datasets, with the largest gains in the most extreme Non-IID cases, such as +5.33 on LEDGAR 10/1.0.","Because each client trains only c_i of l LoRA layers, backward computational cost and communication cost scale down roughly with the allocated fraction, while rank-based alternatives reduce only optimizer-state memory and show large communication or convergence penalties.","The ablation results imply that the geometric pattern matters: Bottleneck allocation consistently beats Triangle, Inverted Triangle, and Uniform, especially under Non-IID data, suggesting that both shallow and deep layers should be kept trainable.","The co-design of RGD-HLA and FIM-HLA beats either component alone, indicating that a warm-start phase protects early training while FIM-based allocation improves later rounds.","Fed-HeLLo degrades gracefully when the proxy dataset is drawn from a different domain, as the Wikipedia-proxy experiment shows, but still outperforms FedRA and HETLoRA in that test."],"supporting_citations":[{"why":"Supplies the random layer-allocation baseline (FedRA) that Fed-HeLLo must beat, and the design choice of training partial LoRA layers per client.","marker":"[45]"},{"why":"Defines LoRA, the parameter-efficient low-rank adaptation method whose layers Fed-HeLLo allocates across clients.","marker":"[25]"},{"why":"Provides the Fisher Information Matrix score used in FIM-HLA to rank layer importance from gradient norms.","marker":"[51]"},{"why":"HETLoRA is the rank-based heterogeneous baseline that Fed-HeLLo compares against and outperforms.","marker":"[10]"},{"why":"FlexLoRA is the SVD-based baseline whose aggregation strategy Fed-HeLLo argues hurts convergence when not all LoRA layers are trainable.","marker":"[3]"},{"why":"FLoRA is the stacking-based baseline used to show that rank-only heterogeneity fails under partial-layer training.","marker":"[50]"},{"why":"Supports the memory-saving rationale by showing that freezing layers removes activation storage and that layer importance can guide which layers to train.","marker":"[41]"},{"why":"Provides the model-freezing motivation for reducing activation and optimizer memory during transformer fine-tuning.","marker":"[1]"}],"fun_headline_variants":["Fed-HeLLo: LoRA allocation by importance beats random in federated learning","Heterogeneous LoRA allocation: train only important layers in federated fine-tuning","Importance-guided LoRA layers cut costs and lift accuracy in federated fine-tuning","Federated fine-tuning: allocate LoRA by importance, not randomly","LoRA importance allocation for efficient federated foundation model fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gains depend on the server's small proxy dataset and early-round gradient norms actually revealing which LoRA layers matter to clients whose data can be extremely non-IID; if the proxy does not resemble client data, allocation drifts toward random and the advantage over FedRA shrinks.","fun_headline_variants_meta":{"raw":{"variants":["Fed-HeLLo: LoRA allocation by importance beats random in federated learning","Heterogeneous LoRA allocation: train only important layers in federated fine-tuning","Importance-guided LoRA layers cut costs and lift accuracy in federated fine-tuning","Federated fine-tuning: allocate LoRA by importance, not randomly","LoRA importance allocation for efficient federated foundation model fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000852,"raw_usage":{"total_tokens":3796,"prompt_tokens":1127,"completion_tokens":2669,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":743,"completion_tokens_details":{"reasoning_tokens":2569}},"tokens_in":743,"tokens_out":2669,"duration_ms":22029,"temperature":1.0,"reasoning_tokens":2569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:56:12.440898+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Fed-HeLLo with FIM scores computed on a proxy dataset deliberately chosen to be unrepresentative of client domains, for example ImageNet images for clients holding medical text, holding everything else fixed, and compare final accuracy against FedRA; the central claim is falsified if Fed-HeLLo's advantage over random allocation disappears or reverses in that setting.","supporting_citations":[{"cited_title":"Fedra: A random allocation strategy for federated tuning to unleash the power of heterogeneous clients","cited_arxiv_id":null,"evidence_quote":"Supplies the random layer-allocation baseline (FedRA) that Fed-HeLLo must beat, and the design choice of training partial LoRA layers per client."},{"cited_title":"Application of computerized adaptive testing to educational problems","cited_arxiv_id":null,"evidence_quote":"Provides the Fisher Information Matrix score used in FIM-HLA to rank layer importance from gradient norms."},{"cited_title":"Heterogeneous lora for federated fine-tuning of on-device foundation models","cited_arxiv_id":null,"evidence_quote":"HETLoRA is the rank-based heterogeneous baseline that Fed-HeLLo compares against and outperforms."},{"cited_title":"Slimfit: Memory-efficient fine-tuning of transformer-based models using training dynamics","cited_arxiv_id":null,"evidence_quote":"Provides the model-freezing motivation for reducing activation and optimizer memory during transformer fine-tuning."}],"review_version":1}