{"id":"b9c242ac-afd6-4a95-bbad-c9882bb6164f","arxiv_id":"2412.17692","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FedTLU selects the top-scoring blocks by update-magnitude consistency each aggregation round and shows modest perplexity gains for federated fine-tuning of small language models.","lead":"FedTLU is a method for federated learning that updates only the most important layers of a language model each round, based on how consistent the layer changes are. It reports lower prediction error than full, random, or last-layer updates, though the gains are small and some comparisons show no clear advantage.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section IV.C compares lower bounds to infer subset beats full updates; that inference is invalid, so the paper's theory does not establish the central claim.","rationale":"The reader's weakest_assumption identifies the unproven link between the Score and Assumption (b); I agree that link is missing. However, I found a more basic correctness problem: even if Assumption (b) held, the proof in Section IV.C does not establish the conclusion. Comparing lower bounds cannot prove an ordering of actual loss reductions, and the condition eta*L >= 2 is the divergence regime rather than a normal fine-tuning regime. This is not merely a missing justification — it is an invalid inference. The empirical evidence is too weak to compensate: no code, no error bars, only three runs, and one table where Random outperforms FedTLU. As submitted, the core rationale is demonstrably unsound, so I recommend REJECT rather than the reader's CONDITIONAL. A revised version that removes or fixes Section IV.C, provides code and error bars, and shows the effect is robust could be reconsidered.","tokens_in":10233,"tokens_out":7171,"duration_ms":72947,"concrete_test":"Simulate the exact one-step comparison of Section IV.C on a two-block quadratic loss L(w) = 0.5*L*(w1^2 + w2^2) with L = 3, eta = 1 (so eta*L >= 2), and choose the subset S to contain the block with gradient contribution A_S = A - delta for delta = 0.05*A. Compute the exact loss reductions Delta_full and Delta_S after one gradient step. If Delta_full > Delta_S despite the subset lower bound being 'tighter', then the theorem's conclusion is falsified. This directly settles whether Section IV.C's inequality is valid.","verdict_should_be":"REJECT","load_bearing_attack":"The theoretical support for FedTLU rests on Section IV.C, but the derivation is logically invalid. From L-smoothness the authors derive lower bounds Delta_all >= eta*A*(1 - eta*L/2) and Delta_S >= eta*(A - delta)*(1 - eta*L/2), where A = ||grad L||^2. They then treat the subset lower bound being larger as proof that the subset update yields greater loss reduction, concluding eta*L >= 2. But comparing lower bounds cannot order the actual quantities: a larger lower bound on one random variable implies nothing about which variable is larger. In the regime eta*L > 2 both lower bounds are negative, making the comparison vacuous, and eta*L >= 2 is precisely the regime where gradient descent on an L-smooth function is not guaranteed to decrease loss. Thus even granting Assumption (b) — which itself is never linked to the Score in Section III — Section IV.C provides no valid reason that selecting top-S blocks by ||Delta W_i||/(sqrt(n_i)*std(Delta W_i)) improves convergence. The remaining support is empirical, but the experiments have no error bars, only three repetitions, and Table III (Transformer/FedAvg) shows Random (1.764) beating FedTLU (1.768) in the noisy scenario, contradicting the claim of consistent outperformance. The central claim therefore rests on an invalid proof plus under-powered experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedTLU, a server-side targeted layer update strategy for federated fine-tuning of language models. In each aggregation round, the server computes a per-layer score Score(W_i) = ||ΔW_i|| / (√n_i · std(ΔW_i)) from the difference between the pre-round global model and the aggregated client updates, sums these scores within blocks, groups blocks by parameter sequence, and updates only the top S blocks per group while freezing all other parameters. The authors claim this improves convergence and test perplexity compared with full, random, and last-layer updates in non-IID settings, and that it is robust to noisy or malicious clients. They provide a theoretical analysis in Section IV intended to show that subset updates can outperform full updates, and they report three-repetition experiments on a standard Transformer (Penn Treebank) and GPT-2 (UDPOS) with FedAvg and FedProx, covering training from scratch, fine-tuning, and noisy-client scenarios.","tokens_in":10528,"tokens_out":4313,"duration_ms":43330,"significance":"If the central claim were established, FedTLU would be a simple, client-agnostic, server-side mechanism for reducing the impact of noisy updates during federated fine-tuning, while also offering communication savings by updating only part of the model. The score is cheap to compute and requires no extra client-side information, which is an attractive practical property. The paper also contains a concrete experimental comparison across several update strategies and datasets. However, the current significance is limited by two serious problems: the theoretical argument in Section IV is invalid as written, and the empirical evidence is underpowered (three repetitions, no error bars) and even contains a direct counterexample to the 'consistently outperforms random' claim in Table III. The idea may still be viable as an empirical contribution, but the paper needs substantial revision of its theory and claims.","major_comments":[{"comment":"The comparison of lower bounds in Section IV.C is logically invalid. From L-smoothness the authors derive Δ_all ≥ ηA(1 − ηL/2) and Δ_S ≥ η(A − δ)(1 − ηL/2), where A = ||∇L(W(t))||², and then treat the fact that the subset lower bound is larger than the full lower bound as proof that the subset update yields greater loss reduction, concluding ηL ≥ 2. A larger lower bound on one quantity does not imply that the quantity itself is larger. Moreover, for ηL > 2 both lower bounds are negative, making the inequalities vacuous, and ηL ≥ 2 is precisely the regime in which gradient descent on an L-smooth nonconvex function is not guaranteed to decrease the loss. This section does not establish that subset updates improve convergence and should be removed or replaced with a valid argument.","section":"Section IV.C"},{"comment":"The theoretical analysis relies on Assumption (b), which states that the selected subset W_S satisfies ||∇_{W_S} L(W)||² ≥ ||∇L(W)||² − δ. This is essentially the property that FedTLU's score is supposed to provide, but the paper never proves that the Score defined in Section III selects layers whose gradient is aligned with the full gradient within a slack δ. The score is based on the L2 norm of parameter differences and their standard deviation, not on gradient alignment. Consequently, even if Section IV.C were corrected, it would apply to an oracle subset selector and not to FedTLU. The connection between the score and Assumption (b) needs to be established, or the theory should be reframed as an empirical heuristic with no formal convergence guarantee.","section":"Section IV.A, Assumption (b)"},{"comment":"Table III (Transformer, FedAvg, noisy/malicious clients) reports FedTLU global perplexity 1.768 versus Random 1.764, and local perplexity 1.765 versus 1.763. That is, Random outperforms FedTLU in this configuration. This directly contradicts the abstract's claim that FedTLU 'consistently outperforms random and last-layer updates' and the contribution bullet in Section I. While Section V acknowledges the negative relative improvement in the text (e.g., −0.23% globally), the abstract and contribution statements remain unqualified. The claim of consistent improvement should be removed or replaced with a precise statement of where FedTLU helps and where it does not.","section":"Table III and Section V"},{"comment":"All experiments are evaluated only three times and the paper reports only averaged minimum test perplexities, without standard deviations, confidence intervals, or per-seed values. Many reported differences are extremely small (for example, Table I, Transformer, FedAvg, portion 0.75: 2.107 versus 2.108 for FedTLU versus Random), so without variance information these differences cannot be distinguished from noise. The authors should provide error bars or standard deviations for the main comparisons, and ideally a paired statistical test over seeds, before claiming empirical superiority, especially for the from-scratch and fine-tuning experiments.","section":"Section V.A and Tables I–III"}],"minor_comments":[{"comment":"The score formula Score(W_i) = ||ΔW_i|| / (√n_i · std(ΔW_i)) is undefined when std(ΔW_i) = 0, which can occur for a frozen or unselected layer; the paper should specify how this case is handled.","section":"Section III"},{"comment":"The description of block grouping is ambiguous: the text says blocks are grouped 'by the sequence of parameters' and Algorithm 1 says 'Group blocks by the sequence of parameters,' but what this means operationally is not formally defined. Please clarify the grouping criterion and how it relates to equal parameter counts.","section":"Section III and Algorithm 1"},{"comment":"The paper reports 'averaged minimum global and local test perplexities' but does not define whether the minimum is taken over training rounds for each seed and then averaged, or computed on the averaged curve. This should be stated precisely because it affects the interpretation of the results.","section":"Section V.A"},{"comment":"The perplexity curves in Figures 1 and 2 show only single trajectories with no indication of variability across the three repetitions; adding error bars or shaded regions would make the figures more informative.","section":"Figures 1 and 2"},{"comment":"The phrase 'extensive experiments' overstates the scope: the evaluation uses two model architectures and two datasets. Please temper the wording to match the actual experimental setup.","section":"Abstract and Section I"},{"comment":"Several references are incomplete or inconsistently formatted (e.g., [8] gives a proceedings venue but no page numbers, and [3] is cited as an OpenAI blog post without a URL). Please check all references against a consistent style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core idea — a server-side, client-agnostic score for selecting a subset of layers to update in federated fine-tuning — is simple and potentially useful, and the empirical study covers reasonable baselines. However, the theoretical section is not merely informal; the inference in Section IV.C is invalid, and Assumption (b) assumes the very property the score is meant to provide. The experiments are underpowered and contain a direct contradiction to the abstract's 'consistently outperforms' claim. I believe the manuscript can be made acceptable only if the theory is removed or substantially replaced, the claims are qualified to match the actual results, and uncertainty information is added for the empirical comparisons. As it stands, the paper overclaims relative to what is demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: FedTLU is a sensible new server-side layer-selection heuristic, but Section IV.C's proof is logically broken and the experiments don't have the statistical power to support the “consistently outperforms” language.\n\nWhat's actually new: the scoring rule Score(W_i) = ||Delta W_i|| / (sqrt(n_i) * std(Delta W_i)) as a basis for picking blocks to update during FL aggregation. I don't know of prior work doing this on the server side with the aggregated difference; the cited baselines use random or last-layer updates. The algorithm is described clearly enough to reimplement, and the experimental protocol is fair: same number of layers/parameters updated for each strategy, three regimes (from scratch, fine-tuning, noisy clients), two models and datasets, and both FedAvg and FedProx aggregators.\n\nWhere the paper falls down:\n\nSection IV.C is not a proof. The authors derive lower bounds on loss reduction for full and subset updates, Delta_all >= ... and Delta_S >= ..., then compare the bounds to infer which actual reduction is larger. That's invalid: a larger lower bound doesn't imply a larger quantity. To make matters worse, the condition they land on, eta*L >= 2, is precisely the regime where the L-smoothness bound is vacuous and gradient descent has no descent guarantee. The theory section also assumes (Assumption b) that the chosen subset contains the effective gradient directions, but never links that assumption to the Score from Section III. So the theoretical support is doubly disconnected.\n\nThe empirical case is modest. In clean fine-tuning, FedTLU beats random by 0.06–0.97% on global perplexity, which is small. In the noisy Transformer/FedAvg case (Table III), random beats FedTLU by 0.23% global, contradicting the abstract's claim. With only three repetitions and no error bars, we can't rule out that most of these differences are noise. No code or data is provided, so a referee can't check.\n\nTo be clear, I don't think the idea is bad. It's cheap, plausible, and worth testing properly. But the current paper overclaims: the theory doesn't prove anything, and the experiments don't yet meet the bar for the headline.\n\nWho's this for: people working on communication-efficient FL fine-tuning and layer-selection heuristics. They might get ideas, but they should not rely on the convergence argument.\n\nRecommendation: send to peer review, but with the expectation of major revision. The authors should remove or replace Section IV.C, add repetitions with error bars and significance tests, report the full curve variability, release code, and tone down the “consistently outperforms” language. If the empirical effect survives that, it's a useful practical result.","headline":"Plausible new server-side layer-selection heuristic, but Section IV.C's proof is invalid and the empirical gains are too small and under-powered to support the central claim.","tokens_in":11031,"tokens_out":4545,"would_cite":false,"duration_ms":39659,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedTLU claims that a server-side score identifies the most critical blocks and that updating only those blocks improves federated fine-tuning of language models under non-IID data.","keywords":["federated learning","layer-wise update","targeted fine-tuning","non-IID data","language models","server-side scoring","noisy clients","robust aggregation"],"falsifier":"Run federated fine-tuning where the global model is known to have task-relevant layers that move little (low score) and noise-dominated layers that move a lot; if FedTLU consistently selects the noisy layers and underperforms random selection, the score's alignment assumption is false. More directly, compute the ratio of the selected subset's squared gradient norm to the full model's squared gradient norm over rounds: if it is often much smaller than $1-\\delta$ for the claimed $\\delta$, the bound in Section IV does not apply.","tokens_in":10018,"feed_emoji":"🧠","tokens_out":6382,"duration_ms":53378,"temperature":0.7,"pith_summary":"FedTLU proposes that in federated fine-tuning of language models, the server should freeze most layers and update only a small set of blocks chosen by a per-layer score, rather than updating all layers or choosing layers at random. The paper's central claim is that this targeted update improves convergence and test perplexity, particularly when clients have non-IID data or when some clients send noisy or malicious updates. The method is attractive because the score is computed entirely on the server from the aggregated weight changes, so clients need no extra computation or information sharing. If the claim holds, federated fine-tuning becomes more robust near a stationary point, where noisy client updates otherwise dominate.","feed_headline":"Server-side score picks which layers to update in federated fine-tuning","feed_subtitle":"Updating only the top-scored blocks cuts test perplexity by up to 7.86% versus random picks under non-IID data.","key_machinery":"The load-bearing object is the per-layer score $\\text{Score}(W_i) = \\frac{\\|\\Delta W_i\\|}{\\sqrt{n_i}\\,\\mathrm{std}(\\Delta W_i)}$ together with the block-selection rule. The numerator rewards layers whose aggregated weights moved a lot, the standard deviation in the denominator penalizes layers whose parameter updates are scattered, and $\\sqrt{n_i}$ normalizes across layers of different sizes. Blocks with identical parameter counts are grouped and the top $S$ blocks per group are updated, making the score a cheap client-agnostic mask over the model. The theoretical support rests on a smoothness bound and Assumption (b), that the selected subset's gradient retains all but $\\delta$ of the full gradient's squared norm.","core_discovery":"The paper introduces a server-side layer score, $\\text{Score}(W_i) = \\frac{\\|\\Delta W_i\\|}{\\sqrt{n_i}\\,\\mathrm{std}(\\Delta W_i)}$, where $\\Delta W_i$ is the aggregated change in layer $i$'s weights in a round. Large aggregate movement with low parameter-level variance marks a layer as both influential and consistent, and layers are grouped into blocks with equal parameter counts so the top $S$ blocks per group can be compared fairly. The core discovery is that updating only those top-scored blocks after aggregation, leaving the rest frozen, yields lower global and local test perplexity than full updates, random block selection, or last-layer-only updates in experiments on Transformer and GPT-2 models under non-IID client splits. The paper also reports that this selectivity blocks noisy or label-shuffled clients from injecting bad updates, with gains up to 7.86% global and 8.27% local perplexity improvements over random selection in those settings. The theoretical analysis argues that when the loss is $L$-smooth and the learning rate satisfies $\\eta L \\ge 2$, updating a well-aligned subset can reduce loss more than a full update.","pith_inferences":["The score is a pure function of aggregated weight movements, so a natural extension is to use it as a drop-in mask for parameter-efficient fine-tuning methods such as adapters or LoRA in FL, where the server could select which modules to aggregate.","The $\\eta L \\ge 2$ condition suggests the benefit should be most visible in late-stage fine-tuning with relatively large learning rates or poorly conditioned losses; a direct test would vary the learning rate and measure the gap between FedTLU and full updates.","If the score truly captures gradient usefulness, similar rankings should correlate with per-layer Fisher information or gradient-norm contributions in centralized settings; checking that correlation is a concrete way to test the mechanism outside federated learning.","The noisy-client experiments use label shuffling; whether the method filters other poisoning patterns, such as targeted backdoors concentrated in low-scoring layers, is an open question that the paper does not address."],"forward_implications":["FedTLU is compatible with existing aggregation rules such as FedAvg and FedProx; the score is computed after aggregation and changes only which blocks overwrite the global model.","Under non-IID client splits, updating 75%, 50%, or 25% of blocks with FedTLU consistently reaches lower minimum perplexity than updating the same portion at random, in both from-scratch and fine-tuning settings.","With 10% of clients given shuffled labels, FedTLU keeps global perplexity below full and random update strategies on GPT-2/UDPOS, where random selection actually worsens perplexity.","Because the same number of layers is updated as in random or last-layer baselines, the gains are attributed to where updates are placed, not to a smaller update budget.","Near stationary points, the paper's bound implies that a well-chosen subset can dominate the full-model update whenever $\\eta L \\ge 2$."],"supporting_citations":[{"why":"Supplies the FedAvg aggregation rule that FedTLU is applied to in the experiments.","marker":"[7]"},{"why":"Supplies the FedProx aggregation rule and the proximal-term formulation used as a second testbed.","marker":"[8]"},{"why":"Introduces the inverse-coefficient-of-variation gradient scoring idea that FedTLU adapts into its layer score.","marker":"[46]"},{"why":"Defines the Transformer architecture used for the Penn Treebank experiments.","marker":"[1]"},{"why":"Defines the GPT-2 architecture used for the UDPOS experiments.","marker":"[3]"},{"why":"Represents the last-layer-only update strategy that FedTLU is compared against.","marker":"[40]"},{"why":"Provides the convergence analysis of FedAvg on non-IID data that motivates the paper's focus on heterogeneity.","marker":"[45]"}],"fun_headline_variants":["Scoring layers cuts federated fine-tuning perplexity by up to 7.86%","Targeted layer updates beat random picks in federated fine-tuning","Server-side score selects vital layers for federated updates","Federated fine-tuning: update only top-scored layers to beat noise","Selective layer updates improve federated model convergence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on the score selecting blocks whose gradient is nearly as effective at reducing global loss as the full gradient; the paper assumes this alignment rather than proving that its score produces such a subset.","fun_headline_variants_meta":{"raw":{"variants":["Scoring layers cuts federated fine-tuning perplexity by up to 7.86%","Targeted layer updates beat random picks in federated fine-tuning","Server-side score selects vital layers for federated updates","Federated fine-tuning: update only top-scored layers to beat noise","Selective layer updates improve federated model convergence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001213,"raw_usage":{"total_tokens":4987,"prompt_tokens":936,"completion_tokens":4051,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":3961}},"tokens_in":552,"tokens_out":4051,"duration_ms":26189,"temperature":1.0,"reasoning_tokens":3961,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:15:50.371155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run federated fine-tuning where the global model is known to have task-relevant layers that move little (low score) and noise-dominated layers that move a lot; if FedTLU consistently selects the noisy layers and underperforms random selection, the score's alignment assumption is false. More directly, compute the ratio of the selected subset's squared gradient norm to the full model's squared gradient norm over rounds: if it is often much smaller than $1-\\delta$ for the claimed $\\delta$, the bound in Section IV does not apply.","supporting_citations":[{"cited_title":"Communication-efficient learning of deep networks from decentralized data,","cited_arxiv_id":null,"evidence_quote":"Supplies the FedAvg aggregation rule that FedTLU is applied to in the experiments."},{"cited_title":"Zico: Zero-shot nas via inverse coefficient of variation on gradients,","cited_arxiv_id":null,"evidence_quote":"Introduces the inverse-coefficient-of-variation gradient scoring idea that FedTLU adapts into its layer score."},{"cited_title":"Fedbert: When federated learning meets pre-training,","cited_arxiv_id":null,"evidence_quote":"Represents the last-layer-only update strategy that FedTLU is compared against."},{"cited_title":"On the convergence of fedavg on non-iid data,","cited_arxiv_id":null,"evidence_quote":"Provides the convergence analysis of FedAvg on non-IID data that motivates the paper's focus on heterogeneity."}],"review_version":1}