{"id":"0bc91780-17e6-417f-89e0-782d3cfbc58e","arxiv_id":"2607.20914","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A federated fine-tuning method combining shared multi-head low-rank bases, singular-value shrinkage, and layer-wise budget allocation to improve accuracy under non-IID client data.","lead":"TRISHUL is a new method for fine-tuning large AI models across many devices without sharing raw data. It stabilizes training when different devices see very different data by trimming noisy directions in each device's update before averaging.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline empirical claim rests on test-set-tuned regularization with no validation split and no error bars; the reported non-IID gains are not yet distinguishable from a selection artifact.","rationale":"I focused on the strongest claim, which is empirical ('best across all datasets and parameter budgets, with gains growing under heterogeneity'), and asked what would have to be true for it to hold. The tables would have to be both fair and statistically stable. They are not demonstrably either: lambda is selected on the same test benchmarks without an explicit validation split, and no error bars are reported despite three-seed averages. This makes the central comparison vulnerable to a selection artifact, and the sensitivity analysis in Section IV-C2 reinforces rather than resolves that worry because it uses the same test numbers. I also considered the reader's weakest assumption about Eq. (8): omitting the trainable scalar s_i does break the claimed exact equivalence between core shrinkage and full-update shrinkage, and Eq. (15) is only as tight as the unreported s_max. That is a real theoretical gap, but it affects the mechanism narrative, not the table values; even a fully correct theory would not validate the empirical headline without a clean evaluation protocol. The proposed concrete test would settle the empirical question directly: if validation-based lambda selection and confidence intervals preserve the reported gaps, the claim is credible; if not, the headline is inflated. Because this concern aligns with the reader's reasons for CONDITIONAL rather than ACCEPT, and because the issue is fixable rather than fatal, the verdict should remain UNCHANGED at CONDITIONAL.","tokens_in":23653,"tokens_out":14439,"duration_ms":139320,"concrete_test":"Release the implementation and data splits, then re-run the full comparison under a strict protocol: reserve a stratified validation split before any tuning, select lambda and learning rates on validation only, and evaluate once on the original test sets. Report means with 95% confidence intervals over at least 10 seeds for TRISHUL, RAVAN, and SCAFFOLD+LoRA on CIFAR-100, 20 Newsgroups, and GLUE under non-IID alpha = 0.3. If TRISHUL's non-IID advantage over RAVAN remains around 3% on CIFAR-100 and 2.4% on 20 Newsgroups with non-overlapping intervals, the headline claim survives; if the gaps shrink below about 1% or overlap, the reported advantage is consistent with test-set selection or noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is an empirical universality claim: TRISHUL is best across all datasets and parameter budgets, with gains growing under heterogeneity. The evidence for that claim is the benchmark tables, so the evaluation protocol is load-bearing. Section IV-A3 states that lambda is selected by grid search over {0.001, 0.01, 0.1, 1.0} on the same benchmark tasks and that lambda = 0.01 is best; no held-out validation split is described anywhere. The sensitivity study in Section IV-C2 and Figure 10 then uses the same test numbers to justify that choice, which makes the selection circular. All tables report only three-seed averages with no standard deviations or confidence intervals, although only three of twenty clients participate per round and local updates are stochastic. Under such conditions, a 0.7% IID gap or even a 3.2% non-IID gap could be within run-to-run variance, and selecting the regularization strength by looking at test accuracy can inflate every table in the same direction. The monotonic trend with alpha in Table VII is suggestive but is generated under the same test-set-selected lambda. If the gains vanish when lambda is chosen on validation data, the central claim would not stand as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"TRISHUL is a federated parameter-efficient fine-tuning (PEFT) framework built on three mechanisms: (i) frozen, shared, orthonormal multi-head low-rank bases that make server-side aggregation of r×r core updates algebraically exact in the ambient parameter space; (ii) client-side nuclear-norm proximal shrinkage (singular value thresholding) applied to the small core matrices before upload; and (iii) a concave water-filling rule that allocates a fixed total head budget across layers using pretrained layer Frobenius norms as capacity scores. The paper proves the aggregation identity, derives a variance bound based on post-threshold rank, solves the water-filling program, and states a convergence theorem for the composite objective. Empirically, it compares TRISHUL against FedIT, FedEx-LoRA, FFA-LoRA, Fed-SB, SCAFFOLD+LoRA, and RAVAN on CIFAR-100, SVHN, 20 Newsgroups, MRQA, and GLUE with LLaMA3.2-1B, reporting consistent gains that grow with non-IID heterogeneity, together with ablations of each prong and direct spectral diagnostics.","tokens_in":23850,"tokens_out":11706,"duration_ms":103971,"significance":"If the empirical claims hold, TRISHUL is a useful contribution to federated PEFT: it offers a communication-neutral way to stabilize heterogeneous client updates, makes a specific spectral-control mechanism operational, and provides diagnostics that go beyond accuracy alone. The controlled experimental design—parameter-matched budgets, all baselines implemented in the same codebase, ablations of each individual prong, a 1B-parameter experiment, and a careful complexity analysis—is a genuine strength. The significance is currently tempered by the evaluation protocol, in particular the test-set-based selection of the regularization coefficient and the absence of error bars, and by several loose ends in the theoretical claims. With those issues fixed, the paper would be a solid contribution to the federated PEFT literature.","major_comments":[{"comment":"The regularization coefficient λ is selected by grid search over {0.001, 0.01, 0.1, 1.0} on the same benchmark test sets that are later used to report results, and the sensitivity study in Table VI and Figure 10 uses the same test numbers to justify the choice λ=0.01. No validation split is described anywhere in the evaluation. Because every main table uses this test-selected λ, the reported gains (e.g., 0.7–0.8% IID and 3.2–3.5% non-IID on CIFAR-100, up to 4.7% at α=0.05 in Table VII) are at risk of being a selection artifact. Additionally, with only three seeds and three participating clients per round, the absence of standard deviations or confidence intervals makes it impossible to tell whether the smaller IID differences are above run-to-run noise. I request a validation-based (or nested) selection procedure and reported error bars for the headline tables; without this, the claim in Section IV-B that TRISHUL is best across all datasets and budgets is not yet supported.","section":"Section IV-A3 and Section IV-C2 (Tables II–IV, VI, VII)"},{"comment":"The claim that nuclear-norm shrinkage of the core matrix is exactly equivalent to shrinkage of the full update omits the trainable scalar s_i. The actual update is s_i B_i H_i A_i, whose nuclear norm is |s_i| ||H_i||_*, so the penalty in Eq. (9) corresponds to an effective shrinkage coefficient λ/|s_i| on the full update, which varies across clients and across rounds unless |s_i| is constrained to be close to 1. The variance bound in Eq. (15) requires |s_i| ≤ s_max, but s_max is never reported and the scalars are reset to 1 on the server after each round (Algorithm 1, line 22), so the shrinkage strength of the uploaded product s_i H_i is not controlled as stated. The paper should either penalize λ |s_i| ||H_i||_*, remove the trainable scalar from the penalty analysis, or report s_max and justify that |s_i| remains near 1 under the training dynamics.","section":"Section III-C, Eq. (8)–(9); Algorithm 1, line 15"},{"comment":"The convergence theorem does not cover the algorithm as written. After aggregation, the server resets all scalars to 1 (Algorithm 1, line 22); this reset is not a proximal step with respect to Ψ in Eq. (35), and the descent inequality in Eq. (40) does not account for the change in θ it induces. In addition, Assumption A.8 requires client sampling probabilities consistent with aggregation weights, but Section IV-A2 states that the server uniformly samples three clients while Eq. (21) aggregates by data-size weights; for unequal local sample sizes these are incompatible. The theorem should be revised to model the scalar reset explicitly (or the reset should be removed from the algorithm) and to state the exact sampling scheme used in the experiments.","section":"Appendix E, Theorem A.5 and Eq. (40)"}],"minor_comments":[{"comment":"The optimality statement for the water-filling allocation is with respect to the paper's own logarithmic utility; the real supporting evidence is the empirical comparison in Table VIII. I recommend framing this as a design choice validated empirically rather than as an independent optimality result.","section":"Section III-D, Eq. (17)–(20)"},{"comment":"The meaning of the 'Budget' column is confusing: the text says the lower budget matches LoRA rank 32, but TRISHUL and RAVAN are listed as budget 110. Please define the parameter-equivalent rank explicitly in terms of trainable parameters and clarify the core rank r used in the sensitivity and ablation plots.","section":"Tables II–III and Section IV-A3"},{"comment":"The scalar clipping bound s_max is a free parameter of the method but is never specified in the experimental setup; please report its value and, ideally, its sensitivity.","section":"Algorithm 1, line 15"},{"comment":"Figures 8 and 9 each contain multiple panels but are referenced by a single figure number; please add (a)–(c) sub-labels and refer to them explicitly in the text.","section":"Figures 8–9"},{"comment":"The proof of Theorem A.5 relies on a 'standard proximal descent inequality' in Eq. (39), but no reference or derivation is given; please provide a citation or a self-contained proof, since the constants are not immediate.","section":"Appendix E, Eq. (39)"},{"comment":"The text says random-normal initialization performs best on 20 Newsgroups, which is consistent with Table X, but the caption should state the dataset-specific best in one sentence for readability.","section":"Table X and Section IV-C6"},{"comment":"The repository reference 'See Trishul Github' provides no URL or identifier; please include a full link or a reproducibility statement.","section":"Footnote 1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the test-set-based selection of λ: if the gains shrink when λ is chosen on a validation split, the central empirical claim would no longer hold as stated. The convergence proof's mismatch with the scalar reset should also be fixed or removed. I do not see a fundamental correctness reason to reject, but the current evaluation protocol and the Eq. (8) omission of s_i need substantial revision before the claims are convincing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on arXiv:2607.20914. The paper combines three established ideas—exact multi-head aggregation from RAVAN, singular-value thresholding as a proximal step for nuclear norm, and water-filling allocation—into a single federated PEFT recipe. The components aren't new, but the combination is, and the paper does a genuinely good job of making the mechanism visible: the spectral diagnostics (principal-angle similarity, effective rank, aggregation variance) actually show that shrinkage aligns client updates, and the ablations isolate each prong's contribution. The 1B-parameter GLUE experiment is a plus. This deserves a serious referee.\n\nThe soft spots are mostly in the evaluation. The biggest one: lambda is selected by grid search on the same benchmark test sets that are used to report the gains, with no validation split described. The sensitivity table (VI) is then used to justify lambda=0.01 using those same test numbers. That's circular, and it means the reported non-IID gains of 3-5 points aren't yet distinguishable from a selection artifact. The lack of error bars makes this worse: three seeds, no standard deviations, and only three of twenty clients participating per round. A 3-point gap under non-IID could be within run-to-run variance. I'd want to see validation-selected lambda and confidence intervals before trusting the headline.\n\nThe math has a smaller issue. Eq. (8) claims nuclear-norm shrinkage of the core H_i is exactly equivalent to shrinkage of the full update, but the trainable scalar s_i is omitted. The full update is s_i B_i H_i A_i, so the actual nuclear norm is |s_i| times ||H_i||_*. The penalty is on ||H_i||_* regardless of s_i, so shrinkage strength is diluted if s_i > 1. They do clip s_i, but don't report s_max, so the variance bound in Eq. (15) may not apply as stated. Fixable, and not fatal to the method.\n\nThe convergence proof in the appendix is a sketch—standard proximal descent with hand-waved drift handling. It's fine as a sanity check, not a real guarantee. The water-filling optimality theorem is close to tautological (it's optimal for the paper's own utility), but the real evidence is Table VIII, which compares against uniform and heuristic allocations, so that's okay.\n\nBottom line: This is a useful paper for people working on federated PEFT, and the spectral-control framing is worth engaging with. But the empirical claims need strengthening: release code and data, add validation-based hyperparameter selection, report error bars, fix Eq. (8). I'd accept it for peer review—it's a legitimate contribution—but I'd push for major revision. Worth a cite? Probably not until the numbers are reproducible. Bring to reading group? Maybe, to discuss the evaluation-protocol issue.","headline":"A coherent three-part recipe for spectral control in federated PEFT, with a plausible mechanism and good diagnostics, but the headline gains are vulnerable to circular hyperparameter selection and missing error bars.","tokens_in":24424,"tokens_out":3288,"would_cite":false,"duration_ms":30125,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C25","65K05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated PEFT under non-IID data fails when client updates are spectrally misaligned, and TRISHUL's three mechanisms—exact shared-basis aggregation, nuclear-norm shrinkage, and water-filled head budgets—fix it without extra communication.","keywords":["Federated learning","Parameter-efficient fine-tuning","Low-rank adaptation","Nuclear norm regularization","Singular value thresholding","Non-IID heterogeneity","Multi-head low-rank bases","Water-filling allocation"],"falsifier":"Set the scalar cap $s_{\\max}$ very large, or remove the cap entirely, in the TRISHUL implementation on CIFAR-100 non-IID with $\\alpha=0.3$, and compare final accuracy and per-head aggregation variance against the reported setting: if accuracy and variance do not degrade, the rank-controlled shrinkage of the core is not what delivers the gains, while if they degrade sharply, the variance bound depends on the unreported cap.","tokens_in":23383,"feed_emoji":"🔱","tokens_out":6887,"duration_ms":54223,"temperature":0.7,"pith_summary":"The paper tries to establish that the main obstacle to federated parameter-efficient fine-tuning under non-IID data is not just optimization drift but spectral inconsistency: different clients learn low-rank updates occupying misaligned subspaces, so averaging them produces high-variance, poorly transferring global updates. It proposes TRISHUL, which combines shared frozen multi-head low-rank bases to make aggregation algebraically exact, nuclear-norm proximal shrinkage via singular value thresholding on small $r\\times r$ core matrices before upload, and a concave water-filling rule that allocates a fixed head budget to layers with larger pretrained capacity. The claim is that this three-pronged control improves convergence, stability, and final accuracy across vision and language benchmarks, including LLaMA3.2-1B on GLUE, with gains that grow as client heterogeneity increases, at no extra per-round communication and under 1% compute overhead. A careful reader would care because if true it identifies a structural, geometry-level fix for federated fine-tuning rather than another drift-correction optimizer.","feed_headline":"Three-pronged spectral control beats federated LoRA on non-IID data","feed_subtitle":"Frozen shared bases, singular-value shrinkage, and water-filled layer budgets stabilize aggregation with under 1% overhead.","key_machinery":"The machinery is the multi-head low-rank parameterization with frozen shared bases: each layer $l$ has $h_l$ heads, each head defined by orthonormal frozen matrices $B_i \\in \\mathbb{R}^{d\\times r}$, $A_i \\in \\mathbb{R}^{r\\times d}$ with mutually orthogonal subspaces across heads, a trainable $r\\times r$ core $H_i$, and a trainable scalar $s_i$. Three mechanisms act on this substrate: exact aggregation, since shared bases let the server average $\\{s_{c,i}H_{c,i}\\}$ and reconstruct the exact mean ambient update; nuclear-norm proximal shrinkage, where each local gradient step is followed by singular value thresholding $\\operatorname{prox}_{\\lambda\\eta\\|\\cdot\\|_*}(H) = U\\operatorname{diag}(\\max(\\sigma_j - \\lambda\\eta, 0))V^\\top$ on the core, exploiting $\\|B_i H_i A_i\\|_* = \\|H_i\\|_*$; and concave water-filling, where head counts $h_l$ maximize $\\sum_l a_l \\log(1+h_l)$ subject to $\\sum_l h_l = h$ with $a_l = \\|W_l\\|_F^2 + \\epsilon$, solved in closed form as $h_l^* = (a_l/\\mu - 1)_+$. The argument's force is that all spectral operations happen on $r\\times r$ cores, so the claims of exactness, variance reduction, and allocation optimality come with negligible cost.","core_discovery":"TRISHUL's central claim is that controlling the singular spectrum of client updates before aggregation, rather than only correcting gradient drift, is what makes federated PEFT robust to heterogeneity. Concretely: because frozen orthonormal bases $B_i$, $A_i$ are shared, averaging the uploaded core products $s_{c,i}H_{c,i}$ recovers exactly the mean ambient update, eliminating the factor-wise averaging bias $\\mathbb{E}[B_c A_c] \\neq \\mathbb{E}[B_c]\\mathbb{E}[A_c]$ of standard LoRA. Because the bases are isometric, penalizing and thresholding the nuclear norm of $H_{c,i}$ is equivalent to spectral shrinkage of the full update, so proximal singular value thresholding zeroes client-specific high-rank modes before upload while preserving dominant shared directions. A concave water-filling allocation then assigns more of the fixed head budget to layers with larger pretrained Frobenius norms. Across CIFAR-100, SVHN, 20 Newsgroups, MRQA, and GLUE with LLaMA3.2-1B, the paper reports that TRISHUL beats federated LoRA baselines, with margins widening from about 3.2% to 4.7% over the strongest baseline as Dirichlet heterogeneity goes from $\\alpha=0.3$ to $\\alpha=0.05$ on CIFAR-100.","pith_inferences":["Editorial inference: the same spectral-shrinkage mechanism may transfer to centralized PEFT, where applying singular value thresholding to core matrices during fine-tuning could act as a regularizer that improves robustness to distribution shift, though the paper only claims federated benefits.","Editorial inference: the variance bound in Eq. (15) suggests a testable scaling law, namely that halving the effective rank $\\rho_i$ via shrinkage should halve the aggregation-variance bound; measuring per-head variance across $\\lambda$ values would directly confirm whether the mechanism itself, rather than the scalar cap, drives the gains.","Editorial inference: if spectrally compact updates admit tighter sensitivity analysis, TRISHUL-style shrinkage could make differentially private federated PEFT cheaper for a given privacy budget; the paper lists DP as future work and does not claim this.","Editorial inference: dynamic allocation that re-solves the water-filling problem during training using running estimates of layer importance may outperform the one-shot pretrained-norm allocation, since the paper itself notes fixed allocation as a limitation."],"forward_implications":["If TRISHUL is right, federated PEFT under non-IID data should be formulated as a spectral-control problem, and methods that only correct gradient drift will remain below methods that shape the update spectrum before aggregation.","The gains should grow monotonically with heterogeneity: on CIFAR-100 the reported margin over the strongest baseline widens from 3.2% at $\\alpha=0.3$ to 4.7% at $\\alpha=0.05$.","Because shrinkage is applied to $r\\times r$ cores, the method preserves the communication budget of the underlying multi-head PEFT protocol and adds less than about 1% per-round compute at 85M to 1B parameter scale.","The spectral diagnostics, including principal-angle similarity, dominant singular-vector similarity, spectral entropy, effective rank, and inter-client aggregation variance, should show improved alignment and lower variance for TRISHUL relative to the multi-head baseline across rounds and heterogeneity levels.","Layer-wise allocation from pretrained Frobenius norms should dominate uniform allocation and heuristic gradient- or curvature-based allocations at zero runtime cost."],"supporting_citations":[{"why":"Supplies the multi-head low-rank parameterization and exact-aggregation substrate that TRISHUL builds on, and the experimental protocol used for fair comparison.","marker":"[4]"},{"why":"Establishes the nuclear norm as the convex surrogate for rank minimization, justifying the penalty on core matrices.","marker":"[7]"},{"why":"Provides the singular value thresholding proximal operator used for the shrinkage step.","marker":"[8]"},{"why":"Defines low-rank adaptation and the factor-wise aggregation bias that TRISHUL corrects.","marker":"[2]"},{"why":"SCAFFOLD is the gradient-variance-reduction baseline that TRISHUL argues is insufficient because it ignores update geometry.","marker":"[6]"},{"why":"FLoRA is a heterogeneous federated LoRA baseline whose stacking-based aggregation TRISHUL contrasts with exact shared-basis aggregation.","marker":"[10]"},{"why":"Supplies the convex water-filling and KKT machinery for the concave allocation rule.","marker":"[9]"},{"why":"Motivates adaptive budget allocation in centralized PEFT, which TRISHUL adapts to layer-wise head allocation in the federated setting.","marker":"[18]"}],"fun_headline_variants":["Frozen bases and spectral shrinkage reduce federated LoRA drift","Three-pronged spectral control outdoes federated LoRA baselines","TRISHUL cuts federated LoRA variance with nuclear norm shrinkage","Spectral control stabilizes non-IID federated fine-tuning","Non-IID? TRISHUL's spectral control beats federated LoRA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that nuclear-norm shrinkage of the core $H_i$ acts with full strength on the uploaded update, but the uploaded object is $s_i B_i H_i A_i$, and the effective shrinkage strength for the actual penalty is $\\lambda\\|H_i\\|_*/|s_i|$, so if the trainable scalar $s_i$ grows large the variance-reduction claim in Eq. (15) can fail; the paper never reports the scalar bound $s_{\\max}$ that Algorithm 1 line 15 refers to.","fun_headline_variants_meta":{"raw":{"variants":["Frozen bases and spectral shrinkage reduce federated LoRA drift","Three-pronged spectral control outdoes federated LoRA baselines","TRISHUL cuts federated LoRA variance with nuclear norm shrinkage","Spectral control stabilizes non-IID federated fine-tuning","Non-IID? TRISHUL's spectral control beats federated LoRA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000759,"raw_usage":{"total_tokens":3433,"prompt_tokens":1071,"completion_tokens":2362,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":687,"completion_tokens_details":{"reasoning_tokens":2270}},"tokens_in":687,"tokens_out":2362,"duration_ms":14991,"temperature":1.0,"reasoning_tokens":2270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:31:36.646276+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Set the scalar cap $s_{\\max}$ very large, or remove the cap entirely, in the TRISHUL implementation on CIFAR-100 non-IID with $\\alpha=0.3$, and compare final accuracy and per-head aggregation variance against the reported setting: if accuracy and variance do not degrade, the rank-controlled shrinkage of the core is not what delivers the gains, while if they degrade sharply, the variance bound depends on the unreported cap.","supporting_citations":[{"cited_title":"Lora: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Defines low-rank adaptation and the factor-wise aggregation bias that TRISHUL corrects."},{"cited_title":"Scaffold: Stochastic controlled averaging for federated learning,","cited_arxiv_id":null,"evidence_quote":"SCAFFOLD is the gradient-variance-reduction baseline that TRISHUL argues is insufficient because it ignores update geometry."},{"cited_title":"Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations,","cited_arxiv_id":null,"evidence_quote":"FLoRA is a heterogeneous federated LoRA baseline whose stacking-based aggregation TRISHUL contrasts with exact shared-basis aggregation."},{"cited_title":"Adalora: Adaptive budget allocation for parameter-efficient fine-tuning,","cited_arxiv_id":null,"evidence_quote":"Motivates adaptive budget allocation in centralized PEFT, which TRISHUL adapts to layer-wise head allocation in the federated setting."}],"review_version":2}