Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning just 0.1% of parameters with step-by-step scalar sync beats full-parameter federated ZO, and a gradient-alignment signal flags the worst Non-IID clients.

desk verdict Promising recipe, flawed theory: the convergence theorem doesn't follow because Lemma B.7 mixes up sparsity density with support size. read the letter →

arxiv 2506.03337 v1 pith:6VZNCGSH submitted 2025-06-03 cs.LG cs.AI

Yide Ran , Wentao Guo , Jingwei Sun , Yanzhou Pan , Xiaodong Yu , Hao Wang , Jianwen Xie , Yiran Chen
show 2 more authors
Denghui Zhang Zhaozhuo Xu
This is my paper
classification cs.LGcs.AI
keywords federatedlearningzeroth-orderoptimizationLLMfine-tuningsparsemaskNon-IIDdataclientdriftGradIPscorehigh-frequencysynchronization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper contends that the right remedy for Non-IID client drift in federated LLM fine-tuning is not a more complex aggregation rule but extreme, targeted sparsity paired with very frequent synchronization. Meerkat restricts zeroth-order updates to a fixed $0.1\%$ of parameters — those with the largest average squared gradients on C4 pre-training data — so clients upload only lists of scalars, and because the server shares the random seeds, it can reconstruct every client's local path at negligible communication cost. The paper proves a convergence bound in which the steady-state error shrinks with the local step count $T$, and shows experimentally that at $T=1$ the Non-IID accuracy gap largely closes, with Meerkat beating full-parameter federated ZO at equal frequency. It also reports a new empirical signal — the GradIP score, the inner product of a client's ZO gradient with the server's pre-training gradient — whose trajectory decays to zero for extreme Non-IID clients but oscillates for IID clients, and uses this signal to early-stop the worst clients and improve the final model.

What carries the argument

Three objects carry the argument. (1) The transferable sparse mask $m \in \{0,1\}^d$ of density $u = 0.1\%$: built once by selecting the top-$u$ parameters by average squared gradient on C4, and assumed (Assumption B.6) to capture a fraction $c \gg u/d$ of every downstream client's gradient norm; it turns a ZO step into the scalar projected gradient $g = (f(w+\varepsilon(z\odot m)) - f(w-\varepsilon(z\odot m)))/(2\varepsilon)$. (2) The virtual path: the server shares random seeds with clients, so from uploaded scalars alone it reconstructs each client's full local trajectory — the infrastructure for both cheap aggregation and client-level analysis. (3) The GradIP score $\langle \nabla f_p, \hat{\nabla} f_k^t \rangle$, whose convergence-versus-oscillation behavior across clients is traced to the per-client ZO estimator variance $\sigma_k^2$: near zero for single-label clients, positive for IID clients. The convergence bounds (Theorems 2.1 and 2.2) formalize the sparsity-frequency trade-off, with a transient term scaling as $(2+u)^2/(TR)$ and a steady-state error scaling as $T/(2+u)$, so operating at small $T$ over many rounds $R$ lowers the error floor.

What would settle it

Measure the gradient coverage $c = \|m \odot \nabla f_k\|^2 / \|\nabla f_k\|^2$ for each downstream task and client at the start of fine-tuning, using first-order gradients on each dataset. If any common task yields $c$ near the random-coordinate baseline $u/d \approx 10^{-3}$, the transferability premise fails for that task and Theorem 2.1 predicts that Meerkat should fall behind full-parameter ZO there. A direct probe is to fine-tune on a dataset far from C4's distribution (for example, code or a specialized domain) and compare Meerkat with full-parameter ZO at equal synchronization frequency: the measured $c$ values would predict, task by task, where the sparse advantage should invert.

Watch

Extended reading notes

Core claim

The paper's central claim is that a static mask of density $u = 0.1\%$, built once from C4 average squared gradients, plus high-frequency scalar zeroth-order synchronization, gives federated LLM fine-tuning a lower convergence error floor and better test accuracy than full-parameter federated ZO under the same synchronization frequency (Theorem 2.1), and that the GradIP trajectory — the inner product between the server's pre-training gradient and each client's ZO-estimated gradient along the reconstructed virtual path — reliably separates extreme Non-IID from IID clients, enabling early stopping that improves the aggregated model (Meerkat-vp, Theorem 2.2). Updating only $0.1\%$ of parameters cuts communication by over 1000$\times$, so clients can synchronize every single step; empirically, at one local step per round Meerkat's Non-IID accuracy matches its IID accuracy on several models, while full-parameter ZO and LoRA baselines trail. The GradIP phenomenon is explained through the ZO estimator's variance: for single-label (extreme Non-IID) clients the relevant covariance vanishes, so gradient norms decay to zero, while IID clients keep fluctuating. Meerkat-vp thresholds two summary statistics of the GradIP trajectory — the initial-to-later ratio and the quiescent step ratio — to flag such clients and restrict them to one local step per round, improving the global model.

Load-bearing premise

The load-bearing premise is Assumption B.6: the one-time mask of density $u = 0.1\%$ chosen from C4 data must satisfy $\|m \odot \nabla f_k\|^2 = c\,\|\nabla f_k\|^2$ with $c \gg u/d$ for every downstream client and task. The paper never measures $c$ on the downstream tasks, so if a client's fine-tuning gradient lives mostly outside the selected coordinates, the sparse update carries no signal and the convergence bound, the communication saving, and the claimed advantage over full-parameter ZO all collapse.

Editorial extensions

If this is right

  • At equal synchronization frequency, Meerkat's fixed $0.1\%$ mask with scalar-gradient exchange outperforms full-parameter federated ZO on average accuracy across seven tasks and three LLMs, with over 1000$\times$ less communication.
  • Pushing synchronization to one local step per round nearly closes the IID-versus-Non-IID accuracy gap on several model and dataset combinations, evidence that high-frequency communication rather than larger local batches is what mitigates Non-IID drift.
  • GradIP trajectories give the server a raw-data-free signal of extreme label heterogeneity: decaying-to-zero GradIP flags single-label clients, while oscillating GradIP flags balanced ones.
  • Early-stopping flagged clients at one local step (Meerkat-vp) improves global test accuracy over both Meerkat and randomly selecting the same number of clients to early-stop, across communication frequencies.
  • The convergence analysis predicts an optimal sparsity level: lowering density $u$ accelerates the transient term but inflates the steady-state error $T/(2+u)$, so the best density depends on how many communication rounds are affordable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the server already reconstructs each client's full local path from scalar uploads, the virtual-path infrastructure could support interventions stronger than binary early stopping — per-client learning-rate scaling, weighted aggregation, or mask re-selection — at zero additional communication cost.
  • The theory's optimal-density prediction implies an untested adaptive scheme: start with extreme sparsity during cheap high-frequency rounds and relax density or refresh the mask later, riding both sides of the $T/(2+u)$ error-floor trade-off.
  • The GradIP variance argument ties the signal specifically to label skew rather than heterogeneity in general; clients with covariate shift but balanced labels may not be flagged, a boundary case the experiments do not cover.
  • Because Assumption B.6 is directly measurable, the paper effectively supplies its own validation protocol: computing per-task gradient coverage $c$ on downstream data would predict, before training, which datasets are safe to fine-tune with a C4-selected mask.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Meerkat, a sparse zeroth-order optimization method for federated fine-tuning of LLMs. Meerkat restricts ZO perturbations to a static mask containing roughly 0.1% of parameters, selected by the largest average squared gradients computed on the C4 pre-training corpus, and clients exchange only scalar projected gradients. The authors claim that the resulting communication savings enable high-frequency synchronization, which reduces Non-IID client drift and yields better accuracy than full-parameter federated ZO under the same synchronization frequency. A second contribution, Meerkat-vp, uses the server's ability to reconstruct client update paths to compute a GradIP score, observes that this score decays for extreme Non-IID clients and oscillates for IID clients, and applies early stopping to clients flagged as extremely Non-IID. The paper provides convergence theorems for both methods (Theorems 2.1 and 2.2), a full proof appendix, and experiments on three LLMs across seven datasets.

Significance. The paper addresses an important practical problem: reducing communication and memory costs in federated LLM fine-tuning, and the experimental campaign is broad, covering three LLMs and several GLUE/SuperGLUE tasks. If the theoretical claims were correct, the result that a 0.1% static mask plus scalar-only synchronization can outperform full-parameter ZO would be a meaningful advance, and the GradIP signal for detecting extreme Non-IID clients is an interesting empirical idea. The paper also includes explicit algorithms and a complete proof appendix, which is a strength. However, the central convergence proof contains a load-bearing units error: Lemma B.7 conflates the sparsity density u with the mask support size, changing the constants in Theorem 2.1 by orders of magnitude. In addition, the key transferability assumption about gradient coverage is never measured, and the VPCS thresholds are tuned on the same datasets used to report Meerkat-vp gains. As a result, the theoretical and VPCS claims are not currently supported, despite the apparently consistent empirical accuracy improvements.

major comments (4)
  1. [Appendix B.2, Lemma B.7; Theorem 2.1] Lemma B.7 is stated with the factor (2+u), but the proof takes the trace of E[ b∇f b∇f^T ] = 2 v v^T + ||v||^2 eId,m, whose diagonal sum is 2||v||^2 + tr(eId,m)||v||^2. Since tr(eId,m) equals the number of nonzero entries of the mask, s = ||m||_0, the lemma should read (2+s)c||∇f||^2, not (2+u)c||∇f||^2. Under the paper's stated density u=0.1% (Section 2.1) and d≈10^9, s≈10^6, so the factor is off by roughly 10^6, and the squared factors that appear in the learning rate and in Theorem 2.1's transient term are off by roughly 10^12. If u is instead interpreted as a count, then full-parameter ZO corresponds to u=d, not u=1, and the repeated comparison against the 'u=1' full-parameter case in Section 2.2 is invalid. Either way, Eq. (2) and Theorem 2.1 do not follow from the proof, and the claimed quantitative benefit of sparsity over full-parameter ZO is unsupported.
  2. [Section 2.1 and Assumption B.6] The transferability claim rests on Assumption B.6, which requires that the fixed C4-derived mask satisfies ||m⊙∇f_k||^2 = c||∇f_k||^2 with c≫u/d for every downstream client and task. This assumption is never measured or verified in the experiments; no downstream-task value of c is reported. If a client's fine-tuning gradient is concentrated outside the selected coordinates, then c is small and Meerkat has no update signal in those coordinates. This is structurally different from the paper's other claims about synchronization frequency and GradIP, and it is load-bearing for Claim 1 (Section 2.3). The paper should either measure c on the actual downstream tasks or provide a direct empirical test of mask transferability across the datasets used in Tables 1 and 8.
  3. [Appendix B.5, Theorem 2.2 comparison] The derivation that E_Meerkat-vp < E_Meerkat relies on several unproven or internally inconsistent ingredients. First, the parameter γ is introduced as 'the clients with balanced data distributions contribute to the global model during training' and is used in the inequality on the sum over K_g, but no formal definition or quantitative relation between γ and K_g, K_b, or c_h is provided. Second, the claim that γ(1+√c_h)^2→0 as c_h→1 is asserted without derivation. Third, the noise-term ratio comparison requires µ(1+√c_h)^2/2 < 1; the paper argues that this holds because µ<1 empirically, but µ<1 only gives µ(1+√c_h)^2/2 < 2, not <1, since (1+√c_h)^2 can approach 4. Thus the conclusion that Meerkat-vp has a strictly smaller error bound than Meerkat is not established by the analysis.
  4. [Algorithm 1, Table 4, and Section 3.3] The VPCS thresholds used to flag extreme Non-IID clients are the initial-to-later ratio ρlater and the quiescent step ratio ρquie. Table 4 reports that these are tuned over ranges [1.5,2,5,10,15] and [0.4,0.5,0.7] on the same tasks and datasets for which Meerkat-vp's accuracy gains are then reported in Tables 6 and Figure 4. No held-out calibration or threshold-sensitivity analysis is provided, and the number of clients flagged is not reported. The improvement of Meerkat-vp over Meerkat and over Random Client Selection could therefore be an artifact of threshold selection rather than evidence for the GradIP phenomenon. In addition, all accuracy tables report a single run without error bars or multiple seeds, so the statistical significance of the reported improvements is unclear.
minor comments (4)
  1. [Section 2.1, Table 2] The notation for u is inconsistent: Table 2 defines u as the sparsity ratio, Section 2.1 says 'set u to 0.1%' and 'select the top u parameters', while Assumption B.6 says the mask has 'exactly u non-zero entries'. Please reconcile these definitions, as the distinction between density and support size is central to the proof.
  2. [Appendix A, paragraph on FedMeZO] The text states 'FedMeZO [15] analyzes the convergence properties of ZO for federated LLM fine-tuning', but reference [15] is the FedAvg non-IID convergence paper by Li et al.; this citation appears mismatched.
  3. [Algorithm 3] The variable m is used both for the global model and for the sparse mask in Algorithm 3, which makes the pseudocode ambiguous. Please rename one of them.
  4. [Throughout] There are numerous typographical spacing errors in the rendered text, such as 'Meerka t', 'Meerka t-vp', and 'Acc' where 'Acc' may be intended, and several equation references are inconsistent (e.g., Theorem B.9 is called 'equation 5'). A careful proofreading pass is needed.

Circularity Check

2 steps flagged · score 5.0 of 10

Theoretical sparsity benefit rests on a density-vs-count equivocation in Lemma B.7, and the VPCS improvement is threshold-fitted; the core sparsity experiments remain independent.

  1. other [Assumption B.6; Lemma B.7; Theorem 2.1 Eq. (2)]
    "Assumption B.6: 'there exists a binary mask m∈{0,1}^d with exactly u non-zero entries'. Lemma B.7 states E_z||b∇f||^2 = (2+u)c||∇f||^2; its proof sums the diagonal of E[zz^T vv^T zz^T] = 2vv^T + ||v||^2 e_{Id,m} and writes '2c||∇f||^2 + uc||∇f||^2 = (2+u)c||∇f||^2'."

    u is defined as a sparsity ratio (Table 2), but the proof obtains (2+u) by taking the trace of the masked identity, which equals the number of nonzero mask entries s=||m||_0. At the paper's u=0.1% and d≈10^9, s≈10^6, so the lemma should read (2+s), not (2+u). Every u-dependent term in Theorem 2.1, including the claimed O((2+u)^2) transient benefit and the 'u=1 full-parameter' baseline, is therefore an artifact of switching between 'u as density' and 'u as count'. The sparsity prediction reduces to this definitional equivocation rather than to the estimator analysis.

  2. fitted input called prediction [Algorithm 1; Appendix C.1 Table 4; Section 3.3]
    "Algorithm 1 takes thresholds 'convergence threshold σ, Initial to later ratio ρ_later, quiescent step ratio ρ_quie'; Table 4 sets 'quiescent step ratio [0.4,0.5,0.7], Initial to later ratio [1.5,2,5,10,15]'; Section 3.3 concludes 'Meerkat-vp consistently outperforms both Meerkat and Random Client Selection'."

    The client-selection rule, and therefore the reported Meerkat-vp gain, is parametrized by ρ_later and ρ_quie, and these thresholds are swept over the listed grids on the same seven benchmarks used for Figure 4/Table 6. The paper does not report a held-out threshold choice or robustness across the grid; it reports the best-case outcome. Thus the 'prediction' that GradIP-based early stopping helps is not an out-of-sample consequence of the GradIP signal but the result of selecting thresholds on the evaluation data, statistically forcing the improvement.

full rationale

Most of the paper is not circular: the C4-derived mask and the direct empirical comparisons against Full-FedZO, Weight Magnitude, and LoRA (Tables 1, 5, 8-10) are self-contained, and the claim that Meerkat outperforms full-parameter ZO at the same synchronization frequency is experimentally tested rather than derived from its inputs. The self-citation to [9] is not load-bearing because the paper includes its own weight-magnitude and LoRA baselines. However, the theoretical derivation is compromised: Lemma B.7's (2+u) factor is obtained by treating the masked identity's trace (support size s) as the density u; the subsequent convergence theorem and the 'u=1 full-parameter' comparison inherit this equivocation, so the predicted benefit of sparsity in Eq. (2) is an artifact of notation, not a consequence of the ZO estimator. Separately, the VPCS thresholds (ρ_later, ρ_quie) are grid-searched on the same datasets used to report Meerkat-vp gains, so that improvement is partly a threshold-selection result. The Meerkat-vp versus Meerkat theoretical comparison also relies on an asserted γ→0 as c_h→1 rather than a derived relation, further weakening that comparison. These are correctness and partial-circularity concerns, but the experimental sparsity core remains independent, so a mid-range score is appropriate.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The central claims rest on standard but strong optimization assumptions, an unverified mask-transferability assumption, and several ad hoc quantities (γ, µ<1, c_h→γ relation) introduced to make the Meerkat-vp comparison go through. No new physical entities are postulated, but the VPCS thresholds act as fitted parameters.

free parameters (5)
  • sparsity density u = 0.001 (0.1%); swept 5e-1 to 5e-5 in Table 7
    The mask density is chosen by the authors and swept; the optimal density is task-dependent, and the theory treats u inconsistently as density and as support count.
  • VPCS thresholds = T_init=20, T_later=20, T_cali=100, sigma=1, rho_quie in {0.4,0.5,0.7}, rho_later in {1.5,2,5,10,15}
    Thresholds are searched over ranges on the same benchmark datasets used to report Meerkat-vp gains; no held-out validation is described.
  • gradient coverage c
    Assumption B.6 postulates ||m⊙∇f_k||² = c||∇f_k||² with c ≫ u/d, but c is never measured; the convergence bound depends on 1/c.
  • balanced-client contribution gamma
    Introduced in Appendix B.5 and Table 2; used to bound the sum over K_g clients by gamma times the full sum and to claim the transient term ratio gamma(1+sqrt(c_h))^2<1; no value or estimator is given.
  • learning rates = [2e-4, 2e-8]
    Reported as ranges, not per model/task, so the exact values used are not reproducible.
assumptions (7)
  • standard math Global loss f(w)=Σ p_k f_k(w) is L-smooth (Assumption B.1).
    Used throughout convergence proofs to bound descent via quadratic upper bound; standard but unverified for LLM fine-tuning losses.
  • standard math Global loss satisfies Polyak-Lojasiewicz inequality with constant µ (Assumption B.2).
    Converts gradient-norm bounds into suboptimality; strong for non-convex LLM landscapes and not demonstrated.
  • domain assumption Heterogeneity bound: ||∇f(θ)-∇f_k(θ)||² ≤ c_h||∇f(θ)||² + σ_h² (Assumption B.3).
    Standard non-IID analysis assumption; no evidence the constants are small or finite in the tested tasks.
  • domain assumption Mask coverage: ||m⊙∇f_k||² = c||∇f_k||² with c ≫ u/d (Assumption B.6).
    This is the transferable-sparsity premise: the C4-selected 0.1% mask must contain most of the gradient norm for every downstream client task. If false, the sparse updates cannot converge. Not measured in the paper.
  • domain assumption For extreme Non-IID single-label clients, softmax probabilities approach the one-hot label so Σ_a≈0; for IID clients Σ_a is positive definite (Appendix B.6).
    Drives the claimed GradIP/norm dichotomy, but the argument is asymptotic and does not imply oscillation for IID clients.
  • ad hoc to paper The relation c_h→1 implies γ→0 and γ(1+√c_h)^2<1 (Appendix B.5).
    γ is not defined precisely and the asserted relation is the step that makes E_Meerkat-vp smaller than E_Meerkat; no derivation is given.
  • ad hoc to paper µ<1 is assumed empirically (Appendix B.5) to make the noise-term ratio µ(1+√c_h)^2/2 < 1.
    Used to claim Meerkat-vp's steady-state noise is smaller; no measurement of µ is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity." pith.science (2026). https://pith.science/paper/6VZNCGSH

@misc{pith2026250603337,
  author       = {Pith},
  title        = {Pith review of: Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6VZNCGSH}},
  note         = {Machine review of arXiv:2506.03337}
}
read the original abstract

Federated Learning enables collaborative fine-tuning of Large Language Models (LLMs) across decentralized Non-Independent and Identically Distributed (Non-IID) clients, but such models' massive parameter sizes lead to significant memory and communication challenges. This work introduces Meerkat, a sparse zeroth-order optimization (ZO) method designed for federated LLM fine-tuning. By limiting fine-tuning to a transferable, static, extremely sparse subset of parameters, Meerkat achieves remarkable communication efficiency, enabling cost-effective high-frequency synchronization. With theoretical analysis and experiments, we show that this high-frequency communication effectively mitigates Non-IID data challenges and leads to superior performance compared to full-parameter ZO. Furthermore, experiment results show that Meerkat outperforms existing sparsity baselines with better performance at the same communication frequency. To further handle Non-IID drift, Meerkat leverages traceable local updates and forms a virtual path for each client. This virtual path mechanism reveals the GradIP phenomenon: the inner products between LLM pre-training gradients maintained by server and client gradients estimated via ZO converges for extreme Non-IID clients but oscillates for IID ones. This distinct behavior provides a signal for identifying clients with extreme data heterogeneity. Using this signal, Meerkat-vp is proposed to analyze GradIP trajectories to identify extreme Non-IID clients and applies early stopping to enhance aggregated model quality. Experiments confirm that Meerkat and Meerkat-vp significantly improve the efficiency and effectiveness of ZO federated LLM fine-tuning.

Figures

Figures reproduced from arXiv: 2506.03337 by the authors.

Figure 1
Figure 1. Meerkat: Sparse zeroth-order optimization for federated LLM fine-tuning workflow. 2 Sparse Zeroth-Order Optimization for Federated LLM Fine￾tuning This section introduces Meerkat, a sparse ZO method for federated LLM fine-tuning, and its upgraded version, Meerkat-vp, which incorporates Virtual Path Client Selection(VPCS) strategy. This strategy leverages the traceable virtual path of client local updates to identify… view at source ↗
Figure 2
Figure 2. This figure compares three methods—Full-FedZO, LoRA-FedZO, and [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Under a density ratio of 5 × 10−3 , we track the GradIP (see Defini￾tion 2.3) over 100 local training steps on the SST-2 dataset using LLaMA-3.2-1B model, comparing a client with IID data to a client with Non-IID data. This section experimentally validates Claim 3 (Sec￾tion 2.5). As established in Section 3.2, GradIP trajecto￾ries provide an effective signal for identifying clients with extremely Non-IID data, exhib… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: This figure compares two methods—Meerkat-vp, Meerkat and Random Client Se￾lection—across three LLMs: LLaMA-3.2-1B, Qwen2-1.5b, and Gemma2-2b. The x-axis shows the local step values (10, 30, 50, 100), while the y-axis indicates the average test accuracy over multiple do…
Figure 5
Figure 5. Figure 5: Meerkat-vp: Each client locally trains with a prescribed statistic step, yielding a sequence of projected gradients. The server leverages a randomly sampled vector z t k from the Gaussian distribution N (0, Id) to reconstruct ∇f t k , and then computes GradIP (see Defi…
Figure 6
Figure 6. Figure 6: Comparison of Full-FedZO, LoRA-FedZO, and [PITH_FULL_IMAGE:figures/full_fig_p052_6.png]
Figure 7
Figure 7. Figure 7: These figures show GradIP (Definition 2.3) curves under IID and Non-IID settings, computed over 100 local training steps on six datasets (WSC, BoolQ, RTE, WIC, AgNews, Yelp) using the Llama-3.2-1B model with density level 5 × 10−3 . An extra BoolQ result is shown for t…
Figure 8
Figure 8. Figure 8: The left panel shows the cosine similarity between locally computed ZO gradients and [PITH_FULL_IMAGE:figures/full_fig_p054_8.png]
Figure 9
Figure 9. Figure 9: These subfigures show GradIP (see Definition [PITH_FULL_IMAGE:figures/full_fig_p054_9.png]
Figure 10
Figure 10. Figure 10: GradIP analysis for different models on the BoolQ dataset under Non-IID and IID [PITH_FULL_IMAGE:figures/full_fig_p055_10.png]
Figure 11
Figure 11. Figure 11: GradIP analysis for different models on the AGNews dataset under Non-IID and IID [PITH_FULL_IMAGE:figures/full_fig_p056_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    RISE applies CountSketch to dual lexical and semantic channels derived from output-layer gradient outer products, cutting data attribution storage by up to 112x and enabling retrospective and prospective influence ana...

Reference graph

Works this paper leans on

44 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [9]

    Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, Beidi Chen, and Zhaozhuo Xu

    Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R. Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, Beidi Chen, and Zhaozhuo Xu. Zeroth-order fine-tuning of llms with extreme sparsity, 2024. URLhttps://arxiv.org/abs/2406.02913

    arXiv 2024

  2. [1]

    Qwen2 technical report. 2024

    2024

  3. [2]

    What- mough, and Venkatesh Saligrama

    Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization, 2021. URLhttps://arxiv.org/abs/2111.04263. 11

    arXiv 2021

  4. [3]

    Curtis, and Jorge Nocedal

    Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale ma- chine learning, 2018. URLhttps://arxiv.org/abs/1606.04838

    arXiv 2018

  5. [4]

    Languagemodels are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, ArvindNeelakantan, PranavShyam, GirishSastry, AmandaAskell, etal. Languagemodels are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  6. [5]

    Fine-grained theoretical analysis of federated zeroth-order optimization.Advances in Neural Information Processing Systems, 36, 2024

    Jun Chen, Hong Chen, Bin Gu, and Hao Deng. Fine-grained theoretical analysis of federated zeroth-order optimization.Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  7. [6]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    arXiv 1905

  8. [7]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

Show all 44 references
  1. [8]

    Communication-efficient stochastic zeroth-order optimization for federated learning.IEEE Transactions on Signal Processing, 70:5058–5073, 2022

    Wenzhi Fang, Ziyi Yu, Yuning Jiang, Yuanming Shi, Colin N Jones, and Yong Zhou. Communication-efficient stochastic zeroth-order optimization for federated learning.IEEE Transactions on Signal Processing, 70:5058–5073, 2022

    2022

  2. [10]

    Pruning large language models with semi-structural adaptive sparse training, 2024

    Weiyu Huang, Yuezhou Hu, Guohao Jian, Jun Zhu, and Jianfei Chen. Pruning large language models with semi-structural adaptive sparse training, 2024. URLhttps://arxiv.org/abs/ 2407.20584

    2024 arXiv

  3. [11]

    Reddi, Sebastian U

    Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning,

  4. [12]

    The winograd schema challenge

    Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012

    2012

  5. [13]

    Federated learning on non-iid data silos: An experimental study, 2021

    Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study, 2021. URLhttps://arxiv.org/abs/2102.02079

    2021 arXiv

  6. [14]

    Federated optimization in heterogeneous networks, 2020

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, 2020. URLhttps://arxiv.org/ abs/1812.06127

    2020 arXiv

  7. [15]

    Ontheconvergence of fedavg on non-iid data, 2020

    XiangLi, KaixuanHuang, WenhaoYang, ShusenWang, andZhihuaZhang. Ontheconvergence of fedavg on non-iid data, 2020. URLhttps://arxiv.org/abs/1907.02189

    2020 arXiv

  8. [16]

    Achieving dimension- free communication in federated learning via zeroth-order optimization, 2024

    Zhe Li, Bicheng Ying, Zidong Liu, Chaosheng Dong, and Haibo Yang. Achieving dimension- free communication in federated learning via zeroth-order optimization, 2024. URLhttps: //arxiv.org/abs/2405.15861. 12

    2024 arXiv

  9. [17]

    On the convergence of zeroth-order federated tuning for large language models

    Zhenqing Ling, Daoyuan Chen, Liuyi Yao, Yaliang Li, and Ying Shen. On the convergence of zeroth-order federated tuning for large language models. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1827–1838, 2024

    2024

  10. [18]

    Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning, 2024

    Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning, 2024. URL https://arxiv.org/abs/2402.15751

    2024

  11. [19]

    Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Alice Oh, Tristan Naumann, Amir Glo...

    2023

  12. [20]

    Deja vu: Contextual sparsity for efficient llms at inference time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shri- vastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient llms at inference time. InInternational Conference on Machine Learning, pages 22137–22176. ...

    2023

  13. [21]

    SPP: Sparsity-preserved parameter-efficient fine-tuning for large language models

    Xudong Lu, Aojun Zhou, Yuhui Xu, Renrui Zhang, Peng Gao, and Hongsheng Li. SPP: Sparsity-preserved parameter-efficient fine-tuning for large language models. InForty-first International Conference on Machine Learning, 2024. URLhttps://openreview.net/forum? id=9Rroj9GIOQ

    2024

  14. [22]

    Fine-tuning language models with just forward passes.Advances in Neural Information Processing Systems, 36:53038–53075, 2023

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes.Advances in Neural Information Processing Systems, 36:53038–53075, 2023

    2023

  15. [23]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. InArtificial intelligence and statistics, pages 1273–1282. PMLR, 2017

    2017

  16. [24]

    Local learning matters: Rethinking data heterogeneity in federated learning

    Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8397–8406, 2022

    2022

  17. [25]

    Wic: the word-in-context dataset for evaluating context-sensitive meaning representations.arXiv preprint arXiv:1808.09121, 2018

    Mohammad Taher Pilehvar and Jose Camacho-Collados. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations.arXiv preprint arXiv:1808.09121, 2018

    2018 arXiv

  18. [26]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  19. [27]

    One-shot sensitivity-aware mixed sparsity pruning for large language models

    Hang Shao, Bei Liu, and Yanmin Qian. One-shot sensitivity-aware mixed sparsity pruning for large language models. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11296–11300. IEEE, 2024

    2024

  20. [28]

    Recursive deep models for semantic compositionality over a senti- ment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a senti- ment treebank. InProceedings of the 2013 conference on empirical methods in natural language proc...

    2013

  21. [29]

    In defense of structural sparse adapters for concurrent llm serving

    Junda Su, Zirui Liu, Zeju Qiu, Weiyang Liu, and Zhaozhuo Xu. In defense of structural sparse adapters for concurrent llm serving. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 4948–4953, 2024

    2024

  22. [30]

    Jingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang, Daguang Xu, Yiran Chen, and Holger R. Roth. Fedbpt: Efficient federated black-box prompt tuning for large language models, 2023. URLhttps://arxiv.org/abs/2310.01467

    2023 arXiv

  23. [31]

    Gemma Team. Gemma. 2024. doi: 10.34740/KAGGLE/M/3301. URLhttps://www.kaggle. com/m/3301

    2024 doi

  24. [32]

    Glue: A multi-task benchmark and analysis platform for natural language under- standing.arXiv preprint arXiv:1804.07461, 2018

    Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language under- standing.arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  25. [33]

    Vincent Poor

    Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. A novel framework for the analysis and design of heterogeneous federated learning.IEEE Transactions on Signal Processing, 69:5234–5249, 2021. doi: 10.1109/TSP.2021.3106104

    2021

  26. [34]

    Structured pruning of large language models

    Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint arXiv:1910.04732, 2019

    1910 arXiv

  27. [35]

    Soft prompt recovers compressed llms, transferably

    Zhaozhuo Xu, Zirui Liu, Beidi Chen, Shaochen Zhong, Yuxin Tang, Jue Wang, Kaixiong Zhou, Xia Hu, and Anshumali Shrivastava. Soft prompt recovers compressed llms, transferably. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, ...

    2024

  28. [36]

    Fedfed: Feature distillation against data heterogeneity in federated learning.Advances in Neural Information Processing Systems, 36, 2024

    Zhiqin Yang, Yonggang Zhang, Yu Zheng, Xinmei Tian, Hao Peng, Tongliang Liu, and Bo Han. Fedfed: Feature distillation against data heterogeneity in federated learning.Advances in Neural Information Processing Systems, 36, 2024

    2024

  29. [37]

    Desirable companion for vertical federated learning: New zeroth-order gradient based algorithm

    Qingsong Zhang, Bin Gu, Zhiyuan Dang, Cheng Deng, and Heng Huang. Desirable companion for vertical federated learning: New zeroth-order gradient based algorithm. InProceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2598–2607, 2021

    2021

  30. [38]

    Character-level convolutional networks for text classification.Advances in neural information processing systems, 28, 2015

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification.Advances in neural information processing systems, 28, 2015

    2015

  31. [39]

    Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen

    Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark. In Forty-fi...

    2024

  32. [40]

    Dynamic sparse no training: Training-free fine-tuning for sparse llms, 2024

    Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free fine-tuning for sparse llms, 2024. URLhttps://arxiv.org/abs/2310.08915

    2024 arXiv

  33. [41]

    Federated learning with non-iid data

    Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. 2018. doi: 10.48550/ARXIV.1806.00582. URLhttps://arxiv. org/abs/1806.00582. 14

    2018 doi arXiv

  34. [42]

    Learn to be efficient: Build structured sparsity in large language models.arXiv preprint arXiv:2402.06126, 2024

    Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Z Morley Mao, Beidi Chen, Fan Lai, and Atul Prakash. Learn to be efficient: Build structured sparsity in large language models.arXiv preprint arXiv:2402.06126, 2024

    2024 arXiv

  35. [43]

    sensitive

    Yang Zhou, Zhuoming Chen, Zhaozhuo Xu, Victoria Lin, and Beidi Chen. Sirius: Contextual sparsity with correction for efficient llms.arXiv preprint arXiv:2409.03856, 2024. 15 Appendix In Section A, we present the related work relevant to this study. In Section B, we present the...

    2024 arXiv

  36. [2021]

    URLhttps://arxiv.org/abs/1910.06378

    1910 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.