Pith. sign in

REVIEW 3 major objections 5 minor 31 references

ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that backpropagation-free, chunk-decoupled LLM fine-tuning converges at the same rate as backpropagation and, in prototype, cuts memory by 26.5% while raising throughput by 4.9%.

desk verdict Real system and Android prototype, but the convergence theorem is proved for a degenerate reverse-KL objective that the implementation doesn't actually minimize. read the letter →

arxiv 2608.07974 v1 pith:2VIZR77Y submitted 2026-08-08 cs.LG cs.DC

Wentao Dai , Xuanran Li , Yuxiang Zhang , Ming Tang , Chao Huang This is my paper · ORCID
classification cs.LGcs.DC
keywords backpropagation-freetrainingpipelineparallelismlocalobjectiveconstructionLLMfine-tuningedgecomputingconvergenceanalysismemoryefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ZeroLock is an algorithm and system for fine-tuning large language models at the network edge without backpropagation. The paper claims that splitting the model into chunks and giving each chunk its own local objective lets all chunks update concurrently, removing the inter-stage waiting and long-lived activation storage that backpropagation forces. It further proves that this decoupled update rule converges at $\tilde{\mathcal{O}}(1/\sqrt{T})$, the same rate as standard stochastic gradient descent on a suitable surrogate objective, differing from backpropagation only by polylogarithmic factors. The paper also reports a working prototype in which ZeroLock uses 26.5% less peak memory and achieves 4.9% higher throughput than a strong pipeline-parallel backpropagation baseline. If these claims hold, edge-device LLM fine-tuning would gain a principled BP-free alternative that trades a polylog factor in theory for large practical gains in memory and pipelining.

What carries the argument

The load-bearing identity is the local-optimum representation of Lemma 1: the minimizer $p^*_k$ of chunk $k$'s local loss can be rewritten as the minimizer of $\langle\nabla L(p_{k-1}), p-p_{k-1}\rangle + \frac{1}{\alpha}D_\psi(p,p_{k-1})$, connecting a purely local objective to the global loss $L$. For the KL special case, Lemma 2 turns this into the closed-form geometric-mean target $p^*_k \propto p_y^\alpha p_{k-1}^{1-\alpha}$. Lemma 3 then shows that the chunk-wise gradient steps compose exactly as a gradient step on the surrogate objective $R^t(\omega;\alpha)$, whose Jacobian is block-diagonal, so the decoupled updates are not an approximation at the level of the surrogate. Lemma 4, driven by the stability assumption, bounds the drift of this surrogate between iterations, and Theorem 1 closes the argument with a standard nonconvex SGD bound.

What would settle it

Run ZeroLock on a real chunked LLM, recording at each iteration the readout distributions $\tilde{p}^t$ and $\tilde{p}^{t+1}$ and the inner product $\langle\nabla_{\tilde{p}} R^t, \tilde{p}^{t+1}-\tilde{p}^t\rangle$; if the positive part of this quantity ever exceeds $C_{\mathrm{fo}}(\alpha)\|\tilde{p}^{t+1}-\tilde{p}^t\|^2$ for the required constant, Assumption 5 is violated and the drift bound—and with it the claimed $\tilde{\mathcal{O}}(1/\sqrt{T})$ rate—fails on that run.

Watch

Extended reading notes

Core claim

The paper's central claim is that local-objective-based decoupled training, previously a heuristic, can be mapped onto a global objective and analyzed as such. Concretely, the paper shows that the optimal update of each chunk's readout distribution is the solution of a composite objective formed by a global-gradient term and a Bregman-divergence penalty (Lemma 1); that for KL divergence this optimum is the normalized geometric mean $p^*_k(l) \propto p_y(l)^\alpha p_{k-1}(l)^{1-\alpha}$ of the target and the upstream output (Lemma 2); and that the stacked chunk-wise parameter updates equal gradient descent on a single surrogate loss $R^t(\omega;\alpha)=\sum_k D_{\mathrm{KL}}(\hat{f}_k(p^t_{k-1};\omega_k)\,\|\,p^{*,\,t}_k(\alpha))$ (Lemma 3). A nonconvex-SGD argument with a drift bound (Lemma 4) then yields Theorem 1: after $T$ iterations the average squared gradient magnitude is $\tilde{\mathcal{O}}(1/\sqrt{T})$, matching the rate of backpropagation up to polylog factors. The same framework yields a chunk-wise performance bound (Proposition 1) showing that near-optimal local updates decrease the final chunk's loss as the number of chunks grows.

Load-bearing premise

The rate proof rests on an unverified stability condition: the surrogate loss must change no faster than quadratically as the readout distribution moves from one iteration to the next, and the paper labels this 'reasonable' without deriving or measuring it.

Editorial extensions

If this is right

  • If the convergence theorem is correct, a model can be split into any number of chunks that update concurrently, and fine-tuning is guaranteed to reach the same worst-case accuracy as backpropagation within a polylog factor of the iteration count.
  • Because only hidden states are exchanged between stages and activations are released after each chunk's local update, the approach supports larger micro-batches before running out of memory and tolerates slower, higher-latency device links.
  • Breaking update locking removes pipeline bubbles at the algorithmic level rather than patching them with scheduling tricks, so the throughput gain is structural rather than tuned.
  • The prototype results show the design is deployable on both multi-GPU servers and mobile phones, and the checkpointing scheme recovers from device failures by replaying only the failed stage's update windows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The theory proves convergence for the KL surrogate objective, but the deployed loss is cross-entropy with a frozen readout head; a controlled experiment that optimizes exactly the geometric-mean KL objective would reveal whether the proven rate transfers to the implemented loss.
  • The stability assumption is the likeliest point of failure: since the constant $C_{\mathrm{fo}}(\alpha)$ is never measured, the drift bound is conditional. Empirically checking the inequality on a few real checkpoints would settle whether the $\tilde{\mathcal{O}}(1/\sqrt{T})$ guarantee holds outside the proof.
  • The block-diagonal Jacobian view suggests the analysis could extend to asynchronous or heterogeneous update schedules, where chunks take different numbers of local steps per global iteration; such a generalization would need the drift term to absorb the added asynchrony.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. ZeroLock proposes a backpropagation-free pipeline-parallel fine-tuning scheme for LLMs in which each transformer chunk is updated with a local loss formed by a frozen readout head and a consistency term, and the paper claims an O~(1/√T) convergence rate that differs from BP only by polylogarithmic factors. The paper also contributes a system design (early forwarding, stage-local checkpoints, state-only inter-stage exchange, failure recovery) and reports prototype measurements on multi-GPU and Android platforms, including a 26.5% memory reduction and 4.9% throughput improvement over PipeDream. The theoretical section maps local chunk updates to a constructed global objective R^t and proves convergence under assumptions that include a nonstandard stability condition.

Significance. The system contribution is concrete and useful: the anonymous repository, the memory decomposition in Fig. 4, the throughput scaling in Figs. 5–6, and the Android deployment in Section IV.D provide evidence that decoupled local-objective training can reduce activation memory and improve pipeline utilization. If the theory were valid, the paper would be the first general-chunking analysis of local-objective LLM training, going beyond the two-chunk analysis of LoPT. However, the main theoretical guarantee is currently about a surrogate objective rather than the implemented loss, so the advertised 'first analytical framework' and the BP-comparable rate are not established; the value of the paper at present rests mostly on the empirical system study.

major comments (3)
  1. [Section II.C, Eq. (14) vs Section III.B.1(a)] The theorem's objective is L_t_k(p;α)=α D_KL(p∥p_y)+(1−α)D_KL(p∥p^t_{k−1}), but the implemented local loss is token-level cross-entropy with one-hot labels. For one-hot p_y, D_KL(p∥p_y) is infinite for any p with positive mass on non-target tokens, so Eq. (14) is degenerate; its minimizer is p_y for every α>0, and the consistency term in the analyzed objective is inactive. The finite training curves in Figs. 3 and 8(b) therefore cannot come from Eq. (14). Lemma 2 and Theorem 1 are derived for Eq. (14), so the O~(1/√T) claim does not apply to the implemented algorithm. Equation (4) also carries a minus sign before Dψ(p,p_y), which is inconsistent with both Eq. (14) and the statement that minimizing (4) equals minimizing cross-entropy.
  2. [Section II.C, Lemma 3 and Theorem 1] The convergence theorem is stated for R^t(ω;α) in Eq. (19), the sum of per-chunk local KL divergences with time-varying targets p*^t_k(α). Lemma 3 constructs R^t so that the ZeroLock update is exactly ω^{t+1}=ω^t−η∇R^t; this is an identity by definition rather than a derivation that local updates optimize the global task loss L(p)=L_Task(p). The abstract's comparison with BP is therefore between a surrogate-objective rate and a global-objective rate. The only link to L is Proposition 1, and its error terms δ_k are not bounded or controlled in the analysis, so the final-chunk global loss decrease is not guaranteed by Theorem 1.
  3. [Section II.C, Assumption 5 (Stability)] Lemma 4's drift bound and hence Theorem 1's rate depend on the quadratic control of the first-order variation of R^t along the moving distribution, ⟨∇_{p̃^t}R(ω^t;α,p̃^t), p̃^{t+1}−p̃^t⟩_+ ≤ C_fo(α)∥p̃^{t+1}−p̃^t∥². This is a nonstandard and nontrivial condition; the paper calls it 'reasonable' but neither derives it from the transformer/LoRA architecture nor verifies it empirically. If it fails, the cumulative drift D_T(α) is no longer upper bounded by a constant times Σ η_t², so the proof's rate does not follow. The assumption needs a derivation or a numerical check, and Theorem 1 should list it explicitly as a condition that must be verified.
minor comments (5)
  1. [Section II.C] The notation p̃^t is used in Assumption 5 and Lemma 4 without a formal definition; define p̃^t=(p^t_0,...,p^t_K) and state its domain explicitly.
  2. [Section II.C, Lemma 2] The statement that p*^t_k(α) is the minimizer of L^t_k(p;α) is asserted 'with proof omitted here'; since this is a short convexity argument, it should be included or referenced.
  3. [Section IV.D and Table II] The heading 'Andriod Phones' should be 'Android Phones,' and Fig. 3's 'negative-likelihood loss' should be 'negative log-likelihood loss.'
  4. [Fig. 1 caption] The caption contains 'microbathch' and 'repetitively,' which should be 'microbatch' and 'respectively.'
  5. [Section IV.C] The recovery-latency definition says 'the during between,' which should be 'the duration between.'

Circularity Check

2 steps flagged · score 5.0 of 10

Theorem 1's convergence rate is for a constructed sum of local losses, and the deployed cross-entropy loss is not the reverse-KL objective analyzed, so the BP-comparable rate is not established for ZeroLock as implemented.

  1. self definitional [Section II.C, Eq. (19), Lemma 3, Theorem 1]
    "Based on Lemma 2 and pt_k = f^k(pt_{k−1};ωt_k), minimizing Lt_k(p;α) for all chunks over dataset D is equivalent to finding the parameter ω that minimizes R^t(ω;α)=E_{(x,y)∼D}[Σ_{k=1}^K D_KL(f^k(pt_{k−1};ω_k)∥p^{*,t}_k(α))]. ... Lemma 3 (Global Update Equivalence). Based on Lemma 2, the update rule of ZeroLock can be represented as follows: ω^{t+1}←ω^t−η_t∇_ω R^t(ω^t;α)."

    R^t is by construction the sum of the chunk-local objectives: Lemma 2 showed each local loss equals D_KL(·∥p*_k) up to an additive constant that does not affect the minimizer. Lemma 3 then represents the local update rule as gradient descent on this same sum, and Theorem 1 bounds ∥∇R^t∥². This is convergence on the constructed local-objective sum, not on the task loss L(p_K) used in Proposition 1; the paper's 'global convergence' and 'differs from BP' conclusion are therefore statements about the objective it defined, not an independent external benchmark.

  2. other [Section II.A Eq. (4) and Section II.C Eq. (14)]
    "Importantly, since minimizing KL divergence is equivalent to minimizing cross-entropy, (4) can be replaced with cross-entropy loss in practice. ... For analytical simplicity, we set the Bregman divergence in (4) and (5) as KL divergence. Then, the local loss is equivalent to Lt_k(p;α)=αD_KL(p∥p_y)+(1−α)D_KL(p∥pt_{k−1}). (14)"

    Cross-entropy is D_KL(p_y∥p)+const (forward KL), not D_KL(p∥p_y) (reverse KL). With one-hot p_y, D_KL(p∥p_y)=∞ for any p with probability outside the target token, so Eq. (14) is degenerate and Eq. (15) collapses to p*=p_y, removing the consistency term's role. The prototype's loss is cross-entropy and its curves in Fig. 3 depend on α. Hence Theorem 1's rate is for the surrogate reverse-KL objective, not for the algorithm actually implemented; the claimed first-principles prediction about ZeroLock does not follow from the analyzed objective.

full rationale

The algebra in Lemmas 1-4 and Theorem 1 is self-contained and checkable; the self-citations to [18] and [31] are not load-bearing in the proof, since the assumptions and propositions are stated and derived in the text. The circularity, insofar as it exists, is at the level of what the theorem is about: R^t is defined as the sum of the local losses, so Lemma 3's 'global update equivalence' is a relabeling of the local update rule, and Theorem 1 proves convergence of that constructed objective. Additionally, the theoretical objective D_KL(p∥p_y) is not the deployed cross-entropy, so the proven rate cannot be transferred to the implemented ZeroLock loss. These issues affect the central claim (BP-comparable convergence for ZeroLock), but they are not hidden fits or imported uniqueness theorems; they are definitional choices and an objective mismatch. Score 5 reflects partial circularity rather than full collapse.

Assumptions & free parameters 2 free parameters · 7 assumptions · 0 invented entities

The central theoretical claims rest on several explicit or implicit premises. Alpha and LoRA ranks are hand-chosen hyperparameters. Assumptions 1-4 and 6 are standard optimization conditions, but they are stated, not verified for TinyLlama. Assumption 5 is nonstandard and load-bearing. The readout-head interpretability premise is necessary for local losses to be meaningful. The implicit finiteness of D_KL(p||p_y) for one-hot targets is violated by the actual cross-entropy implementation, and this mismatch undermines Lemma 2.

free parameters (2)
  • alpha (α) = 0.5 default; 1.0 ablation
    Hand-chosen weight between task and consistency terms in Eq (3). The bound in Proposition 1 requires α < 1/β, and the geometric-mean target in Lemma 2 depends on α.
  • LoRA rank and scaling factor = rank 4, scale 16 (server); rank 8, scale 16 (mobile)
    Hyperparameters for the LoRA adapters. They affect model capacity and all reported efficiency numbers but are not fitted in the theory.
assumptions (7)
  • standard math Bregman divergence properties and KKT conditions for simplex-constrained optimization
    Used in Lemma 1 and Proposition 1 proofs; standard mathematical tools.
  • domain assumption Assumption 1: global objective L is β-smooth relative to ψ
    Invoked in Proposition 1 and the chunkwise inequality in Eq (11); plausible but not verified for LLM fine-tuning.
  • domain assumption Assumptions 2-4: bounded gradient, Lipschitz readout, and smoothness of R^t
    Standard SGD analysis conditions, stated but not verified for tinyLlama or the constructed objective.
  • ad hoc to paper Assumption 5: first-order variation of R^t along p~t is quadratically controlled
    Nonstandard, labeled 'reasonable', and load-bearing for Lemma 4 and Theorem 1. Not derived or empirically checked.
  • standard math Assumption 6: mini-batch sampling is unbiased with bounded variance
    Standard SGD sampling assumption, reasonable under random mini-batching.
  • domain assumption Each chunk's readout head output p_k is a meaningful next-token distribution for the task loss
    The entire local-objective construction depends on intermediate hidden states being interpretable by the frozen head or a small adapter; stated in Section III.B but not theoretically justified.
  • ad hoc to paper D_ψ(p, p_y) is finite for one-hot target p_y (with KL this is D_KL(p||p_y))
    Implicit in Eq (4) and Lemma 2. This fails for one-hot targets under standard KL, and the implemented cross-entropy loss is not the same objective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling." pith.science (2026). https://pith.science/paper/2VIZR77Y

@misc{pith2026260807974,
  author       = {Pith},
  title        = {Pith review of: ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2VIZR77Y}},
  note         = {Machine review of arXiv:2608.07974}
}
abstract

Large language model (LLM) fine-tuning at the edge adapts the model to scenario-specific data while preserving privacy. Although existing studies proposed pipeline parallelism to address the limited memory and computing resources of edge devices, they commonly rely on backpropagation (BP) training, which has a fundamental limitation of update locking and could experience severe throughput and memory bottlenecks. In this work, we propose a BP-free algorithm, called ZeroLock, that decouples the model updates into independent chunk updates by local objective construction. It breaks the update locking of BP and hence can improve throughput at the algorithm level and lower memory usage by reducing activation storage. To the best of our knowledge, we provide the first theoretical framework for such local objective construction-based approaches under general model chunk division by mapping local objectives to the global objective. We prove that ZeroLock has a convergence rate of $\tilde{\mathcal{O}}(1/\sqrt{T})$, which differs from BP only by polylogarithmic factors. We design a system for ZeroLock and build real-world prototypes, incorporating techniques such as early forwarding and failure recovery for efficient and robust implementation. Experiments on the prototype show that compared to BP-based baselines, ZeroLock reduces the memory by 26.5% and improves throughput by 4.9%.

Figures

Figures reproduced from arXiv: 2608.07974 by the authors.

Figure 1
Figure 1. Throughput with (a) BP baseline and (b) ZeroLock; memory with (c) BP [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. System overview with one coordinator and multiple executors, each [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. (a) Accuracy and (b) negative-likelihood loss. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: (a) and (c), maximum stage memory usage under [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 4
Figure 4. Figure 4: Peak GPU memory usage and the memory usage of each component [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 7
Figure 7. Figure 7: CUDA activity: (a) GPipe; (b) 1F1B; (c) PipeDream; (d) ZeroLock. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 27 canonical work pages

  1. [18]

    Depth-progressive monotonic learning without global backpropagation,

    C. Ye, R. Ye, Y . Zhang, and M. Tang, “Depth-progressive monotonic learning without global backpropagation,” inProc. ICML, 2026

    work page 2026

  2. [1]

    End-to-end test-time training for long context,

    A. Tandon, K. Dalal, X. Li, D. Koceja, M. Rød, S. Buchanan, X. Wang, J. Leskovec, S. Koyejo, T. Hashimotoet al., “End-to-end test-time training for long context,”arXiv preprint arXiv:2512.23675, 2025

    arXiv 2025

  3. [2]

    Cross-subject eeg signals-based emotion recognition using contrastive learning,

    A. M. Alghamdi, M. U. Ashraf, A. A. Bahaddad, K. A. Almarhabi, W. A. Al Shehri, and A. Daraz, “Cross-subject eeg signals-based emotion recognition using contrastive learning,”Scientific Reports, vol. 15, no. 1, p. 28295, 2025

    work page 2025

  4. [3]

    GPipe: Efficient training of giant neural networks using pipeline parallelism,

    Y . Huang, Y . Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam, Q. V . Le, Y . Wu, and Z. Chen, “GPipe: Efficient training of giant neural networks using pipeline parallelism,” inProc. NeurIPS, Vancouver, BC, Canada, Dec. 2019

    work page 2019

  5. [4]

    PipeDream: Generalized pipeline parallelism for DNN training,

    D. Narayanan, A. Harlap, A. Phanishayee, V . Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “PipeDream: Generalized pipeline parallelism for DNN training,” inProc. ACM SOSP, Huntsville, ON, Canada, Oct. 2019

    work page 2019

  6. [5]

    Confidant: Customizing transformer-based llms via collaborative training on mobile devices,

    Y . Chen, Y . Yan, S. Ge, Y . Qin, Y . Zheng, Q. Yang, S. He, Z. Shi, J. Chen, and Y . Shu, “Confidant: Customizing transformer-based llms via collaborative training on mobile devices,” inProc. ACM MobiCom, 2025

    work page 2025

  7. [6]

    Memory-efficient pipeline-parallel DNN training,

    D. Narayanan, A. Phanishayee, K. Shi, X. Chen, and M. Zaharia, “Memory-efficient pipeline-parallel DNN training,” inProc. ICML, Virtual Conf., Jul. 2021

    work page 2021

  8. [7]

    DAPPLE: A pipelined data parallel approach for training large models,

    S. Fan, Y . Rong, C. Meng, Z. Cao, S. Wang, Z. Zheng, C. Wu, G. Long, J. Yang, L. Xia, L. Diao, X. Liu, and W. Lin, “DAPPLE: A pipelined data parallel approach for training large models,” inProc. ACM PPoPP, Virtual Conf., Feb.–Mar. 2021

    work page 2021

Show all 31 references
  1. [8]

    Efficient large-scale language model training on GPU clusters using Megatron-LM,

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V . A. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on GPU clusters using Megatron-LM,” inProc. ACM/IEEE SC, S...

    2021

  2. [9]

    Chimera: Efficiently training large-scale neural networks with bidirectional pipelines,

    S. Li and T. Hoefler, “Chimera: Efficiently training large-scale neural networks with bidirectional pipelines,” inProc. ACM/IEEE SC, St. Louis, MO, USA, Nov. 2021

    2021

  3. [10]

    Zero bubble (almost) pipeline parallelism,

    P. Qi, X. Wan, G. Huang, and M. Lin, “Zero bubble (almost) pipeline parallelism,” inProc. ICLR, Vienna, Austria, May 2024

    2024

  4. [11]

    Universal checkpointing: A flexible and efficient distributed checkpointing system for large-scale DNN training with reconfigurable parallelism,

    X. Lian, S. A. Jacobs, L. Kurilenko, M. Tanaka, S. Bekman, O. Ruwase, and M. Zhang, “Universal checkpointing: A flexible and efficient distributed checkpointing system for large-scale DNN training with reconfigurable parallelism,” inProc. USENIX ATC, Boston, MA, USA, Jul. 2025

    2025

  5. [12]

    Sparse checkpointing for fast and reliable MoE training,

    S. Gandhi and C. Kozyrakis, “Sparse checkpointing for fast and reliable MoE training,” inProc. USENIX NSDI, Renton, W A, USA, May 2026

    2026

  6. [13]

    Attack of the bubbles: Straggler-resilient pipeline parallelism for large model training,

    T. Wu, L. Cao, H. Lu, X. Jiang, Y . Yu, S. Yang, G. Yang, J. Wang, L. Qu, L. Zhang, and W. Wang, “Attack of the bubbles: Straggler-resilient pipeline parallelism for large model training,” inProc. USENIX NSDI, Renton, W A, USA, May 2026

    2026

  7. [14]

    CollaPipe: Adaptive segment-optimized pipeline parallelism for collabo- rative LLM training in heterogeneous edge networks,

    J. Chen, X. Deng, Z. Xiong, S. Guo, X. Qiu, P. Wang, and D. Niyato, “CollaPipe: Adaptive segment-optimized pipeline parallelism for collabo- rative LLM training in heterogeneous edge networks,”arXiv:2509.19855, Sep. 2025

    2025

  8. [15]

    SWARM parallelism: Training large models can be surprisingly communication- efficient,

    M. Ryabinin, T. Dettmers, M. Diskin, and A. Borzunov, “SWARM parallelism: Training large models can be surprisingly communication- efficient,” inProc. ICML, Honolulu, HI, USA, Jul. 2023

    2023

  9. [16]

    Petals: Collaborative infer- ence and fine-tuning of large models,

    A. Borzunov, D. Baranchuk, T. Dettmers, M. Ryabinin, Y . Belkada, A. Chumachenko, P. Samygin, and C. Raffel, “Petals: Collaborative infer- ence and fine-tuning of large models,” inProc. ACL Syst. Demonstrations, Toronto, ON, Canada, Jul. 2023

    2023

  10. [17]

    Decoupled parallel backpropagation with convergence guarantee,

    Z. Huo, B. Gu, qian Yang, and H. Huang, “Decoupled parallel backpropagation with convergence guarantee,” inProc. ICML, Jul. 2018

    2018

  11. [19]

    Beyond- backpropagation training: Methods, applications, and perspectives,

    R. Ye, C. Ye, C. Huang, M. Tang, and Y . Liu, “Beyond- backpropagation training: Methods, applications, and perspectives,” TechRxiv, vol. 2026, no. 0103, 2026. [Online]. Available: https: //www.techrxiv.org/doi/abs/10.36227/techrxiv.176740426.63642005/v1

    2026

  12. [20]

    Direct feedback alignment provides learning in deep neural networks,

    A. Nøkland, “Direct feedback alignment provides learning in deep neural networks,”Proc. NeurIPS, 2016

    2016

  13. [21]

    Deep learning without weight transport,

    M. Akrout, C. Wilson, P. Humphreys, T. Lillicrap, and D. B. Tweed, “Deep learning without weight transport,” inProc. NeurIPS, vol. 32, 2019

    2019

  14. [22]

    Fine-tuning language models with just forward passes,

    S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora, “Fine-tuning language models with just forward passes,” in Proc. NeurIPS, 2023

    2023

  15. [23]

    Curvzo: Adaptive curvature-guided sparse zeroth-order optimization for efficient llm fine-tuning,

    S. Wang, Z. Chen, and M. Tang, “Curvzo: Adaptive curvature-guided sparse zeroth-order optimization for efficient llm fine-tuning,” inProc. ICML, 2026

    2026

  16. [24]

    Noprop: Training neural networks without back-propagation or forward-propagation,

    Q. Li, Y . W. Teh, and R. Pascanu, “Noprop: Training neural networks without back-propagation or forward-propagation,” inConference on Lifelong Learning Agents. PMLR, 2026, pp. 525–544

    2026

  17. [25]

    Predictive coding approximates backprop along arbitrary computation graphs,

    B. Millidge, A. Tschantz, and C. L. Buckley, “Predictive coding approximates backprop along arbitrary computation graphs,”Neural Computation, vol. 34, no. 6, pp. 1329–1368, 2022

    2022

  18. [26]

    Faster multi-GPU training with PPLL: A pipeline parallelism framework leveraging local learning,

    X. Guo, C. Xu, G. Guo, F. Zhu, C. Cai, P. Wang, X. Wei, J. Su, and J. Gao, “Faster multi-GPU training with PPLL: A pipeline parallelism framework leveraging local learning,”arXiv:2411.12780, Nov. 2024

    2024 arXiv

  19. [27]

    Idle no more: Boosting distributed pipeline training via FluidPipe,

    M. Aljahdali and M. Canini, “Idle no more: Boosting distributed pipeline training via FluidPipe,” inProc. IEEE ICDCSW, Glasgow, U.K., Jul. 2025

    2025

  20. [28]

    Scpl: Enhancing neural network training throughput with decoupled local losses and model parallelism,

    M.-Y . Ho, C.-K. Wang, Y .-T. Lin, and H.-H. Chen, “Scpl: Enhancing neural network training throughput with decoupled local losses and model parallelism,”ACM Trans. Manage. Inf. Syst., vol. 17, no. 2, Jun. 2026

    2026

  21. [29]

    Rethinking local learning: A cheaper and faster recipe for LLM post-training,

    H. Shi, T. Han, P. Wang, Z. Wang, X. Yang, and J. Su, “Rethinking local learning: A cheaper and faster recipe for LLM post-training,” arXiv:2605.04913, May 2026

    2026 arXiv

  22. [30]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inProc. ICLR, Virtual Conf., Apr. 2022

    2022

  23. [31]

    Convergence analysis of split federated learning on heterogeneous data,

    P. Han, C. Huang, G. Tian, M. Tang, and X. Liu, “Convergence analysis of split federated learning on heterogeneous data,”Advances in Neural Information Processing Systems, vol. 37, pp. 103 476–103 544, 2024

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.