Lion Cub: Minimizing Communication Overhead in Distributed Lion

Brian Van Essen; Nikoli Dryden; Rio Yokota; Satoki Ishikawa; Tal Ben-Nun

REVIEW 3 major objections 7 minor 13 references

Lion Cub: Minimizing Communication Overhead in Distributed Lion

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Lion Cub shows that the Lion optimizer's sign-based updates can be quantized and momentum synchronization mostly skipped, cutting distributed training's communication enough for up to 5.1x end-to-end speedups while matching full-precision…

desk verdict A solid systems paper on compressing Lion's update vector: the L1 quantization is genuinely new, but the 5–5.1x speedup headline is not backed by the paper's own displayed numbers. read the letter →

arxiv 2411.16462 v2 pith:RSGXXTLR submitted 2024-11-25 cs.LG cs.DC

Satoki Ishikawa , Tal Ben-Nun , Brian Van Essen , Rio Yokota , Nikoli Dryden This is my paper · ORCID

classification cs.LGcs.DC

keywords distributeddeeplearningLionoptimizercommunicationcompressiongradientquantizationmomentumsynchronizationallreduceheavy-tailedgradientsGPTtraining

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Distributed deep learning spends a large and growing share of its time moving data between workers, especially over Ethernet-class networks. This paper argues that the Lion optimizer is unusually well suited to fixing that: its update vectors are already the output of a sign operation, so they can be communicated in far fewer bits. The authors combine an optimized 1-bit allreduce, a new L1 quantization that survives heavy-tailed updates, and selective momentum synchronization into a method they call Lion Cub. Across GPT-style language models and vision models, Lion Cub preserves convergence while cutting communication volume enough to deliver up to 5.1x faster end-to-end training in low-bandwidth settings. The practical claim is that Lion can become a communication-efficient default for distributed training, not just a quality-competitive optimizer.

What carries the argument

Lion Cub's central object is Lion's update vector before the final sign: each worker forms $c_{i,t}=\beta_1 m_{i,t-1}+(1-\beta_1)\nabla f_i(\theta_{t-1})$ and the only thing workers exchange is a quantized sum of these vectors, after which a global sign is taken. The paper contributes three mechanisms around that object. First, a 1-bit compressed allreduce tailored to majority voting, where signs are packed as 0/1 and exact zeros are forced to $+1$ or $-1$ on alternating iterations to stop error accumulation. Second, L1 quantization, $Q_1(x)=\mathrm{clamp}(\mathrm{round}((2^{n-1}-1)x/M_1(x)),2^{n-1}-1)$ with $M_1$ the mean absolute value, chosen because standard $\ell^\infty$ normalization collapses on Lion's heavy-tailed updates. Third, an empirical finding that inter-worker momentum variance concentrates in the input embedding and output layers, so momentum can be synchronized only there or not at all. The $\alpha$-$\beta$ communication-cost model ties these together, showing when each allreduce variant wins.

What would settle it

A decisive test would be to train the same GPT architecture with a small vocabulary or tied input/output embeddings, so the first and last layers no longer dominate momentum variance, and check whether Lion Cub's selective momentum synchronization still matches full-precision Lion's validation loss; if variance appears in middle layers and validation loss diverges, the momentum-sync premise fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the Lion optimizer can be made communication-efficient without sacrificing its convergence behavior. Lion's pre-sign update vectors are heavy-tailed rather than uniform, so the standard max-norm quantizer $Q_\infty$ collapses most entries to zero; the authors replace it with L1 quantization, normalizing by the mean absolute value and clamping, which keeps the sign of most updates intact. On the communication side they implement a 1-bit compressed allreduce and a p-bit allreduce tailored to the sign structure, and they show that exact zeros must be represented (for instance by alternating sign) or training diverges. For momentum, they find that worker-to-worker variance is concentrated almost entirely in the input embedding and output layers, so that with a sufficiently large $\beta_2$ momentum need not be synchronized at all, and with smaller $\beta_2$ synchronizing only those layers suffices. Combined as Lion Cub, these techniques deliver up to 5.1x faster wall-clock training than standard Lion in a 1 Gbps Ethernet setting while matching its validation loss or accuracy on GPT-style, ResNet, and DeiT models.

Load-bearing premise

The load-bearing premise is empirical: that worker-to-worker variation in Lion's momentum is concentrated almost entirely in the input embedding and output layers, so that skipping most momentum synchronization or limiting it to those layers preserves convergence across architectures, depths, and data shardings.

Editorial extensions

If this is right

On 1 Gbps Ethernet, Lion Cub cuts the communication share of training time from about 74% to roughly half that, and end-to-end training is up to 5.1x faster than standard Lion while matching validation loss or accuracy.
With a large enough $\beta_2$ (e.g., 0.99), distributed Lion can run with no momentum synchronization at all; with smaller $\beta_2$, synchronizing only the input and output layers is enough to maintain convergence.
Standard max-norm quantization is a poor fit for Lion's heavy-tailed update distribution, and L1 quantization restores convergence where 1-bit and $\ell^\infty$ quantization diverge or degrade.
With more than a handful of workers, the bit width needed to represent a majority vote grows, so 1-bit Lion Cub can become slower than a 4-bit or 8-bit p-bit allreduce; the paper's guidance is to use wider L1 quantization when the worker count is large.
Even on high-bandwidth InfiniBand, Lion Cub reduces iteration time compared to standard Lion, although the gains are smaller than on Ethernet.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the layerwise momentum-variance pattern generalizes, one could precompute a static communication mask that never syncs middle layers, saving decision overhead and making the method trivially compatible with federated or edge settings where only selected layers are transmitted.
L1 quantization can be read as an implicit, automatic gradient-clipping step (normalize by mean absolute value, then clamp); a natural next test is whether Lion Cub's empirical stability transfers to settings with provably heavy-tailed noise, such as federated learning.
The tie-handling rule (forcing zeros to alternate signs) suggests that 1-bit Lion Cub's stability depends on the fraction of tied majority votes; a sweep across vocabularies and shard counts could predict precisely when the 2-bit fallback becomes necessary.
Because the paper's experiments use relatively homogeneous data shards and small clusters, the most direct open test is whether the input/output-layer concentration of momentum variance persists under heterogeneous data and at 100+ workers; if not, selective synchronization would need to become adaptive rather than static.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Punchline: this is a solid systems paper. The genuinely new piece is L1 quantization of Lion's update vectors, motivated by the heavy-tailed distribution of updates; the layerwise momentum variance analysis is also useful. The 5.1x speedup headline, though, is not backed by the paper's own displayed numbers. In the Ethernet 1.5B case the largest ratio is 2600/574 = 4.53x, against a 16-bit Lion baseline, not the 32-bit "standard Lion" referenced in the claim. That overstatement is load-bearing because it is the abstract's central promise.

What the paper does well: it gives a clear cost model comparing parameter-server, direct allreduce, and 1-bit compressed allreduce for majority voting; the L1 quantization is simple, efficient, and consistently improves sign-matching over L∞ and logarithmic schemes; and the observation that momentum synchronization can be restricted to input/output layers at large β2 is empirically convincing and practically valuable. The authors also disclose their main limitations: modest scale, no convergence theory, and the fact that 1-bit packing overhead can eat the gains.

Soft spots: single seeds, no error bars, no released code, and no convergence theory—all acknowledged. The selective-momentum-sync recipe rests on the assumption that variance concentrates in input/output layers, validated for two architectures and one data-sharding scheme; that generalization is the weakest empirical link. The L1 choice was partly made after seeing it win on the same dataset families, though it is then tested on held-out tasks, so the circularity is minor. Finally, the 5.1x claim appears to be an arithmetic or reporting error; at minimum it needs a 32-bit baseline for the 1.5B model.

Who should read: anyone working on communication-efficient training for low-bandwidth clusters. It deserves a serious referee: the core idea is sound, the evaluation is broad, and the flaws are fixable. I would send it out with a request for code, corrected speedup numbers, and preferably multi-seed variance.

Referee Report

3 major / 7 minor

Summary. The paper studies communication reduction for the Lion optimizer in distributed data-parallel training. It proposes (i) 1-bit and p-bit allreduce implementations specialized for sign-based Lion updates, (ii) an L1 quantization scheme for the update vector that is robust to the heavy-tailed distributions observed in Lion updates, and (iii) selective or deferred momentum synchronization based on layerwise measurements of inter-worker momentum variance. The method, called Lion Cub, is evaluated on GPT models (350M–1558M), ResNet-18/50, and DeiT variants under 1 Gbps Ethernet and 200 Gbps InfiniBand, with the reported headline of up to 5.1x end-to-end speedups over standard Lion in low-bandwidth settings. The paper also states limitations: no large-scale confirmation and no formal convergence guarantees.

Significance. If the empirical claims hold, the paper makes a useful practical contribution to low-bandwidth distributed training. Its strengths are the communication-cost analysis of majority-vote implementations for Lion, the systematic study of the update-vector distribution, the layerwise momentum-variance analysis, and the breadth of evaluated architectures. The paper does not provide machine-checked proofs or release code, and the central speedup claim is not consistently supported by the displayed timing data. The momentum-synchronization guidance is an empirical generalization over a narrow set of configurations. The core ideas are plausible and worth publishing after the specific claims are corrected and the evidence is tightened.

major comments (3)

[§6.1, Fig. 15] The claimed "up to 5.1× speedups" is not supported by the paper's own timing data. The largest Ethernet ratio visible in Fig. 15 is GPT-1558M: 2600 ms / 574 ms = 4.53×, and the baseline there is labeled "Lion (16bit)", not the 32-bit "standard Lion" used in the claim. Other displayed ratios are lower: GPT-774M on 4 nodes is 2033/701 = 2.90×, GPT-350M is 672/242 = 2.78×, and Table 4's 1 Gbps column gives at most 1.0/0.3 = 3.3×. The abstract's "up to 5×" and §6.1's "up to 5.1×" therefore need an explicit measurement of the 32-bit Lion baseline for the 1558M Ethernet configuration, or a correction of the headline to a number directly supported by the data.
[§4.2 and App. C.2] The contribution bullet in §1 states that L1 quantization "offers superior performance", but the paper's own appendix contradicts this. App. C.2 reports that on ImageNet, L0 quantization achieves accuracy on par with or slightly above L1, making L0 the best-performing scheme, and that on CIFAR5m at learning rate 6e-5, L0 surpasses both L1 and standard Lion. The main text should either restrict the L1 claim to the GPT/low-bandwidth regime where it is demonstrated, or present a careful analysis of when L1 is preferable to L0 rather than asserting general superiority.
[§5.1, Fig. 6, Fig. 16] The recommendation to eliminate or restrict momentum synchronization rests on the empirical premise that inter-worker momentum variance is concentrated in the input embedding and output layers, based on GPT-730M and ResNet-18 with one data-sharding scheme. The paper itself acknowledges in §7 and App. A.2 that there are no formal convergence guarantees for the compressed or partially synchronized variants. If a different architecture or sharding spreads momentum variance more broadly across layers, the "no sync" or "Input+Output" variants of Lion Cub could diverge. This load-bearing generalization should be tested on more depths/shardings or the recommendation should be softened accordingly.

minor comments (7)

[App. D.2] The text refers to "Figure hoge" as a placeholder; this must be replaced with the actual figure number for the 735M and 1.5B GPT experiments.
[App. E.1] The sentence "The results are presented in Figure 4" appears to reference the wrong figure; the three-bandwidth timing results are described in §6.1 and Table 4, not in Figure 4.
[App. C.2] The sentence "As shown in Table 3, L1 quantization yields the lowest training loss during GPT pre-training" should refer to Table 2, which reports validation loss for different quantization functions; Table 3 reports majority-vote sign statistics.
[§6.1] The text says "when training the 730M parameter model on 2 nodes", but the surrounding experiments and figures use 774M (and App. D.2 uses 735M); the model size should be made consistent.
[§5.2] There is a typo in "a uint8 can can only support 0/1 summation"; the duplicated "can" should be removed.
[Eq. (1)] The rounding operator in Eq. (1) is written as "round" without specifying whether it is deterministic or stochastic; since §2 defines stochastic rounding for the standard Q∞ quantizer, the comparison in Table 2 and Fig. 4 should use the same rounding convention for both schemes.
[Table 4] Table 4 does not state which model and batch size the per-step times refer to, and the unit "1step (s)" is written inconsistently relative to the milliseconds used in Fig. 15; please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the quantization and momentum-synchronization results are empirical evaluations, not conclusions built into their definitions or inputs.

full rationale

The paper's claimed derivation chain is not circular. The L1 quantizer (Eq. 1) is defined independently of the convergence results, and its adoption is justified by direct comparisons (Table 2, Figures 4 and 12) against 1-bit, L0, and L∞ alternatives; the paper discloses that L0 sometimes matches or beats L1 (ImageNet, Fig. 13; CIFAR5m, Fig. 14), which shows the choice is not forced by the method's definition. The momentum-synchronization strategy is motivated by direct measurements of layerwise inter-worker momentum variance (Figs. 6, 16, and 17) and then tested (Fig. 5), rather than being assumed from the variance data or from a fitted parameter. Citations to external prior work (Liu et al. 2024, Tang et al. 2021, Chen et al. 2024) supply the baseline algorithm and compressed-allreduce primitive; they are not self-citations carrying the load. The only in-paper caveats are acknowledged limitations: Sec. 7 says large-scale confirmation is pending, and App. A.2 says no formal convergence guarantees are provided. Whether the reported 'up to 5.1x' speedup is fully supported by the displayed timing figures is a measurement-reporting concern, not a circularity concern, since the timing numbers are measured outcomes rather than consequences of the method's definitions. The choice of L1 over L0 after observing results on the same task family is a model-selection consideration, not a definitional or fitted-parameter circularity, and both variants are reported. No step reduces, by construction or by self-citation, to its own inputs.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two empirical distributional assumptions (heavy-tailed updates, layer-concentrated momentum variance) and two ad hoc techniques (alternating-sign tie-breaking, chosen sync interval). There are no fitted constants in the quantization itself, but the sync interval and the selective layer choices are hand-set and not ablated.

free parameters (1)

momentum_sync_interval = 10 iterations
The paper synchronizes momentum or the head layer every 10 iterations in the beta2=0.95 experiments (Figures 5 and 8). The interval is chosen by hand, no ablation of the interval is reported, and the convergence results for the sync variants depend on this unexamined constant.

assumptions (5)

standard math Alpha-beta communication cost model
Used in Table 1 and Figure 2 to compare parameter server, direct allreduce, and 1-bit compressed allreduce costs; standard in the collective-communication literature (Thakur et al. 2005, Chan et al. 2007).
domain assumption Lion update vectors are heavy-tailed, approximately Laplace distributed
Figure 3 shows the update vector distribution for a GPT model; this motivates why Q-infinity fails and L1 normalization helps. If the distribution were closer to uniform, the paper's central quantization claim would not hold.
domain assumption Clipping before quantization compensates heavy-tailed gradient noise
Appendix A.2 argues L1 quantization behaves like clipping and cites FAT-Clipping, EPISODE, and TQSGD for the benefits; the paper does not prove this for Lion.
ad hoc to paper Inter-worker momentum variance is concentrated in input and output layers
Established only by variance measurements in Figure 6 (GPT) and Figure 16 (ResNet18); this underpins the selective momentum-synchronization strategy and is the load-bearing empirical premise for skipping all-layer sync.
ad hoc to paper Alternating-sign tie-breaking prevents 1-bit error accumulation
Figure 10 shows naive 1-bit allreduce diverges while alternating 0 to +1 or -1 on odd and even iterations stabilizes training; no theory explains why the alternating rule avoids accumulation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Lion Cub: Minimizing Communication Overhead in Distributed Lion." pith.science (2026). https://pith.science/paper/RSGXXTLR

@misc{pith2026241116462,
  author       = {Pith},
  title        = {Pith review of: Lion Cub: Minimizing Communication Overhead in Distributed Lion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RSGXXTLR}},
  note         = {Machine review of arXiv:2411.16462}
}

read the original abstract

Communication overhead is a key challenge in distributed deep learning, especially on slower Ethernet interconnects, and given current hardware trends, communication is likely to become a major bottleneck. While gradient compression techniques have been explored for SGD and Adam, the Lion optimizer has the distinct advantage that its update vectors are the output of a sign operation, enabling straightforward quantization. However, simply compressing updates for communication and using techniques like majority voting fails to lead to end-to-end speedups due to inefficient communication algorithms and reduced convergence. We analyze three factors critical to distributed learning with Lion: optimizing communication methods, identifying effective quantization methods, and assessing the necessity of momentum synchronization. Our findings show that quantization techniques adapted to Lion and selective momentum synchronization can significantly reduce communication costs while maintaining convergence. We combine these into Lion Cub, which enables up to 5x speedups in end-to-end training compared to Lion. This highlights Lion's potential as a communication-efficient solution for distributed training.

Figures

Figures reproduced from arXiv: 2411.16462 by the authors.

**Figure 1.** Overview of our communication algorithm. We propose two synchronization methods for Lion’s update vector: 1-bit allreduce and p-bit allreduce. 1-bit allreduce provides fast communication but may compromise convergence. Additionally, the frequent packing and unpacking of 1-bit data can introduce overhead. In contrast, p-bit allreduce requires more communication time than 1-bit allreduce but typically converges more q… view at source ↗

**Figure 3.** The distribution of the update vector is not uniform. The distribution represents the values of the update vector before quantization (sign). Left: The values do not follow a uniform distribution; rather, they approximate a Laplace distribution (not a Gaussian distribution). Right: The norm of the update vector is heavy-tailed, with the maximum significantly exceeding the mean. In a direct allreduce implementation… view at source ↗

**Figure 2.** Communication costs with our performance model. We illustrate the communication costs from [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Figures from the paper (11 more)

**Figure 4.** Figure 4: shows the accuracy of quantizing Lion updates with different methods in terms of whether the signs of the updates match or flip (introducing error). Standard Q∞ stochastic quantization clips most values to zero, resulting in a low match and flip rate. One approach to a…

**Figure 5.** Figure 5: (Left) Momentum synchronization or L1 quantization is necessary when β2 is small. Momentum is never synchronized for “None”; all other settings synchronize momentum every 10 iterations. When β2 is large, gradients are stable across workers, and synchronization is not…

**Figure 6.** Figure 6: There is significant variance between workers for the first (embedding) and last layers. We plot the maximum standard deviation between workers for each layer, observed for each element, while training GPT (730M) on OpenWebText. synchronization is not required to main…

**Figure 8.** Figure 8: Validation loss with GPT (774M) when training with 8 workers on 2 nodes in our Ethernet configuration. Lion Cub reduces training time compared to standard Lion and can maintain comparable convergence. Note in 4-bit Lion Cub, workers only send 1/-1 values, whereas in 8-…

**Figure 9.** Figure 9: Validation accuracy for training ResNet-18 on CIFAR100 and fine-tuning DeiT-Tiny on CIFAR100 using 8 workers on 2 nodes in our lower-bandwidth configuration. Interestingly, if you look at the training curve per iteration, even 1-bit allreduce closely mirrors Lion with…

**Figure 10.** Figure 10: 1-bit allreduces, unable to represent exact zeros, diverge without adjustments. The naive implementation of Lion using a 1-bit allreduce, which does not transmit exact zeros, leads to error accumulation and causes the loss to diverge. However, when zeros are converted…

**Figure 11.** Figure 11: Smaller p values enhance LpQuantization under heavy-tailed noise. 0 2000 4000 iter 70 80 90 100 Accuracy Rate Sign Matching Normal D-Lion(β2 =0.95) L1 Quantization(β2 =0.95) Normal D-Lion(β2 =0.99) L1 Quantization(β2 =0.99) 0 2000 4000 iter 0.0 2.5 5.0 7.5 Error Rate …

**Figure 14.** Figure 14: CIFAR ablation study of quantization function. Accuracy of ResNet-18 on CIFAR100 and CIFAR5m using L∞, L0, and L1 quantization. While all three schemes yield nearly identical results on CIFAR100, the choice of quantization method produces clear performance differences…

**Figure 15.** Figure 15: Breakdown of runtime performance when training GPT. Lion Cub significantly reduces communication volume, leading to reduced communication time and faster end-to-end iterations. “Comm” includes both communication time and quantization overheads. indicated by this graph…

**Figure 19.** Figure 19: Validation loss on OpenwebText with GPT-2(1.5B). We trained a 1.5B parameter GPT model on OpenWebText using a cluster equipped with 4 H100 GPUs per node. With low-latency environments, this approach significantly reduces training time. adapted it to Lion. We identifie…

**Figure 22.** Figure 22: Breakdown of runtime performance when training ResNet and DeiT. All results use 2 nodes and our “Ethernet” configuration and “Comm” includes both communication time and quantization overheads. Lion Cub significantly reduces communication and overall iteration time com…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

[3]

While this rate decreases with 8 workers (as ties are less likely), it is still significant

With 4 workers, over 30% of the parameters have an update where the majority vote is tied, leading to an erroneous update depending on how it is transmitted. While this rate decreases with 8 workers (as ties are less likely), it is still significant. Indeed, as Figure 10 demonstrates, these errors can lead to loss spikes and training divergence, and so ar...

work page
[4]

Note that when a high-speed interconnect (e.g

The 1-bit LionCub implementation uses an AlltoAll algorithm, which in some cases yields lower throughput than the 4-bit (and higher) LionCub variants that use AllReduce. Note that when a high-speed interconnect (e.g. 200 Gbps InfiniBand) is available, the cost of quantization and other computations can dominate overall performance. Conversely, when limite...

work page 2000
[5]

Nishino, R

URL https: //www.nextplatform.com/2024/06/02/ nvidia-unfolds-gpu-interconnect-roadmaps-out-to-2027/ . Nishino, R. and Loomis, S. H. C. CuPy: A NumPy- compatible library for NVIDIA GPU calculations. 31st conference on neural information processing systems, 151 (7),

work page 2024
[6]

Communication-efficient distributed deep learning: A comprehensive survey

Tang, Z., Shi, S., Wang, W., Li, B., and Chu, X. Communication-efficient distributed deep learning: A comprehensive survey. arXiv preprint arXiv:2003.06307,

arXiv 2003
[7]

Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning

Yan, G., Li, T., Xiao, Y ., Hou, H., and Song, L. Im- proved quantization strategies for managing heavy- tailed gradients in distributed learning. arXiv preprint arXiv:2402.01798,

work page Pith review arXiv
[11]

CIFAR5m Figure 14 presents an ablation study of quantization functions for ResNet-18 trained on CIFAR5m and CIFAR100

A more detailed analysis of the accuracy–efficiency trade-off between L0 and L1 quantization is left for future work. CIFAR5m Figure 14 presents an ablation study of quantization functions for ResNet-18 trained on CIFAR5m and CIFAR100. Note that CIFAR-5m consists of six million synthetic, CIFAR-10-style images generated by a DDPM-based model1. On CIFAR100...

work page 2000
[12]

We trained a 735M parameter GPT model on OpenWebText with 16 GPUs using a cluster equipped with 4 H100 GPUs per node

and 15 Lion Cub: Minimizing Communication Overhead in Distributed Lion 0 20000 40000 Training Time 3 × 100 4 × 100 5 × 100 Val Loss 2=0.99 Lion (32bit) Lion Cub (8bit) Lion Cub (8bit, w/o L1) Lion Cub (1bit) Figure 18: Validation loss on OpenWebText with GPT-2 (735MB) when training with 4 nodes. We trained a 735M parameter GPT model on OpenWebText with 16...

work page 2024
[280]

Ethernet

for 50,000 steps . We used the Lion optimizer with a base learning rate of 3e-5; in addition, we scaled the head, embedding, and scalar parameter groups to 3e-3, 3e-2, and 3e-3 respectively. The betas were set to ( β1 = 0.8, β2 = 0.95), weight decay to 1e-2, and no gradient clipping was applied. We used a training sequence length of 12,288 tokens and a va...

work page 2017

Show all 13 references

[2019]

Petals: Collaborative inference and fine-tuning of large models

Borzunov, A., Baranchuk, D., Dettmers, T., Ryabinin, M., Belkada, Y ., Chumachenko, A., Samygin, P., and Raffel, C. Petals: Collaborative inference and fine-tuning of large models. arXiv:2209.01188,

arXiv
[2020]

P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S

Zhang, J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 2020a. Zhang, Z., Chang, C., Lin, H., Wang, Y ., Arora, R., and Jin, X. Is network the bot...

2014
[2021]

A., and Van Essen, B

Dryden, N., Moon, T., Jacobs, S. A., and Van Essen, B. Communication quantization for data-parallel training of deep neural networks. In 2016 2nd Workshop on Machine Learning in HPC Environments (MLHPC),

2016
[2023]

Morgan, T. P. NVIDIA unfolds GPU, interconnect roadmaps out to 2027,

2027
[2024]

Kaplan, J., McCandlish, S., Henighan, T., Brown, T

URL https://www.infinibandta.org/ infiniband-roadmap/. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

2001 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [3]

While this rate decreases with 8 workers (as ties are less likely), it is still significant

With 4 workers, over 30% of the parameters have an update where the majority vote is tied, leading to an erroneous update depending on how it is transmitted. While this rate decreases with 8 workers (as ties are less likely), it is still significant. Indeed, as Figure 10 demonstrates, these errors can lead to loss spikes and training divergence, and so ar...

work page

[2] [4]

Note that when a high-speed interconnect (e.g

The 1-bit LionCub implementation uses an AlltoAll algorithm, which in some cases yields lower throughput than the 4-bit (and higher) LionCub variants that use AllReduce. Note that when a high-speed interconnect (e.g. 200 Gbps InfiniBand) is available, the cost of quantization and other computations can dominate overall performance. Conversely, when limite...

work page 2000

[3] [5]

Nishino, R

URL https: //www.nextplatform.com/2024/06/02/ nvidia-unfolds-gpu-interconnect-roadmaps-out-to-2027/ . Nishino, R. and Loomis, S. H. C. CuPy: A NumPy- compatible library for NVIDIA GPU calculations. 31st conference on neural information processing systems, 151 (7),

work page 2024

[4] [6]

Communication-efficient distributed deep learning: A comprehensive survey

Tang, Z., Shi, S., Wang, W., Li, B., and Chu, X. Communication-efficient distributed deep learning: A comprehensive survey. arXiv preprint arXiv:2003.06307,

arXiv 2003

[5] [7]

Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning

Yan, G., Li, T., Xiao, Y ., Hou, H., and Song, L. Im- proved quantization strategies for managing heavy- tailed gradients in distributed learning. arXiv preprint arXiv:2402.01798,

work page Pith review arXiv

[6] [11]

CIFAR5m Figure 14 presents an ablation study of quantization functions for ResNet-18 trained on CIFAR5m and CIFAR100

A more detailed analysis of the accuracy–efficiency trade-off between L0 and L1 quantization is left for future work. CIFAR5m Figure 14 presents an ablation study of quantization functions for ResNet-18 trained on CIFAR5m and CIFAR100. Note that CIFAR-5m consists of six million synthetic, CIFAR-10-style images generated by a DDPM-based model1. On CIFAR100...

work page 2000

[7] [12]

We trained a 735M parameter GPT model on OpenWebText with 16 GPUs using a cluster equipped with 4 H100 GPUs per node

and 15 Lion Cub: Minimizing Communication Overhead in Distributed Lion 0 20000 40000 Training Time 3 × 100 4 × 100 5 × 100 Val Loss 2=0.99 Lion (32bit) Lion Cub (8bit) Lion Cub (8bit, w/o L1) Lion Cub (1bit) Figure 18: Validation loss on OpenWebText with GPT-2 (735MB) when training with 4 nodes. We trained a 735M parameter GPT model on OpenWebText with 16...

work page 2024

[8] [280]

Ethernet

for 50,000 steps . We used the Lion optimizer with a base learning rate of 3e-5; in addition, we scaled the head, embedding, and scalar parameter groups to 3e-3, 3e-2, and 3e-3 respectively. The betas were set to ( β1 = 0.8, β2 = 0.95), weight decay to 1e-2, and no gradient clipping was applied. We used a training sequence length of 12,288 tokens and a va...

work page 2017