Pith. sign in

REVIEW 4 major objections 9 minor 31 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Change coordinates, not precision: whitening gradients before FP8

2026-07-09 09:00 UTC pith:KLPCBTFK

load-bearing objection K-FAC whitening repurposed for FP8 gradient communication coordinates — mechanistically clean, downstream evidence thin the 4 major comments →

arxiv 2607.07494 v1 pith:KLPCBTFK submitted 2026-07-08 cs.DC cs.LG

GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining

Jieying Wang , Shuyuan Fan , Mingkai Zheng , Zhao Zhang This is my paper
classification cs.DC cs.LG
keywords communicationgiftlow-precisiongradientgradientspretrainingeuclideangeometry-aware
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the dominant source of error in low-precision gradient communication for LLM pretraining is not the number of bits per se, but a mismatch between the anisotropic shape of the gradient distribution and the axis-aligned grid of the quantization format. When gradients are long and stretched in certain directions (as they typically are in Euclidean parameter coordinates), a single FP8 scale factor must stretch to cover the largest direction, leaving smaller directions severely distorted. The authors propose GIFT, which transforms gradients into a near-isotropic coordinate system using a K-FAC-style local curvature metric before quantizing and communicating them, then maps the result back. The key finding is that this coordinate change, applied selectively to the most vulnerable layers and approximated cheaply via a rank-32 input-side factor, makes FP8 communication substantially more faithful to the FP32 reference than direct Euclidean FP8 — improving downstream task preservation while retaining most of the communication speedup.

Core claim

The paper's central discovery is that nearly all the fidelity benefit of full two-sided K-FAC whitening for FP8 gradient communication is captured by the input-side factor alone, and that this factor can be further compressed to rank 32 and deployed on a small subset of layers without meaningful degradation. In a one-step FP8 round-trip test, the input-side transform alone reduced relative reconstruction error by roughly 67% over the Euclidean baseline, matching the full two-sided K-FAC transform almost exactly. This means the practical geometry needed to protect low-precision communication is far lower-dimensional and cheaper to compute than the full second-order curvature would suggest.

What carries the argument

The K-FAC block-diagonal approximation to the Fisher information matrix, which decomposes each layer's local curvature into an input-side factor A and an output-side factor G. GIFT uses only the input-side factor A, further approximated at rank 32, to whiten the gradient before FP8 quantization and communication, then applies the inverse transform afterward. The optimizer itself is never modified — the geometry is used purely as a temporary communication coordinate.

Load-bearing premise

The input-side K-FAC factor, approximated at rank 32 and refreshed every 50 training steps, is assumed to capture the dominant anisotropic geometry of the gradient throughout training. If the curvature changes faster than the 50-step refresh interval or if rank 32 is insufficient for larger models, the whitening becomes stale and the fidelity benefit degrades.

What would settle it

Measure the FP8 round-trip reconstruction error of GIFT versus the Euclidean baseline as a function of steps since the last factor-A refresh. If at step 49 (just before refresh) the error matches or exceeds the Euclidean baseline, the stale-factor assumption is violated and the method's benefit is contingent on a refresh interval that may not generalize.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If coordinate choice matters more than raw bit-width, then future FP4 gradient communication (when hardware supports it) should benefit from the same whitening principle, since the anisotropy mismatch only worsens at lower bit-widths.
  • The finding that input-side geometry alone captures nearly all the benefit suggests that output-side curvature, which is more expensive to maintain, may be dispensable for communication purposes even if it matters for optimization.
  • Selective deployment based on numerical vulnerability profiling implies that the cost of geometry-aware communication can be tuned per model and per training recipe, rather than requiring a fixed universal overhead.
  • The gap between validation-loss similarity and downstream-task divergence suggests that communication fidelity affects learned representations in ways that standard training-loss monitoring does not capture, raising questions about what loss curves actually measure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the input-side factor A is updated every 50 steps but the true curvature rotates faster (e.g., during sharp loss landscape transitions or learning-rate warmup phases), the whitening transform could become stale and actually increase distortion relative to Euclidean FP8. A test would be to measure reconstruction error as a function of steps-since-last-refresh during high-curvature training phase
  • The vulnerability profiling identifies fc2 layers as most sensitive in one model profile, but the generality of this finding to other architectures (e.g., non-Llama models, mixture-of-experts) is untested. If the vulnerable-layer set is architecture-specific, the profiling step becomes a mandatory per-model cost rather than a one-time setup.
  • The 7.6% end-to-end speedup is measured at 64 GPUs; if the geometry-transform overhead is compute-bound while communication savings are bandwidth-bound, the net speedup could either grow (communication dominates at larger scale) or shrink (transform overhead grows with model size) in ways not extrapolated by the current data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 9 minor

Summary. This paper proposes GIFT, a geometry-informed coordinate transformation for FP8 gradient communication in distributed LLM pretraining. The core idea is to whiten gradients using an input-side K-FAC factor before FP8 quantization and communication, so that quantization error is distributed more isotropically across gradient directions. The authors progressively simplify the full two-sided K-FAC transform to: (1) input-side only, (2) rank-32 low-rank approximation, and (3) selective deployment on the 13 most numerically vulnerable fc2 layers. They evaluate on Llama-300M and Llama-600M with the Muon optimizer, showing up to 67.4% reduction in FP8 round-trip reconstruction error over the Euclidean baseline, a 7.6% end-to-end pretraining speedup over FP32 on 64 GH200 Superchips, and a more favorable downstream task preservation profile (7/14 tasks vs FP32) compared to direct Euclidean FP8 (5/14 tasks).

Significance. The paper addresses a practically important problem: gradient communication is a well-known scaling bottleneck in LLM pretraining, and naive FP8 quantization introduces direction-dependent distortion under anisotropic gradient geometry. The coordinate-transformation perspective is novel relative to prior compression work (QSGD, PowerSGD, 1-bit Adam, SDP4Bit) that modifies quantization rules, sparsity, or communication patterns. The round-trip error analysis (Tables I–II) is mechanistically clean and provides a falsifiable, parameter-light justification for the input-side simplification. The systems evaluation on real GH200 hardware with end-to-end pretraining and a concrete 7.6% speedup measurement is a strength. The method is complementary to existing optimizers and communication collectives, which is a practically useful design property. However, the downstream task evaluation that supports the central practical claim is statistically fragile (single seed, 14 binary comparisons, no significance testing), which limits the strength of the evidence for the headline claim.

major comments (4)
  1. §VI.D, Table III: The central practical claim—that geometry-aware communication improves the downstream preservation profile over Euclidean FP8—rests on single-seed runs evaluated via a binary 'wins vs FP32' count over 14 tasks. The 7-vs-5 margin (600M) and 7-vs-4 margin (300M) could easily arise from noise given the per-task variance visible in the table (e.g., 600M CB-ACC: GIFT 0.3036 vs Euclidean 0.4107; RCD-F1: GIFT 0.8197 vs Euclidean 0.8271; WINO: GIFT 0.5257 vs Euclidean 0.5320). Without error bars, multiple seeds, or any significance test, the downstream advantage is not established at conventional confidence. The paper acknowledges this in §VI.G ('additional seeds and a larger benchmark suite' needed), but the claim in the abstract and conclusion is stated without this qualification. At minimum, the paper should either (a) run at least 3 seeds and report mean ± std for each task
  2. §V.E, Algorithm 1: The K-FAC input-side factor A is updated every K=50 steps, but no sensitivity analysis is provided for this choice. The reader's concern about staleness is legitimate: if the curvature changes faster than the 50-step refresh interval, the whitening transform may become stale and the fidelity benefit could degrade. The paper should either provide an ablation over K (e.g., K ∈ {10, 25, 50, 100}) or cite evidence that the input-side activation statistics are stable over 50-step windows for the models and training recipes studied.
  3. §V.C, Figure 3: The layer-selection procedure (top 13 vulnerable fc2 layers) is profiled on the 600M model during the first 100 training steps. It is unclear whether this vulnerability ranking is stable throughout training or whether it reflects only early-training dynamics. If the vulnerable layer set shifts as training progresses, the fixed selection may miss layers that become vulnerable later. The paper should discuss whether the profiling window is representative of the full training run, or ideally re-profile at multiple training checkpoints to verify stability.
  4. §VI.D, Table III (300M row): The full K-FAC variant wins 6/14 tasks vs FP32, while the selective GIFT design wins 7/14. The paper frames this as supporting the selective design, but the 6-vs-7 difference over 14 tasks is within noise. More importantly, on several individual tasks full K-FAC substantially outperforms GIFT (e.g., CB-ACC: full K-FAC 0.4107 vs GIFT 0.2321; MUL-RC: full K-FAC 0.5140 vs GIFT 0.4420). The claim that selective deployment is superior to full K-FAC is not well-supported by this data and should be softened or supported by additional evidence.
minor comments (9)
  1. §I: The abstract states 'improving the downstream task preservation profile over direct Euclidean FP8 communication.' Given the statistical fragility discussed above, this should be qualified (e.g., 'in our single-seed experiments, improving...').
  2. §II.A: The notation 'g^(r)' in Eq. (1) uses a superscript that could be confused with exponentiation; consider using a subscript or explicit indexing.
  3. §IV.A, Eq. (14): The notation L^{-⊤}_A is used but not explicitly defined as (L_A^{-1})^⊤. A brief clarification would help readers unfamiliar with this convention.
  4. §V.B, Table II: The 'block-A-32' and 'block-A-64' methods are mentioned but not clearly defined in the text. A brief description of the block-diagonal approximation structure would improve reproducibility.
  5. §VI.A: The choice of different sequence lengths for the two models (4096 for 300M, 2048 for 600M) is explained as a memory constraint, but this means the two models are not directly comparable. This should be noted more prominently when interpreting cross-model trends.
  6. §VI.E: The memory overhead for 600M (8.98%) is notably higher than for 300M (3.33%). The paper should briefly explain why the overhead scales non-linearly with model size (e.g., is it because the number of selected layers or the factor dimensions scale differently?).
  7. Table III: The 'LAMB-STD' column header appears to differ in format from other headers (period vs hyphen). Consistency would improve readability.
  8. §VI.D: The paper states 'BF16 matches FP32 on 7 out of 14 tasks, but its absolute task values are slightly lower overall.' This claim of 'slightly lower overall' should be supported by an aggregate metric (e.g., average across tasks) rather than stated qualitatively.
  9. References: The Muon optimizer [8] is cited as a web page rather than a peer-reviewed publication. If a preprint or formal report is available, it should be cited accordingly; if not, the citation format should note it as a software/technical report.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive report. The referee raises four major points, all of which concern the strength of empirical evidence supporting our claims: (1) the downstream task evaluation lacks multiple seeds and significance testing; (2) no sensitivity analysis for the K-FAC refresh interval K=50; (3) the layer-vulnerability profiling window may not be representative of the full training run; and (4) the claim that selective GIFT is superior to full K-FAC is not well-supported by the 6-vs-7 task count. We agree with the substance of all four points and will revise the manuscript accordingly. Specifically, we will run additional seeds with mean±std reporting, add a K-sensitivity ablation, re-profile vulnerability at multiple checkpoints, and soften the selective-vs-full-K-FAC claim. We also note that the mechanistic round-trip error analysis (Tables I–II) and the systems-level speedup measurement are independent of these concerns and remain valid as stated.

read point-by-point responses

  1. Referee: §VI.D, Table III: The central practical claim—that geometry-aware communication improves the downstream preservation profile over Euclidean FP8—rests on single-seed runs evaluated via a binary 'wins vs FP32' count over 14 tasks. The 7-vs-5 margin (600M) and 7-vs-4 margin (300M) could easily arise from noise given the per-task variance visible in the table. Without error bars, multiple seeds, or any significance test, the downstream advantage is not established at conventional confidence. The paper acknowledges this in §VI.G but the claim in the abstract and conclusion is stated without this qualification.

    Authors: The referee is correct. The downstream evaluation as presented is single-seed with binary win counting and no significance testing, which is insufficient to establish the claimed advantage at conventional confidence. We will address this by running at least 3 seeds for each method (FP32, Euclidean FP8, GIFT) on both model sizes and reporting mean ± std for each task. We will also apply paired significance tests (e.g., Wilcoxon signed-rank) across tasks. The abstract and conclusion will be revised to include appropriate qualifications (e.g., 'in our experiments' or 'under the seeds tested') rather than stating the downstream advantage as an unqualified fact. We note that the mechanistic round-trip error analysis (Tables I–II) showing 67.4% reduction in reconstruction error is independent of the downstream evaluation and is not affected by this concern. The 7.6% end-to-end speedup is a systems measurement that does not depend on seed variance. revision: yes

  2. Referee: §V.E, Algorithm 1: The K-FAC input-side factor A is updated every K=50 steps, but no sensitivity analysis is provided for this choice. The reader's concern about staleness is legitimate: if the curvature changes faster than the 50-step refresh interval, the whitening transform may become stale and the fidelity benefit could degrade. The paper should either provide an ablation over K or cite evidence that the input-side activation statistics are stable over 50-step windows.

    Authors: We agree that a sensitivity analysis for K is needed and absent from the current manuscript. We will add an ablation over K ∈ {10, 25, 50, 100} measuring both the round-trip reconstruction error (as in Tables I–II) and, if compute permits, validation loss trajectories. Our intuition is that input-side activation second-order statistics (A = E[aa^T]) are relatively stable over 50-step windows because they depend on input distributions rather than rapidly changing gradient dynamics, but this should be verified empirically rather than asserted. If the ablation shows that shorter intervals improve fidelity meaningfully, we will report the cost-quality tradeoff explicitly. We will also add a brief discussion of why input-side statistics may be more stable than output-gradient-side statistics, which was part of our motivation for the input-only simplification. revision: yes

  3. Referee: §V.C, Figure 3: The layer-selection procedure (top 13 vulnerable fc2 layers) is profiled on the 600M model during the first 100 training steps. It is unclear whether this vulnerability ranking is stable throughout training or whether it reflects only early-training dynamics. If the vulnerable layer set shifts as training progresses, the fixed selection may miss layers that become vulnerable later. The paper should discuss whether the profiling window is representative of the full training run, or ideally re-profile at multiple training checkpoints to verify stability.

    Authors: This is a valid concern. The current profiling is limited to the first 100 training steps, and we have not verified that the vulnerability ranking is stable throughout training. We will re-profile the vulnerability ranking at multiple checkpoints (e.g., early, mid, and late training) for the 600M model and report whether the top-13 layer set changes. If the set is stable, we will state this explicitly with the supporting evidence. If the set shifts, we will discuss the implications: (a) a fixed selection may be suboptimal for later training, and (b) periodic re-selection could be a practical extension. We will also add an explicit caveat in §V.C noting that the profiling window is early-training and that the stability assumption is empirically tested rather than theoretically guaranteed. revision: yes

  4. Referee: §VI.D, Table III (300M row): The full K-FAC variant wins 6/14 tasks vs FP32, while the selective GIFT design wins 7/14. The paper frames this as supporting the selective design, but the 6-vs-7 difference over 14 tasks is within noise. More importantly, on several individual tasks full K-FAC substantially outperforms GIFT (e.g., CB-ACC: full K-FAC 0.4107 vs GIFT 0.2321; MUL-RC: full K-FAC 0.5140 vs GIFT 0.4420). The claim that selective deployment is superior to full K-FAC is not well-supported by this data and should be softened or supported by additional evidence.

    Authors: The referee is correct on both counts. The 6-vs-7 difference is within noise, and on several individual tasks full K-FAC outperforms selective GIFT. The current framing overstates the evidence. We will soften the claim from 'supporting the selective design' to something like 'the selective design achieves a comparable downstream profile to full K-FAC while incurring lower computational cost, which is the primary practical motivation for the selective approach.' The argument for selective deployment is better justified on cost grounds (lower overhead) than on downstream-quality grounds, and we will make this distinction explicit. We will also add the multi-seed evaluation for the full K-FAC variant on 300M, which will provide a fairer comparison. If the multi-seed results show no significant difference between full K-FAC and selective GIFT on downstream tasks, we will frame the selective design purely as a cost-reduction measure that preserves the fidelity benefit, not as a quality improvement. revision: yes

Circularity Check

0 steps flagged

No significant circularity: the core derivation (whitening reduces quantization error) follows from standard K-FAC and Fisher geometry, not from a self-citation chain or fitted-to-target construction.

full rationale

The paper's central mechanistic claim—that whitening gradients via K-FAC factors before FP8 quantization reduces reconstruction error relative to Euclidean quantization—follows directly from the standard Fisher metric and K-FAC approximation (Eqs. 6–13), which are attributed to external work (Martens & Grosse 2015, Amari 1998). The K-FAC factors A and G are computed from the model's own activation and gradient statistics during training, not fitted to downstream task metrics. The design simplifications (input-side only, rank-32, top-13 layers) are selected via round-trip reconstruction error profiling (Tables I–II, Figure 3), which is an independent mechanistic metric, not the downstream task results they later evaluate against. The downstream task comparison (Table III) is presented as an empirical consequence, not as a derived prediction. The rank-32 and layer-selection choices are tuned on reconstruction error, which is a different quantity from the downstream wins-vs-FP32 metric, so this is not a case of fitting to the target. There is minor self-referentiality in that the simplification roadmap (Figure 2) is the paper's own design process, but this is standard iterative engineering, not circular reasoning. No step in the derivation chain reduces to its own inputs by construction. The statistical fragility of the downstream evaluation (single seed, 14 binary comparisons) is a correctness risk, not a circularity issue.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The free parameters (rank 32, 13 layers, K=50) are all empirically tuned on the specific 300M/600M Llama models used in the study. The authors are transparent that these are not universal constants. The axioms are standard K-FAC assumptions repurposed for communication. No new entities are invented.

free parameters (4)
  • Rank of low-rank approximation = 32
    Chosen empirically in §V.B based on Table II round-trip error analysis for the profiled model. Not claimed as universal.
  • Number of selected vulnerable layers = 13
    Chosen by profiling numerical vulnerability on 600M model (§V.C, Figure 3). Authors note 'the number 13 is not intended as a model-independent constant.'
  • Factor update frequency K = 50
    K-FAC input-side factors refreshed every 50 training steps (§V.E, Algorithm 1). No sensitivity analysis provided.
  • Layer selection threshold = top 13 fc2 layers
    Determined by the visible drop in vulnerability score after the top 13 entries in Figure 3.
axioms (4)
  • standard math The Fisher information matrix provides a valid local metric for parameter perturbations in neural-network optimization.
    Standard information geometry result (Amari 1998, cited [9][10]). Used in §II.B, Eq. 2.
  • domain assumption The K-FAC block-diagonal approximation (layerwise, Kronecker-factored) is a tractable and sufficiently accurate approximation of the Fisher for the purpose of defining communication coordinates.
    §II.B, Eq. 3-4. The paper does not prove this approximation is adequate for communication; it relies on empirical round-trip error (Table I-II) to justify it.
  • domain assumption Gradients in LLM pretraining are highly anisotropic, causing direction-dependent quantization distortion under uniform scaling.
    §I, Figure 1. Empirically observed but not formally proven for all models/training stages.
  • ad hoc to paper The input-side K-FAC factor A captures the dominant anisotropic structure relevant for FP8 quantization error.
    §V.A, Table I. Justified empirically for the specific 600M model profiled, but the output-side factor G is discarded based on this single profile.

pith-pipeline@v1.1.0-glm · 20584 in / 2529 out tokens · 460805 ms · 2026-07-09T09:00:43.230850+00:00 · methodology

0 comments
read the original abstract

Gradient communication is a primary scaling bottleneck in large language model (LLM) pretraining. Communicating gradients in low-precision formats, such as FP8 and NVFP4, can significantly reduce the communication volume. Existing methods quantize gradients via linear or nonlinear mappings in Euclidean space, often degrading model performance because highly anisotropic gradients incur direction-dependent distortion. We present GIFT, a geometry-informed gradient scaling method that performs low-precision communication in geometry-aware coordinates. By transforming gradients into a near-isotropic space before quantization, GIFT makes low-precision representations substantially more faithful to their high-precision counterparts. GIFT only changes the coordinate system used for low-precision gradient communication and does not change the optimizer, training recipe, communication collective, or low-precision format. We also develop a simplified geometry-aware transformation algorithm with low-rank approximation and selective application to balance the computation overhead and communication reduction. We examine the empirical convergence of GIFT using Llama-300M and Llama-600M models. Our results show that GIFT reduces the end-to-end pretraining time of Llama-600M by 7.6% on 64 NVIDIA GH200 Superchips, while improving the downstream task preservation profile over direct Euclidean FP8 communication under the same optimizer and communication path.

Figures

Figures reproduced from arXiv: 2607.07494 by Jieying Wang, Mingkai Zheng, Shuyuan Fan, Zhao Zhang.

Figure 1
Figure 1. Figure 1: Two-dimensional projection of real gradient samples [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Roadmap of the practical simplifications used to turn [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Pretraining-time structure of GIFT. Most layers use the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Step-time improvement over FP32 communication as [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Validation loss during pretraining for the 300M model [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages · 22 internal anchors

  1. [1]

    Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

    T. Ben-Nun and T. Hoefler, “Demystifying parallel and distributed deep learning: An in-depth concurrency analysis,”ACM Computing Surveys, vol. 52, no. 4, pp. 65:1–65:43, 2019. [Online]. Available: https://doi.org/10.48550/arXiv.1802.09941

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1802.09941 2019

  2. [2]

    Bandwidth optimal all-reduce algorithms for clusters of workstations,

    P. Patarasuk and X. Yuan, “Bandwidth optimal all-reduce algorithms for clusters of workstations,”Journal of Parallel and Distributed Computing, vol. 69, no. 2, pp. 117–124, 2009. [Online]. Available: https://doi.org/10.1016/j.jpdc.2008.09.002

    work page doi:10.1016/j.jpdc.2008.09.002 2009

  3. [3]

    Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression,

    J. Song, J. Yim, J. Jung, H. Jang, H.-J. Kim, Y . Kim, and J. Lee, “Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression,” inProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. Association for Computing Machinery, 2023, pp. 56...

    work page doi:10.1145/3575693.3575712 2023

  4. [4]

    FP8-LM: Training FP8 Large Language Models

    H. Peng, K. Wu, Y . Wei, G. Zhao, Y . Yang, Z. Liu, Y . Xiong, Z. Yang, B. Ni, J. Hu, R. Li, M. Zhang, C. Li, J. Ning, R. Wang, Z. Zhang, S. Liu, J. Chau, H. Hu, and P. Cheng, “FP8-LM: Training FP8 large language models,”arXiv preprint arXiv:2310.18313, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2310.18313

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2310.18313 2023

  5. [5]

    COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training

    H. Xi, H. Cai, L. Zhu, Y . Lu, K. Keutzer, J. Chen, and S. Han, “COAT: Compressing optimizer states and activation for memory-efficient FP8 training,”arXiv preprint arXiv:2410.19313, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2410.19313

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2410.19313 2024

  6. [6]

    Decoupled Weight Decay Regularization,

    I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,”

    work page

  7. [7]

    Decoupled Weight Decay Regularization

    [Online]. Available: https://doi.org/10.48550/arXiv.1711.05101

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1711.05101

  8. [8]

    SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training

    J. Jia, C. Xie, H. Lu, D. Wang, H. Feng, C. Zhang, B. Sun, H. Lin, Z. Zhang, X. Liu, and D. Tao, “SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training,” 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2410.15526

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2410.15526 2024

  9. [9]

    Muon: An optimizer for hidden layers in neural networks,

    K. Jordan, Y . Jin, V . Boza, Y . Jiacheng, F. Cesista, L. Newhouse, and J. Bernstein, “Muon: An optimizer for hidden layers in neural networks,”

    work page

  10. [10]

    Available: https://kellerjordan.github.io/posts/muon/

    [Online]. Available: https://kellerjordan.github.io/posts/muon/

    work page

  11. [11]

    Amari,Information Geometry and Its Applications

    S.-i. Amari,Information Geometry and Its Applications. Springer Tokyo, 2016. [Online]. Available: https://doi.org/10.1007/ 978-4-431-55978-8

    work page 2016

  12. [12]

    Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer

    ——, “Natural gradient works efficiently in learning,”Neural Computation, vol. 10, no. 2, pp. 251–276, 1998. [Online]. Available: https://doi.org/10.1162/089976698300017746

    work page doi:10.1162/089976698300017746 1998

  13. [13]

    Optimizing Neural Networks with Kronecker-factored Approximate Curvature

    J. Martens and R. Grosse, “Optimizing neural networks with kronecker-factored approximate curvature,” inProceedings of the 32nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 37. PMLR, 2015, pp. 2408–2417. [Online]. Available: https://doi.org/10.48550/arXiv.1503.05671

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1503.05671 2015

  14. [14]

    Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V . A. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on gpu clusters using megatron-lm,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2104.04473 2021

  15. [15]

    Openwebtext cor- pus,

    A. Gokaslan, V . Cohen, E. Pavlick, and S. Tellex, “Openwebtext cor- pus,” http://Skylion007.github.io/OpenWebTextCorpus, 2019. [Online]. Available: http://Skylion007.github.io/OpenWebTextCorpus

    work page 2019

  16. [16]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,”arXiv preprint arXiv:1909.08053,

    work page internal anchor Pith review Pith/arXiv arXiv 1909

  17. [17]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    [Online]. Available: https://doi.org/10.48550/arXiv.1909.08053

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1909.08053 1909

  18. [18]

    GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

    Y . Huang, Y . Cheng, A. Bapna, O. Firat, D. Chen, M. X. Chen, H. Lee, J. Ngiam, Q. V . Le, Y . Wu, and Z. Chen, “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” inAdvances in Neural Information Processing Systems, vol. 32, 2019. [Online]. Available: https://doi.org/10.48550/arXiv.1811.06965

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1811.06965 2019

  19. [19]

    Memory-Efficient Pipeline-Parallel DNN Training

    D. Narayanan, A. Phanishayee, K. Shi, X. Chen, and M. Zaharia, “Memory-efficient pipeline-parallel dnn training,” inProceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 139. PMLR, 2021, pp. 7937–7947. [Online]. Available: https://doi.org/10.48550/arXiv.2006.09503

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2006.09503 2021

  20. [20]

    ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He, “Zero: Memory optimizations toward training trillion parameter models,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2020, pp. 1–16. [Online]. Available: https://doi.org/10.48550/arXiv.1910.02054

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1910.02054 2020

  21. [21]

    ZeRO++: Extremely Efficient Collective Communication for Giant Model Training

    G. Wang, H. Qin, S. A. Jacobs, C. Holmes, S. Rajbhandari, O. Ruwase, F. Yan, L. Yang, and Y . He, “ZeRO++: Extremely efficient collective communication for giant model training,”arXiv preprint arXiv:2306.10209, 2023. [Online]. Available: https://doi.org/10.48550/ arXiv.2306.10209

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  22. [22]

    Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures

    R. Eschenhagen, A. Immer, R. E. Turner, F. Schneider, and P. Hennig, “Kronecker-factored approximate curvature for modern neural network architectures,” inAdvances in Neural Information Processing Systems, vol. 36, 2023, pp. 33 624–33 655. [Online]. Available: https://doi.org/10.48550/arXiv.2311.00636

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2311.00636 2023

  23. [23]

    QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding

    D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. V ojnovic, “Qsgd: Communication-efficient sgd via gradient quantization and encoding,” inAdvances in Neural Information Processing Systems, vol. 30, 2017. [Online]. Available: https://doi.org/10.48550/arXiv.1610.02132

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1610.02132 2017

  24. [24]

    Deep gradient compression: Reducing the communication bandwidth for distributed training,

    Y . Lin, S. Han, H. Mao, Y . Wang, and W. J. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,” inInternational Conference on Learning Representations,

    work page

  25. [25]

    Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

    [Online]. Available: https://doi.org/10.48550/arXiv.1712.01887

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1712.01887

  26. [26]

    PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization

    T. V ogels, S. P. Karimireddy, and M. Jaggi, “Powersgd: Practical low- rank gradient compression for distributed optimization,” inAdvances in Neural Information Processing Systems, vol. 32, 2019. [Online]. Available: https://doi.org/10.48550/arXiv.1905.13727

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1905.13727 2019

  27. [27]

    1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed

    H. Tang, S. Gan, A. A. Awan, S. Rajbhandari, C. Li, X. Lian, J. Liu, C. Zhang, and Y . He, “1-bit adam: Communication efficient large-scale training with adam’s convergence speed,” inProceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 139. PMLR, 2021, pp. 10 118–10 129. [Online]. Availa...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2102.02888 2021

  28. [28]

    FP8 Formats for Deep Learning

    P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu, N. Mellempudi, S. Oberman, M. Shoeybi, M. Siu, and H. Wu, “Fp8 formats for deep learning,”arXiv preprint arXiv:2209.05433, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2209.05433

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2209.05433 2022

  29. [29]

    LoCo: Low-Bit Communication Adaptor for Large-scale Model Training

    X. Xie, Z. Lin, K.-C. Toh, and P. Zhou, “Loco: Low- bit communication adaptor for large-scale model training,” arXiv preprint arXiv:2407.04480, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2407.04480

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2407.04480 2024

  30. [30]

    Mixed Precision Training

    P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, “Mixed precision training,” inInternational Conference on Learning Representations, 2018. [Online]. Available: https: //doi.org/10.48550/arXiv.1710.03740

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1710.03740 2018

  31. [31]

    Training and inference of large language models using 8-bit floating point

    S. P. Perez, Y . Zhang, J. Briggs, C. Blake, J. Levy-Kramer, P. Balanca, C. Luschi, S. Barlow, and A. Fitzgibbon, “Training and inference of large language models using 8-bit floating point,” arXiv preprint arXiv:2309.17224, 2023. [Online]. Available: https: //doi.org/10.48550/arXiv.2309.17224

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2309.17224 2023