Pith. sign in

REVIEW 3 major objections 6 minor 19 references

Lossless Compression for LLM Tensor Incremental Snapshots

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that byte-grouping plus block-adaptive Huffman coding (BG-LMC) compresses LLM tensor incremental snapshots at least as well as BZ2 while running about an order of magnitude faster, and that a 16-core version reaches 2.78…

desk verdict Useful empirical study of LLM checkpoint delta compression, but the paper never defines what a 'shard-delta' is, so the headline numbers can't be reproduced or trusted until that is pinned down. read the letter →

arxiv 2505.09810 v1 pith:QKBZ3NYO submitted 2025-05-14 cs.LG

Daniel Waddington , Cornel Constantinescu This is my paper · ORCID
classification cs.LG
keywords losslesscompressionLLMcheckpointingincrementalsnapshotsbyte-groupingHuffmancodingrun-lengthencodingbfloat16tensordelta
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Training large language models forces periodic checkpoints of tensor state, often hundreds of gigabytes, and the time and bandwidth to write them limits how often they can be taken. This paper argues that the copy can be made much smaller and cheaper if each checkpoint is encoded as the change from the previous one, and that this delta compresses well with a scheme built from byte-grouping, run-length encoding, and block-adaptive Huffman coding. The authors measure over 28 TiB of tensor data from six public models and report that BG-LMC matches or beats the best general-purpose engine (BZ2) in compression ratio while encoding roughly 30 times faster. The practical stake is that checkpoints can become more frequent without stealing training time, because less CPU and less storage bandwidth are consumed per snapshot.

What carries the argument

The central mechanism is byte-grouping: for 16-bit values, each value's most-significant byte is copied into one contiguous stream and the least-significant byte into another (four streams for 32-bit values), so the stable high-order bytes form long runs of equal symbols. Run-length encoding collapses those runs, and Huffman coding with a codebook recomputed every 64 KiB block adapts to the slowly shifting byte statistics. The load-bearing insight is that floating-point weight deltas are not random bytes: the most-significant bytes of a delta are almost always zero once training converges, while the least-significant bytes keep flipping, so separating bytes by significance converts stability into compressibility.

What would settle it

Take a single BLOOM shard, form a delta by element-wise arithmetic subtraction between consecutive checkpoints, serialize the bfloat16 bytes, and run BG-LMC and BZ2 on identical hardware: if BG-LMC's compression ratio is no longer at or below BZ2's, or its encode-time advantage shrinks to less than roughly 5x, the paper's central claim fails. The same experiment should verify that Huffman codebook overhead per 64 KiB block is included in both ratio and throughput numbers.

Watch

Extended reading notes

Core claim

On shard-deltas taken from six public LLM checkpoint sequences, byte-grouping of bfloat16 and float32 tensors exposes the fact that high-order bytes stabilize during training while low-order bytes keep fluctuating. A compressor that groups bytes by significance, applies run-length encoding to the resulting runs of repeated symbols, and then Huffman-codes each 64 KiB block with a fresh codebook comes within a few percent of the per-block entropy floor. The authors report BG-LMC mean compression ratios below 0.6 on BLOOM shards (0.519 to 0.600 across shard types), slightly better than BG-BZ2 (0.538 to 0.627), while encoding about 30x faster. A data-parallel OpenMP implementation reaches 2.78 GiB/s compression and 3.76 GiB/s decompression on 16 cores, with a compression ratio of 0.597 that beats parallel gzip (0.623) and parallel BZ2 (0.628).

Load-bearing premise

The paper's compression ratios and speeds are computed on 'shard-deltas' whose exact construction is never defined (bitwise XOR versus arithmetic subtraction, and how the resulting bytes are grouped), so if real incremental checkpoints produce different delta statistics, the measured advantage over BZ2 would not transfer to practice.

Editorial extensions

If this is right

  • Incremental checkpointing can store deltas at roughly half the raw size on BLOOM-class bfloat16 models, with ratios improving as training converges (weight shards fall below 0.1).
  • Compression time ceases to be the bottleneck: single-threaded BG-LMC encodes at about 300 MB/s, around 30x faster than BG-BZ2, so more frequent checkpoints fit into the same epoch window.
  • A 16-core parallel implementation (PLMC) sustains 2.78 GiB/s compression and 3.76 GiB/s decompression, exceeding parallel gzip and BZ2 in both throughput and compression ratio.
  • Because the compression is lossless, recovery restores bit-identical tensor values, avoiding the silent errors that lossy checkpoint compression could introduce.
  • The byte-grouping plus RLE plus Huffman recipe generalizes across bfloat16 and float32 models, since the stability ordering of byte significance is intrinsic to IEEE floating-point formats.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If real training systems form deltas by arithmetic subtraction rather than bitwise XOR, the low-order mantissa bytes will still fluctuate but the exponent and sign bytes should stabilize even more, so BG-LMC's advantage likely holds; however, the paper's measured numbers depend on the exact, unspecified delta construction.
  • A natural next step is an online checkpointing pipeline that writes a compressed delta every few steps and periodically writes a full snapshot; the 128 MiB buffered parallel design already points toward streaming operation.
  • The gap between BG-LMC and the entropy floor grows as data becomes more compressible because Huffman coding uses integral bits per symbol, suggesting a hybrid that switches to an arithmetic coder on near-converged shards could squeeze out a few more percent.
  • The reported throughputs are CPU-only; on GPU clusters the compression can overlap with network transfer, so the end-to-end savings depend on whether the bottleneck is CPU, storage bandwidth, or network bandwidth in a given checkpoint system.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper investigates lossless compression of incremental LLM checkpoint snapshots. It analyzes checkpoint data from six Hugging Face model sequences (over 28 TiB total), studies how tensor bit patterns and cross-step deltas evolve, and proposes Language Model Compressor (LMC), a scheme combining byte-grouping, run-length encoding, and block-adaptive Huffman coding. The empirical sections report that the byte-grouped variant BG-LMC slightly outperforms BG-BZ2 in compression ratio on BLOOM data while encoding roughly 30x faster, and that a 16-thread parallel implementation (PLMC) reaches 2.78 GiB/s compression and 3.76 GiB/s decompression throughput. The paper is an empirical systems contribution and makes no formal correctness claims.

Significance. If the reported numbers hold, the contribution is practically valuable: it addresses a real bottleneck in LLM training by reducing checkpoint size and CPU time, potentially enabling more frequent checkpointing. The paper's strengths are its real-world data coverage, the concrete bit-level analysis of why tensor deltas are compressible, and the detailed BLOOM measurements in Appendix A.3. The central risks are reproducibility (the delta construction is never specified), statistical robustness (means without variance), and the narrow comparator set; these risks are addressable in revision.

major comments (3)
  1. [§5.1 and Appendix A.3] The paper never defines the byte-level input that is compressed. Section 3.1 uses XOR only as an analytical tool for bit-flip ratios, and Section 4.2 describes byte-grouping without stating the element width, byte order, or how deltas between consecutive checkpoints are produced and grouped. Nor is the compression ratio direction defined (the values in Appendix A.3, e.g., 0.565, suggest compressed/original), and it is unclear whether the ratio includes the per-block Huffman codebook overhead. Because the reported BG-LMC advantage over BG-BZ2 is only about 0.02-0.03 (0.565 vs 0.593 for dense_4h_to_h, 0.600 vs 0.627 for self_attention.dense), an alternate delta construction or accounting convention could plausibly change the conclusion. Please specify the delta operation, the grouping layout, the ratio definition, and the overhead treatment.
  2. [§5.1 and Appendix A.3] All headline ratios and throughputs are reported as means across shard-deltas without standard deviations, per-step series, or a clear statement of how many items each mean covers; the word_embeddings row uses 20 steps while the weight rows use 200. The BG-LMC vs BG-BZ2 differences are small enough (e.g., 0.600 vs 0.627) that variance could be material. Please report per-shard or per-step statistics, or at least min-max ranges, and clarify how the mean is aggregated over the layers represented by 'X' in the shard names.
  3. [§5.2 and §6] The claim that LMC beats 'the best alternative' is supported only against BZ2, gzip, DEFLATE, and LZ4; zstd (including its parallel variant pzstd) is absent even though FCBench [17] is cited, and no specialized floating-point compressor is benchmarked. A common zstd preset often achieves better ratios than BZ2 at much higher speed, and including it is necessary to substantiate the 'best alternative' wording and the practical, systems-level contribution.
minor comments (6)
  1. [Title] The title in the manuscript reads 'LLM T ENSOR'; remove the unintended space so it reads 'LLM Tensor'.
  2. [Throughout] The word 'compressability' appears in Sections 3.1, 5.1, and 7; it should be 'compressibility'.
  3. [§5.1] The text refers to 'Appendix 3' when presenting decompression results; this should be 'Appendix A.3'.
  4. [§4.2] The phrase 'as in paragraph 4' is ambiguous and should be replaced with a specific reference to the RLE description in Section 4.
  5. [§4.1 and Figures 9-10] The term 'BG-entropy' is used without a definition; please state explicitly that it denotes the measured first-order entropy of byte-grouped data using the block-adaptive probabilities.
  6. [§4.2] The algorithm description is underspecified: it is not stated how the 'maximum 64KB' block size is chosen in practice, how each block's Huffman codebook is serialized, or how tensor shapes that are not multiples of the element width are handled. A pseudocode listing or a precise normative description would aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LMC's compression ratios and throughputs are direct measurements against external baselines.

full rationale

I walked the derivation chain: Section 3 analyzes bit flips in real checkpoint data, Section 4.1 computes entropy directly from observed byte histograms, and Section 4.2 implements LMC as a block-adaptive Huffman coder plus RLE, whose near-entropy behavior is a standard property of Huffman coding rather than a fitted prediction. Section 5 reports measured compression ratios and throughputs for LMC and for external engines (BZ2, LZ4, DEFLATE, gzip) on the same public checkpoint data. No equation is defined in terms of the result it is said to predict, no fitted parameter is back-solved into a ratio formula, and no self-citation or imported uniqueness theorem is used as load-bearing support. The unspecified construction of shard-deltas and the absence of a holdout design are reproducibility and generalization concerns, not circularity, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard information theory, a domain assumption about byte stability in trained tensors, an unspecified delta representation, and representativeness of six public datasets. The hand-picked block and buffer sizes are the only explicit numerical choices; no fitted constants enter the compression ratio formula. No new physical or mathematical entities are introduced; LMC and PLMC are software artifacts.

free parameters (2)
  • Adaptive Huffman block size = 64 KiB (maximum)
    Chosen by hand in Sections 4.1 and 4.2; determines codebook overhead and adaptivity. No sensitivity analysis is reported, and the same datasets used for evaluation motivated this choice.
  • PLMC input buffer size = 128 MiB (default)
    Section 4.3 uses this default buffer size for parallel segmentation. Authors state throughput is insensitive to buffer size, but the measurements are made at this setting and no sweep is shown.
assumptions (5)
  • standard math Shannon entropy is a lower bound for lossless symbol-wise compression.
    Section 4.1 uses entropy as the ideal compression ratio; this is standard information theory.
  • standard math Huffman coding is optimal to within one bit per symbol.
    Section 4.2 motivates the use of Huffman over arithmetic coding; this is a standard result.
  • domain assumption In LLM checkpoint tensors, most-significant bytes are more stable across training steps than least-significant bytes.
    Established empirically in Section 3.1 (Figures 4 to 7) and used as the design basis for byte-grouping; assumed to generalize beyond the six datasets.
  • domain assumption Delta encoding between consecutive checkpoints is a valid representation of checkpoint data for compression.
    Section 3.1 and the experimental results use shard-deltas; the full checkpoint files are not compressed directly, and the delta operation is not specified.
  • domain assumption The six Hugging Face checkpoint sequences are representative of LLM training checkpoint data.
    Table 1 lists the datasets; all downstream conclusions generalize only if these models and training schedules are representative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lossless Compression for LLM Tensor Incremental Snapshots." pith.science (2026). https://pith.science/paper/QKBZ3NYO

@misc{pith2026250509810,
  author       = {Pith},
  title        = {Pith review of: Lossless Compression for LLM Tensor Incremental Snapshots},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKBZ3NYO}},
  note         = {Machine review of arXiv:2505.09810}
}
read the original abstract

During the training of Large Language Models (LLMs), tensor data is periodically "checkpointed" to persistent storage to allow recovery of work done in the event of failure. The volume of data that must be copied during each checkpoint, even when using reduced-precision representations such as bfloat16, often reaches hundreds of gigabytes. Furthermore, the data must be moved across a network and written to a storage system before the next epoch occurs. With a view to ultimately building an optimized checkpointing solution, this paper presents experimental analysis of checkpoint data used to derive a design that maximizes the use of lossless compression to reduce the volume of data. We examine how tensor data and its compressibility evolve during model training and evaluate the efficacy of existing common off-the-shelf general purpose compression engines combined with known data optimization techniques such as byte-grouping and incremental delta compression. Leveraging our analysis we have built an effective compression solution, known as Language Model Compressor (LMC), which is based on byte-grouping and Huffman encoding. LMC offers more compression performance than the best alternative (BZ2) but with an order-of-magnitude reduction in the time needed to perform the compression. We show that a 16-core parallel implementation of LMC can attain compression and decompression throughput of 2.78 GiB/s and 3.76 GiB/s respectively. This increase in performance ultimately reduces the CPU resources needed and provides more time to copy the data to the storage system before the next epoch thus allowing for higher-frequency checkpoints.

Figures

Figures reproduced from arXiv: 2505.09810 by the authors.

Figure 1
Figure 1. Tensor Data Shards Evolving Inputs Strengths Sum Transform Output [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. LLM Floating Point Formats Name Model Size (params) Tokens Type Chkpt Size Chkpt Count Shard Splits bloom 176B 366B BF16 329GiB 20 72 amber 6.74B 1.25T BF16 13GiB 358 3 transnormer 7B 1.4T BF16 28GiB 18 11 multiberts 110M 160M FP32 441MiB 28 1 olmo 7B 2.5T FP32 26GiB 558 1 gpt2 124M 4M FP32 1.4GiB 2000 1 [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Sample Self-Attention Weights over Time from BLOOM Data Set [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Bit Distribution for BLOOM Self-Attention QKV Weights Shard 0 [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: XOR Delta Bit Distribution for BLOOM Self-Attention QKV Weights Shard 0 [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Bit changes for bfloat16 values -0.25 to 0.25 modified in 0.005 increments [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Example Byte-Grouping of four 16-bit values [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Entropy Comparison with Off-the-shelf Engines on BLOOM Data [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Comparison of BZ2, LMC, BG-LMC compression ratios vs. entropy of BLOOM data [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: Summary of Encoder Performance 4.3 Parallel LMC (PLMC) We implemented a simple data-parallel version of the BG-LMC algorithm (referred to as PLMC) that iteratively reads byte-grouped data into an input buffer and uses OpenMP to implement multi-threaded compression and…
Figure 12
Figure 12. Figure 12: Compressability over time for 16 BLOOM shards [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 13
Figure 13. Figure 13: Comparison of PBZ2, PIGZ, PLMC compression throughput [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 14
Figure 14. Figure 14: Comparison of PBZ2, PIGZ, PLMC decompression throughput (MiB/s) [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 7 canonical work pages

  1. [18]

    Lossless and Near-Lossless Compression for Foundation Models

    Moshik Hershcovitch et al. Lossless and Near-Lossless Compression for Foundation Models . 2024. arXiv: 2404.15198 [cs.LG]. URL: https://arxiv.org/abs/2404.15198

    arXiv 2024

  2. [17]

    FCBench: Cross-Domain Benchmarking of Lossless Compression for Floating-Point Data

    Xinyu Chen et al. “FCBench: Cross-Domain Benchmarking of Lossless Compression for Floating-Point Data”. In: Proc. VLDB Endow. 17.6 (May 2024), pp. 1418–1431. ISSN : 2150-8097. DOI: 10.14778/3648160.3648180. URL: https://doi.org/10.14778/3648160.3648180

    arXiv 2024

  3. [1]

    Efficient Decoding of Prefix Codes

    Daniel S. Hirschberg and Debra A. Lelewer. “Efficient Decoding of Prefix Codes”. In: Communications of the ACM 33 (1990), pp. 449–459

    work page 1990

  4. [2]

    Michael Burrows and David J. Wheeler. A Block-Sorting Lossless Data Compression Algorithm. Tech. rep. 124. Digital Systems Research Center, May 1994

    work page 1994

  5. [3]

    Peter Deutsch

    L. Peter Deutsch. DEFLATE Compressed Data Format Specification version 1.3. RFC 1951. May 1996. DOI: 10.17487/RFC1951. URL: https://www.rfc-editor.org/info/rfc1951

    doi:10.17487/rfc1951 1951

  6. [4]

    Lessons Learned from the Analysis of System Failures at Petascale: The Case of Blue Waters

    Catello Di Martino et al. “Lessons Learned from the Analysis of System Failures at Petascale: The Case of Blue Waters”. In: 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks. 2014, pp. 610–621. DOI: 10.1109/DSN.2014.62

    work page doi:10.1109/dsn.2014.62 2014

  7. [5]

    Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms

    Nikhil Buduma and Nicholas Locascio. Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms. 1st. O’Reilly Media, Inc., 2017. ISBN : 1491925612

    work page 2017

  8. [6]

    Failures in large scale systems: long-term measurement, analysis, and implications

    Saurabh Gupta et al. “Failures in large scale systems: long-term measurement, analysis, and implications”. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. SC ’17. Denver, Colorado: Association for Computing Machinery, 2017. ISBN : 9781450351140. DOI: 10.1145/3126908.3126937. URL: https://do...

    arXiv 2017

Show all 19 references
  1. [7]

    Development of a Lossless Data Compression Algorithm for Multichannel Environmental Monitoring Systems

    Hussein Sh. Mogahed and Alexey G. Yakunin. “Development of a Lossless Data Compression Algorithm for Multichannel Environmental Monitoring Systems”. In: 2018 XIV International Scientific-Technical Conference on Actual Problems of Electronics Instrument Engineering (APEIE). 201...

    2018

  2. [8]

    Bfloat16 Processing for Neural Networks

    Neil Burgess et al. “Bfloat16 Processing for Neural Networks”. In: 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH). 2019, pp. 88–91. DOI: 10.1109/ARITH.2019.00022

    2019

  3. [9]

    Brown et al

    Tom B. Brown et al. Language Models are Few-Shot Learners . 2020. arXiv: 2005 . 14165 [cs.CL]. URL: https://arxiv.org/abs/2005.14165

    2020 arXiv

  4. [10]

    CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recom- mendation with Partial Recovery

    Kiwan Maeng et al. CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recom- mendation with Partial Recovery. 2020. arXiv: 2011.02999 [cs.LG]. URL: https://arxiv.org/abs/2011. 02999

    2020 arXiv

  5. [11]

    CheckFreq: Frequent, Fine-Grained DNN Checkpointing

    Jayashree Mohan, Amar Phanishayee, and Vijay Chidambaram. “CheckFreq: Frequent, Fine-Grained DNN Checkpointing”. In: 19th USENIX Conference on File and Storage Technologies (FAST 21). USENIX Association, Feb. 2021, pp. 203–216. ISBN : 978-1-939133-20-5. URL: https://www.usenix...

    2021

  6. [12]

    The MultiBERTs: BERT Reproductions for Robustness Analysis

    Thibault Sellam et al. “The MultiBERTs: BERT Reproductions for Robustness Analysis”. In: CoRR abs/2106.16163 (2021). arXiv: 2106.16163. URL: https://arxiv.org/abs/2106.16163. 11 Lossless Compression for LLM Tensor Incremental Snapshots

    2021 arXiv

  7. [13]

    Check-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models

    Assaf Eisenman et al. “Check-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models”. In: 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). Renton, W A: USENIX Association, Apr. 2022, pp. 929–943.ISBN : 978-1-939133-27-4. U...

    2022

  8. [14]

    Time Series Compression Survey

    Giacomo Chiarot and Claudio Silvestri. “Time Series Compression Survey”. In: ACM Comput. Surv. 55.10 (Feb. 2023). ISSN : 0360-0300. DOI: 10.1145/3560814. URL: https://doi.org/10.1145/3560814

    2023 doi

  9. [15]

    GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints

    Zhuang Wang et al. “GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints”. In: Proceedings of the 29th Symposium on Operating Systems Principles. SOSP ’23. Koblenz, Germany: Association for Computing Machinery, 2023, pp. 364–381. ISBN : 979840070229...

    2023

  10. [16]

    BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

    BigScience Workshop et al. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. 2023. arXiv: 2211.05100 [cs.CL]

    2023 arXiv

  11. [19]

    A Comprehensive Survey of Compression Algorithms for Language Models

    Seungcheol Park et al. A Comprehensive Survey of Compression Algorithms for Language Models. 2024. arXiv: 2401.15347 [cs.CL]. URL: https://arxiv.org/abs/2401.15347. 12 Lossless Compression for LLM Tensor Incremental Snapshots Appendix A A.1 Data Set Descriptions • Bloom - BigS...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.