Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Encoding LLM-generated text with an LLM's next-token probabilities drives arithmetic coding to lossless compression ratios above 20x, far above Gzip's roughly 3x.

desk verdict The benchmark is useful, but the paper's lossless claim is unverified because arithmetic coding requires byte-identical model behavior that the paper itself says is not guaranteed. read the letter →

arxiv 2505.06297 v1 pith:KAVM4RMQ submitted 2025-05-07 cs.LG cs.CL

Yu Mao , Holger Pirk , Chun Jason Xue This is my paper · ORCID
classification cs.LGcs.CL
keywords losslesscompressionLLM-generatedtextarithmeticcodingnext-tokenpredictionentropyratiolargelanguagemodelsneural
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that LLM-generated text is so predictable to LLMs because the models themselves are trained to guess the next token, and that this predictability can be turned into a lossless compressor. Driving arithmetic coding with an LLM's next-token probabilities yields compression ratios from 14.62 to 23.80 across eight LLM-generated datasets, far above Gzip's roughly 3x and the roughly 3x to 12.6x range of earlier neural compressors. The central identity is the chain-rule factorization $P(X)=\prod_t P(x_t \mid x_{

What carries the argument

The central object is an autoregressive language model acting as a probability oracle for an arithmetic coder. The model computes $P(x_t \mid x_{<t})$ for each token via a transformer; the arithmetic coder maps the sequence of conditional probabilities to an interval, emitting a binary fraction whose length is about $\sum_t -\log_2 P(x_t \mid x_{<t})$ (Eqs. 3-5). This works because the next-token objective used in pretraining is the same objective that minimizes code length: a better predictor means smaller KL divergence between predicted and true distributions, hence shorter codes. The paper replaces the small recurrent or transformer predictors used in prior neural compressors with large pretrained LLMs, which is what pushes the ratios into the 20x range.

What would settle it

Encode one text on machine A and decode on machine B (or with a different software build) and compare the result to the original text; any mismatch would invalidate the lossless claim. A simpler necessary check: encode the same file twice with identical weights but different floating-point settings and see whether the two bitstreams are byte-identical, which they must be for the method to work.

Watch

Extended reading notes

Core claim

The paper establishes that next-token prediction is itself a compression engine. An LLM assigns a conditional probability $P(x_t \mid x_{<t})$ to each token; arithmetic coding uses these probabilities to narrow an interval and produces a bitstream whose length is close to $\sum_t -\log_2 P(x_t \mid x_{<t})$. On eight datasets of LLM-written text, this scheme achieves lossless compression ratios from 14.62 to 23.80, versus 2.73 to 5.70 for Gzip, 4.24 to 9.76 for LZMA, and 2.95 to 12.56 for the neural baselines. The advantage persists across general-purpose LLMs from 1B to 14B parameters, grows with chunk size up to a saturation point, and is largest when the compressor's training domain matches the text's domain; a 1B math-specialized model compresses math text about as well as a 14B general model. The paper further shows the effect is specific to machine-written text: the same model compresses LLM-generated movie reviews far better than human-written reviews, and the gap widens with context length. The paper explicitly notes in Section 4.4 that LLM inference is not guaranteed deterministic across runs, a necessary condition for the arithmetic decoder to reconstruct the original text.

Load-bearing premise

The load-bearing premise is that the same LLM can reproduce exactly the same token probabilities when decoding as when encoding; the paper's own deterministic-inference discussion (Section 4.4) says this may not hold because of floating-point rounding, routing choices, and hardware differences, and the paper never tests it.

Editorial extensions

If this is right

  • LLM-generated corpora can be stored losslessly at more than 20x compression with an open-weight LLM as the codebook, a large margin over the roughly 3x of Gzip.
  • The practical recipe for maximum compression is to use the largest available base (non-instruction-tuned) model, the longest context chunk that still shows gains (around 128 to 256 tokens), and, when the text is domain-specific, a model fine-tuned on that domain.
  • The method's compression ratio stays stable as file size grows, so it fits small and streaming workloads where dictionary compressors cannot accumulate repetition.
  • The gap between LLM text and human text compression means the value of this approach will increase as the proportion of machine-generated text in stored data grows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical standardized LLM codec is one logical next step: if the model weights and the inference routine are fixed as part of the format, the compression scheme becomes interoperable and its ratios reproducible, much as a codec specification fixes a decoder.
  • Because the compressed length is the LLM's own surprise about the text, the same pipeline yields a distribution-based signal for whether a text is machine-written; this is an inference not made in the paper but directly testable with the reported setup.
  • The reported dependence on domain alignment suggests that specialized open-weight compressors could approach or exceed closed-model compression on their own domains, making the 20x result achievable in practice without access to the original generating model.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes using LLMs as probability models for arithmetic coding to losslessly compress LLM-generated text. It analyzes the redundancy of LLM-generated data through n-gram statistics, entropy, and mutual information, then compares conventional and neural compressors with an LLM-driven arithmetic coder. Across 8 datasets and 14 LLMs, the authors report compression ratios up to 23.80x, far exceeding Gzip's roughly 3x performance on the same data. The framework is standard: factorize the sequence probability with the chain rule, use an LLM to estimate conditional token probabilities, and feed those probabilities to an arithmetic coder. The paper also studies the effects of chunk size, model scale, dataset scale, instruction tuning, and domain-specific fine-tuning on compression ratio.

Significance. If the lossless round trip is actually realized, this would be a strong empirical demonstration that open-weight LLMs' next-token probabilities match the statistics of LLM-generated text far better than conventional compressors, with clear implications for storing the growing volume of synthetic text. The paper gives a clean, parameter-free derivation of the code-length estimator in Eqs. (3)-(5), evaluates a broad set of models and datasets, and provides a natural explanation for why domain-tuned and instruction-tuned models behave differently. The main weakness is that the headline ratios are currently conditional: arithmetic coding is lossless only when the decoder reproduces exactly the same probability tables as the encoder, and the paper does not demonstrate that this holds. As a measurement of LLM predictability, the results are interesting; as a compression claim, they require additional verification.

major comments (3)
  1. [§4.3, §4.4] The losslessness guarantee of arithmetic coding requires the decoder to reproduce byte-identical conditional probabilities P(xt|x<t) at every step. Section 4.4 explicitly concedes that LLM inference is not fully deterministic across runs due to floating-point precision, MoE routing, and hardware differences, stating that 'even a near-zero temperature may not always yield perfectly consistent results.' The paper reports no encode-decode round trip, no bit-exactness check, and no specification of how a decompressor would reproduce the same logits (checkpoint, tokenizer, precision, batch size, attention implementation). This is load-bearing: if the probabilities differ at any step, the arithmetic coder diverges and the 'lossless >20x' claim fails. The authors should either demonstrate deterministic probability reproduction on their hardware or describe an archive format that ships the probability tables or model configuration, and then verify a complete round trip.
  2. [§5.3, Table 5, §5.1.3] Table 5 reports 'Ours' ratios up to 23.80x and the text says arithmetic coding was implemented, but the paper does not state whether S_compressed is the length of an actual arithmetic-coded bitstream or the ideal code length from Eq. (4). Because Eq. (4) defines length as -log2 P, the reported ratio is, by construction, a cross-entropy measurement; a real coder adds flushing, byte-alignment, and tokenization overhead, and Table 5 omits these. The table also does not state the chunk size, context length, or dataset subsample size used; §5.2.4 only says 'unless otherwise specified Llama-3.1-8B.' Since compression ratios depend strongly on chunk size (Fig. 5), the headline numbers are not reproducible as reported. Specify the exact codec, overhead, and settings, and report measured file sizes for at least the main configuration.
  3. [§5.2.4, §5.3] The comparison of 'Ours' to the baselines is not fully fair as presented because the authors do not state the hardware, runtime, or model inference settings used for the LLM-based method, while the baseline numbers come from standard compressor executables. Compression ratio alone is not enough: the paper should report compressed size as actually produced by the coder and, ideally, provide code or a detailed artifact description so that the numbers in Table 5 can be reproduced. Without this, the empirical claim that the method 'achieves' these ratios remains an estimate rather than a demonstrated file-based result.
minor comments (6)
  1. [Abstract, §3.3] The abstract labels Gzip the 'state-of-the-art lossless compressor,' but the paper's own Table 3 shows that LZMA and Zstd outperform Gzip on all three pilot datasets and NNCP outperforms all dictionary methods. The comparison should be reframed as 'widely used' or should compare against the strongest baselines, including LZMA and NNCP.
  2. [Introduction, Related Work] The paper claims to be the 'first systematic investigation' of LLM-generated data compression and the 'first to propose leveraging LLMs to compress LLM-generated synthetic data,' but related work [14, 23, 40] already applies LLMs and foundation models to lossless text compression. The novelty claim should be narrowed to the specific setting of LLM-generated data and qualified against these prior works.
  3. [§5.3, Table 5] Table 5 contains several typos and inconsistencies, including the column header 'Artical' instead of 'Article' and inconsistent naming of 'Artical' in the table versus 'Article' in the text; also, §5.4 and §5.5 contain 'Peformance' for 'Performance.'
  4. [Fig. 1, §1] The quadratic extrapolation predicting '16,000 TB within the next decade' is presented without confidence intervals, data-point details, or model diagnostics; it is motivational and should be softened or moved to the introduction with appropriate caveats.
  5. [§5.7.2] The statement that DeepSeek-Coder-1.3B performs comparably to Llama1B 'may be due to insufficient training' is speculative; either provide supporting evidence such as training corpus size or remove the speculation.
  6. [§5.6, Fig. 7] The claim that the proposed method is 'stable' with dataset scale is based on a narrow range (file sizes 10 to 80 units) and no error analysis; the units are unspecified and should be stated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper implements a standard LLM-driven arithmetic coder and measures compression ratios on held-out LLM-generated datasets; the fixed pretrained models are not fitted to the test data.

full rationale

The claimed derivation chain runs from Eq. (3) (chain-rule factorization), through Eq. (4) (self-information as code length), to arithmetic coding in Sec. 4.3 and the ratio definition in Eq. (18). This is the standard source-coding argument, not a circular one: for a fixed pretrained LLM used as the probability model, the compressed length is by definition the sum of -log2 P(x_t | x_<t) (up to coder overhead), and the compression ratio is therefore a measurement of that model's predictive accuracy on the test text. The LLM compressors are not trained or fine-tuned on the evaluation datasets, and the generating models (GPT-3, GPT-4, Mixtral, etc.) differ from the open-weight compression models, so the high ratios are an empirical finding about cross-model predictability rather than a fitted parameter renamed as a prediction. The only self-citations (TRACE [34], PAC [35]) appear as baseline compressors and are not load-bearing for the central claim. The paper's own Sec. 4.4 caveat about floating-point nondeterminism, and the absence of an explicit encode/decode round-trip, are reproducibility and correctness concerns about whether a deployed decoder can reproduce the same probability tables; they are not instances of a derivation reducing to its own inputs. Under the rule that only quotable reductions count as circularity, no circular step can be identified.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The compression pipeline rests on standard information theory (arithmetic coding with a correct probability model is exact, Eqs. 3 to 8) and on two unverified domain assumptions: deterministic reproduction of the LLM's probability distribution at decode time (contradicted in spirit by Sec. 4.4), and sufficient distributional overlap between the open-weight compressors used and the closed models that generated the datasets. The headline ratio is a cross-entropy measurement over data sampled from the LLM family, so no fitted constants are involved, but the absence of a generator-as-compressor baseline means the upper bound of the method is never measured.

free parameters (3)
  • chunk size for main results = not stated in paper
    The compression ratio depends on chunk size (Sec. 5.4), but the 'Ours' row in Table 5 does not state which chunk size produced the headline 14x to 24x numbers.
  • dataset subsample size = not stated
    Fig. 7 varies file size up to 80 (units unlabeled), implying subsampling; the exact volumes compressed for Table 5 are not reported, so ratios may not be directly comparable across rows.
  • quadratic growth projection coefficients = not reported
    The Introduction predicts 16,000 TB of LLM-generated data within a decade from a quadratic polynomial regression on Huggingface dataset counts; coefficients and fit range are absent. Peripheral to the central claim.
assumptions (3)
  • standard math Arithmetic coding with a shared probability model is a lossless scheme; encoded length equals the sum of -log2 P(xt | x<t) (Eqs. 3 to 5).
    Sections 4.1 to 4.3 restate Shannon entropy and arithmetic coding; this is standard and correct.
  • domain assumption The LLM produces byte-identical probability distributions at encode time and decode time, with identical tokenizer and weights.
    Section 4.3.2 requires the decoder to repeat the same interval updates; Section 4.4 itself argues LLMs are non-deterministic across runs due to floating-point precision, MoE routing, and hardware, so this assumption is load-bearing and unverified.
  • domain assumption Text generated by closed models (GPT-3.5, GPT-4, Mixtral, LongWriter) is well approximated by the distributions of open-weight compressors (Llama, Qwen, DeepSeek-Coder).
    The 20x ratios depend on cross-model overlap; the paper never compresses with the actual generating model, so this overlap is only measured indirectly across a set of open models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction." pith.science (2026). https://pith.science/paper/KAVM4RMQ

@misc{pith2026250506297,
  author       = {Pith},
  title        = {Pith review of: Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KAVM4RMQ}},
  note         = {Machine review of arXiv:2505.06297}
}
read the original abstract

As large language models (LLMs) continue to be deployed and utilized across domains, the volume of LLM-generated data is growing rapidly. This trend highlights the increasing importance of effective and lossless compression for such data in modern text management systems. However, compressing LLM-generated data presents unique challenges compared to traditional human- or machine-generated content. Traditional machine-generated data is typically derived from computational processes or device outputs, often highly structured and limited to low-level elements like labels or numerical values. This structure enables conventional lossless compressors to perform efficiently. In contrast, LLM-generated data is more complex and diverse, requiring new approaches for effective compression. In this work, we conduct the first systematic investigation of lossless compression techniques tailored specifically to LLM-generated data. Notably, because LLMs are trained via next-token prediction, we find that LLM-generated data is highly predictable for the models themselves. This predictability enables LLMs to serve as efficient compressors of their own outputs. Through extensive experiments with 14 representative LLMs and 8 LLM-generated datasets from diverse domains, we show that LLM-based prediction methods achieve remarkable compression rates, exceeding 20x, far surpassing the 3x rate achieved by Gzip, a widely used general-purpose compressor. Furthermore, this advantage holds across different LLM sizes and dataset types, demonstrating the robustness and practicality of LLM-based methods in lossless text compression under generative AI workloads.

Figures

Figures reproduced from arXiv: 2505.06297 by the authors.

Figure 1
Figure 1. Trend of increasing LLM-generated data. law, and software development. Authors use LLMs to co-author novels and generate narratives, as seen in Ayad Akhtar’s play “Mc￾Neal” [1]. In healthcare, they aid in diagnostics, automate documen￾tation, and enhance medical education [50]. In the legal domain, LLMs assist in drafting legal documents and summarizing cases [7]. Software development has also benefited from LLM-pow… view at source ↗
Figure 2
Figure 2. N-gram frequency count for LLM-generated synthetic data. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. LLM-generated data distribution. 2.3 Characteristics of LLM-Generated Text vs. Other Datasets We begin by analyzing the characteristics of different types of data, aiming to gain insights that can guide more effective compres￾sion strategies. The characteristics of LLM-generated text differ significantly from traditional structured datasets used in database benchmarks. While machine-generated datasets such as TPC-H … view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: General process of Neural-based Compression. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Compression ratio comparison across different Llama models. Larger models achieve higher compression ratios, [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Model scale vs. compression ratio. Larger models consistently demonstrate higher compression ratios, indicating that [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Compression ratio variations with increasing [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Performances of domain-specific models on Math [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Compression ratio comparison between human [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Joint Lossless Compression and Steganography for Medical Images via Large Language Models

    eess.IV 2025-08 reject novelty 4.0 of 10

    A joint lossless compression and steganography framework for medical images that splits bit planes into a VAE-compressed global part and an LLM-compressed local part, embedding secret messages in the local part.

  2. EDPC: Accelerating Lossless Compression via Lightweight Probability Models and Decoupled Parallel Dataflow

    cs.IT 2025-07 conditional novelty 4.0 of 10

    EDPC reports state-of-the-art lossless compression ratios across text, image, float, and benchmark datasets while running about 2.7x faster than the strongest prior learned compressor, PAC.

Reference graph

Works this paper leans on

60 extracted references · 27 canonical work pages · cited by 2 Pith papers

  1. [1]

    Ayad Akhtar. 2024. The Role of AI in Creative Writing: A Case Study on McNeal. The Atlantic (2024). https://www.theatlantic.com/magazine/archive/2024/11/ mcneal-play-akhtar-downey-artificial-intelligence/679949/

    work page 2024

  2. [2]

    Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs. arXiv preprint arXiv:2408.07055 (2024)

    arXiv 2024

  3. [3]

    Fabrice Bellard. 2023. NNCP: Neural Network Compression Program. https: //bellard.org/nncp/. Accessed: 2024-02-19

    work page 2023

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    work page 2018

  5. [5]

    Bryant, Randy H

    Randal E. Bryant, Randy H. Katz, and Edward D. Lazowska. 2008. Big-Data Computing: Creating Revolutionary Breakthroughs in Commerce, Science, and Society. (2008). https://cra.org/ccc/wp-content/uploads/sites/2/2015/05/Big_ Data.pdf Accessed: 2025-03-01

    work page 2008

  6. [6]

    Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712 (2023)

    arXiv 2023

  7. [7]

    Carrell and R

    D. Carrell and R. Smith. 2024. Legal AI: Applications and Risks of Large Language Models in Law. Harvard Law Review 137 (2024), 112–130. https://www.reuters.com/technology/artificial-intelligence/ai- hallucinations-court-papers-spell-trouble-lawyers-2025-02-18/

    work page 2024

  8. [8]

    M. et al. Chen. 2021. Evaluating Large Language Models for Code Generation. arXiv preprint arXiv:2107.03374 (2021). https://arxiv.org/abs/2107.03374

    arXiv 2021

Show all 60 references
  1. [9]

    Yann Collet. 2013. Finite State Entropy. https://github.com/Cyan4973/ FiniteStateEntropy. Accessed: 2025-03-01

    2013

  2. [10]

    Y. Collet. 2016. Zstd github repository from facebook. https://github.com/ facebook/zstd

    2016

  3. [11]

    Transaction Processing Performance Council. 2011. TPC-H. http://www.tpc. org/tpch/default.asp

    2011

  4. [12]

    D. Cox. 2016. Syntactically informed text compression with recurrent neural networks. arXiv preprint arXiv:1608.02893 (2016)

    2016 arXiv

  5. [13]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  6. [14]

    Delétang, A

    G. Delétang, A. Ruoss, P. Duquenne, E. Catt, T. Genewein, C. Mattern, J. Grau- Moya, L. K. Wenliang, M. Aitchison, L. Orseau, M. Hutter, and J. Veness. 2024. Language modeling is compression. In International Conference on Learning Representations (ICLR)

    2024

  7. [15]

    Peter Deutsch. [n. d.]. GZIP file format specification version 4.3. RFC 1952 ([n. d.])

    1952

  8. [16]

    Dmitry Lepikhin et al. 2021. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. In Proceedings of NeurIPS. https://arxiv. org/abs/2006.16668

    2021 arXiv

  9. [17]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical Neural Story Generation. In Proceedings of ACL. https://arxiv.org/abs/1805.04833

    2018 arXiv

  10. [18]

    Markus Freitag and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.01806 (2017)

    2017 arXiv

  11. [19]

    Goyal, K

    M. Goyal, K. Tatwawadi, S. Chandak, and I. Ochoa. 2019. Deepzip: Lossless data compression using recurrent neural networks. In Data Compression Conference (DCC)

    2019

  12. [20]

    contamination

    Andrew Gray. 2023. ChatGPT "contamination": estimating the prevalence of LLMs in the scholarly literature. https://en.wikipedia.org/wiki/Generative_ artificial_intelligence

    2023

  13. [21]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guant- ing Chen, Xiao Bi, Y. Wu, Y.K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang

  14. [22]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. In Proceedings of ICLR. https://arxiv. org/abs/1904.09751

    2020 arXiv

  15. [23]

    Huang, J

    Y. Huang, J. Zhang, Z. Shan, and J. He. 2024. Compression represents intelligence linearly. arXiv preprint arXiv:2404.09937 (2024)

    2024 arXiv

  16. [24]

    D. A. Huffman. 1952. A Method for the Construction of Minimum-Redundancy Codes. Vol. 40. 1098–1101 pages. doi:10.1109/JRPROC.1952.273884 Accessed: 2024-11-07

    1952

  17. [25]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-Coder Technical Report. arXiv preprint arXiv:2409.12186 (2024)

    2024 arXiv

  18. [26]

    IBM Corporation. 2013. Bringing Big Data and Business Analytics to the Enter- prise. (2013). https://public.dhe.ibm.com/software/os/systemz/pdf/Bringing_ Big_Data_and_Business_Analytics_to_the_Enterprise.pdf Accessed: 2025-03-01

    2013

  19. [27]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton

  20. [28]

    Prabha Kannan. 2023. How Much Research Is Being Written by Large Language Models? https://en.wikipedia.org/wiki/Generative_artificial_intelligence

    2023

  21. [29]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Opti- mization. In International Conference on Learning Representations

    2015

  22. [30]

    Sunjun Kweon, Junu Kim, Jiyoun Kim, Sujeong Im, Eunbyeol Cho, Seongsu Bae, Jungwoo Oh, Gyubok Lee, Jong Hak Moon, Seng Chan You, Seungjin Baek, Chang Hoon Han, Yoon Bin Jung, Yohan Jo, and Edward Choi. 2023. Publicly Shareable Clinical Large Language Model Built on Synthetic C...

    2023 arXiv

  23. [31]

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society. arXiv:2303.17760 [cs.AI]

    2023 arXiv

  24. [32]

    Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. 2024. Rho-1: Not All Tokens Are What You Need. arXiv:2404.07965 [cs.CL]

    2024 arXiv

  25. [33]

    Jiang Liu, Jialian Wu, Prakamya Mishra, Zicheng Liu, Sudhanshu Ranjan, Pratik Prabhanjan Brahma, Yusheng Su, Gowtham Ramesh, Peng Sun, Zhe Li, Dong Li, Lu Tian, and Emad Barsoum. 2024. AMD-OLMo: A series of 1B lan- guage models trained from scratch by AMD on AMD Instinct ™ MI2...

    2024

  26. [34]

    Yu Mao, Yufei Cui, Tei-Wei Kuo, and Chun Jason Xue. 2022. Trace: A fast transformer-based general-purpose lossless compressor. In Proceedings of the ACM Web Conference 2022. 1829–1838

    2022

  27. [35]

    Yu Mao, Jingzong Li, Yufei Cui, and Jason Chun Xue. 2023. Faster and stronger lossless compression with optimized autoregressive framework. In 2023 60th ACM/IEEE Design Automation Conference (DAC). IEEE, 1–6

    2023

  28. [36]

    Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. 2024. OpenELM: An Efficient Language Model Family with Open Training and Inference Framework. a...

    2024 arXiv

  29. [37]

    Meta. 2024. Introducing Llama 3.1: Our most capable models to date. https: //ai.meta.com/blog/meta-llama-3-1/

    2024

  30. [38]

    Meta. 2024. Llama 3.2: Revolutionizing edge AI and vision with open, customiz- able models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge- mobile-devices/

    2024

  31. [39]

    Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah

  32. [40]

    Mittu, Y

    F. Mittu, Y. Bu, A. Gupta, A. Devireddy, A. E. Ozdarendeli, A. Singh, and G. Anumanchipalli. 2024. Finezip: Pushing the limits of large language models for practical lossless text compression. arXiv preprint arXiv:2409.17141 (2024)

    2024 arXiv

  33. [41]

    Andreas Bentzen Winje Nicolai Thorer Sivesind. 2023. Human-vs-Machine

    2023

  34. [42]

    arXiv:2402.14830 [cs.CL]

    Orca-Math: Unlocking the potential of SLMs in Grade School Math. arXiv:2402.14830 [cs.CL]

    arXiv

  35. [43]

    OpenAI. 2023. ChatGPT. https://en.wikipedia.org/wiki/ChatGPT

    2023

  36. [44]

    OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]

    2023 arXiv

  37. [45]

    OpenAI. 2022. ChatGPT. https://chat.openai.com/chat/

    2022

  38. [46]

    Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. https://qwenlm. github.io/blog/qwen2.5/

    2024

  39. [47]

    Brian Thompson, Mehak Dhaliwal, Peter Frisch, Tobias Domhan, and Marcello Federico. 2024. A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism. https://en.wikipedia.org/wiki/Generative_artificial_ intelligence

    2024

  40. [48]

    Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bell system technical journal 27, 3 (1948), 379–423

    1948

  41. [49]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems . 5998–6008

    2017

  42. [50]

    Wang and J

    Y. Wang and J. Lee. 2023. Applications of Large Language Models in Medical Diagnostics and Education. Journal of Medical AI Research 45 (2023), 23–35. https://pmc.ncbi.nlm.nih.gov/articles/PMC11396764/

    2023

  43. [51]

    Hugo Touvron, Albert Jiang, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint (2023). https://arxiv.org/abs/2302.13971

    2023 arXiv

  44. [53]

    Li Xu, Yanqing He, Guanghui Li, Wei Li, and Wen Shi. 2021. Big Data-Driven Smart Manufacturing: Challenges, Advances and Future Trends. IEEE Transactions on Industrial Informatics 17, 3 (2021), 2274–2284. Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Trovato et al

    2021

  45. [54]

    Witten et al. 1987. Arithmetic coding for data compression. Commun. ACM 30, 6 (1987), 520–540

    1987

  46. [55]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. Qwen2.5-Math Technical Report: Toward Mathematical Expert Model ...

    2024 arXiv

  47. [56]

    Hailin Zhang, Penghao Zhao, Xupeng Miao, Yingxia Shao, Zirui Liu, Tong Yang, and Bin Cui. 2023. Experimental Analysis of Large-Scale Learnable Vector Storage Compression. Proc. VLDB Endow.17, 4 (Dec. 2023), 808–822. doi:10.14778/3636218. 3636234

    2023 doi

  48. [57]

    Vishaal Y. 2025. Synthetic Code Generations Dataset. https://huggingface.co/ datasets/VishaalY/synthetic-code-generations

    2025

  49. [58]

    Ziv et al. 1977. A universal algorithm for sequential data compression. IEEE Transactions on information theory 23, 3 (1977), 337–343

    1977

  50. [60]

    Lixi Zhou, Jiaqing Chen, Amitabh Das, Hong Min, Lei Yu, Ming Zhao, and Jia Zou. 2022. Serving deep learning models with deduplication from relational databases. 15, 10 (June 2022), 2230–2243. doi:10.14778/3547305.3547325

    2022

  51. [1991]

    Neural Computation 3, 1 (03 1991), 79–87

    Adaptive Mixtures of Local Experts. Neural Computation 3, 1 (03 1991), 79–87

    1991

  52. [2024]

    https://arxiv.org/abs/2401.14196

    DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence. https://arxiv.org/abs/2401.14196

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.