REVIEW 15 cited by
LLMZip: Lossless Text Compression using Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We provide new estimates of an asymptotic upper bound on the entropy of English using the large language model LLaMA-7B as a predictor for the next token given a window of past tokens. This estimate is significantly smaller than currently available estimates in \cite{cover1978convergent}, \cite{lutati2023focus}. A natural byproduct is an algorithm for lossless compression of English text which combines the prediction from the large language model with a lossless compression scheme. Preliminary results from limited experiments suggest that our scheme outperforms state-of-the-art text compression schemes such as BSC, ZPAQ, and paq8h.
Forward citations
Cited by 15 Pith papers
-
Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
Long-context training shifts language models from parametric knowledge to context reliance, producing an inverted-U in pretraining performance and context addiction in supervised fine-tuning.
-
LLM-based Source Code Compression via Thresholded Symbol Ranking
Bounding LLM next-token ranks to T=63 (or T=1) with escaped exceptions beats unbounded LLM ranking by up to 37% ratio and 40% speed on source code, and beats general-purpose compressors by up to 82% ratio.
-
StateSMix: Online Lossless Compression via Mamba State Space Models and Sparse N-gram Context Mixing
An online-trained Mamba SSM mixed with sparse n-gram logit bias beats xz on enwik8 up to 10MB without pre-trained weights or a GPU.
-
LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs
A frozen LLM can process very long contexts by recurrently compressing them with a Perceiver and injecting the compressed memory through gated cross-attention, with query-dependent compression boosting QA performance.
-
Separate Source Channel Coding Is Still What You Need: An LLM-based Rethinking
LLM-based arithmetic coding plus ECCT-enhanced LDPC decoding makes separate source and channel coding competitive with, and in these tests superior to, joint source-channel coding for text under a total-energy comparison.
-
LMM-driven Semantic Image-Text Coding for Ultra Low-bitrate Learned Image Compression
A single LMM generates and compresses image captions for ultra-low-bitrate learned image compression, improving LPIPS BD-rate by 41.58% over MISC.
-
PMKLC: Parallel Multi-Knowledge Learning-based Lossless Compression for Large-Scale Genomics Database
PMKLC is a parallel multi-knowledge learning-based genomic compressor that achieves the best average compression ratio and throughput among tested baselines, with gains over the strongest baseline under 1%.
-
DualComp: End-to-End Learning of a Unified Dual-Modality Lossless Compressor
A unified RWKV-based model compresses both images and text losslessly with a shared vocabulary, modality-specific contextual layers, and mixture-of-experts routing, matching or improving on much larger SOTA compressor...
-
ToDMA: Large Model-Driven Massive Token Communications for Semantic Multiple Access
ToDMA lets many devices share one wireless channel by transmitting token indices from a common codebook, recovering collisions with compressed sensing and masked-token prediction from pretrained models.
-
GPT as a Monte Carlo Language Tree: A Probabilistic Perspective
GPT models trained on a corpus are shown to approximate a tree of empirical next-token probabilities derived from the same corpus, with alignment increasing with model size.
-
L3TC: Leveraging RWKV for Learned Lossless Low-Complexity Text Compression
L3TC, a 3.2M-parameter RWKV-based compressor with an outlier-bypass tokenizer, saves roughly 50% of gzip's bitrate on enwik9 and decodes at megabyte-per-second rates.
-
Channel-Agnostic Semantic Compression for Bandwidth-Limited Visual Communication
RQ-NAC, a residual-quantized image codec with n-gram arithmetic coding, reports 671 times compression over uncompressed dashcam frames at modest reconstruction quality (SSIM 0.74, PSNR 23.6 dB).
-
Joint Lossless Compression and Steganography for Medical Images via Large Language Models
A joint lossless compression and steganography framework for medical images that splits bit planes into a VAE-compressed global part and an LLM-compressed local part, embedding secret messages in the local part.
-
EDPC: Accelerating Lossless Compression via Lightweight Probability Models and Decoupled Parallel Dataflow
EDPC reports state-of-the-art lossless compression ratios across text, image, float, and benchmark datasets while running about 2.7x faster than the strongest prior learned compressor, PAC.
-
An Enhanced Text Compression Approach Using Transformer-based Language Models
Removing vowels before LZW compression yields high compression ratios, but the resulting text cannot be restored without a large transformer, making the claimed state-of-the-art comparison unfair.
Discussion (0). Continue with ORCID to comment.