Pith. sign in

REVIEW 4 major objections 6 minor 23 references

StateSMix: Online Lossless Compression via Mamba State Space Models and Sparse N-gram Context Mixing

T0 review · 4 major / 6 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Download PDF Read on arXiv ↗

Pith's one-line read A tiny online-trained Mamba model plus sparse n-grams beats xz on Wikipedia text up to 10 MB without any pre-trained weights or GPU.

desk verdict Solid pure-C online Mamba+n-gram compressor that honestly beats xz on small enwik8 files, but internal bpb/size mismatches and unreadable tables make the headline margins hard to trust until fixed. read the letter →

arxiv 2605.02904 v1 submitted 2026-04-05 cs.LG cs.ITmath.IT

Roberto Tacconelli This is my paper
classification cs.LGcs.ITmath.IT
keywords losslesscompressionstatespacemodelsMambaonlinelearningarithmeticcodingn-gramBPEtokenisationcontextmixing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

StateSMix shows that lossless compression can be done entirely online: a small Mamba-style state-space model is initialised from scratch on the file being compressed and trained token-by-token, so no external weights need to be shipped. Sparse n-gram tables (from bigrams up to 32-grams) then inject exact local and long-range pattern counts as a softmax-invariant logit bias that is scaled by the neural model’s own uncertainty. On the enwik8 Wikipedia benchmark the system reaches 2.123–2.162 bits per byte on 1–10 MB excerpts, beating the strong classical compressor xz -9e by up to 8.7 %. Ablations establish that the online SSM is the main engine (already better than xz alone on 3 MB) while the n-grams supply a complementary few-percent gain. The whole pipeline is pure C with AVX2, needs no GPU, and keeps every bit of model knowledge inside the compressed stream.

What carries the argument

Softmax-invariant sparse logit bias: n-gram counts update only the non-zero-count tokens by δj = λ log(1 + cj/α), which is equivalent to a Bayesian likelihood update of the SSM prior and is further scaled by an entropy-adaptive factor so that n-grams dominate only when the neural model is uncertain.

What would settle it

Re-run the identical pure-C binary, with the published hyper-parameters frozen, on a non-Wikipedia English corpus of 1–10 MB (for example Project Gutenberg novels or news wire) and check whether the compressed size still undercuts xz -9e by a comparable margin.

Watch

Extended reading notes

Core claim

A lightweight Mamba SSM trained online from random initialisation on a single file, mixed with sparse n-gram logit biases under entropy-adaptive scaling, produces a fully self-contained lossless compressor that outperforms xz -9e on enwik8 excerpts up to 10 MB; the SSM alone already accounts for a 46.6 % size reduction over a frequency baseline and beats xz without any n-gram tables.

Load-bearing premise

The many hand-tuned mixing strengths, learning rates and warm-up schedules that were fitted on Wikipedia excerpts will still beat xz on other kinds of natural-language files of similar size.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. StateSMix is a fully self-contained lossless compressor that trains a small Mamba-style SSM (DM=32, NL=2) online on the file being compressed and mixes it with sparse n-gram hash tables (orders 2–8, 16, 32) via a softmax-invariant logit bias and entropy-adaptive scaling, then encodes with range arithmetic coding. No pre-trained weights or GPU are required; the system is pure C with AVX2. On enwik8 the authors report beating xz -9e on 1–10 MB excerpts (claimed ~8.7%/5.4%/0.7%), with ablations arguing that the SSM alone drives most of the gain (~46.6% over a frequency baseline and already below xz on 3 MB) while n-grams add a complementary ~4.1%. The paper also reports speed (~2k tok/s with OpenMP), memory, and a theoretical reading of the bias as a Bayesian update and of the architecture as a neural PPM/PAQ hybrid.

Significance. If the reported ratios hold under clean, reproducible evaluation, the work is a useful systems contribution: it shows that a tiny online Mamba model plus sparse exact-context tables can match or beat a strong classical compressor (xz -9e) on moderate-size natural-language files without external weights or GPU. Strengths include a clear self-contained design, a carefully derived sparse logit-bias update (Eq. 10), entropy-adaptive mixing (Eq. 11), and ablations that attempt to separate SSM vs n-gram contributions. Open-source pure-C code is a practical plus. The niche (portable online neural compression up to ~10 MB) is narrower than LLM-based compressors but is of genuine engineering interest.

major comments (4)
  1. Load-bearing result tables (Tables 1, 4, 5 and related) are rendered as unreadable glyph blocks in the manuscript, so compressed sizes, bpb, Δ vs xz, and ablation rows cannot be verified from the text. The central claim (beats xz by stated margins; SSM alone 840 KB vs xz 852 KB; full ~806 KB / 4.1% gain) rests on these numbers. Please restore machine-readable tables with absolute compressed bytes, bpb, and exact xz baselines for every row.
  2. Internal numeric inconsistency on headline ratios: the opening abstract reports 2.123 / 2.149 / 2.162 bpb on 1/3/10 MB, while the second abstract-like paragraph reports 2.161 bpb on 10 MB and 2.130 bpb on 100 MB; body text cites 840 KB / 852 KB / 806 KB on enwik83M. These must be reconciled to a single audited set of sizes (including header overhead) before the 8.7%/5.4%/0.7% claims can be accepted.
  3. Evaluation and hyperparameter selection are confined to enwik8 (lambda sweep and ablations on enwik83M; Sec. 4.5.3–4.5.4, Sec. 6). Free parameters include per-order λ/α (aggressive α=0.001 for 16/32-gram), entropy schedule (β, H0, smin/smax), Adam/label-smoothing/warm-up, and architecture width. Without at least one held-out domain (e.g., non-Wikipedia text, source code, or binary-ish data) or a frozen-hyperparameter transfer experiment, the claim of superiority on “natural language text up to ~10 MB” is not yet supported beyond the tuning corpus.
  4. Table 4 / Sec. 6.2: the crossover to xz superiority (~30 MB) and the 100 MB result are discussed qualitatively, but without a clear, consistent size table (and without reporting xz sizes side-by-side for 100 MB in recoverable form) the scaling narrative and the “practical niche” conclusion remain hard to audit. Please add a single consolidated results table covering 1/3/10/100 MB for StateSMix variants and xz -9e.
minor comments (6)
  1. Several tables (parameter counts, n-gram λ/α, speed/memory, per-order hit rates) are similarly corrupted; fix encoding so all numeric content is legible.
  2. Clarify whether reported bpb includes the Rice-coded vocab map and arithmetic-coder finish overhead, and report header size separately for small files where it is non-negligible.
  3. Sec. 5.2 Bayesian reading of Eq. (10) is standard and helpful; state explicitly that λ, α are free hyperparameters, not estimated posteriors.
  4. Related work: briefly note how StateSMix differs from PAQ/CMIX bit-level mixers beyond the high-level analogy in Sec. 5.5 (token-level vs bit-level, online SSM vs large static model zoo).
  5. Minor prose inconsistencies between the two abstract blocks and the introduction (e.g., 2.161 vs 2.162 bpb on 10 MB; ~120K active parameters) should be unified.
  6. Code link is given; for reproducibility, pin commit/hash and list exact compile flags and the xz version used for baselines.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical compressor evaluated against external baselines; softmax/Bayesian identities are standard, not self-referential fits.

full rationale

StateSMix is an empirical systems paper. Its load-bearing claims are measured compressed sizes (bpb) on enwik8 versus external tools (xz -9e and other classical/neural compressors), plus ablations that isolate SSM vs n-gram components. Those outcomes are not forced by construction from the method’s inputs: the model is trained online from scratch and the archive is scored against independent codecs. The only formal identities used—softmax translation invariance (Sec. 4.5.1) and the Bayesian reading of the sparse logit bias (Sec. 5.2, Eqs. 10/15–17)—are standard mathematical facts that justify an efficient implementation of a design choice; they do not define the reported bpb numbers. Hyperparameters (λ, α, entropy schedule, Adam settings) are hand-chosen and partly swept on enwik83M, which is a generalization/overfitting concern, not a circular derivation in which a fitted quantity is renamed a prediction. The sole self-citation (Nacrith [23]) appears only in the bibliography and is not load-bearing for any uniqueness claim or central result. No uniqueness theorem is imported, no ansatz is smuggled via prior author work, and no known empirical law is merely renamed. Score 0 with empty steps is therefore the correct outcome.

Assumptions & free parameters 6 free parameters · 5 assumptions · 3 invented entities

Load-bearing content is mostly engineering choices and standard coding theory, not new physical entities. The central performance claim rests on many free hyperparameters tuned on enwik8, on the assumption that identical online updates keep encoder and decoder distributions matched, and on enwik8 as a proxy for natural-language files. No machine-checked proofs; evidence is empirical.

free parameters (6)
  • SSM width/depth (DM=32, NL=2, DS=16, DI=64)
    Architecture size chosen for CPU feasibility; directly determines capacity and the ~120K active-parameter claim.
  • Adam η=0.002, β1/β2, Gclip=5.0, label smoothing ε=0.12
    Online training hyperparameters that control how fast the SSM adapts; not derived from theory.
  • Chunk size C=32 and warm-up niter schedule (8/4/2)
    Truncated-BPTT and early-chunk iteration counts chosen by hand to bootstrap before n-grams fill.
  • Per-order n-gram λ and α (incl. α=0.001 for 16/32-gram)
    Strength and smoothing of logit bias; paper reports a lambda sweep on enwik83M, so values are fit to the evaluation domain.
  • Entropy-adaptive scale (β=0.6, H0=5.5 nats, smin=0.2, smax=2.5)
    Controls when n-grams override the SSM; hand-set confidence mapping.
  • Auxiliary biases (BLZ=1.5, λr=0.05, λc=0.1) and hash size 2^24 / probe depth 8
    Extra context models and table capacity are free design knobs affecting both ratio and the 6.1GB memory footprint.
assumptions (5)
  • standard math Shannon source coding / arithmetic coding: expected length equals cross-entropy of the predictor; identical encoder/decoder model updates yield lossless decode.
    Invoked in Sec. 3.1 and Sec. 4.8 as the foundation of the compress-by-predict pipeline.
  • standard math Softmax is translation-invariant: softmax(ℓ+c·1)=softmax(ℓ), justifying sparse non-zero-only logit updates.
    Sec. 4.5.1; used to make high-order n-gram tables computationally feasible.
  • domain assumption Online Adam on chunk cross-entropy with truncated BPTT is a sufficient online learner for file-specific structure at DM=32.
    Sec. 3.3 and 4.4; not proved, only supported by the reported bpt curves.
  • domain assumption enwik8 excerpts are representative of the practical niche of natural-language files up to ~10MB.
    All main claims and the xz crossover narrative (Sec. 6–7) rest on this single corpus family.
  • ad hoc to paper Aggressive low-α long-range n-grams (16/32-gram) improve ratio without harmful overconfidence after entropy scaling.
    Sec. 4.5.3; justified by a sweep on enwik83M rather than a general bound.
invented entities (3)
  • Softmax-invariant sparse n-gram logit-bias update (Eq. 10)
    purpose: Inject exact context counts into SSM logits while touching only non-zero-count tokens.
    Core mixing mechanism of StateSMix; derived from softmax invariance plus a smoothed count likelihood, but the specific sparse implementation and order set are paper-specific.
  • Entropy-adaptive n-gram scale s(H) (Eq. 11)
    purpose: Down-weight n-gram bias when the SSM is confident and up-weight it when uncertain.
    Analytic single-weight mixer presented as a simplified PAQ-style gate; parameters chosen in this work.
  • StateSMix online Mamba+n-gram compressor architecture
    purpose: Fully self-contained lossless compressor with no external weights.
    The system as a whole is the paper’s product; evidence is the enwik8 measurements, not an external independent test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StateSMix: Online Lossless Compression via Mamba State Space Models and Sparse N-gram Context Mixing." pith.science (2026). https://pith.science/paper/2605.02904

@misc{pith2026260502904,
  author       = {Pith},
  title        = {Pith review of: StateSMix: Online Lossless Compression via Mamba State Space Models and Sparse N-gram Context Mixing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2605.02904}},
  note         = {Machine review of arXiv:2605.02904}
}
read the original abstract

We present StateSMix, a fully self-contained lossless compressor that couples an online-trained Mamba-style State Space Model (SSM) with sparse n-gram context mixing and arithmetic coding. The model is initialised from scratch and trained token-by-token on the file being compressed, requiring no pre-trained weights, no GPU, and no external dependencies. The SSM (DM=32, NL=2, approximately 120K active parameters per file) provides a continuously-updated probability estimate over BPE tokens, while nine sparse n-gram hash tables (bigram through 32-gram, 16M slots each) add exact local and long-range pattern memorisation via a softmax-invariant logit-bias mechanism that updates only non-zero-count tokens. An entropy-adaptive scaling mechanism modulates the n-gram contribution based on the SSM's predictive confidence, preventing over-correction when the neural model is already well-calibrated. On the standard enwik8 benchmark, StateSMix achieves 2.123 bpb on 1 MB, 2.149 bpb on 3 MB, and 2.162 bpb on 10 MB, beating xz -9e (LZMA2) by 8.7%, 5.4%, and 0.7% respectively. Ablation experiments establish the SSM as the dominant compression engine: it alone accounts for a 46.6% size reduction over a frequency-count baseline and beats xz without any n-gram component, while n-gram tables provide a complementary 4.1% gain through exact context memorisation. OpenMP parallelisation of the training loop yields 1.9x speedup on 4 cores. The system is implemented in pure C with AVX2 SIMD and processes approximately 2,000 tokens per second on commodity x86-64 hardware.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 4 linked inside Pith

  1. [1]

    C. E. Shannon. A mathematical theory of communication.Bell System Technical Journal, 27(3):379–423, 1948

    1948

  2. [2]

    Rissanen and G

    J. Rissanen and G. G. Langdon. Arithmetic cod- ing.IBM Journal of Research and Development, 23(2):149–162, 1979

    1979

  3. [3]

    Ziv and A

    J. Ziv and A. Lempel. A universal algorithm for sequential data compression.IEEE Transactions on Information Theory, 23(3):337–343, 1977

    1977

  4. [4]

    J. G. Cleary and I. H. Witten. Data compres- sion using adaptive coding and partial string matching.IEEE Transactions on Communica- tions, 32(4):396–402, 1984

    1984

  5. [5]

    I. H. Witten, R. M. Neal, and J. G. Cleary. Arith- metic coding for data compression.Communica- tions of the ACM, 30(6):520–540, 1987

    1987

  6. [6]

    D. A. Huffman. A method for the construction of minimum-redundancy codes.Proceedings of the IRE, 40(9):1098–1101, 1952

    1952

  7. [7]

    M. Mahoney. Adaptive weighting of context mod- els for lossless data compression. Technical report, Florida Institute of Technology, 2005. 9

    2005

  8. [8]

    B. Knoll. CMIX: A lossless data compressor using neural networks. http://www.byronknoll.com/ cmix.html, 2024

    2024

Show all 23 references
  1. [9]

    Burrows and D

    M. Burrows and D. J. Wheeler. A block-sorting lossless data compression algorithm. Technical report SRC-124, Digital Equipment Corporation, 1994

    1994

  2. [10]

    R. F. Rice. Some practical universal noiseless coding techniques. Technical report JPL-79-22, Jet Propulsion Laboratory, 1979

    1979

  3. [11]

    R. E. Kalman. A new approach to linear filter- ing and prediction problems.Journal of Basic Engineering, 82(1):35–45, 1960

    1960

  4. [12]

    Schmidhuber

    J. Schmidhuber. A fixed size storageO(n3)time complexity learning algorithm for fully recurrent continually running networks.Neural Computa- tion, 4(2):243–248, 1992

    1992

  5. [13]

    A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Represen- tations, 2022

    2022

  6. [14]

    Gu and T

    A. Gu and T. Dao. Mamba: Linear-time se- quence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  7. [15]

    Dao and A

    T. Dao and A. Gu. Transformers are SSMs: Gen- eralized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    2024 arXiv

  8. [16]

    F. Bellard. NNCP v2: Lossless data compression with transformer. https://bellard.org/nncp/, 2021

    2021

  9. [17]

    F. Bellard. ts_zip: Text compression using a large language model. https://bellard.org/ ts_zip/, 2023

    2023

  10. [18]

    Delétang, A

    G. Delétang, A. Ruoss, P.-A. Duquenne, E. Catt, T. Genewein, C. Mattern, J. Grau-Moya, L. K. Wenliang, M. Aitchison, L. Orseau, M. Hutter, and J. Veness. Language modeling is compres- sion. InInternational Conference on Learning Representations, 2024

    2024

  11. [19]

    Huang et al

    S. Huang et al. FineZip: Pushing the limits of large language models for practical lossless text compression.arXiv preprint arXiv:2409.17141, 2024

    2024 arXiv

  12. [20]

    DeepZip: Lossless data compression using recur- rent neural networks

    S.Goyal, K.Tatwawadi, S.Chandak, andI.Ochoa. DeepZip: Lossless data compression using recur- rent neural networks. InData Compression Con- ference, 2019

    2019

  13. [21]

    C. S. Valmeekam, H. Scheinin, H. Zhang, and D. Kalathil. LLMZip: Lossless text compres- sion using large language models.arXiv preprint arXiv:2306.04050, 2023

    2023 arXiv

  14. [22]

    M. Mahoney. Large text compression benchmark. http://mattmahoney.net/dc/text.html, 2011

    2011

  15. [23]

    Tacconelli

    R. Tacconelli. Nacrith: Neural lossless com- pression via ensemble context modeling and high-precision CDF coding.arXiv preprint arXiv:2602.19626, 2026. 10

    2026

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.