L3TC: Leveraging RWKV for Learned Lossless Low-Complexity Text Compression

Dajiang Zhou; Guo Lu; Junxuan Zhang; Li Song; Shihao Wang; Yan Zhao; Zhengxue Cheng

REVIEW 3 major objections 5 minor 1 cited by

L3TC: Leveraging RWKV for Learned Lossless Low-Complexity Text Compression

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 200K-parameter neural text compressor matches larger learned models and decodes at MB/s.

desk verdict L3TC is a solid engineering contribution with a real speed-measurement gap: the headline decode speed is model-only throughput, not end-to-end, and the arithmetic coder may dominate. read the letter →

arxiv 2412.16642 v2 pith:CX4G62UX submitted 2024-12-21 cs.CL cs.AIcs.ITcs.MMmath.IT

Junxuan Zhang , Zhengxue Cheng , Yan Zhao , Shihao Wang , Dajiang Zhou , Guo Lu , Li Song This is my paper · ORCID

classification cs.CLcs.AIcs.ITcs.MMmath.IT

keywords losslesstextcompressionRWKVlearnedoutlier-awaretokenizerhigh-rankreparameterizationarithmeticcodinglow-complexityinferenceenwik9

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that learned lossless text compression can be made practical by choosing an architecture whose inference is fast, restricting tokenization to frequent tokens, and spending extra model capacity only during training. It introduces L3TC, built on a small RWKV model, an outlier-aware tokenizer that lets rare tokens bypass prediction and encoding, and a high-rank reparameterization that merges extra branches away at inference time. The paper reports a 48% bit saving over gzip, compression ratios comparable to learned compressors that are 50 times larger, and decoding throughput up to 1.30 MB/s on a phone and 4.35 MB/s on an A100. A sympathetic reader would care because this points toward neural compression that could run in real time on ordinary hardware rather than only on large servers.

What carries the argument

The load-bearing object is the smallest tested RWKV model combined with the outlier-aware tokenizer and the high-rank reparameterization (HiRA). RWKV's linear attention keeps a fixed-size state, so per-token inference time does not grow with sequence length; the outlier-aware tokenizer merges frequent subwords into a 16K-token vocabulary and routes remaining rare tokens to a raw-byte bypass, lowering the number of model forward passes; HiRA adds trainable high-rank branches to the R/K/V matrices that are merged into a single matrix at inference, improving training capacity without adding multiply-accumulate operations.

What would settle it

Run L3TC-200K end to end on enwik9 on an iPhone 12, timing a full decode that includes the arithmetic decoder, the outlier-aware detokenizer, and file I/O, and compare the wall-clock throughput to the reported 1.30 MB/s; if the measured end-to-end speed falls well below 1 MB/s, the practical-speed claim for the compressor as a whole does not hold.

Watch

Extended reading notes

Core claim

The central claim is that a low-complexity learned text compressor can match the compression performance of much larger learned models without paying their inference cost. The evidence combines three components: RWKV, a recurrent architecture with a fixed-length hidden state, gives the best speed-to-compression trade-off among tested backbones; an outlier-aware tokenizer with a 16K vocabulary and 0.999 coverage cuts the number of tokens the model must score while coding rare characters directly as UTF-8 bytes; and high-rank reparameterization trains R, K, and V branches that are summed into the main weights so inference has a single path. On enwik9, L3TC-3.2M compresses to 16.23% of the original size versus 32.26% for gzip, and its adjusted compression ratio, which includes model size, is the best among all compared learned compressors.

Load-bearing premise

The headline megabyte-per-second decoding speed is measured as batch size times average bytes per token divided by batch inference time, with the arithmetic coder, tokenizer, and file input/output excluded; if those components take non-negligible time, the whole-compressor throughput is lower than reported.

Editorial extensions

If this is right

Compression performance on a fixed corpus can be improved without raising inference cost by spending train-time capacity on reparameterized branches.
Subword tokenization that deliberately leaves rare tokens to a bypass can lower both compressed size and decoding latency, because the model scores fewer tokens.
A recurrent backbone with constant-memory inference is a better starting point than a transformer for on-device neural compression.
For small corpora, model size dominates total storage, so L3TC's low parameter count makes its adjusted compression ratio the relevant practical metric.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The outlier-bypass idea could transfer to byte-level image or audio compression, where a small model handles frequent symbols and a raw bypass covers the tail.
End-to-end timing that includes the arithmetic coder, tokenizer, and file I/O would make the speed comparison with classical compressors like zstd fair; the paper's metric isolates the neural model.
If the speed claim holds across devices, a natural next test is whether the design transfers to other domains where a 16K vocabulary may not cover the frequent-token tail.
The high-rank reparameterization could apply to other linear-attention or RNN-based predictors, since it only changes training and merges away at inference.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this paper is a well-executed engineering effort showing that a 3.2M-parameter RWKV can reach ~16% CR on enwik9, beating gzip by half and matching small transformer compressors. The real news is that it proposes a practical low-complexity learned text compressor with a sensible escape-token tokenizer and a training-only high-rank reparameterization. That is genuinely useful for on-device compression.

The paper does several things well. The backbone comparison (Transformer, Transformer-XL, RWKV) is thorough and the choice of RWKV is justified by speed at equal size. The outlier-aware tokenizer is a principled way to handle rare tokens: keep a 16K vocab covering 99.9% of characters, send the rest as raw UTF-8. The HiRA trick is simple and the ablation shows it buys ~0.85% CR without adding inference cost. Compression numbers are internally consistent across model sizes and datasets.

The main soft spot is the decode-speed claim. Section 4.3 defines speed as batch_size * bytes_per_token / model_inference_time. That excludes arithmetic decoding, tokenization and I/O. The stress-test note is right: arithmetic decoding is sequential per symbol, while the model forward pass is batch-amortized. So with batch 256, the model cost per token drops, but the arithmetic coder cost scales with batch size. For a 200K model with sub-millisecond batch inference, even a few microseconds per symbol could dominate. The paper's Eq. 2 hand-wave that the coder is 'usually smaller' needs a number. Without an end-to-end measurement, the 'fastest among all learned compressors' and 'MB/s on mobile' claims are not supported as written.

A second, smaller issue: the abstract says compression is 'comparable to other learned compressors.' That is true for small transformer models, but cmix and nncp (in supplementary) reach ~11% CR. So the comparison is only fair within the low-complexity niche. The paper should say that.

Who should read this: people building on-device text compression or studying efficient LLM inference. It is not an information-theory breakthrough, but it is a solid, reproducible data point. I would give it peer review, but the authors should run a proper end-to-end decoder benchmark. If the speed claim survives that, this becomes a solid paper; if not, it's still a useful compression experiment but with a weaker practical hook.

Referee Report

3 major / 5 minor

Summary. The paper proposes L3TC, a learned lossless text compressor built around a small RWKV model. Three components are introduced: an outlier-aware tokenizer that restricts the vocabulary to frequent tokens and sends rare tokens as raw UTF-8 bytes; a high-rank reparameterization (HiRA) scheme that adds trainable low-complexity branches and merges them into the main weights at inference; and an architecture-level comparison motivating the choice of RWKV as the backbone. Experiments on enwik8/enwik9 report compression ratios around 16--18% on enwik9 (about 48--50% bit saving versus gzip), lower ACR than large pretrained LLM-based compressors, model sizes from 200K to 12M parameters, and decoding speeds up to 1.30 MB/s on an iPhone 12 ANE and 4.35 MB/s on an A100. The paper claims that L3TC is the fastest among all learned compressors and offers real-time decode speeds.

Significance. If the speed claim is substantiated end-to-end, L3TC would be a practically valuable result: it demonstrates that a sub-1M-parameter neural model can approach the compression of much larger learned systems while running on mobile hardware. The compression-ratio experiments are internally consistent, the ACR metric usefully accounts for model-size overhead, and the ablation studies for vocabulary size, coverage, rank, and branch count are informative. The proposed outlier-aware tokenizer and HiRA reparameterization are simple and plausible ideas, and the paper does not appear to have any definitional circularity: the reported ratios are measured outputs on held-out text, with hyperparameters selected from ablations. However, the central practical contribution currently rests on a speed measurement that excludes arithmetic coding, tokenization, and I/O, so the headline speed advantage is not yet established.

major comments (3)

[Section 4.3, Table 5; Section 3.1, Eq. (2)] The central decoding-speed claim is computed from model batch throughput only. The definition in Section 4.3 multiplies batch size by average bytes per token and divides by batch inference time, with no measurement of the arithmetic coder, the tokenizer, or file I/O. The justification in Eq. (2) that arithmetic coding cost is 'usually smaller' than model inference is not supported by any measurement or citation, and it is especially questionable at the batch sizes used here: the RWKV forward pass amortizes model cost across a batch of 256 or 2048 sequences, whereas arithmetic decoding is inherently sequential per symbol and its cost scales with batch size rather than amortizing. Consequently, the abstract and conclusion claims of 'real-time decoding speeds up to megabytes per second' and 'fastest among all learned compressors' are overstated on the current evidence. The authors should report end-to-end encoder and decoder wall-clock throughput, including arithmetic coding and tokenization, and should qualify the headline numbers accordingly if only model inference is being measured.
[Section 4.1, Section 4.3, Table 5] The throughput measurement uses parallel batch processing of 2048-byte chunks, but the paper does not specify how this maps to the decoding of a single compressed stream. For a single file, either the stream must be split into independent chunks with context resets and additional framing overhead, or the speed applies only to concurrently decoding many independent files. Neither the chunk overhead nor the single-stream latency is measured or discussed. Since lossless compression of a single text file is the natural use case, the authors should state the intended deployment scenario and report the corresponding end-to-end throughput; otherwise the MB/s figure is ambiguous and cannot be compared directly with classical single-stream compressors such as gzip.
[Section 3.3, Eq. (3), Table 3, Fig. 5] The outlier-aware tokenizer transmits unknown tokens as raw UTF-8 bytes, but the paper does not provide a quantitative breakdown of the compressed output between in-vocabulary arithmetic-coded tokens and bypassed outliers for the final models. Fig. 5 reports the unknown ratio as a percentage, but Eq. (3) and the CR/ACR numbers in Table 3 would be easier to verify if the actual bit cost of the bypassed outliers were reported separately. This is not a challenge to the measured CR values, but it is needed to attribute the tokenizer's benefit correctly and to check that the bypass is not accidentally hiding a significant fraction of the data.

minor comments (5)

[Abstract] The abstract contains a typo: 'Besides,L3TC' should read 'Besides, L3TC'.
[References] The gzip reference is attributed to 'Pasco, R. C. 1996', but RFC 1952 is authored by P. Deutsch. Please correct the citation.
[Fig. 5 and Section 3.3] The term 'SPM-BPE' is used in the left sub-figure of Fig. 5 but is not defined in the main text; please define it (presumably SentencePiece BPE) at first use.
[Table 3 and Section 4.2] The claim of '50x reduction in model parameters with comparable compression performance' is supported by comparing L3TC-3.2M with the 169M models listed in Table 1, but Table 3 does not include those 169M baselines. Adding the 169M learned compressors to Table 3, or at least a cross-reference, would make the comparison transparent.
[Section 4.4, Fig. 7] The batch-size saturation plot in Fig. 7 is informative, but the y-axis label 'Decoding Speed (KB/s)' should be qualified as 'model-inference throughput' unless the measurement includes the entropy coder and tokenizer.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: compression results are measured benchmark outputs and design choices are validated empirically; the speed-metric concern is an under-specified benchmark, not a circular derivation.

full rationale

The paper's central claims are empirical, not definitional. Compression ratios in Tables 3 and 4 are measured outputs on enwik8/enwik9 of a trained model combined with an arithmetic coder, and the reported ACR separately adds model size as an external accounting rule; no claim is obtained by substituting a fitted parameter into the quantity it is supposed to predict. Hyperparameters such as coverage 0.999, vocabulary size 16K, HiRA rank 4, and one branch are selected from ablations on enwik8 and then applied to the benchmark, which is ordinary model selection rather than a forced prediction. The architectural comparison in Table 1 is an experiment comparing RWKV, Transformer, and Transformer-XL under matched settings, and the reparameterization identity W = W0 + sum_m A_m B_m is an algebraic merge of training-time branches, not a result whose conclusion is an input. Citations to RWKV, Delétang et al., LoRA, and other prior work are external; no load-bearing conclusion rests on a self-citation or on a uniqueness theorem from the present authors. The one substantive weakness, that the Section 4.3 decoding speed is model-only throughput and excludes the arithmetic coder, tokenizer, and I/O, is a benchmark-completeness issue rather than circularity: it does not make the reported speed true by construction or reduce any compression-ratio result to its own inputs. Accordingly, no circular step meeting the evidentiary standard is present.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claims depend on six hand-chosen hyperparameters (coverage, vocabulary size, HiRA rank and branch count, chunk length, batch size) and on four background assumptions about entropy coding and the benchmark. The trained network weights are fitted artifacts produced by the paper's pipeline; they are not counted as free parameters here because the report ledger targets constants chosen ad hoc rather than the learned model itself.

free parameters (6)

tokenizer coverage threshold = 0.999
Chosen from Fig. 3 as the value with lowest bits per byte on enwik8; directly controls how many characters bypass the neural model as raw bytes.
vocabulary size = 16K
Chosen from Fig. 5 as the best trade-off between sequence length and model capacity; larger vocabularies (32K, 64K) degrade or slow inference.
HiRA rank multiplier = 4
Ablation Fig. 6a shows compression ratio decreases with rank; rank 4 chosen as the tested maximum.
HiRA branch count = 1
Ablation Fig. 6a shows diminishing gains beyond one branch; one branch used for all main results.
context chunk length = 2048 bytes
Set following Deletang et al.; larger contexts increase compute and the paper does not sweep this.
device batch size = 256
Fig. 7 shows throughput saturates near batch 256 on iPhone ANE; used for reported mobile speeds.

assumptions (4)

standard math Maximum-likelihood training is equivalent to minimizing entropy coding cost
Invoked in Section 3.1 with Eq. (1); it is the theoretical basis for using a language model as a compressor.
domain assumption Arithmetic coding can achieve the model's cross-entropy with negligible overhead
Section 3.1 assumes the entropy coder is not the bottleneck; the paper does not report end-to-end coder time.
domain assumption Batched decoding of independent 2048-byte chunks is a faithful proxy for sequential decompression speed
Section 4.3 derives speed from batch throughput; this assumes no inter-chunk dependencies and that the arithmetic decoder can keep up.
domain assumption enwik9, which contains all of enwik8, is an acceptable held-out test for generalization
Section 4.1 trains on enwik8 and evaluates on full enwik9; the first 100MB of the test file is training data, a known benchmark convention that the paper acknowledges only through the overfitting discussion.

how reviews work

0 comments

Cite this review

Pith. "Pith review of L3TC: Leveraging RWKV for Learned Lossless Low-Complexity Text Compression." pith.science (2026). https://pith.science/paper/CX4G62UX

@misc{pith2026241216642,
  author       = {Pith},
  title        = {Pith review of: L3TC: Leveraging RWKV for Learned Lossless Low-Complexity Text Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CX4G62UX}},
  note         = {Machine review of arXiv:2412.16642}
}

read the original abstract

Learning-based probabilistic models can be combined with an entropy coder for data compression. However, due to the high complexity of learning-based models, their practical application as text compressors has been largely overlooked. To address this issue, our work focuses on a low-complexity design while maintaining compression performance. We introduce a novel Learned Lossless Low-complexity Text Compression method (L3TC). Specifically, we conduct extensive experiments demonstrating that RWKV models achieve the fastest decoding speed with a moderate compression ratio, making it the most suitable backbone for our method. Second, we propose an outlier-aware tokenizer that uses a limited vocabulary to cover frequent tokens while allowing outliers to bypass the prediction and encoding. Third, we propose a novel high-rank reparameterization strategy that enhances the learning capability during training without increasing complexity during inference. Experimental results validate that our method achieves 48% bit saving compared to gzip compressor. Besides, L3TC offers compression performance comparable to other learned compressors, with a 50x reduction in model parameters. More importantly, L3TC is the fastest among all learned compressors, providing real-time decoding speeds up to megabytes per second. Our code is available at https://github.com/alipay/L3TC-leveraging-rwkv-for-learned-lossless-low-complexity-text-compression.git.

Figures

Figures reproduced from arXiv: 2412.16642 by the authors.

**Figure 2.** Overall architecture of our proposed Learned Lossless Low-Complexity Text Compression ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Performance with different coverage values. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Proposed high-rank reparameterization method. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Discussion on the outlier-aware tokenizer. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 7.** Figure 7: Decoding speeds with different batch sizes. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

A Survey of RWKV
cs.CL 2024-12 conditional novelty 3.0 of 10

A review of the RWKV architecture, its versions, applications, benchmarks, and open-source ecosystem; it presents no new experimental results.

Reference graph

Works this paper leans on

21 extracted references · 6 canonical work pages · cited by 1 Pith paper

[7]

arXiv:2404.09937

Compres- sion Represents Intelligence Linearly. arXiv:2404.09937. Huffman, D. A

arXiv
[9]

arXiv:2404.03626

Training LLMs over Neurally Compressed Text. arXiv:2404.03626. Loshchilov, I.; and Hutter, F

arXiv
[10]

arXiv:1711.05101

Decoupled Weight De- cay Regularization. arXiv:1711.05101. MacKay., D. J. C., ed

arXiv
[14]

arXiv:2402.18376

Tokenization Is More Than Compression. arXiv:2402.18376. Sennrich, R.; Haddow, B.; and Birch, A

arXiv
[17]

arXiv:2306.04050

LLMZip: Loss- less Text Compression using Large Language Models. arXiv:2306.04050. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I

arXiv
[20]

HiRA is configured with a single bypass branch and a rank set to four times the main branch’s dimension, which has been proved as the optimal configuration in the main paper

Notably, all these experiments use models with 200K parameters. HiRA is configured with a single bypass branch and a rank set to four times the main branch’s dimension, which has been proved as the optimal configuration in the main paper. Model MACs CR(%) RWKV-200K 143.6K 24.36 RWKV-200K + Linear 160.9K 24.18 (-0.18) RWKV-200K + Linear + HiRA 160.9K 23.51...

work page 2021
[21]

However, these models typically involve excessive computational complex- ity

It can be observed that advanced compressors like CMIX and NNCP achieve impressive compression per- formance, with compression ratios of approximately 11%, which is comparable to RWKV-1.5B model. However, these models typically involve excessive computational complex- ity. CMIX and NNCP require 7.2 and 2.8 days, respectively, to decode 1 GB of text (with ...

work page 2024
[1948]

Bell Syst

A mathematical theory of communi- cation. Bell Syst. Tech. J. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi `ere, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lam- ple, G. 2023a. LLaMA: Open and Efficient Foundation Lan- guage Models. arXiv preprint arXiv:2302.13971. Touvron, H.; M...

arXiv

Show all 21 references

[2001]

https://xiph.org/flac/

Free Lossless Audio Codec. https://xiph.org/flac/. Accessed: 2024-08-09. Ziv, A., Jacob; Lempel

2024
[2003]

Cambridge University Press

Information theory, inference, and learning algorithms. Cambridge University Press. Mahoney, M. 2006a. PAQ8H: A High Compression Ra- tio Data Compression Program. http://mattmahoney.net/dc/ paq8h.zip. Accessed: 2024-08-09. Mahoney, M. 2006b. Text8 Dataset. http://mattmahoney. ...

2024
[2009]

In 2009 IEEE conference on computer vision and pattern recognition, 248–255

Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248–255. Ieee. Ding, X.; Zhang, X.; Ma, N.; Han, J.; Ding, G.; and Sun, J

2009
[2013]

ArXiv:1311.2540

Asymmetric numeral systems: entropy cod- ing combining speed of Huffman coding with compression rate of arithmetic coding. ArXiv:1311.2540. Goldman, O.; Caciularu, A.; Eyal, M.; Cao, K.; Szpektor, I.; and Tsarfaty, R

arXiv
[2015]

In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), 5206–5210

Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), 5206–5210. IEEE. Pasco., R. C

2015
[2016]

arXiv:1508.07909

Neural Machine Translation of Rare Words with Subword Units. arXiv:1508.07909. Seward, J

arXiv
[2017]

arXiv:1706.03762

At- tention Is All You Need. arXiv:1706.03762. Xiph.Org Foundation

arXiv
[2019]

arXiv:1901.02860

Transformer-XL: Atten- tive Language Models Beyond a Fixed-Length Context. arXiv:1901.02860. Del´etang, G.; Ruoss, A.; Duquenne, P.-A.; Catt, E.; Ge- newein, T.; Mattern, C.; Grau-Moya, J.; Wenliang, L. K.; Aitchison, M.; Orseau, L.; Hutter, M.; and Veness, J

1901 arXiv
[2020]

arXiv preprint arXiv:2001.08361

Scaling laws for neural language mod- els. arXiv preprint arXiv:2001.08361. Knoll, B

2001 arXiv
[2021]

arXiv:2106.09685

LoRA: Low-Rank Adap- tation of Large Language Models. arXiv:2106.09685. Huang, Y .; Zhang, J.; Shan, Z.; and He, J

arXiv
[2022]

arXiv preprint arXiv:2203.15556

Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. Howard, P. G.; and Vitter, J. S

arXiv
[2023]

arXiv:2305.13048

RWKV: Reinventing RNNs for the Transformer Era. arXiv:2305.13048. Schmidt, C. W.; Reddy, V .; Zhang, H.; Alameddine, A.; Uzan, O.; Pinter, Y .; and Tanner, C

arXiv
[2024]

arXiv:2403.06265

Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Perfor- mance. arXiv:2403.06265. Hochreiter, S.; and Schmidhuber, J

arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [7]

arXiv:2404.09937

Compres- sion Represents Intelligence Linearly. arXiv:2404.09937. Huffman, D. A

arXiv

[2] [9]

arXiv:2404.03626

Training LLMs over Neurally Compressed Text. arXiv:2404.03626. Loshchilov, I.; and Hutter, F

arXiv

[3] [10]

arXiv:1711.05101

Decoupled Weight De- cay Regularization. arXiv:1711.05101. MacKay., D. J. C., ed

arXiv

[4] [14]

arXiv:2402.18376

Tokenization Is More Than Compression. arXiv:2402.18376. Sennrich, R.; Haddow, B.; and Birch, A

arXiv

[5] [17]

arXiv:2306.04050

LLMZip: Loss- less Text Compression using Large Language Models. arXiv:2306.04050. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I

arXiv

[6] [20]

HiRA is configured with a single bypass branch and a rank set to four times the main branch’s dimension, which has been proved as the optimal configuration in the main paper

Notably, all these experiments use models with 200K parameters. HiRA is configured with a single bypass branch and a rank set to four times the main branch’s dimension, which has been proved as the optimal configuration in the main paper. Model MACs CR(%) RWKV-200K 143.6K 24.36 RWKV-200K + Linear 160.9K 24.18 (-0.18) RWKV-200K + Linear + HiRA 160.9K 23.51...

work page 2021

[7] [21]

However, these models typically involve excessive computational complex- ity

It can be observed that advanced compressors like CMIX and NNCP achieve impressive compression per- formance, with compression ratios of approximately 11%, which is comparable to RWKV-1.5B model. However, these models typically involve excessive computational complex- ity. CMIX and NNCP require 7.2 and 2.8 days, respectively, to decode 1 GB of text (with ...

work page 2024

[8] [1948]

Bell Syst

A mathematical theory of communi- cation. Bell Syst. Tech. J. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi `ere, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lam- ple, G. 2023a. LLaMA: Open and Efficient Foundation Lan- guage Models. arXiv preprint arXiv:2302.13971. Touvron, H.; M...

arXiv