Text-Preserving Lossy Text Compression: A Study of Strategic Deletion and LLM Reconstruction

Junhong Tong; Jun Li; Yuchun Zou

arxiv: 2605.29000 · v1 · pith:7RX6KD45 · submitted 2026-05-27 · cs.CL

Text-Preserving Lossy Text Compression: A Study of Strategic Deletion and LLM Reconstruction

Yuchun Zou , Junhong Tong , Jun Li This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-29 12:41 UTCgrok-4.3pith:7RX6KD45 record.json open to challenge →

classification cs.CL

keywords lossy text compressionstrategic deletionLLM reconstructionword frequencysemantic compressionQLoRA fine-tuningBBC News datasetretention rate

0 comments

The pith

Word-frequency deletion serves as a competitive low-cost baseline for lossy semantic text compression that matches expensive semantic methods at high compression rates.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper studies lossy semantic text compression in which an encoder deletes selected words and an LLM reconstructs the original content from the retained skeleton. It benchmarks deletion strategies ranging from uniform and length-based rules to frequency-guided, entropy-based, and hybrid approaches on the BBC News dataset at retention rates from 10 to 90 percent. A simple static word-frequency lookup proves competitive with far more expensive semantic deletion methods while running much faster at the encoder, and it becomes especially robust when most of the text is removed. Semantic and hybrid strategies show their advantage mainly at milder compression levels, and a QLoRA-tuned local model can match large cloud LLMs for reconstruction. The framework transfers to additional English and Chinese data, though the single best deletion rule varies by dataset.

Core claim

Lossy semantic text compression by strategic deletion followed by LLM reconstruction demonstrates that word-frequency-guided deletion using only a static lookup remains competitive with semantic methods such as GPT-2 surprisal and hybrids, particularly at the lowest retention rates, while semantic and hybrid methods perform best at mild-to-moderate compression; QLoRA fine-tuning additionally produces a local decoder competitive with Gemini 2.0 Flash, and the overall approach transfers across domains although the optimal deletion rule is dataset-dependent.

What carries the argument

Word-frequency-guided deletion (WordFreq), a static frequency lookup that decides which words to retain before LLM reconstruction from the skeleton.

If this is right

WordFreq deletion stays competitive with semantic methods yet requires only a static lookup and runs far faster at the encoder.
Semantic and hybrid deletion strategies deliver their largest improvements at mild-to-moderate compression levels.
Word-frequency deletion proves more robust than semantic alternatives at the lowest retention rates.
QLoRA fine-tuning produces a local decoder that is competitive with Gemini 2.0 Flash in decoder-only tests.
The deletion-and-reconstruction framework transfers across English and Chinese domains, with the best rule remaining dataset-dependent.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Extreme compression scenarios may not require access to large models for the deletion decision itself.
Joint training of the deletion policy together with the reconstructor could improve results beyond the current separate-stage design.
The approach could support bandwidth-constrained settings such as mobile or edge deployment if reconstruction fidelity holds.
Dataset dependence of the best rule points toward the value of adaptive or learned deletion policies across varied text types.

Load-bearing premise

LLM reconstruction from the retained skeleton reliably preserves semantic content without systematic distortion.

What would settle it

A controlled comparison in which human raters or a semantic similarity metric show that reconstructions from WordFreq-deleted text are consistently less faithful to the original meaning than reconstructions from surprisal-based deletions at the same retention rate.

Figures

Figures reproduced from arXiv: 2605.29000 by Junhong Tong, Jun Li, Yuchun Zou.

**Figure 1.** Figure 1: BERTScore F1 vs. retention rate in two stacked panels. Top: word-level deletion methods. Bottom: representative semantic and hybrid methods. because it shows that some of the hybrid advantage is not merely stylistic fluency but better retention of factual anchor tokens. Third, at the most aggressive settings, all methods lose most entity mentions, but the stronger semantic methods remain slightly more r… view at source ↗

read the original abstract

Traditional lossless text compression preserves every byte, but its gains on natural language are often modest in realistic operating regimes. We study \emph{lossy semantic text compression}, where the encoder strategically deletes parts of the text and a large language model (LLM) reconstructs the original content from the retained skeleton. We benchmark a progression of deletion strategies, including uniform step deletion, word-length-guided deletion (WordLen), word-frequency-guided deletion (WordFreq), LP-optimized deletion (Opt), entropy-based deletion using GPT-2 surprisal, and hybrid methods that combine frequency and surprisal signals. Evaluation on the BBC News dataset across retention rates $\r_{keep} \in [0.1,0.9]$ shows three main findings. First, WordFreq is a strong low-cost baseline: despite using only a static frequency lookup, it remains competitive with much more expensive semantic methods while being far faster at the encoder. Second, semantic and hybrid methods provide their clearest gains at mild-to-moderate compression, whereas word-frequency deletion is often more robust at the lowest retention rates. Third, QLoRA fine-tuning yields a strong local decoder that is competitive with Gemini 2.0 Flash and is often strongest in decoder-only comparisons. Additional English and Chinese experiments show that the overall framework transfers across domains, while the best deletion rule remains dataset-dependent.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Word-frequency deletion holds up as a strong cheap baseline against semantic methods for LLM reconstruction, but the unspecified quality metric makes the competitiveness claims hard to verify.

read the letter

The main takeaway is that simple word-frequency deletion performs competitively with more complex semantic and hybrid deletion rules when an LLM reconstructs the deleted text, especially at low retention rates on the BBC News data.

The paper runs a straightforward empirical comparison. It tests uniform deletion, word-length, word-frequency, LP-optimized, GPT-2 entropy, and hybrid rules across retention rates from 0.1 to 0.9. It also adds QLoRA fine-tuning for a local decoder and checks transfer to Chinese data. The reported pattern is that frequency is fast and robust at aggressive compression while semantic signals help more at moderate rates, and the fine-tuned decoder competes with Gemini in some settings.

This is useful as a practical benchmark because it shows encoder cost can stay low without losing too much on the reconstruction side. The progression of methods and the dataset-dependent note on best rules give a clear picture of trade-offs.

The soft spot is the evaluation. The abstract states comparative findings but never names the reconstruction metric, reports no variance or significance numbers, and supplies no error analysis. Without knowing whether they used ROUGE, BERTScore, perplexity, or human ratings, it is difficult to judge if WordFreq's robustness reflects real semantic preservation or something else. The stress-test note flags exactly this gap, and the abstract alone does not resolve it.

This work is for people building efficient NLP pipelines or testing deletion heuristics in practice. A reader wanting new theory or large-scale claims will not find them here.

I would send it to peer review if the full paper supplies the missing metric details and reproducibility information, because the core comparison is concrete enough to be worth referee time.

Referee Report

2 major / 2 minor

Summary. The paper examines lossy semantic text compression via strategic deletion of text segments followed by LLM-based reconstruction of the original content. It evaluates a range of deletion policies—uniform step deletion, WordLen, WordFreq, LP-optimized (Opt), GPT-2 entropy-based, and frequency-surprisal hybrids—on the BBC News dataset at retention rates r_keep from 0.1 to 0.9. The central empirical claims are that WordFreq remains competitive with far more expensive semantic methods while being faster, that semantic/hybrid methods excel at mild-to-moderate compression while WordFreq is more robust at the lowest retention rates, and that QLoRA fine-tuning produces a local decoder competitive with Gemini 2.0 Flash. Additional experiments indicate the framework transfers to other English and Chinese domains, though the best deletion rule is dataset-dependent.

Significance. If the reconstruction quality metric and implementation details are supplied and the reported competitiveness holds under standard semantic metrics with proper controls for variance and statistical significance, the work would offer practical guidance on low-cost deletion heuristics for LLM-assisted compression pipelines. The observation that a static frequency baseline can outperform or match semantic methods at aggressive compression rates is potentially useful for resource-constrained settings, and the QLoRA decoder result is a concrete engineering contribution.

major comments (2)

[Evaluation / Abstract] Evaluation section (and abstract): No reconstruction quality metric is defined or referenced (ROUGE, BERTScore, perplexity, human judgment, or otherwise), nor is any error analysis, variance across runs, or validation that the metric tracks semantic fidelity rather than surface overlap or LLM priors. This directly undermines the ability to verify the headline claims about WordFreq competitiveness and the differential performance of semantic vs. frequency methods at different retention rates.
[Method / Experiments] § on LP-optimized and entropy-based methods: The abstract states that LP-optimized deletion (Opt) and entropy-based deletion using GPT-2 surprisal are implemented and compared, yet supplies no description of the exact LP formulation, the surprisal computation pipeline, or how these are made tractable at scale. Without these details the reported superiority or inferiority of these methods cannot be reproduced or assessed.

minor comments (2)

[Abstract / Notation] The retention-rate notation is written inconsistently as r_keep and \r_{keep}; standardize throughout.
[Experiments] No mention of the number of runs, random seeds, or confidence intervals for the comparative results; add these to all tables/figures reporting method rankings.

Simulated Author's Rebuttal

2 responses · 0 unresolved

Thank you for the constructive referee report on our manuscript. We address each major comment below and will revise the paper to enhance clarity, reproducibility, and verifiability of the results.

read point-by-point responses

Referee: [Evaluation / Abstract] Evaluation section (and abstract): No reconstruction quality metric is defined or referenced (ROUGE, BERTScore, perplexity, human judgment, or otherwise), nor is any error analysis, variance across runs, or validation that the metric tracks semantic fidelity rather than surface overlap or LLM priors. This directly undermines the ability to verify the headline claims about WordFreq competitiveness and the differential performance of semantic vs. frequency methods at different retention rates.

Authors: We agree that the reconstruction quality metric requires explicit definition, along with supporting analyses. The revised manuscript will clearly specify the metric employed (e.g., ROUGE, BERTScore, or perplexity), include variance reporting across runs, provide error analysis, and add validation demonstrating correlation with semantic fidelity rather than surface-level or prior-based artifacts. These changes will directly support verification of the claims regarding WordFreq competitiveness and performance differences across retention rates. revision: yes
Referee: [Method / Experiments] § on LP-optimized and entropy-based methods: The abstract states that LP-optimized deletion (Opt) and entropy-based deletion using GPT-2 surprisal are implemented and compared, yet supplies no description of the exact LP formulation, the surprisal computation pipeline, or how these are made tractable at scale. Without these details the reported superiority or inferiority of these methods cannot be reproduced or assessed.

Authors: We agree that additional methodological detail is necessary for reproducibility. The revised version will include the precise linear programming formulation for the Opt deletion policy, the full GPT-2 surprisal computation pipeline, and implementation steps showing how both methods are rendered tractable at the scale of the BBC News experiments. This will enable readers to assess and replicate the comparisons between methods. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical benchmarking on external dataset

full rationale

The paper is an empirical benchmarking study that evaluates deletion strategies (WordFreq, WordLen, entropy-based, hybrid, etc.) by measuring reconstruction quality after LLM decoding on the public BBC News dataset across retention rates. No equations, fitted parameters, or derivations are presented whose outputs are then relabeled as predictions. No self-citations are invoked as load-bearing uniqueness theorems or ansatzes. All reported competitiveness claims rest on direct experimental measurements rather than any reduction to the paper's own inputs, so the derivation chain is empty and the work is self-contained.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

Empirical benchmarking study; the central claims rest on the domain assumption that LLMs can perform semantic reconstruction from partial text and on standard evaluation practices in NLP, with no new mathematical free parameters or invented entities required.

free parameters (1)

retention rates r_keep
Discrete values in [0.1, 0.9] chosen for benchmarking; not fitted to optimize any reported outcome.

axioms (1)

domain assumption Large language models can reconstruct semantically coherent text from strategically deleted input at varying retention rates.
The entire lossy compression framework depends on this holding sufficiently for the reported competitiveness claims to be meaningful.

pith-pipeline@v0.9.1-grok · 5776 in / 1413 out tokens · 52236 ms · 2026-06-29T12:41:10.799336+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

34 extracted references · 7 canonical work pages · 1 internal anchor

[1]

The Llama 3 Herd of Models

HanLP: Han language processing toolkit. https: //github.com/hankcs/HanLP/tree/doc-zh. BigScience Data. 2022. Roots chinese traditional wikipedia dataset. https://huggingface.co/ datasets/bigscience-data/roots_zh-tw_ wikipedia. Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Li Kevi...

work page internal anchor Pith review Pith/arXiv arXiv 2022
[2]

InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353

Compressing context to enhance inference ef- ficiency of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353. As- sociation for Computational Linguistics. Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. InProceedings of the 2019 Conference on Emp...

2023
[3]

InAdvances in Neural Information Processing Systems

Joint autoregressive and hierarchical priors for learned image compression. InAdvances in Neural Information Processing Systems. Igor Pavlov. 1999. LZMA SDK (software development kit).https://7-zip.org/sdk.html. Alexander M. Rush, Sumit Chopra, and Jason Weston

1999
[4]

InProceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 379–389

A neural attention model for abstractive sen- tence summarization. InProceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 379–389. Shavbo Salehi, Melike Erol-Kantarci, and Dusit Niyato

2015
[5]

In2025 IEEE Symposium on Computers and Communications (ISCC), pages 1–6

LLM-enabled data transmission in end-to-end semantic communication. In2025 IEEE Symposium on Computers and Communications (ISCC), pages 1–6. IEEE. Salesforce Research. 2022. Wikitext language model- ing dataset.https://huggingface.co/datasets/ Salesforce/wikitext. Abigail See, Peter J. Liu, and Christopher D. Manning

2022
[6]

arXiv preprint arXiv:2306.04050 , year=

Get to the point: Summarization with pointer- generator networks. InProceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Computa- tional Linguistics. Julian Seward. 1996. bzip2 and libbzip2. https:// sourceware.org/bzip2/. Robyn Speer. 2022. ...

work page arXiv 1996
[7]

WF-3class and WF-Opt remain strong zero- shot baselines on Wikipedia without any domain-specific recalibration, especially once compression becomes aggressive

Frequency methods still generalize well. WF-3class and WF-Opt remain strong zero- shot baselines on Wikipedia without any domain-specific recalibration, especially once compression becomes aggressive
[8]

Semantic gains are concentrated at the mildest rates.Entropy is strongest at rkeep = 0.9, and Hybrid- 0.3 is the best hybrid at rkeep = 0.7 , but the semantic family does not control the lower-retention regime
[9]

Hybrid gains are weaker than on BBC News.On Wikipedia, direct frequency– surprisal interpolation is only marginally help- ful and becomes unstable once the skeleton gets sparse, likely because many encyclopedic terms are rare globally but predictable locally
[10]

Llama SFT improves clearly over Llama base and also beats the zero-shot decoder through- out the table, but the margin is smaller than on the main BBC News benchmark

Fine-tuning helps, but the gain is moderate. Llama SFT improves clearly over Llama base and also beats the zero-shot decoder through- out the table, but the margin is smaller than on the main BBC News benchmark
[11]

Step remains the weakest deletion family once compression is non-trivial.Its deteri- oration mirrors the main-paper pattern, con- firming that uniform deletion does not transfer well across domains. B Cross-Domain Evaluation: Reddit To further assess cross-domain robustness, we eval- uate our deletion strategies on a Reddit comments dataset (MentionBroker...

2023
[12]

Many methods cluster near the top of the table at rkeep = 0.9 , reflecting the shorter, more repetitive structure of conversational text

Reddit is unusually easy at high retention. Many methods cluster near the top of the table at rkeep = 0.9 , reflecting the shorter, more repetitive structure of conversational text
[13]

Frequency methods still generalize, but se- mantic methods are more competitive than in other English domains.WF-3class and WF-Opt remain strong, yet the best semantic and hybrid methods match or exceed them at most rates
[14]

Entropy and Hybrid rows use GPT-2 surprisal (same pipeline as BBC News experiments)

Conversational text benefits from contex- tual signals.Unlike Wikipedia, Reddit often favors surprisal-based ranking or hybridiza- tion, suggesting that local contextual redun- 11 Table 7: BERTScore F1 on Wikipedia (Salesforce/wikitext). Entropy and Hybrid rows use GPT-2 surprisal (same pipeline as BBC News experiments). Llama base and Llama SFT report th...

work page arXiv
[15]

WordLen transfers much better here than on BBC News.It remains surprisingly strong into the middle regime, indicating that conver- sational text is more tolerant of length-based deletions
[16]

Llama SFT is strongest throughout the ta- ble.Fine-tuned Llama leads at all five re- ported rates and substantially improves over Llama base, with the clearest gains in the middle-to-low retention regime
[17]

Opt (w/ polish)

Character-level targeting is strong only when retention is high.WF-3class (char.) is excellent at mild compression, but its ad- vantage collapses once the skeleton becomes sparse, where token-level methods recover more gracefully. ROUGE-L and CER.Tables 9 and 10 report ROUGE-L and CER on the same Reddit test set. These lexical diagnostics sharpen the Redd...

work page arXiv 1903
[18]

Opt (w/o polish)

The overall framework transfers well to 13 Table 11: BERTScore F1 (n= 200 , bert-base-chinese) on Chinese official news text for Gemini 2.0 Flash zero-shot reconstruction, with additional local Qwen base/SFT decoder results from the updated SFT workbook. “Opt (w/o polish)” = LP-optimized deletion without fluency correction; “Opt (w/ polish)” = LP-optimize...

work page arXiv
[19]

Word Frequency is strongest at rkeep = 0.7 and 0.5, so the main English pattern largely carries over here

Frequency dominates the middle regime. Word Frequency is strongest at rkeep = 0.7 and 0.5, so the main English pattern largely carries over here
[20]

Semantic and hybrid methods matter most at aggressive compression.Once retention becomes very low, Hybrid-0.3 and Entropy- LP overtake the frequency-family baselines
[21]

Chinese SFT helps, but does not overturn the zero-shot ranking.Qwen SFT is much stronger than the unfine-tuned local decoder, yet it generally remains below the best Gemini zero-shot deletion setups
[22]

Chinese Wikipedia We additionally evaluate on Chinese Wikipedia articles to assess whether the same trends hold be- yond formal news text

The low-retention regime is harder than in English.The Chinese curves fall further at rkeep = 0.1, consistent with higher informa- tion density per character. Chinese Wikipedia We additionally evaluate on Chinese Wikipedia articles to assess whether the same trends hold be- yond formal news text. The dataset consists of n=200 held-out chunks (≤512 charact...

2022
[23]

Semantic bucket allocation helps most at mild compression.Entropy-in-FreqBuckets is strongest at rkeep = 0.9 and 0.7, so Chinese Wikipedia behaves more like an encyclopedic stress test than a news dataset
[24]

Word Length remains useful in the middle regime.At rkeep = 0.5, it still edges out the other zero-shot methods
[25]

Entropy-LP is the strongest low-retention method.At rkeep = 0.3 and 0.1, semantic bucket allocation works better than the direct hybrids
[26]

Chinese Wikipedia is less favorable to local fine-tuning.Qwen SFT improves substan- tially over the base decoder, but it does not surpass the strongest Gemini zero-shot meth- ods
[27]

The Chinese news and Chinese Wikipedia stories are meaningfully different.News favors Hybrid-0.3 at the lowest rates, whereas Chinese Wikipedia favors Entropy-LP, sug- gesting that encyclopedic text benefits more from semantic allocation than from direct frequency–surprisal interpolation. 14 Table 12: BERTScore F1 ( n= 200 , bert-base-chinese) on Chinese ...

work page arXiv 2024
[28]

Zhihu has a different best-method pro- file from the other Chinese datasets.At very high retention, Entropy-in-FreqBuckets is strongest, while the middle regime looks more favorable to Word Length
[29]

Length and frequency heuristics still dom- inate the middle regime.The semantic and hybrid methods do not control rkeep = 0.7 or 0.5here
[30]

Entropy-LP is strongest once retention be- comes aggressive.At rkeep = 0.3 and 0.1, semantic LP overtakes both Opt (w/ polish) and the direct hybrids
[31]

Zhihu benefits the most from local fine- tuning.Qwen SFT is the strongest method from rkeep = 0.9 through 0.3, indicating that this conversational domain is especially favor- able to local adaptation
[32]

Opt (w/ polish)

Hybrid gains are present but secondary. The direct hybrids improve on some low- retention frequency baselines, but none sur- pass Entropy-LP. D Detailed Metrics This section reports additional metrics for the main English benchmark, BBC News. CER grows roughly linearly with the deletion rate up to rkeep = 0.5 and then accelerates, as at high deletion rate...

work page arXiv 2048
[33]

Our work is com- plementary: we target thelossyregime where full recovery is not required and the goal is maximizing a semantic fidelity metric at a given compression rate

further applies this idea directly to lossless text compression using arithmetic coding condi- tioned on LLM probabilities. Our work is com- plementary: we target thelossyregime where full recovery is not required and the goal is maximizing a semantic fidelity metric at a given compression rate. Neural and Learned Compression.Learned compression of images...

2018
[34]

The DeepSC system (Xie et al., 2021) trains a joint source-channel encoder for English sentences, achieving near-lossless se- mantic recovery even at low SNR

aim to transmit themeaningof a message at minimal bandwidth. The DeepSC system (Xie et al., 2021) trains a joint source-channel encoder for English sentences, achieving near-lossless se- mantic recovery even at low SNR. LLM-enabled semantic communication (Salehi et al., 2025; Xiao et al., 2024) extends this idea to generation-based receivers. Our framewor...

2021

[1] [1]

The Llama 3 Herd of Models

HanLP: Han language processing toolkit. https: //github.com/hankcs/HanLP/tree/doc-zh. BigScience Data. 2022. Roots chinese traditional wikipedia dataset. https://huggingface.co/ datasets/bigscience-data/roots_zh-tw_ wikipedia. Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Li Kevi...

work page internal anchor Pith review Pith/arXiv arXiv 2022

[2] [2]

InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353

Compressing context to enhance inference ef- ficiency of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353. As- sociation for Computational Linguistics. Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. InProceedings of the 2019 Conference on Emp...

2023

[3] [3]

InAdvances in Neural Information Processing Systems

Joint autoregressive and hierarchical priors for learned image compression. InAdvances in Neural Information Processing Systems. Igor Pavlov. 1999. LZMA SDK (software development kit).https://7-zip.org/sdk.html. Alexander M. Rush, Sumit Chopra, and Jason Weston

1999

[4] [4]

InProceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 379–389

A neural attention model for abstractive sen- tence summarization. InProceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 379–389. Shavbo Salehi, Melike Erol-Kantarci, and Dusit Niyato

2015

[5] [5]

In2025 IEEE Symposium on Computers and Communications (ISCC), pages 1–6

LLM-enabled data transmission in end-to-end semantic communication. In2025 IEEE Symposium on Computers and Communications (ISCC), pages 1–6. IEEE. Salesforce Research. 2022. Wikitext language model- ing dataset.https://huggingface.co/datasets/ Salesforce/wikitext. Abigail See, Peter J. Liu, and Christopher D. Manning

2022

[6] [6]

arXiv preprint arXiv:2306.04050 , year=

Get to the point: Summarization with pointer- generator networks. InProceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Computa- tional Linguistics. Julian Seward. 1996. bzip2 and libbzip2. https:// sourceware.org/bzip2/. Robyn Speer. 2022. ...

work page arXiv 1996

[7] [7]

WF-3class and WF-Opt remain strong zero- shot baselines on Wikipedia without any domain-specific recalibration, especially once compression becomes aggressive

Frequency methods still generalize well. WF-3class and WF-Opt remain strong zero- shot baselines on Wikipedia without any domain-specific recalibration, especially once compression becomes aggressive

[8] [8]

Semantic gains are concentrated at the mildest rates.Entropy is strongest at rkeep = 0.9, and Hybrid- 0.3 is the best hybrid at rkeep = 0.7 , but the semantic family does not control the lower-retention regime

[9] [9]

Hybrid gains are weaker than on BBC News.On Wikipedia, direct frequency– surprisal interpolation is only marginally help- ful and becomes unstable once the skeleton gets sparse, likely because many encyclopedic terms are rare globally but predictable locally

[10] [10]

Llama SFT improves clearly over Llama base and also beats the zero-shot decoder through- out the table, but the margin is smaller than on the main BBC News benchmark

Fine-tuning helps, but the gain is moderate. Llama SFT improves clearly over Llama base and also beats the zero-shot decoder through- out the table, but the margin is smaller than on the main BBC News benchmark

[11] [11]

Step remains the weakest deletion family once compression is non-trivial.Its deteri- oration mirrors the main-paper pattern, con- firming that uniform deletion does not transfer well across domains. B Cross-Domain Evaluation: Reddit To further assess cross-domain robustness, we eval- uate our deletion strategies on a Reddit comments dataset (MentionBroker...

2023

[12] [12]

Many methods cluster near the top of the table at rkeep = 0.9 , reflecting the shorter, more repetitive structure of conversational text

Reddit is unusually easy at high retention. Many methods cluster near the top of the table at rkeep = 0.9 , reflecting the shorter, more repetitive structure of conversational text

[13] [13]

Frequency methods still generalize, but se- mantic methods are more competitive than in other English domains.WF-3class and WF-Opt remain strong, yet the best semantic and hybrid methods match or exceed them at most rates

[14] [14]

Entropy and Hybrid rows use GPT-2 surprisal (same pipeline as BBC News experiments)

Conversational text benefits from contex- tual signals.Unlike Wikipedia, Reddit often favors surprisal-based ranking or hybridiza- tion, suggesting that local contextual redun- 11 Table 7: BERTScore F1 on Wikipedia (Salesforce/wikitext). Entropy and Hybrid rows use GPT-2 surprisal (same pipeline as BBC News experiments). Llama base and Llama SFT report th...

work page arXiv

[15] [15]

WordLen transfers much better here than on BBC News.It remains surprisingly strong into the middle regime, indicating that conver- sational text is more tolerant of length-based deletions

[16] [16]

Llama SFT is strongest throughout the ta- ble.Fine-tuned Llama leads at all five re- ported rates and substantially improves over Llama base, with the clearest gains in the middle-to-low retention regime

[17] [17]

Opt (w/ polish)

Character-level targeting is strong only when retention is high.WF-3class (char.) is excellent at mild compression, but its ad- vantage collapses once the skeleton becomes sparse, where token-level methods recover more gracefully. ROUGE-L and CER.Tables 9 and 10 report ROUGE-L and CER on the same Reddit test set. These lexical diagnostics sharpen the Redd...

work page arXiv 1903

[18] [18]

Opt (w/o polish)

The overall framework transfers well to 13 Table 11: BERTScore F1 (n= 200 , bert-base-chinese) on Chinese official news text for Gemini 2.0 Flash zero-shot reconstruction, with additional local Qwen base/SFT decoder results from the updated SFT workbook. “Opt (w/o polish)” = LP-optimized deletion without fluency correction; “Opt (w/ polish)” = LP-optimize...

work page arXiv

[19] [19]

Word Frequency is strongest at rkeep = 0.7 and 0.5, so the main English pattern largely carries over here

Frequency dominates the middle regime. Word Frequency is strongest at rkeep = 0.7 and 0.5, so the main English pattern largely carries over here

[20] [20]

Semantic and hybrid methods matter most at aggressive compression.Once retention becomes very low, Hybrid-0.3 and Entropy- LP overtake the frequency-family baselines

[21] [21]

Chinese SFT helps, but does not overturn the zero-shot ranking.Qwen SFT is much stronger than the unfine-tuned local decoder, yet it generally remains below the best Gemini zero-shot deletion setups

[22] [22]

Chinese Wikipedia We additionally evaluate on Chinese Wikipedia articles to assess whether the same trends hold be- yond formal news text

The low-retention regime is harder than in English.The Chinese curves fall further at rkeep = 0.1, consistent with higher informa- tion density per character. Chinese Wikipedia We additionally evaluate on Chinese Wikipedia articles to assess whether the same trends hold be- yond formal news text. The dataset consists of n=200 held-out chunks (≤512 charact...

2022

[23] [23]

Semantic bucket allocation helps most at mild compression.Entropy-in-FreqBuckets is strongest at rkeep = 0.9 and 0.7, so Chinese Wikipedia behaves more like an encyclopedic stress test than a news dataset

[24] [24]

Word Length remains useful in the middle regime.At rkeep = 0.5, it still edges out the other zero-shot methods

[25] [25]

Entropy-LP is the strongest low-retention method.At rkeep = 0.3 and 0.1, semantic bucket allocation works better than the direct hybrids

[26] [26]

Chinese Wikipedia is less favorable to local fine-tuning.Qwen SFT improves substan- tially over the base decoder, but it does not surpass the strongest Gemini zero-shot meth- ods

[27] [27]

The Chinese news and Chinese Wikipedia stories are meaningfully different.News favors Hybrid-0.3 at the lowest rates, whereas Chinese Wikipedia favors Entropy-LP, sug- gesting that encyclopedic text benefits more from semantic allocation than from direct frequency–surprisal interpolation. 14 Table 12: BERTScore F1 ( n= 200 , bert-base-chinese) on Chinese ...

work page arXiv 2024

[28] [28]

Zhihu has a different best-method pro- file from the other Chinese datasets.At very high retention, Entropy-in-FreqBuckets is strongest, while the middle regime looks more favorable to Word Length

[29] [29]

Length and frequency heuristics still dom- inate the middle regime.The semantic and hybrid methods do not control rkeep = 0.7 or 0.5here

[30] [30]

Entropy-LP is strongest once retention be- comes aggressive.At rkeep = 0.3 and 0.1, semantic LP overtakes both Opt (w/ polish) and the direct hybrids

[31] [31]

Zhihu benefits the most from local fine- tuning.Qwen SFT is the strongest method from rkeep = 0.9 through 0.3, indicating that this conversational domain is especially favor- able to local adaptation

[32] [32]

Opt (w/ polish)

Hybrid gains are present but secondary. The direct hybrids improve on some low- retention frequency baselines, but none sur- pass Entropy-LP. D Detailed Metrics This section reports additional metrics for the main English benchmark, BBC News. CER grows roughly linearly with the deletion rate up to rkeep = 0.5 and then accelerates, as at high deletion rate...

work page arXiv 2048

[33] [33]

Our work is com- plementary: we target thelossyregime where full recovery is not required and the goal is maximizing a semantic fidelity metric at a given compression rate

further applies this idea directly to lossless text compression using arithmetic coding condi- tioned on LLM probabilities. Our work is com- plementary: we target thelossyregime where full recovery is not required and the goal is maximizing a semantic fidelity metric at a given compression rate. Neural and Learned Compression.Learned compression of images...

2018

[34] [34]

The DeepSC system (Xie et al., 2021) trains a joint source-channel encoder for English sentences, achieving near-lossless se- mantic recovery even at low SNR

aim to transmit themeaningof a message at minimal bandwidth. The DeepSC system (Xie et al., 2021) trains a joint source-channel encoder for English sentences, achieving near-lossless se- mantic recovery even at low SNR. LLM-enabled semantic communication (Salehi et al., 2025; Xiao et al., 2024) extends this idea to generation-based receivers. Our framewor...

2021