REVIEW 4 major objections 4 minor 27 references
ICPC: In-context Prompt Compression with Faster Inference
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that prompt compression can be done with a small pretrained encoder instead of a second large language model, making compression several times faster while matching or slightly exceeding the output quality of LLM-based…
desk verdict Plausible fast compression heuristic, but the paper never measures task performance, so the central claim is unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the information function in Eq. (1), $L(x_i) = \alpha \sum_{n=-k, n \ne 0}^{k} \mathrm{sim}(x_{i+n}, x_i) + \log p(x_i \mid x_{i,k})$, where $x_{i,k}$ is the context window around $x_i$ with $x_i$ itself removed. The first term punishes a word that closely resembles its neighbors, treating it as redundant; the second term gives a low score to words the encoder can predict from context, again marking them as carrying little new information. Words, phrases, or clauses are ranked by $L(x_i)$ and all units at or above the $p$-th percentile of the loss distribution are removed. The percentile threshold makes the amount of deletion adaptive to the input rather than fixed.
What would settle it
Take a fixed set of prompts with downstream answers, delete exactly one token at a time, and measure how much task accuracy drops; if the tokens with the highest $L(x_i)$ from Eq. (1) are the ones whose removal most damages accuracy, the score is measuring predictability rather than task relevance.
Extended reading notes
Core claim
The paper's central claim is that a pretrained transformer encoder of roughly 110 million parameters can replace a large language model as the prompt compressor. The encoder's masked-language-model probability for each word, combined with the cosine similarity between that word and its local context, is used as an information score; words with the highest scores are deleted. On the paper's own experiments, this yields compressed prompts whose BLEU, ROUGE, and BERTScore values match or slightly exceed those of LLM-based compressors at the same compression ratios, while compression time drops from tens of milliseconds to about 10-17 milliseconds. The paper takes this as evidence that context information from a small encoder suffices for prompt compression, eliminating the memory overhead of running a second large model.
Load-bearing premise
The method assumes that a word is redundant if a pretrained encoder can predict it from its surrounding words and it closely resembles those words, so deleting such words does not remove task-relevant information.
Editorial extensions
If this is right
- Long prompts can be shortened before the LLM sees them, so attention cost scales with the compressed length rather than the original.
- The paper reports an increase on BLEU from 42.6 to 43.4 at ratio 0.6, pointing to compression without fidelity loss.
- Compression time drops from tens of milliseconds to about 10-17 milliseconds, roughly a 3-4x speedup over LLM-based compressors.
- The same formula works across six different encoder families, so the method is not tied to one model's pretraining scheme.
Reading between the lines
- A natural testable extension is to measure end-to-end latency and downstream task accuracy after the compressed prompt is fed to the target LLM; the paper reports compression time in isolation.
- Because the deletion threshold is a global percentile over a chunk, an adaptive per-sentence threshold might better protect rare but essential tokens.
- The local window $k$ in Eq. (1) cannot see cross-sentence dependencies; coupling the score with a coarse sentence-level salience model could extend the method to documents whose key information spans many sentences.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ICPC, a prompt-compression method that uses a pretrained transformer encoder to score each lexical unit by a weighted combination of its masked-language-model probability and cosine similarity to surrounding words, then removes units whose loss exceeds a percentile threshold. The authors evaluate compression fidelity with BLEU, ROUGE, METEOR, and BERTScore on Wikipedia, arXiv, and Reddit passages, compare several encoder backbones, and report compression-time measurements. They conclude that ICPC improves performance and speed on NLP tasks.
Significance. The speed improvement (Table 2: 10.3 ms vs 45.2 ms at ratio 0.8 for LLMLingua) is a plausible practical benefit if it holds across hardware. However, the paper's central claim of better task performance is not tested: all reported quality metrics measure similarity to the original text, not success on downstream tasks. The scoring function behind the method is also asserted rather than validated. As a result, the contribution currently reduces to a fast lossy text filter whose effect on actual prompt-based tasks is unknown. There is no code release, and the method is not fully specified (e.g., the weight alpha is never reported), which further limits reproducibility.
major comments (4)
- [Abstract, §5, §4.2, Tables 1 and 3] The abstract and conclusion state that ICPC achieves 'better performance' on 'different types of NLP tasks,' but no downstream task is evaluated. Tables 1 and 3 report BLEU, ROUGE, METEOR, and BERTScore between the compressed and original texts; these are compression-fidelity metrics. The corpora in §4.1.1 are unlabeled long-text collections, and §4.1.1 explicitly justifies dropping SQuAD and PIQA because they are 'not appropriate'—yet without such task benchmarks, the claim that compressed prompts improve task performance does not follow. Please add evaluations on downstream tasks such as question answering, classification, or summarization, or revise the claims to compression fidelity only.
- [§3.2, Eq. (1)] The loss L(x_i) is a linear combination of cosine similarity and log MLM probability, but no derivation or independent validation is provided. The method's behavior rests entirely on this ranking; if it is wrong, the compressor deletes tokens needed by the downstream task. Add ablations against each component, a probability-only baseline, and ideally a check that high-L removal preserves task accuracy.
- [§4.1, §3.3] Algorithm hyperparameters are not reported: the scalar alpha in Eq. (1), the context window k, the lexical-unit granularity, and the percentile p are never given numerical values. The text says parameters are tuned (§4.1) but does not say on which data or to which values; if tuned on the same corpora used for the reported comparison, the small improvements over LLMLingua could be due to overfitting rather than method quality.
- [§4.2, Table 1] The observed differences over LLMLingua are small (e.g., BLEU 42.7 vs 42.1 at ratio 0.8; METEOR 45.3 vs 45.2), and no error bars, confidence intervals, or significance tests are given despite a statement about 'multiple runs' in §4.1. Moreover, the sentence 'ICPC saw the BLEU score increase with ratio 0.6 on metric BLEU from 42.6 to 43.4' misreads the table: 42.6 and 43.4 are METEOR scores for Selective Context and ICPC, respectively, while the BLEU scores at ratio 0.6 are 37.8 (LLMLingua) and 38.0 (ICPC).
minor comments (4)
- [Table 2] The column header 'Training time (ms)' conflicts with the caption and text, which describe compression time; please correct this inconsistency.
- [§3.1] The term 'participle' (as in 'participle-based filtering') is nonstandard; if segmentation is meant, use 'segmenting' or 'parsing.'
- [Figure 1] Figure 1 is referenced in §4.4 but appears missing or incomplete in the manuscript; if it is meant to be included, ensure it is rendered.
- [Appendix A.1, Table 6] Approximate inference times for BERT vs GPT-3 are given without specifying hardware; since the speed claim is a core advantage, provide a direct measured comparison on the same hardware.
Circularity Check
No significant circularity; similarity-based evaluation aligns with the compression objective but no derivation step reduces to its inputs.
full rationale
The derivation chain is self-contained and non-circular. Equation (1) defines a deletion score from a pretrained encoder's MLM probability log p(xi | xi,k) plus the average cosine similarity to its context, and the threshold rule in Eqs. (3)-(4) selects tokens below a percentile of that score. Nothing in this chain is fitted to the reported BLEU/ROUGE/BERTScore values, and no fitted parameter is renamed as a prediction. The comparisons against Random Deletion, Selective Context, and LLMLingua are external baselines, and the cited references, including the authors' own survey [2], are not load-bearing for the method's construction. The only mild concern is that the evaluation metrics measure similarity to the original text, which aligns with the compression objective, and Section 4.1.1 explicitly avoids task benchmarks by saying 'existing benchmarks such as SQuAD and Piqa are mostly single-round question-answer datasets with short question length, which is not appropriate to evaluate our proposed method.' Consequently, the abstract's claim of 'better performance on different types of NLP tasks' is not established by the presented evidence; however, that is a validity gap, not a circular reduction. Hyperparameters are tuned on the same datasets, which can inflate scores, but this does not make any reported quantity equal to the input by construction. A score of 1 reflects these minor evaluation-alignment concerns rather than any derivation-by-definition.
Assumptions & free parameters
free parameters (4)
- alpha =
Not reported
- compression ratio p =
0.8, 0.6, 0.4
- context window size k =
Not reported
- lexical unit granularity =
Not reported
assumptions (4)
- domain assumption MLM probability p(xi | xi,k) measures how much information is lost when xi is removed.
- domain assumption Cosine similarity between embeddings of nearby words captures semantic redundancy.
- domain assumption Pretrained transformer encoders transfer to arbitrary domains such as Reddit and arXiv abstracts.
- domain assumption Removing all units with loss above the p-th percentile preserves the information a downstream task needs.
Cite this review
Pith. "Pith review of ICPC: In-context Prompt Compression with Faster Inference." pith.science (2026). https://pith.science/paper/YWMQM5NA
@misc{pith2026250101625,
author = {Pith},
title = {Pith review of: ICPC: In-context Prompt Compression with Faster Inference},
year = {2026},
howpublished = {\url{https://pith.science/paper/YWMQM5NA}},
note = {Machine review of arXiv:2501.01625}
}
read the original abstract
Despite the recent success of Large Language Models (LLMs), it remains challenging to feed LLMs with long prompts due to the fixed size of LLM inputs. As a remedy, prompt compression becomes a promising solution by removing redundant tokens in the prompt. However, using LLM in the existing works requires additional computation resources and leads to memory overheads. To address it, we propose ICPC (In-context Prompt Compression), a novel and scalable prompt compression method that adaptively reduces the prompt length. The key idea of ICPC is to calculate the probability of each word appearing in the prompt using encoders and calculate information carried by each word through the information function, which effectively reduces the information loss during prompt compression and increases the speed of compression. Empirically, we demonstrate that ICPC can effectively compress long texts of different categories and thus achieve better performance and speed on different types of NLP tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023
arXiv 2023
-
[2]
Beyond efficiency: A systematic survey of resource-efficient large language models
Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, et al. Beyond efficiency: A systematic survey of resource-efficient large language models. arXiv preprint arXiv:2401.00625, 2024
arXiv 2024
-
[3]
Piqa: Reasoning about physical commonsense in natural language
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020
2020
-
[4]
Generating long sequences with sparse transformers
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019
arXiv 1904
-
[5]
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018
arXiv 2018
-
[6]
A survey on large language models: Applications, challenges, limitations, and practical usage
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023
2023
-
[7]
Deberta: Decoding-enhanced bert with disentan- gled attention
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentan- gled attention. arXiv preprint arXiv:2006.03654, 2020
arXiv 2006
-
[8]
Lora: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021
arXiv 2021
Show all 27 references
-
[9]
The distribution of the flora in the alpine zone
Paul Jaccard. The distribution of the flora in the alpine zone. 1. New phytologist, 11(2):37–50, 1912
1912
-
[10]
Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel
JP Kincaid. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Chief of Naval Technical Training, 1975
1975
-
[11]
Albert: A lite bert for self-supervised learning of language representations
Z Lan. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019
1909 arXiv
-
[12]
Compressing context to enhance inference efficiency of large language models
Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.06201, 2023
2023 arXiv
-
[13]
Prompt compression for large language models: A survey
Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Collier. Prompt compression for large language models: A survey. arXiv preprint arXiv:2410.12388, 2024
2024 arXiv
-
[14]
Rouge: A package for automatic evaluation of summaries
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004
2004
-
[15]
Roberta: A robustly optimized bert pretraining approach
Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364, 2019
1907 arXiv
-
[16]
Large language models: A survey
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024
2024 arXiv
-
[17]
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002
2002
-
[18]
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020
2020
-
[19]
Know what you don’t know: Unanswerable questions for squad
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822, 2018
2018 arXiv
-
[20]
Benefits and challenges of variable compression ratio (vcr)
Martyn Roberts. Benefits and challenges of variable compression ratio (vcr). Technical report, SAE Technical Paper, 2003
2003
-
[21]
A mathematical theory of communication
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379– 423, 1948
1948
-
[22]
A statistical interpretation of term specificity and its application in retrieval
Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21, 1972. 7 ICPC: In-context Prompt Compression with Faster Inference A PREPRINT
1972
-
[23]
Llama 2: Open foundation and fine-tuned chat models
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023
2023 arXiv
-
[24]
Xlnet: Generalized autoregressive pretraining for language understanding
Zhilin Yang. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019
1906 arXiv
-
[25]
Bertscore: Evaluating text generation with bert
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019
1904 arXiv
-
[26]
A survey of large language models
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023
2023 arXiv
-
[27]
A survey on model compression for large language models
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12:1556–1577, 2024. 8 ICPC: In-context Prompt Compression with Faster Inference A PREPRINT A Appendix...
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.