{"id":"d496b974-f317-410b-80cf-d62e49930c0d","arxiv_id":"2501.01625","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"ICPC compresses long prompts with a small encoder scoring word redundancy and predictability, running faster than LLM-based compressors but with only marginal, unvalidated quality gains.","lead":"ICPC is a prompt compression method that uses a small pretrained encoder, such as BERT, to decide which words in a long prompt can be deleted before the prompt is sent to a large language model. The authors report that it compresses text several times faster than existing methods, though the measured quality gains over baselines are small and the evaluation does not test any real downstream task.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central performance claim rests on similarity-to-original metrics; no downstream task accuracy is measured, so 'better performance on NLP tasks' is unsupported.","rationale":"The reader's weakest_assumption focuses on the unvalidated ranking formula in Eq. (1). My concern is closely related but more fundamental: the experiments never measure downstream task performance, so neither Eq. (1) nor the central quality claim is tested. The reader's rationale already mentions the absence of downstream task evaluations, but their stated weakest assumption is the Eq. (1) scoring function. I therefore partially agree: the reader identifies a real weakness, yet the decisive gap is the evaluation metric mismatch, not merely the loss function's fidelity. The concrete test I propose would directly close that gap by measuring task accuracy on compressed prompts and comparing against the baselines used in the paper. If the test shows no task-level advantage, the abstract's claim of 'better performance on NLP tasks' collapses, and the paper's contribution reduces to a possible speedup with unspecified quality trade-offs. This supports the reader's REJECT verdict, because as written the evidence does not establish the central claim. I keep the verdict at REJECT rather than UNVERDICTED because the paper presents its similarity scores as evidence of task performance, which is a category error that cannot be fixed without rerunning the experiments; there is no positive evidence of downstream task superiority among existing numbers.","tokens_in":911,"tokens_out":1005,"duration_ms":34822,"concrete_test":"Run a fixed long-context task suite, e.g., 20-shot MMLU or NaturalQuestions-style open-domain QA, using compressed prompts as the context for a frozen LLM (like GPT-3.5 or Llama-2-7B). Compare ICPC against the original full prompt, Random Deletion, Selective Context, and LLMLingua at compression ratios 0.8, 0.6, and 0.4, with at least 3 random seeds and 95% confidence intervals. If ICPC does not match or exceed full-context task accuracy and does not outperform LLMLingua within the error bars on a majority of tasks, then the paper's 'better performance on NLP tasks' claim is unsupported.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's abstract and conclusion claim that ICPC achieves 'better performance' on 'different types of NLP tasks,' and Section 4.2 highlights a BLEU increase from 42.6 to 43.4. However, all quantitative results in Tables 1 and 3 are lexical and semantic similarity scores between the compressed text and the original text (BLEU, ROUGE, BERTScore, etc.). No experiment feeds the compressed prompts into a downstream task such as question answering, text classification, or summarization and measures task accuracy. The datasets described in Section 4.1.1 (Wikipedia, arXiv, Reddit) are unlabeled long-text collections, not task benchmarks. Consequently, even if Eq. (1) perfectly selected tokens that preserve the original wording, that would only establish compression fidelity, not that the compressed prompt retains the information needed for a specific task. The ranking assumption in Eq. (1) that MLM probability plus cosine similarity measures task-relevant information loss is therefore never directly validated. Moreover, the reported improvements over LLMLingua are small (often under 1 metric point) and no error bars or significance tests are provided, so the claimed advantage could be noise. This missing downstream evaluation is the load-bearing gap: without it, the central claim that ICPC improves performance on NLP tasks does not follow from the presented evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ICPC, a prompt-compression method that uses a pretrained transformer encoder to score each lexical unit by a weighted combination of its masked-language-model probability and cosine similarity to surrounding words, then removes units whose loss exceeds a percentile threshold. The authors evaluate compression fidelity with BLEU, ROUGE, METEOR, and BERTScore on Wikipedia, arXiv, and Reddit passages, compare several encoder backbones, and report compression-time measurements. They conclude that ICPC improves performance and speed on NLP tasks.","tokens_in":8207,"tokens_out":6309,"duration_ms":59291,"significance":"The speed improvement (Table 2: 10.3 ms vs 45.2 ms at ratio 0.8 for LLMLingua) is a plausible practical benefit if it holds across hardware. However, the paper's central claim of better task performance is not tested: all reported quality metrics measure similarity to the original text, not success on downstream tasks. The scoring function behind the method is also asserted rather than validated. As a result, the contribution currently reduces to a fast lossy text filter whose effect on actual prompt-based tasks is unknown. There is no code release, and the method is not fully specified (e.g., the weight alpha is never reported), which further limits reproducibility.","major_comments":[{"comment":"The abstract and conclusion state that ICPC achieves 'better performance' on 'different types of NLP tasks,' but no downstream task is evaluated. Tables 1 and 3 report BLEU, ROUGE, METEOR, and BERTScore between the compressed and original texts; these are compression-fidelity metrics. The corpora in §4.1.1 are unlabeled long-text collections, and §4.1.1 explicitly justifies dropping SQuAD and PIQA because they are 'not appropriate'—yet without such task benchmarks, the claim that compressed prompts improve task performance does not follow. Please add evaluations on downstream tasks such as question answering, classification, or summarization, or revise the claims to compression fidelity only.","section":"Abstract, §5, §4.2, Tables 1 and 3"},{"comment":"The loss L(x_i) is a linear combination of cosine similarity and log MLM probability, but no derivation or independent validation is provided. The method's behavior rests entirely on this ranking; if it is wrong, the compressor deletes tokens needed by the downstream task. Add ablations against each component, a probability-only baseline, and ideally a check that high-L removal preserves task accuracy.","section":"§3.2, Eq. (1)"},{"comment":"Algorithm hyperparameters are not reported: the scalar alpha in Eq. (1), the context window k, the lexical-unit granularity, and the percentile p are never given numerical values. The text says parameters are tuned (§4.1) but does not say on which data or to which values; if tuned on the same corpora used for the reported comparison, the small improvements over LLMLingua could be due to overfitting rather than method quality.","section":"§4.1, §3.3"},{"comment":"The observed differences over LLMLingua are small (e.g., BLEU 42.7 vs 42.1 at ratio 0.8; METEOR 45.3 vs 45.2), and no error bars, confidence intervals, or significance tests are given despite a statement about 'multiple runs' in §4.1. Moreover, the sentence 'ICPC saw the BLEU score increase with ratio 0.6 on metric BLEU from 42.6 to 43.4' misreads the table: 42.6 and 43.4 are METEOR scores for Selective Context and ICPC, respectively, while the BLEU scores at ratio 0.6 are 37.8 (LLMLingua) and 38.0 (ICPC).","section":"§4.2, Table 1"}],"minor_comments":[{"comment":"The column header 'Training time (ms)' conflicts with the caption and text, which describe compression time; please correct this inconsistency.","section":"Table 2"},{"comment":"The term 'participle' (as in 'participle-based filtering') is nonstandard; if segmentation is meant, use 'segmenting' or 'parsing.'","section":"§3.1"},{"comment":"Figure 1 is referenced in §4.4 but appears missing or incomplete in the manuscript; if it is meant to be included, ensure it is rendered.","section":"Figure 1"},{"comment":"Approximate inference times for BERT vs GPT-3 are given without specifying hardware; since the speed claim is a core advantage, provide a direct measured comparison on the same hardware.","section":"Appendix A.1, Table 6"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the reader is right that the headline claim outruns the evidence. Everything in Tables 1 and 3 measures similarity between the compressed text and the original; no downstream task accuracy is reported. So \"better performance on NLP tasks\" does not follow from the presented numbers.\n\nThat said, the paper is not worthless. The specific loss in Eq. (1) — windowed cosine similarity plus an MLM log-probability term — is a genuine variation on existing importance-based pruning, and using a small encoder instead of an LLM as the scorer is a sensible practical direction. The speed numbers in Table 2 are plausible and match intuition: a 110M-parameter BERT should be much faster than a GPT-scale scorer. The method is simple enough to reimplement quickly, which is a point in its favor.\n\nThe soft spots are real and load-bearing. All quantitative evidence is compression fidelity, not task retention. Section 4.1.1 says SQuAD and PIQA are too short, then compiles unlabeled long-text collections; that explains why there are no task labels, but it does not justify skipping task evaluation entirely. If ICPC deletes tokens that preserve BLEU against the original, we still do not know whether a downstream model can answer a question or classify a document from the compressed prompt. The margins over LLMLingua are small (BLEU 42.7 vs 42.1 at ratio 0.8; 38.0 vs 37.8 at ratio 0.6), no error bars or significance tests are given, and hyperparameters alpha, k, and granularity are either unreported or tuned on the evaluation sets. LLMLingua-2, the most relevant encoder-based baseline, is neither cited nor compared. The stress-test concern holds up: the central performance claim is unsupported.\n\nWho is this for? A practitioner looking for a cheap and fast compression heuristic might try it, but only after seeing task-level validation. As written, I would not cite it. For peer review, I would desk reject or send back for major revision; there is a real method here, but the evaluation needs to be rebuilt with downstream task accuracy, error bars, held-out hyperparameter selection, and a direct comparison with LLMLingua-2. If those land, the speed angle could become useful.","headline":"Plausible fast compression heuristic, but the paper never measures task performance, so the central claim is unsupported.","tokens_in":8797,"tokens_out":2803,"would_cite":false,"duration_ms":27038,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that prompt compression can be done with a small pretrained encoder instead of a second large language model, making compression several times faster while matching or slightly exceeding the output quality of LLM-based…","keywords":["prompt compression","in-context learning","masked language modeling","transformer encoder","inference speed","text redundancy","BERT","long-context efficiency"],"falsifier":"Take a fixed set of prompts with downstream answers, delete exactly one token at a time, and measure how much task accuracy drops; if the tokens with the highest $L(x_i)$ from Eq. (1) are the ones whose removal most damages accuracy, the score is measuring predictability rather than task relevance.","tokens_in":7743,"feed_emoji":"⚡","tokens_out":12175,"duration_ms":105380,"temperature":0.7,"pith_summary":"The paper proposes ICPC, a prompt-compression method that uses a small pretrained transformer encoder rather than a large language model to decide which words in a long prompt are redundant. It claims this makes compression several times faster while preserving, and in some comparisons slightly improving, the similarity between the compressed prompt and the original as measured by BLEU, ROUGE, and BERTScore. The motivation is that attention cost grows with prompt length, so shortening prompts before inference should let LLMs handle longer contexts with less memory and latency. The paper aims to establish that encoder-based importance scoring is a viable, lightweight alternative to LLM-based prompt compression.","feed_headline":"Prompt compression with small encoders is 3-4x faster","feed_subtitle":"ICPC matches LLM-based compression scores while cutting compression time from ~50 ms to ~10 ms per prompt.","key_machinery":"The machinery is the information function in Eq. (1), $L(x_i) = \\alpha \\sum_{n=-k, n \\ne 0}^{k} \\mathrm{sim}(x_{i+n}, x_i) + \\log p(x_i \\mid x_{i,k})$, where $x_{i,k}$ is the context window around $x_i$ with $x_i$ itself removed. The first term punishes a word that closely resembles its neighbors, treating it as redundant; the second term gives a low score to words the encoder can predict from context, again marking them as carrying little new information. Words, phrases, or clauses are ranked by $L(x_i)$ and all units at or above the $p$-th percentile of the loss distribution are removed. The percentile threshold makes the amount of deletion adaptive to the input rather than fixed.","core_discovery":"The paper's central claim is that a pretrained transformer encoder of roughly 110 million parameters can replace a large language model as the prompt compressor. The encoder's masked-language-model probability for each word, combined with the cosine similarity between that word and its local context, is used as an information score; words with the highest scores are deleted. On the paper's own experiments, this yields compressed prompts whose BLEU, ROUGE, and BERTScore values match or slightly exceed those of LLM-based compressors at the same compression ratios, while compression time drops from tens of milliseconds to about 10-17 milliseconds. The paper takes this as evidence that context information from a small encoder suffices for prompt compression, eliminating the memory overhead of running a second large model.","pith_inferences":["A natural testable extension is to measure end-to-end latency and downstream task accuracy after the compressed prompt is fed to the target LLM; the paper reports compression time in isolation.","Because the deletion threshold is a global percentile over a chunk, an adaptive per-sentence threshold might better protect rare but essential tokens.","The local window $k$ in Eq. (1) cannot see cross-sentence dependencies; coupling the score with a coarse sentence-level salience model could extend the method to documents whose key information spans many sentences."],"forward_implications":["Long prompts can be shortened before the LLM sees them, so attention cost scales with the compressed length rather than the original.","The paper reports an increase on BLEU from 42.6 to 43.4 at ratio 0.6, pointing to compression without fidelity loss.","Compression time drops from tens of milliseconds to about 10-17 milliseconds, roughly a 3-4x speedup over LLM-based compressors.","The same formula works across six different encoder families, so the method is not tied to one model's pretraining scheme."],"supporting_citations":[{"why":"The competing context-compression method whose design ICPC most directly extends; it supplies the baseline at the same compression ratios and the notion of grouping tokens into participle units.","marker":"[12]"},{"why":"The pretrained masked-language-model encoder used in the main experiments; it provides the $p(x_i \\mid x_{i,k})$ term of the information function.","marker":"[5]"},{"why":"A large language model cited as an example of what existing compression methods require and what ICPC avoids.","marker":"[1]"},{"why":"Another large language model whose memory and speed costs motivate the move to encoder-based compression.","marker":"[23]"},{"why":"The survey that documents the quadratic attention cost of long prompts, establishing why prompt compression matters.","marker":"[13]"},{"why":"The BLEU metric used to score compressed prompts against originals in the headline comparison.","marker":"[17]"},{"why":"The BERTScore metric used to check that compressed text retains semantic similarity beyond surface n-gram overlap.","marker":"[25]"}],"fun_headline_variants":["Small encoder compresses prompts 4x faster, same quality","Prompt compression via 110M encoder: faster, no LLM needed","Drop the LLM: tiny encoder trims prompts in milliseconds","ICPC: small model, big speedup for prompt compression","Prompt slimming gets 3-4x faster with a 110M encoder"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a word is redundant if a pretrained encoder can predict it from its surrounding words and it closely resembles those words, so deleting such words does not remove task-relevant information.","fun_headline_variants_meta":{"raw":{"variants":["Small encoder compresses prompts 4x faster, same quality","Prompt compression via 110M encoder: faster, no LLM needed","Drop the LLM: tiny encoder trims prompts in milliseconds","ICPC: small model, big speedup for prompt compression","Prompt slimming gets 3-4x faster with a 110M encoder"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1557,"prompt_tokens":847,"completion_tokens":710,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":618}},"tokens_in":463,"tokens_out":710,"duration_ms":5583,"temperature":1.0,"reasoning_tokens":618,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:23:21.895171+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed set of prompts with downstream answers, delete exactly one token at a time, and measure how much task accuracy drops; if the tokens with the highest $L(x_i)$ from Eq. (1) are the ones whose removal most damages accuracy, the score is measuring predictability rather than task relevance.","supporting_citations":[],"review_version":1}