Pith. sign in

REVIEW 4 major objections 5 minor 12 references

MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scores

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A BERT-based prompt compressor gains a lasting edge over LLMLingua-2 by training against inter-class similarity and by blending per-class outlier scores into its retention decision.

desk verdict Worth a serious look, but the claims outrun the protocol: alpha is tuned on test data and the 'consistently outperforms' is contradicted by several LongBench subtasks. read the letter →

arxiv 2504.16786 v1 pith:CDKMPFTY submitted 2025-04-23 cs.CL cs.LG

Fengwei Zhou , Jiafei Song , Wenjin Jason Li , Gengjian Xue , Zhikang Zhao , Yichao Lu , Bailin Na This is my paper · ORCID
classification cs.CLcs.LG
keywords promptcompressionlong-contextunderstandingBERTtokenclassificationover-smoothinginter-classcosinesimilaritylossoutlierscoresZ-scoreedgedeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MOOSComp tries to establish that two inexpensive additions to a BERT-based token classifier improve task-agnostic long-context prompt compression. First, an extra training loss directly reduces the cosine similarity between "preserve" and "discard" token representations in the final layer, countering the over-smoothing that makes tokens hard to tell apart. Second, at compression time every token receives a Z-score outlier score computed within its predicted class, and the final retention score blends the classifier's probability with that normalized outlier score. The paper reports that the combination outperforms LLMLingua-2 on in-domain summarization and on out-of-domain long-context understanding and reasoning benchmarks, while adding almost no compression overhead and enabling up to 3.3x speedup on a smartphone at 4x compression.

What carries the argument

The two load-bearing mechanisms are the inter-class cosine similarity loss L_CS(phi), the average cosine similarity between preserve-class and discard-class token representations at the last layer, and the retention metric m_k = $\alpha$ * p_k + (1 - $\alpha$) * s_norm_k, where s_norm_k is the within-class normalized Z-score norm of token k. The loss directly attacks over-smoothing at the layer the classifier reads, while the outlier score gives statistically rare tokens within their predicted class a second chance to survive compression.

What would settle it

Take a reading-comprehension prompt engineered so that the answer-bearing tokens are among the most typical tokens in their class (low Z-score norms) while frequent filler tokens are made unusual; if MOOSComp with alpha below 0.5 still retains the fillers and drops the answers on a few hundred samples, the rarity-importance link underlying Eq. (7) is falsified.

Watch

Extended reading notes

Core claim

The paper claims that a token-classification prompt compressor can be improved on two fronts. During training, it adds an inter-class cosine similarity loss that directly minimizes the average cosine similarity between final-layer representations of tokens labeled "preserve" and tokens labeled "discard", shrinking the inter-class overlap that remains after standard fine-tuning. At compression time, it splits tokens by the classifier's predicted class, computes per-class Z-scores of the last-layer token vectors, normalizes the Euclidean norm of each Z-score to [0, 1], and retains tokens by the metric m_k = alpha * p_k + (1 - alpha) * s_norm_k. With both changes, MOOSComp keeps more rare but answer-bearing tokens and separates the two classes better, which the paper argues improves task-agnostic compression across black-box API models and local models.

Load-bearing premise

The load-bearing premise is that a token whose last-layer BERT representation is far from its predicted class's mean, measured by the Euclidean norm of its per-feature Z-score, is a token worth preserving because it carries task-relevant rarity; if statistical rarity in representation space does not line up with task importance, the outlier term injects noise instead of signal.

Editorial extensions

If this is right

  • On the MeetingBank summarization test with GPT-3.5-Turbo, MOOSComp reports higher Rouge and BERTScore than LLMLingua-2 at 3x compression while using a similar token budget.
  • On LongBench with GPT-3.5-Turbo, single-document QA rises from 29.8 to 35.3 at the 2K-token constraint, and multi-document QA from 33.1 to 35.5.
  • The outlier scoring adds only about 0.01 seconds of compression cost over LLMLingua-2 on a V100 GPU, since it is a single Z-score normalization pass over the last layer.
  • Ablations show that either mechanism alone improves over LLMLingua-2, and combining both gives the best results.
  • The outlier mechanism does not help code completion, which the paper attributes to the inherently structural nature of code texts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If rarity in representation space is a proxy for task importance, the same per-class Z-score trick could be ported to any token-scoring compressor, replacing expensive LLM-based perplexity scoring with a nearly free pass over the final layer.
  • The per-class normalization is likely necessary because training now separates class means, so a single global Z-score would flag tokens mostly by class membership rather than within-class rarity; the ablation comparing per-class versus pooled outlier scores supports this reading.
  • At higher compression ratios, rare-token effects should be stronger, since ordinary high-probability tokens fill the budget; a compression-ratio sweep beyond 5x could reveal whether outlier weighting should grow as the budget shrinks.
  • Code completion being the failure case suggests that tasks where information is carried by structure rather than lexical rarity need a different rescue mechanism, perhaps preserving syntax-special tokens with outlier weights set to zero.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MOOSComp, a task-agnostic token-level prompt compressor built on LLMLingua-2. Two modifications are introduced: an inter-class cosine similarity loss applied to the last-layer token representations during training (Eq. 2, combined with the cross-entropy loss in Eq. 4), and an outlier-score mechanism during compression that blends a per-class Z-score norm with the classifier's preservation probability (Eqs. 5-7). Experiments on MeetingBank, LongBench, GSM8K, BBH, and latency measurements on a V100 GPU and a smartphone NPU/GPU report improved accuracy over LLMLingua-2 at comparable compression ratios and speedups up to 3.3x on a mobile device.

Significance. If the reported results hold, MOOSComp would be a genuinely useful, low-overhead improvement over LLMLingua-2: the anti-over-smoothing loss is simple, requires no extra inference-time computation, and is well motivated by the layer-wise cosine-similarity analysis in Figure 2; the outlier score is cheap and does not add hyperparameters beyond alpha. The ablation study in Table 7 gives evidence that both mechanisms contribute positively. However, the central empirical claim is currently weakened by the evaluation protocol: alpha in Eq. (7) is tuned on 20% of each test set before reporting full-set numbers, which makes the reported gains optimistic and conflicts with the paper's task-agnostic framing. In addition, the claim that MOOSComp 'consistently outperforms' LLMLingua-2 is not supported by the paper's own Table 2, where MOOSComp is below LLMLingua-2 on three subtasks, and the authors explicitly concede that the outlier mechanism does not benefit code completion. With no code, data, or error bars, the reader cannot separate the effect of the proposed mechanisms from the effect of test-set tuning.

major comments (4)
  1. [Section 4.2, Eq. (7)] The hyperparameter alpha in the retention metric m_k = alpha * p_k + (1 - alpha) * s_norm_k is selected using 20% of each test set and then applied to the full dataset. This is stated explicitly for MeetingBank ('we determine the optimal alpha ... using 20% of the test data and then apply it to the entire dataset') and again for out-of-domain tasks ('we still use 20% of the data to select the optimal alpha'). Because alpha directly changes which tokens are kept, this procedure fits a task-specific parameter to the test distribution and can inflate MOOSComp's advantage over baselines whose parameters are fixed. The reported alpha values also vary by task (0.5 for MeetingBank, 0.7-0.8 for LongBench), indicating that the gains are not achieved by a single task-agnostic setting. I ask the authors to rerun the evaluation with alpha fixed across all tasks, or tuned on a separate validation split that is not part of the reported test set, and to report results across a range of alpha values. Without this, the central outperformance claim is not established.
  2. [Table 2] The abstract and Section 4.2 claim that MOOSComp 'consistently outperforms' the state-of-the-art task-agnostic hard prompt methods, but Table 2 contains multiple counterexamples at the 2,000- and 3,000-token constraints: on Code at 2K, MOOSComp scores 53.8 versus LLMLingua-2's 58.9; on MultiDoc at 3K, 38.4 versus 38.7; on FewShot at 3K, 67.0 versus 69.6. The text itself acknowledges that the outlier mechanism does not benefit code completion. The claim of consistent superiority should be qualified accordingly, and the paper should analyze why the method loses on these subtasks rather than asserting universality.
  3. [Table 2, footnote 1] The code-completion row on LongBench uses GPT-4o-mini as the target model, whereas the other rows in Table 2 use GPT-3.5-Turbo, because the authors report that GPT-3.5-Turbo achieves only 49.3 accuracy on the original prompt. This makes the code row non-comparable to the other rows in the same table. The change of target model should be disclosed in the table itself, and the comparison should either use the same target model across all rows or clearly separate the code-completion results into a different evaluation block.
  4. [Table 2, 3,000-token constraint] Under the '3,000-token constraint', MOOSComp uses only 2,934 tokens at a 4x compression ratio, while LLMLingua-2 uses 3,392 tokens at a 3x ratio and other baselines use up to 3,421 tokens. These are not matched retained-token budgets, so part of the accuracy difference may be due to the different amount of context available to the target model rather than to token-selection quality. The same issue appears in less severe form in other rows. Please report matched-token comparisons or clarify why the constraint is interpreted differently for MOOSComp.
minor comments (5)
  1. [Introduction] There is a grammatical error: 'it still face challenges' should read 'it still faces challenges.' Also, 'SOTA' should be expanded at first use.
  2. [Table 6] The abstract states 'a speedup of 3.3x at a 4x compression ratio' on a smartphone, but Table 6 shows 3.3x only for the 6.7K-token example; the 2.7K-token example achieves only 1.4x at 4x compression. Please state the setup and example length when reporting the headline speedup.
  3. [Appendix B] Training details report the optimizer, learning rate, batch size, and epoch count, but not weight decay, warmup, or gradient clipping. Providing these settings would improve reproducibility.
  4. [Section 4.3, Table 7] The ablation table reports single numbers without error bars or multiple seeds. Given that some differences between variants are small (e.g., 34.9 vs. 33.0 on SingleDoc), it would be helpful to report standard deviations or significance tests, especially since alpha may have been tuned separately for each variant.
  5. [Figure 4] The compressed-text examples contain long rows of probability numbers that are difficult to read. Highlighting the preserved tokens with color or bolding, as the caption implies, would make the qualitative comparison much clearer.

Circularity Check

2 steps flagged · score 6.0 of 10

Anti-over-smoothing validation is tautological and alpha is tuned on the test set, making reported gains partially self-fulfilling.

  1. self definitional [Sec. 3.3, Eq. (2); Sec. 4.2 (Impact of Inter-Class Cosine Similarity Loss), Fig. 3]
    "Then the inter-class cosine similarity loss is L_CS(φ) := S_L. ... When β is larger, the similarity in the last layer decreases drastically. This shows that adjusting β effectively alleviates the over-smoothing issue."

    Eq. (2) defines the training loss as exactly the same last-layer inter-class cosine similarity S_L that Eq. (1) uses as the operational measure of over-smoothing. Therefore, Figure 3's demonstration that larger β lowers the similarity is guaranteed by minimizing that very objective; it is not independent evidence that a distinct over-smoothing problem is solved. The downstream classification and benchmark gains are separate and non-circular, but this specific validation reduces to the training objective by definition.

  2. fitted input called prediction [Sec. 4.2, In-Domain and Out-of-Domain Evaluation; Eq. (7)]
    "We determine the optimal α in Eq. (7) using 20% of the test data and then apply it to the entire dataset. ... For each task, we still use 20% of the data to select the optimal α."

    The retention metric m_k = α·p_k + (1−α)·s_norm contains α, and the paper selects α on a 20% subset of each test set before reporting results on the entire test set of the same benchmark. The reported 'superior performance' is therefore a fitted result rather than a task-agnostic out-of-sample prediction: the balance between classifier probability and outlier score is chosen after seeing test data, which inflates and partially forces the benchmark gains. This is furthered by α varying per task (0.5 on MeetingBank, 0.7–0.8 on LongBench).

full rationale

The paper is not deeply circular overall: it contains no load-bearing self-citations, and the outlier-score heuristic plus the downstream ablations are independent empirical components. However, two specific steps are circular or self-referential. First, Section 3.3 defines the training loss L_CS(φ) as exactly the last-layer inter-class cosine similarity S_L, the same quantity Section 3.2 uses to operationalize over-smoothing; Section 4.2's Figure 3 then confirms that larger β lowers S_L, which is true by construction and not independent evidence that over-smoothing is mitigated. Second, the retention metric in Eq. (7) includes α, and the paper selects α on 20% of the test data of each benchmark before reporting results on that same benchmark. The reported MOOSComp gains are therefore partly fitted, not task-agnostic out-of-sample predictions, and the 'consistently outperforms' claim is not established by the reported protocol. These are partial circularities; the core downstream comparisons still retain some independent content, so the score is 6.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two fitted hyperparameters (beta and alpha) and three domain assumptions: the quality of GPT-4-generated training labels, the task-relevance of representation outliers, and the sufficiency of last-layer BERT representations. No new physical or conceptual entities are introduced.

free parameters (2)
  • beta (inter-class cosine similarity loss weight) = 0.001
    Selected based on validation accuracy of the compressor; controls the strength of the anti-over-smoothing loss in Eq. (4).
  • alpha (outlier score weight) = 0.5 for MeetingBank, 0.7-0.8 for LongBench/GSM8K/BBH
    Tuned on 20% of each evaluation set and then applied to the full set, as stated in Section 4.2. This is a test-set-fitted hyperparameter that affects the reported gains.
assumptions (3)
  • domain assumption GPT-4-distilled MeetingBank compression labels are a reliable training signal for token-level preservation.
    Inherited from LLMLingua-2 and used in Section 3.1; the compressor is trained on this dataset, so the validity of the entire method depends on these labels generalizing to out-of-domain tasks.
  • domain assumption Rare tokens in the compressor's representation space are task-critical and should be preserved.
    The core premise of the outlier mechanism in Section 3.4; if rarity does not correlate with task importance, the outlier term adds noise rather than signal.
  • domain assumption Last-layer BERT representations contain sufficient information for both token classification and outlier detection.
    Both the classifier and the Z-score outlier computation operate on final-layer representations h^L, as defined in Eqs. (2) and (5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scores." pith.science (2026). https://pith.science/paper/CDKMPFTY

@misc{pith2026250416786,
  author       = {Pith},
  title        = {Pith review of: MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scores},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CDKMPFTY}},
  note         = {Machine review of arXiv:2504.16786}
}
read the original abstract

Recent advances in large language models have significantly improved their ability to process long-context input, but practical applications are challenged by increased inference time and resource consumption, particularly in resource-constrained environments. To address these challenges, we propose MOOSComp, a token-classification-based long-context compression method that enhances the performance of a BERT-based compressor by mitigating the over-smoothing problem and incorporating outlier scores. In the training phase, we add an inter-class cosine similarity loss term to penalize excessively similar token representations, thereby improving the token classification accuracy. During the compression phase, we introduce outlier scores to preserve rare but critical tokens that are prone to be discarded in task-agnostic compression. These scores are integrated with the classifier's output, making the compressor more generalizable to various tasks. Superior performance is achieved at various compression ratios on long-context understanding and reasoning benchmarks. Moreover, our method obtains a speedup of 3.3x at a 4x compression ratio on a resource-constrained mobile device.

Figures

Figures reproduced from arXiv: 2504.16786 by the authors.

Figure 1
Figure 1. An overview of the proposed MOOSComp. Training Phase (Top): The compressor, containing a BERT-based encoder and a classifier in series, is trained for binary token classification with the cross-entropy loss and the inter-class cosine similarity loss to mitigate over-smoothing. Compression Phase (Bottom): Each token’s classification score is combined with an outlier score to enhance the preservation of rare but impor… view at source ↗
Figure 2
Figure 2. Inter-class cosine similarity of different layers [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Inter-class cosine similarity of different layers [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Examples of the original and compressed prompts from the LongBench dataset, with the preserved words [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 5 canonical work pages

  1. [5]

    In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16409– 16423, Toronto, Canada

    MeetingBank: A benchmark dataset for meet- ing summarization. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16409– 16423, Toronto, Canada. Association for Computa- tional Linguistics. Huiqiang Jiang, Qianhui Wu, , Xufang Luo, Dong- sheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    work page

  2. [6]

    In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1658–1677, Bangkok, Thailand

    LongLLMLingua: Accelerating and enhanc- ing LLMs in long context scenarios via prompt com- pression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1658–1677, Bangkok, Thailand. Association for Computational Linguistics. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and ...

    work page 2023

  3. [8]

    In Proceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing, pages 6342–6353, Singa- pore

    Compressing context to enhance inference ef- ficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing, pages 6342–6353, Singa- pore. Association for Computational Linguistics. Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Col- lier. 2024b. Prompt compression for large language models: ...

    arXiv 2023

  4. [9]

    arXiv preprint arXiv:2407.00079

    Mooncake: A kvcache-centric disaggre- gated architecture for llm serving. arXiv preprint arXiv:2407.00079. Sridhar Ramaswamy, Rajeev Rastogi, and Kyuseok Shim. 2000. Efficient algorithms for mining outliers from large data sets. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 427–438. Han Shi, Jiahui Gao, Hang X...

    arXiv 2000

  5. [11]

    arXiv preprint arXiv:2403.05530

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv prepr...

    arXiv 2023

  6. [12]

    auto- regressive

    using PyTorch 2.1.2 on an A100-80G GPU. We divide the dataset into 80% for training and 20% for validation. The compressor is trained for 10 epochs by Adam optimizer with a batch size of 10 and a learning rate of 1× 10−5. After each epoch, we test the model on the validation set and ulti- mately select the model with the highest validation accuracy. The t...

    work page 2024

  7. [2009]

    ACM comput- ing surveys (CSUR), 41(3):1–58

    Anomaly detection: A survey. ACM comput- ing surveys (CSUR), 41(3):1–58. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. Adapting language models to ...

    arXiv 2023

  8. [2020]

    In International Conference on Learning Representations

    ALBERT: A lite bert for self-supervised learn- ing of language representations. In International Conference on Learning Representations. Luchang Li, Sheng Qian, Jie Lu, Lunxi Yuan, Rui Wang, and Qin Xie. 2024a. Transformer- lite: High-efficiency deployment of large language models on mobile phone gpus. arXiv preprint arXiv:2403.20041. Yucheng Li, Bo Dong,...

    arXiv

Show all 12 references
  1. [2021]

    arXiv preprint arXiv:2110.14168

    Training verifiers to solve math word prob- lems. arXiv preprint arXiv:2110.14168. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cro...

    arXiv

  2. [2022]

    In International Conference on Machine Learning, pages 20827–20840

    Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pages 20827–20840. PMLR. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou,...

    2023

  3. [2023]

    In The Eleventh International Conference on Learning Representa- tions

    ContraNorm: A contrastive learning perspec- tive on oversmoothing and beyond. In The Eleventh International Conference on Learning Representa- tions. Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. Lm- infinite: Zero-shot extreme length ge...

    2024

  4. [2024]

    Preprint, arXiv:2404.00456

    Quarot: Outlier-free 4-bit inference in rotated llms. Preprint, arXiv:2404.00456. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi...

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.