{"id":"778ef68c-ef17-4a8b-a9de-1ff2b7c1cc8e","arxiv_id":"2504.16786","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Adding an inter-class cosine similarity loss and Z-score outlier scores to LLMLingua-2's BERT compressor improves long-context compression quality and on-device speed.","lead":"MOOSComp improves a BERT-based prompt compressor by adding a training loss that separates token classes and by scoring rare tokens as outliers during compression. It reports better long-context task scores than LLMLingua-2 and up to 3.3x speedup on a phone.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set tuning of alpha in Eq. (7) inflates reported gains; the central outperformance claim is not established until evaluations are rerun with alpha fixed or tuned on a separate validation split.","rationale":"I agree with the reader that MOOSComp is plausible and the mechanisms are clearly described; the ablation and speedup measurements are useful. However, the central empirical claim rests on benchmark tables whose reported gains are produced by tuning alpha on a subset of each test set. This is a protocol problem, not a theoretical disagreement: it affects essentially every reported accuracy number, whereas the outlier-assumption concern would only affect the outlier mechanism. The reader mentioned this issue in the rationale but placed the formal weakest_assumption on the outlier rarity assumption; I see the test-set alpha tuning as more load-bearing because it directly compromises the evidence for the headline claim. A fixed-alpha or proper-validation rerun would settle it. The verdict remains CONDITIONAL: the paper should be accepted only if the authors supply code/data and show the main comparison survives a legitimate alpha-selection protocol.","tokens_in":21978,"tokens_out":4794,"duration_ms":43716,"concrete_test":"Re-run the LongBench, GSM8K, BBH, and MeetingBank evaluations with alpha fixed at a single value (e.g., 0.7) chosen without access to any test labels, or tuned only on a separate validation split (e.g., MeetingBank validation). Then compare per-subtask scores against LLMLingua-2 at matched compressed-token budgets and report standard deviations. If MOOSComp does not beat LLMLingua-2 on a majority of subtasks under this protocol, the reported gains are an artifact of test-set tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the evaluation protocol for the hyperparameter alpha in Eq. (7). In Section 4.2, the authors state that 'We determine the optimal alpha ... using 20% of the test data and then apply it to the entire dataset,' and for out-of-domain tasks, 'we still use 20% of the data to select the optimal alpha.' Because the final retention metric m_k = alpha * p_k + (1 - alpha) * s_norm_k is evaluated on the same benchmark after alpha is tuned on a subset of that benchmark, the test set is being used to fit a task-specific parameter. This contradicts the paper's task-agnostic framing and can inflate MOOSComp's advantage: the reported alpha values vary by task (0.5 on MeetingBank, 0.7-0.8 on LongBench tasks), and the top-token selection is sensitive to this balance. The published results also show non-uniform gains even with this protocol: Table 2 has MOOSComp below LLMLingua-2 on Code at 2K (53.8 vs 58.9), MultiDoc at 3K (38.4 vs 38.7), and FewShot at 3K (67.0 vs 69.6), so 'consistently outperforms' is not supported by the data as reported. The absence of code, data, and error bars makes it impossible to separate the effect of the proposed mechanisms from the effect of test-set tuning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MOOSComp, a task-agnostic token-level prompt compressor built on LLMLingua-2. Two modifications are introduced: an inter-class cosine similarity loss applied to the last-layer token representations during training (Eq. 2, combined with the cross-entropy loss in Eq. 4), and an outlier-score mechanism during compression that blends a per-class Z-score norm with the classifier's preservation probability (Eqs. 5-7). Experiments on MeetingBank, LongBench, GSM8K, BBH, and latency measurements on a V100 GPU and a smartphone NPU/GPU report improved accuracy over LLMLingua-2 at comparable compression ratios and speedups up to 3.3x on a mobile device.","tokens_in":22209,"tokens_out":5092,"duration_ms":47165,"significance":"If the reported results hold, MOOSComp would be a genuinely useful, low-overhead improvement over LLMLingua-2: the anti-over-smoothing loss is simple, requires no extra inference-time computation, and is well motivated by the layer-wise cosine-similarity analysis in Figure 2; the outlier score is cheap and does not add hyperparameters beyond alpha. The ablation study in Table 7 gives evidence that both mechanisms contribute positively. However, the central empirical claim is currently weakened by the evaluation protocol: alpha in Eq. (7) is tuned on 20% of each test set before reporting full-set numbers, which makes the reported gains optimistic and conflicts with the paper's task-agnostic framing. In addition, the claim that MOOSComp 'consistently outperforms' LLMLingua-2 is not supported by the paper's own Table 2, where MOOSComp is below LLMLingua-2 on three subtasks, and the authors explicitly concede that the outlier mechanism does not benefit code completion. With no code, data, or error bars, the reader cannot separate the effect of the proposed mechanisms from the effect of test-set tuning.","major_comments":[{"comment":"The hyperparameter alpha in the retention metric m_k = alpha * p_k + (1 - alpha) * s_norm_k is selected using 20% of each test set and then applied to the full dataset. This is stated explicitly for MeetingBank ('we determine the optimal alpha ... using 20% of the test data and then apply it to the entire dataset') and again for out-of-domain tasks ('we still use 20% of the data to select the optimal alpha'). Because alpha directly changes which tokens are kept, this procedure fits a task-specific parameter to the test distribution and can inflate MOOSComp's advantage over baselines whose parameters are fixed. The reported alpha values also vary by task (0.5 for MeetingBank, 0.7-0.8 for LongBench), indicating that the gains are not achieved by a single task-agnostic setting. I ask the authors to rerun the evaluation with alpha fixed across all tasks, or tuned on a separate validation split that is not part of the reported test set, and to report results across a range of alpha values. Without this, the central outperformance claim is not established.","section":"Section 4.2, Eq. (7)"},{"comment":"The abstract and Section 4.2 claim that MOOSComp 'consistently outperforms' the state-of-the-art task-agnostic hard prompt methods, but Table 2 contains multiple counterexamples at the 2,000- and 3,000-token constraints: on Code at 2K, MOOSComp scores 53.8 versus LLMLingua-2's 58.9; on MultiDoc at 3K, 38.4 versus 38.7; on FewShot at 3K, 67.0 versus 69.6. The text itself acknowledges that the outlier mechanism does not benefit code completion. The claim of consistent superiority should be qualified accordingly, and the paper should analyze why the method loses on these subtasks rather than asserting universality.","section":"Table 2"},{"comment":"The code-completion row on LongBench uses GPT-4o-mini as the target model, whereas the other rows in Table 2 use GPT-3.5-Turbo, because the authors report that GPT-3.5-Turbo achieves only 49.3 accuracy on the original prompt. This makes the code row non-comparable to the other rows in the same table. The change of target model should be disclosed in the table itself, and the comparison should either use the same target model across all rows or clearly separate the code-completion results into a different evaluation block.","section":"Table 2, footnote 1"},{"comment":"Under the '3,000-token constraint', MOOSComp uses only 2,934 tokens at a 4x compression ratio, while LLMLingua-2 uses 3,392 tokens at a 3x ratio and other baselines use up to 3,421 tokens. These are not matched retained-token budgets, so part of the accuracy difference may be due to the different amount of context available to the target model rather than to token-selection quality. The same issue appears in less severe form in other rows. Please report matched-token comparisons or clarify why the constraint is interpreted differently for MOOSComp.","section":"Table 2, 3,000-token constraint"}],"minor_comments":[{"comment":"There is a grammatical error: 'it still face challenges' should read 'it still faces challenges.' Also, 'SOTA' should be expanded at first use.","section":"Introduction"},{"comment":"The abstract states 'a speedup of 3.3x at a 4x compression ratio' on a smartphone, but Table 6 shows 3.3x only for the 6.7K-token example; the 2.7K-token example achieves only 1.4x at 4x compression. Please state the setup and example length when reporting the headline speedup.","section":"Table 6"},{"comment":"Training details report the optimizer, learning rate, batch size, and epoch count, but not weight decay, warmup, or gradient clipping. Providing these settings would improve reproducibility.","section":"Appendix B"},{"comment":"The ablation table reports single numbers without error bars or multiple seeds. Given that some differences between variants are small (e.g., 34.9 vs. 33.0 on SingleDoc), it would be helpful to report standard deviations or significance tests, especially since alpha may have been tuned separately for each variant.","section":"Section 4.3, Table 7"},{"comment":"The compressed-text examples contain long rows of probability numbers that are difficult to read. Highlighting the preserved tokens with color or bolding, as the caption implies, would make the qualitative comparison much clearer.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The test-set tuning of alpha is the main gate for publication; it is fixable by re-running experiments with a proper validation split or a fixed alpha and by reporting sensitivity. The paper would also benefit from releasing code and configuration files, since the direct comparison with LLMLingua-2 relies on numbers taken from the prior paper. I do not see a circularity problem with the outlier score itself: it is an auxiliary heuristic computed from the compressor's own representations, and the ablation study supports its contribution, but the evaluation protocol needs to be clean before the central claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Key takeaway: this is a sensible incremental advance over LLMLingua-2, and it deserves a serious look, but the headline 'consistently outperforms' is not backed by the evaluation as reported. The contribution is clear: an inter-class cosine similarity loss (Eq. 2) to counter over-smoothing in the BERT compressor, and a per-class Z-score outlier score blended into the retention metric (Eq. 7). Neither is groundbreaking, but the diagnostic Figure 2 is a nice piece of evidence, and the ablations in Table 7 show each component adds something. The smartphone latency numbers are also a practical plus.\n\nNow the soft spots. The biggest is that alpha in Eq. (7) is selected using 20% of each test set before reporting full-set scores. The paper states this openly, which is honest, but it undercuts the task-agnostic claim: alpha varies by task (0.5 on MeetingBank, 0.7-0.8 on LongBench), and without error bars or a fixed-alpha sensitivity analysis you cannot tell whether the gains come from the mechanism or from per-benchmark fitting. The second soft spot is the 'consistently outperforms' wording. Table 2 itself shows MOOSComp below LLMLingua-2 on Code at 2K (53.8 vs 58.9), MultiDoc at 3K (38.4 vs 38.7), and FewShot at 3K (67.0 vs 69.6). The authors acknowledge the code failure in Section 4.2, which is to their credit, but it is still a contradiction with the abstract's claim.\n\nThe math is simple and sound; Eqs. (2)-(7) are standard, and there is no circularity. The Z-score is a heuristic, not a fitted surrogate, so the circularity burden is correctly zero. The citation pattern is normal for this line of work--the paper builds directly on LLMLingua-2 and cites relevant prior work. No self-citation red flags.\n\nWho this is for: people working on task-agnostic prompt compression or efficient LLM inference at the edge. It is a modest but real step; I would not reshape a research program around it, but I would take the mobile-latency evidence seriously.\n\nRecommendation: deserve a serious referee, but the revision should fix the alpha protocol--either fix alpha across all tasks or tune it on a separate validation split--and add variance estimates. If the gains survive that re-run, the paper is a solid conditional accept.","headline":"Worth a serious look, but the claims outrun the protocol: alpha is tuned on test data and the 'consistently outperforms' is contradicted by several LongBench subtasks.","tokens_in":22814,"tokens_out":3333,"would_cite":false,"duration_ms":27543,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A BERT-based prompt compressor gains a lasting edge over LLMLingua-2 by training against inter-class similarity and by blending per-class outlier scores into its retention decision.","keywords":["prompt compression","long-context understanding","BERT token classification","over-smoothing","inter-class cosine similarity loss","outlier scores","Z-score","edge deployment"],"falsifier":"Take a reading-comprehension prompt engineered so that the answer-bearing tokens are among the most typical tokens in their class (low Z-score norms) while frequent filler tokens are made unusual; if MOOSComp with alpha below 0.5 still retains the fillers and drops the answers on a few hundred samples, the rarity-importance link underlying Eq. (7) is falsified.","tokens_in":21734,"feed_emoji":"✂️","tokens_out":6656,"duration_ms":56401,"temperature":0.7,"pith_summary":"MOOSComp tries to establish that two inexpensive additions to a BERT-based token classifier improve task-agnostic long-context prompt compression. First, an extra training loss directly reduces the cosine similarity between \"preserve\" and \"discard\" token representations in the final layer, countering the over-smoothing that makes tokens hard to tell apart. Second, at compression time every token receives a Z-score outlier score computed within its predicted class, and the final retention score blends the classifier's probability with that normalized outlier score. The paper reports that the combination outperforms LLMLingua-2 on in-domain summarization and on out-of-domain long-context understanding and reasoning benchmarks, while adding almost no compression overhead and enabling up to 3.3x speedup on a smartphone at 4x compression.","feed_headline":"Outlier scores keep rare tokens alive in compressed prompts","feed_subtitle":"Penalizing similar token classes and rescoring rare tokens lifts long-context accuracy and reaches 3.3x phone speedup.","key_machinery":"The two load-bearing mechanisms are the inter-class cosine similarity loss L_CS(phi), the average cosine similarity between preserve-class and discard-class token representations at the last layer, and the retention metric m_k = $\\alpha$ * p_k + (1 - $\\alpha$) * s_norm_k, where s_norm_k is the within-class normalized Z-score norm of token k. The loss directly attacks over-smoothing at the layer the classifier reads, while the outlier score gives statistically rare tokens within their predicted class a second chance to survive compression.","core_discovery":"The paper claims that a token-classification prompt compressor can be improved on two fronts. During training, it adds an inter-class cosine similarity loss that directly minimizes the average cosine similarity between final-layer representations of tokens labeled \"preserve\" and tokens labeled \"discard\", shrinking the inter-class overlap that remains after standard fine-tuning. At compression time, it splits tokens by the classifier's predicted class, computes per-class Z-scores of the last-layer token vectors, normalizes the Euclidean norm of each Z-score to [0, 1], and retains tokens by the metric m_k = alpha * p_k + (1 - alpha) * s_norm_k. With both changes, MOOSComp keeps more rare but answer-bearing tokens and separates the two classes better, which the paper argues improves task-agnostic compression across black-box API models and local models.","pith_inferences":["If rarity in representation space is a proxy for task importance, the same per-class Z-score trick could be ported to any token-scoring compressor, replacing expensive LLM-based perplexity scoring with a nearly free pass over the final layer.","The per-class normalization is likely necessary because training now separates class means, so a single global Z-score would flag tokens mostly by class membership rather than within-class rarity; the ablation comparing per-class versus pooled outlier scores supports this reading.","At higher compression ratios, rare-token effects should be stronger, since ordinary high-probability tokens fill the budget; a compression-ratio sweep beyond 5x could reveal whether outlier weighting should grow as the budget shrinks.","Code completion being the failure case suggests that tasks where information is carried by structure rather than lexical rarity need a different rescue mechanism, perhaps preserving syntax-special tokens with outlier weights set to zero."],"forward_implications":["On the MeetingBank summarization test with GPT-3.5-Turbo, MOOSComp reports higher Rouge and BERTScore than LLMLingua-2 at 3x compression while using a similar token budget.","On LongBench with GPT-3.5-Turbo, single-document QA rises from 29.8 to 35.3 at the 2K-token constraint, and multi-document QA from 33.1 to 35.5.","The outlier scoring adds only about 0.01 seconds of compression cost over LLMLingua-2 on a V100 GPU, since it is a single Z-score normalization pass over the last layer.","Ablations show that either mechanism alone improves over LLMLingua-2, and combining both gives the best results.","The outlier mechanism does not help code completion, which the paper attributes to the inherently structural nature of code texts."],"supporting_citations":[{"why":"Defines the LLMLingua-2 compressor architecture, MeetingBank compression training data, and binary token classification setup that MOOSComp extends.","marker":"Pan et al., 2024"},{"why":"Provides MeetingBank, the source of the compression training set and the in-domain summarization test.","marker":"Hu et al., 2023"},{"why":"Supplies the xlm-roberta-large encoder used as the compressor's feature extractor.","marker":"Conneau et al., 2020"},{"why":"Documents over-smoothing in BERT, motivating the inter-class cosine similarity loss.","marker":"Shi et al., 2022"},{"why":"Supplies the Z-score outlier methodology whose normalized norms become the outlier term in Eq. (7).","marker":"Anusha et al., 2019"},{"why":"Earlier task-agnostic selection method that retains high-information tokens, giving the rarity-as-importance idea a precedent.","marker":"Li et al., 2023"},{"why":"LLMLingua's iterative high-perplexity token selection is the baseline family and another precedent for preserving rare tokens.","marker":"Jiang et al., 2023"},{"why":"LongBench is the long-context benchmark used for out-of-domain understanding evaluation.","marker":"Bai et al., 2024"}],"fun_headline_variants":["Outlier scores rescue rare tokens in compressed prompts","Anti-smoothing loss sharpens token classification for compression","MOOSComp: rare token survival via outlier scores","3.3x faster mobile compression with MOOSComp","Two fixes lift long-context compression accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a token whose last-layer BERT representation is far from its predicted class's mean, measured by the Euclidean norm of its per-feature Z-score, is a token worth preserving because it carries task-relevant rarity; if statistical rarity in representation space does not line up with task importance, the outlier term injects noise instead of signal.","fun_headline_variants_meta":{"raw":{"variants":["Outlier scores rescue rare tokens in compressed prompts","Anti-smoothing loss sharpens token classification for compression","MOOSComp: rare token survival via outlier scores","3.3x faster mobile compression with MOOSComp","Two fixes lift long-context compression accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000376,"raw_usage":{"total_tokens":1984,"prompt_tokens":904,"completion_tokens":1080,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":1005}},"tokens_in":520,"tokens_out":1080,"duration_ms":9701,"temperature":1.0,"reasoning_tokens":1005,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:55:19.557399+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a reading-comprehension prompt engineered so that the answer-bearing tokens are among the most typical tokens in their class (low Z-score norms) while frequent filler tokens are made unusual; if MOOSComp with alpha below 0.5 still retains the fillers and drops the answers on a few hundred samples, the rarity-importance link underlying Eq. (7) is falsified.","supporting_citations":[],"review_version":1}