Pith. sign in

REVIEW 9 cited by

SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.12094 v6 pith:XE55HDQM submitted 2024-12-16 cs.CL cs.AIcs.LG

Guoxuan Chen , Han Shi , Jiawei Li , Yihang Gao , Xiaozhe Ren , Yimeng Chen , Xin Jiang , Zhenguo Li
show 2 more authors
Weiyang Liu Chao Huang
This is my paper
classification cs.CLcs.AIcs.LG
keywords tokenssepllmlanguageseparatoracrosscompressingeffectivelyinference
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) have exhibited exceptional performance across a spectrum of natural language processing tasks. However, their substantial sizes pose considerable challenges, particularly in computational demands and inference speed, due to their quadratic complexity. In this work, we have identified a key pattern: certain seemingly meaningless separator tokens (i.e., punctuations) contribute disproportionately to attention scores compared to semantically meaningful tokens. This observation suggests that information of the segments between these separator tokens can be effectively condensed into the separator tokens themselves without significant information loss. Guided by this insight, we introduce SepLLM, a plug-and-play framework that accelerates inference by compressing these segments and eliminating redundant tokens. Additionally, we implement efficient kernels for training acceleration. Experimental results across training-free, training-from-scratch, and post-training settings demonstrate SepLLM's effectiveness. Notably, using the Llama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on the GSM8K-CoT benchmark while maintaining comparable performance. Furthermore, in streaming settings, SepLLM effectively processes sequences of up to 4 million tokens or more while maintaining consistent language modeling capabilities.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Metaphor Tracer: A Theory-Informed Analysis of Hidden States

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Hidden-state aggregator and differentiator scores, frozen on one text, track within-text organization across models and align with engineered registers and psychoanalytic marks while dissociating from information and ...

  2. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  3. SAGE: Selective Attention-Guided Extraction for Token-Efficient Document Indexing

    cs.DB 2026-04 unverdicted novelty 6.0 of 10

    SAGE is a training-free context reduction method that converts attention signals from a small LLM into a differential relevance heatmap to select top units for downstream QA, achieving competitive accuracy at 10% toke...

  4. LightThinker++: From Reasoning Compression to Memory Management

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    LightThinker++ adds explicit adaptive memory management and a trajectory synthesis pipeline to LLM reasoning, cutting peak token use by ~70% while gaining accuracy in standard and long-horizon agent tasks.

  5. ChunkLLM: A Lightweight Pluggable Framework for Accelerating LLMs Inference

    cs.CL 2025-09 conditional novelty 6.0 of 10

    ChunkLLM adds lightweight chunk-boundary and chunk-attention adapters to frozen LLMs, keeping ~98% of long-context quality with ~49% KV cache and up to 4.48x speedup on 120K-token generation.

  6. CaliDrop: KV Cache Compression with Calibration

    cs.CL 2025-07 conditional novelty 6.0 of 10

    CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.

  7. OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference

    cs.CL 2025-07 conditional novelty 6.0 of 10

    OrthoRank selects tokens for computation by their orthogonality to the sink token in normalized hidden states, yielding better perplexity and accuracy than layer pruning at matched sparsity without additional training.

  8. EARN: Efficient Inference Acceleration for LLM-based Generative Recommendation by Register Tokens

    cs.IR 2025-07 conditional novelty 6.0 of 10

    EARN inserts learnable register tokens at both ends of a user prompt, prunes prompt tokens after early layers, and achieves up to 3.79x faster LLM-based recommendation inference with comparable or better accuracy.

  9. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

    cs.CL 2025-02 unverdicted novelty 6.0 of 10

    NSA is a hardware-aligned sparse attention mechanism that enables end-to-end trainable long-context modeling by combining coarse token compression with fine-grained selection.

Pith tools