{"id":"3830e0a7-04de-46c3-a0ab-782800f70bab","arxiv_id":"2412.12094","paper_version":6,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Keeping only initial, separator (punctuation), and neighboring tokens in the KV cache preserves LLM performance while cutting cache size by more than half on GSM8K chain-of-thought.","lead":"SepLLM speeds up large language models by keeping only a few special tokens, such as punctuation and spaces, in memory and discarding the rest, while matching the accuracy of the full model on reasoning benchmarks. The method offers a simple plug-and-play way to cut memory and compute for long-context inference, with applications to streaming and on-device deployment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'separator compression' mechanism is underdetermined: matched-KV ablations do not rule out the high-frequency attention-sink explanation the authors themselves note in Appendix H.","rationale":"The reader's weakest assumption is the same one I identify as most load-bearing: the fixed separator set must be sufficient long-range anchors, and the separators must actually store compressed segment information. The paper's matched-KV ablations and FixLLM comparisons are good controls and show that separators outperform random or fixed anchors, but they do not distinguish 'separator hidden state encodes the segment' from 'separator positions are convenient high-frequency attention sinks at segment boundaries.' The authors themselves raise the frequency-based explanation in Appendix H, so this is not an external criticism but an unresolved internal ambiguity. The universal-approximation theorem issue flagged by the reader is real but less load-bearing because the main practical claim is empirical; the single-run issue is a reporting weakness. What would settle the central claim is a direct measurement of the information content of separator hidden states. If the probe test shows no content advantage, the method may still work as a sparse-attention heuristic, but the paper's title and mechanism would be overstated. Since the reader already reached CONDITIONAL on grounds that include this concern, my read does not change the verdict.","tokens_in":26749,"tokens_out":7613,"duration_ms":75181,"concrete_test":"Run a probe experiment on Llama-3-8B-Instruct. Sample ~1,000 long sequences (e.g., PG19 and GSM8K-CoT). At a middle layer, collect hidden states at (i) the last separator token of each segment and (ii) a fixed-interval token chosen so that both positions have the same number of preceding tokens. Train linear probes to predict a segment-level fact, such as whether a specific number or entity name appears in the segment just closed, or to reconstruct a low-rank representation of the segment's content. Compare probe accuracy/AUC for separator states versus control states. If separator states decode segment content significantly better than fixed-interval states, the compression hypothesis is supported; if not, SepLLM's gains should be attributed to anchor placement and token frequency rather than to separator-borne compression, and the central mechanism claim should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that separator tokens compress their enclosing segments, so discarding all non-separator, non-neighbor, non-initial KV is near-lossless. The evidence for this mechanism is indirect: attention heatmaps show high scores on separators, and ablations show that removing separator KV hurts more than removing fixed-interval KV. Both observations are also predicted by a weaker hypothesis stated in Appendix H: separators are extremely high-frequency tokens whose embeddings act as generic attention sinks. Under that hypothesis, separators are useful anchors because of token frequency and segment-boundary placement, not because their hidden states store a compressed summary of the segment. The matched-KV comparisons with StreamingLLM and FixLLM control for the number of anchors, but a fixed-interval anchor at position k also has access to the full preceding prefix, so position alone explains part of the benefit. The mechanism claim therefore is underdetermined by the reported experiments. This matters because the paper builds on the mechanism to claim that SepLLM is a 'native sparse attention' mechanism and to justify training-from-scratch with a mask that forces compression into separators; if the true mechanism is anchor frequency, the English-only results may not transfer to tokenizers or languages with different separator distributions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports that separator tokens (commas, periods, spaces, newlines, and similar) receive disproportionately high attention in pretrained LLMs, and hypothesizes that these tokens compress the information of the text segments they delimit. On this basis it proposes SepLLM, a sparse-attention method that retains only three groups of KV-cache entries: a few initial tokens, all separator tokens, and a local window of neighboring tokens. The method is evaluated in three regimes: training-free inference (Llama-3-8B on GSM8K-CoT and MMLU, achieving a KV-cache reduction of over 50% at near-vanilla accuracy), training from scratch (Pythia-160m on the Pile), and post-training (Pythia-1.4B), plus a streaming variant that keeps a bounded cache and runs on sequences up to 4M tokens (PG19). The controls include a matched-KV StreamingLLM baseline, a fixed-interval anchor baseline (FixLLM), and standard KV-compression baselines (H2O, SnapKV, PyramidKV). An appendix contributes a universal-approximation theorem for an encoder-based variant of SepLLM, adapted from Yun et al. (2020).","tokens_in":26896,"tokens_out":32355,"duration_ms":260586,"significance":"Conditional on the results, this is a practically useful contribution: SepLLM is a simple, plug-and-play KV-retention rule that requires no importance scoring, integrates into both training-from-scratch and fine-tuning, and ships with a hardware-efficient kernel and released code. The experimental program is broad and internally consistent. I credit in particular the matched-KV StreamingLLM control (Table 1), the FixLLM fixed-interval control (Table 17), which rules out the trivial alternative that any uniformly placed sparse anchor would work, the systematic ablations of cache hyperparameters (s, w, c, separator list, initial tokens, positional shifting), and the range of backbones (Pythia-6.9B/12B, Llama-3-8B base/instruct, Falcon-40B). The main caveat is interpretive: the experiments establish that separator positions are effective sparse anchors, but they do not directly establish that separator hidden states contain compressed summaries of the preceding segments. That caveat affects the framing claims in Section 6 ('native sparse attention') and the title more than the validity of the method as an engineering contribution.","major_comments":[{"comment":"The compression mechanism is underdetermined by the reported experiments. The evidence offered for the claim that 'information of the segments... can be effectively condensed into the separator tokens themselves' is (i) attention heatmaps (Figure 2; Appendix A shows layers 0-2, head 0 only), (ii) the removal ablation against StreamingLLM at matched KV (Table 1), and (iii) the fixed-interval control FixLLM (Table 17, Appendix I). All three are also predicted by the rival hypothesis that the paper itself states in Appendix H: separators are extremely high-frequency tokens whose keys act as generic attention sinks and whose positions coincide with syntactic boundaries. Because a fixed-interval anchor at position k is also computed from the full prefix in the training-free setting, FixLLM compares anchor positions but cannot distinguish 'separator hidden states summarize the preceding segment' from 'separator positions are good places for prefix-computed anchors'; the Needle-in-a-Haystack result (Appendix F) is likewise consistent with both readings. Consequently, the sentence in §4.2 ('This indicates that the KV of separators indeed encapsulates information contained within their respective segments') overclaims what Table 1 shows. Since the mechanism is used to justify the training mask (§3.1) and the 'native sparse attention' framing (§6), I ask the authors to either (a) add direct evidence—for example, a frequency-matched non-separator anchor (a common content token at the same positions), a probe of whether separator KVs encode segment content, or an evaluation on tokenizers/languages with different separator frequency—or (b) restate the contribution as the empirical finding that separator positions are effective sparse anchors, and correspondingly soften the abstract, title, and Section 6.","section":"§3.1, §4.2, §4.6/FixLLM, Appendix H"},{"comment":"Theorem 5.1, as stated in the main text, claims universal approximation for every f in F over arbitrary inputs, but the proof in Appendix J relies on an assumption that appears only in the appendix's class definition: 'for at most s successive tokens, a special token will appear in the sequence.' This bounded-gap assumption is used in the token-transmission step of Lemma K.4 ('the last token can attend to the nearest special tokens with the help of neighboring tokens, requiring at most ceil(s/l) layers'), so the construction does not cover inputs without special tokens or with arbitrarily large separator gaps. The main-text statement omits this restriction, and the class notation T^{2,1,4}_{Sep} does not expose the parameters l and s, so the theorem as stated overclaims what the proof establishes. In addition, the theorem is proven for an encoder-based variant with bidirectional local windows, whereas the evaluated SepLLM is a causal decoder; the paper should state explicitly how the encoder version bears on the decoder architecture used in the experiments. Please restate Theorem 5.1 with the separator-regularity assumption made explicit and the layer-count dependence on n, l, s, and delta given.","section":"§5 and Appendices J-K"}],"minor_comments":[{"comment":"The post-training section presents only loss curves in the main text; the downstream evidence for post-training appears in Appendix D (Table 14) but for a different setting (LongAlpaca fine-tuning of Llama-3-8B rather than continued pretraining of Pythia-1.4B). Please add downstream-task results for the post-trained Pythia-1.4B checkpoint (for example, the tasks used in Table 2) or make the relationship between the two experiments explicit.","section":"§4.4, Figure 6"},{"comment":"The perplexity protocol is underspecified: please state how the test text is chunked, whether caches are reset at chunk or book boundaries, how positions are assigned after compression, and whether perplexity is computed on ground-truth next tokens or on model-generated continuations. The absolute values reported (Vanilla 302.6/1090.8 on PG19 at 20K/64K; StreamingLLM around 31-38 for Llama-3-8B) are far outside the ranges typically reported for this model, so these details are needed for reproducibility.","section":"Tables 4, 5, and 11-13"},{"comment":"The introduction states that SepLLM 'reduces computational costs by 28% and training time by 26% while achieving the same training loss,' but Table 10 reports 2524.45 ms to 1648.11 ms per iteration, which is a ~35% wall-clock reduction; please reconcile the 26% figure or identify the setting it refers to.","section":"Introduction and Table 10"},{"comment":"The attention-visualization evidence shows only head 0 of layers 0-2; because the paper claims a consistent pattern across the model, an aggregate statistic (for example, the fraction of heads and layers with high separator-attention mass) would be more convincing than three individual attention maps.","section":"Appendix A and Figure 2"},{"comment":"Please unify naming and notation: 'StrmLLM' in the tables versus 'StreamingLLM' in the text, 'streamingLLM' in §4.5, and the typo 'SegLLM' in the Figure 4 caption; also align the mask notation in Eq. (1) ('Mul(Q, K^T M)') with the elementwise definition in Eq. (2).","section":"Throughout, Figure 4, Eq. (1)"},{"comment":"The separator-choice ablation fixes n=256 while the separator list shrinks, so the rows with fewer separators consume less KV budget; a budget-matched comparison (increasing n in the reduced-separator rows) would isolate the effect of separator identity more cleanly, as the StrmLLM (n=380) row does for the no-separator case.","section":"Table 16, Appendix G"},{"comment":"The second explanatory paragraph contains a non-sequitur: 'separators need to be generated by the language model very frequently. Therefore, their attention values with respect to any other token cannot be too small.' Output-token frequency does not imply high attention mass on that token as a key; this argument should be removed or reformulated, particularly since it appears in the very appendix that discusses alternative explanations.","section":"Appendix H"}],"recommendation":"major_revision","confidential_remarks":"If the manuscript is the ICML 2025 camera-ready version (per the footer on page 1), the editors should ask the authors to state explicitly what the journal version adds beyond that publication. I also flag the citation pattern in Appendix I: the sentence on sparsity cites Chen et al. (2025a,b) and Zheng et al. (2025), which are the authors' own works on recommendation systems and softmax adjustment and are unrelated to the present topic; this looks like padding and should be checked. On substance, the empirical contribution is solid; my recommendation is driven by the overstated mechanism interpretation and the gap between Theorem 5.1 and its proof, both of which are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SepLLM is a genuinely useful empirical paper. The core idea is simple: when generating, keep only initial tokens, a local window, and separator tokens (commas, periods, newlines), and drop everything else. On GSM8K-CoT with Llama-3-8B it keeps about 47% of the KV cache and stays within a point of full attention. The matched-KV comparison against StreamingLLM matters: at equal cache budget, SepLLM beats StreamingLLM by a wide margin (77.18 vs 71.42 on GSM8K). Even better, the FixLLM control, retaining one token every k positions instead of separators, does substantially worse, which rules out the lazy hypothesis that any set of anchors works. That is a real contribution, and the paper executes it across training-free, training-from-scratch, post-training, and streaming settings up to 4M tokens. The training-from-scratch results on Pythia-160m are especially credible because the model genuinely learns to use the mask.\n\nThe soft spots are real but not load-bearing. First, the mechanistic claim that separators 'compress segment information' is underdetermined. The evidence, high attention on separators and the FixLLM comparison, is also consistent with the attention-sink/frequency explanation the authors themselves float in Appendix H: separators are extremely frequent tokens with well-conditioned embeddings, so they serve as generic anchors. A control using other high-frequency tokens (e.g., 'the', 'a') would have sharpened the mechanism. As it stands, the practical method works, but the interpretive gloss should be toned down. Second, the main-text Theorem 5.1 states universal approximation without the assumption, present in Appendix J, that a special token appears at least every s positions. The proof relies on that assumption, so the main-text statement is technically stronger than what is proved. Minor, but sloppy. Third, all results are single runs without error bars. For large gaps this is fine; for the near-tie between SepLLM (77.18) and Vanilla (77.26), it means the 'comparable performance' claim is qualitative.\n\nOverall, this is a paper I would cite and would want a serious referee to look at carefully. The method is simple enough to adopt in practice, and the empirical work is broad and honest. The mechanism needs one more control experiment, and the theory section needs to state its assumptions honestly. Neither issue undermines the core practical result.","headline":"A practical KV-cache compression method with strong matched-KV controls, but the 'separator as segment summary' mechanism is overclaimed and the theory section is sloppy.","tokens_in":687,"tokens_out":1247,"would_cite":true,"duration_ms":31452,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Separator tokens act as compressed segment summaries, so a transformer can drop most of its KV cache and still match full attention.","keywords":["KV cache compression","separator tokens","sparse attention","efficient transformers","long context inference","streaming inference","attention sinks","token pruning"],"falsifier":"Take a long passage with a fact embedded mid-sentence between commas, then ask a training-free SepLLM to retrieve that fact from a prompt of thousands of tokens; if accuracy drops sharply relative to full attention while a same-budget StreamingLLM also fails, then separator hidden states alone are not carrying segment content in that regime.","tokens_in":26470,"feed_emoji":"⚡","tokens_out":7360,"duration_ms":60252,"temperature":0.7,"pith_summary":"This paper claims that punctuation and whitespace tokens—commas, periods, newlines—are not noise but compressed summaries of the text segments they close. On that basis it builds SepLLM, a transformer attention mask that lets each token see only a few initial tokens, all separator tokens, and a local window of neighbors, while discarding every other key-value pair. With Llama-3-8B this halves the KV cache on the GSM8K-CoT benchmark (down to 47.36% of the original) while keeping accuracy at full-attention levels, and it extends streaming inference to millions of tokens under a bounded cache. A sympathetic reader would care because inference memory and attention cost scale with kept tokens, so a fixed-token compression that preserves accuracy makes long-context deployment cheaper.","feed_headline":"Punctuation can replace half the KV cache in Llama-3-8B","feed_subtitle":"SepLLM keeps only initial, separator, and neighboring tokens, matching full-attention accuracy on GSM8K at 47% cache.","key_machinery":"The machinery is a binary attention mask $M$ that zeroes out every attention pair except those between a query and (i) the first $a$ initial tokens, (ii) the separator tokens that have already appeared, and (iii) the $n$ nearest preceding tokens; the paper also implements a Sep-Attention kernel for sparse matrix multiplication. Separators are the load-bearing element: the list $\\{\\texttt{.}, \\texttt{,}, \\texttt{?}, \\texttt{!}, \\texttt{;}, \\texttt{:}, \\texttt{ }, \\texttt{\\t}, \\texttt{\\n}\\}$ defines the segment boundaries, and during training the mask forces the model to condense each segment's content into the corresponding separator's key and value. For streaming, four cache blocks (initial, separator, local window, past window) keep total usage under capacity $c$; once full, past-window non-separators are dropped and separator KV moves into the separator cache, giving a periodic cache evolution whose average usage tends to $(w+c+a+s)/2$, strictly below $c$. The universal-approximation appendix shows that the class of SepLLM networks with $H=2$, $d_h=1$, $d_f=4$ can approximate any continuous sequence-to-sequence function.","core_discovery":"The central claim is that information within a segment of text is functionally stored in the hidden state of the token that ends the segment, so a model does not need to attend to the segment's interior once it has seen the separator. SepLLM operationalizes this with a data-dependent sparse mask that keeps three classes of tokens: initial attention sinks, separator tokens from a fixed set of nine punctuation and whitespace types, and the closest neighbors of the current token. The authors support the compression hypothesis with attention visualizations showing separators receiving disproportionately high scores, removal experiments in which deleting separator KV hurts reasoning, and a fixed-interval variant that attends at regular intervals and performs worse. In training-free mode on Llama-3-8B, SepLLM retains 47.36% of the original KV cache and achieves 77.18% on GSM8K-CoT versus 77.79% for full attention; in streaming mode the cache stays bounded and perplexity remains stable out to millions of tokens.","pith_inferences":["Editorial inference: if separator hidden states really serve as RNN-like memory, tokenizers that insert explicit boundary tokens could make the compression more robust than relying on the nine-natural-punctuation list.","Editorial inference: the fixed nine-token separator set is an English-centric inductive bias; languages with different punctuation, or code with operators as boundaries, would need a different anchor set, and the method's performance there is a direct testable extension.","Editorial inference: the compression hypothesis implies a hierarchy of segment granularity—short separators (commas) inside longer ones (periods)—and a natural next step would be checking whether attention to higher-level separators carries more global information than lower-level ones.","Editorial inference: comparing SepLLM's cache against a learned policy that picks boundary tokens by attention score would separate the value of \"separator\" as a linguistic category from the value of \"high-attention token\" generally."],"forward_implications":["With the Llama-3-8B backbone, the KV cache drops below half (47.36%) on GSM8K-CoT with no meaningful accuracy loss, which directly reduces the memory cost per decoded token.","SepLLM is trainable from scratch and via post-training, unlike many KV-compression methods, so its sparse mask can be baked into pretraining and fine-tuning rather than applied only at inference.","In streaming settings the cache stays bounded by capacity $c$ while processing millions of tokens, with average usage tending to $(w+c+a+s)/2$, so long dialogues and document streams no longer require unbounded memory.","Training with the SepLLM mask cuts FLOPs by roughly 30% and wall-clock time per iteration by about a third while achieving the same training loss as full attention.","The universal approximation theorem says a small SepLLM network can approximate arbitrary continuous sequence functions, so the sparse pattern is not inherently capacity-limited."],"supporting_citations":[{"why":"Supplies the attention-sink and initial-token mechanism plus the StreamingLLM baseline that SepLLM extends and compares against.","marker":"(Xiao et al., 2024b)"},{"why":"Provides the Llama-3-8B backbone used in the training-free and streaming evaluations.","marker":"(Dubey et al., 2024)"},{"why":"Defines the GSM8K-CoT benchmark on which the headline 47.36% KV ratio is measured.","marker":"(Cobbe et al., 2021)"},{"why":"Defines MMLU, the second training-free benchmark used to show separators preserve broad reasoning.","marker":"(Hendrycks et al., 2021)"},{"why":"Supplies the Pythia backbones and training configurations for training-from-scratch and post-training experiments.","marker":"(Biderman et al., 2023)"},{"why":"Provides the deduplicated Pile dataset used for training and post-training.","marker":"(Gao et al., 2020)"},{"why":"Provides the proof strategy (quantization, contextual mapping, value mapping) used in the universal approximation appendix.","marker":"(Yun et al., 2020)"}],"fun_headline_variants":["SepLLM compresses segments into separators, halving KV cache","Punctuation tokens store sentence info: SepLLM exploits it","SepLLM: keep only punctuation, neighbors, and attention sinks","Half the KV cache: SepLLM compresses segments into separators","Separator tokens are memory: SepLLM drops interior tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a fixed set of nine separator tokens is sufficient as long-range anchors for every input—that every fact a token needs to recall sits in the hidden state of a comma, period, or similar separator—so inputs where important information is not adjacent to a separator (code, formulas, punctuation-sparse prose) may not be compressible this way.","fun_headline_variants_meta":{"raw":{"variants":["SepLLM compresses segments into separators, halving KV cache","Punctuation tokens store sentence info: SepLLM exploits it","SepLLM: keep only punctuation, neighbors, and attention sinks","Half the KV cache: SepLLM compresses segments into separators","Separator tokens are memory: SepLLM drops interior tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3128,"prompt_tokens":961,"completion_tokens":2167,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":2075}},"tokens_in":577,"tokens_out":2167,"duration_ms":13924,"temperature":1.0,"reasoning_tokens":2075,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:16:14.335671+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a long passage with a fact embedded mid-sentence between commas, then ask a training-free SepLLM to retrieve that fact from a prompt of thousands of tokens; if accuracy drops sharply relative to full attention while a same-budget StreamingLLM also fails, then separator hidden states alone are not carrying segment content in that regime.","supporting_citations":[],"review_version":1}