{"id":"025f748c-3e09-45f6-8df2-c6ebd44e386d","arxiv_id":"2507.08045","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Krul dynamically selects per-conversation cross-layer KV cache compression from attention similarity patterns, cutting TTFT by 1.28x to 2.68x and KV storage by 1.33x to 2.35x with less than 1% average accuracy loss on LongBench.","lead":"Krul is an LLM inference system that chooses a different KV cache compression plan for each conversation, based on how similar attention patterns are across model layers. It claims up to 2.68x faster time-to-first-token and up to 2.35x smaller KV cache storage than fixed-compression and recompute-loading baselines, while keeping answer quality roughly unchanged.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy evaluation may leak the simulated future user input into compression-strategy selection, so Table 4 may not test the preemptive setting that the central claim depends on.","rationale":"The reader's weakest-assumption analysis focused on the I-R layer hypothesis being validated only on five queries with Qwen1.5-7B and then extrapolated to other models. That is a real concern about generalization of the preemptive selector. However, the more load-bearing issue is whether the evaluation even exercises the preemptive constraint: if compression-strategy selection uses attention weights computed on the full context including the simulated future user input, then Table 4 does not measure what the system would do in a real multi-turn restoration, where the future input is unknown at compression time. This is not an ad hominem or a stylistic objection; it is a question about the validity of the principal accuracy evidence. The paper's own Challenge 1 makes future-input unpredictability the core difficulty, so the evaluation protocol must respect that boundary. The text's silence on which prompt feeds Algorithm 1 is therefore a critical gap. Because the concern is addressable by a rerun and an explicit statement of the protocol, the appropriate verdict remains conditional acceptance with an added requirement, which matches the reader's CONDITIONAL verdict; hence UNCHANGED. The reader and I partially overlap: we both worry about the preemptive selector's validity, but the reader emphasizes the small validation set and cross-model extrapolation, whereas I emphasize the possible leakage of future input into strategy selection within the accuracy experiments themselves. A corrected evaluation would also help settle the reader's concern, since running the historical-only protocol on multiple models would show whether the I-R characterization holds without the future query in the input.","tokens_in":20128,"tokens_out":7834,"duration_ms":88725,"concrete_test":"Rerun the Table 4 accuracy protocol on Qwen1.5-7B with Algorithm 1 executed on the historical prefix only (first L-128 tokens) to select the compression strategy; then append the 128-token simulated user input and decode using the LongBench evaluation harness. Compare the resulting average accuracy against the reported Krul row and against Baseline. If the average loss relative to Baseline exceeds the claimed <1% (or exceeds the Krul-vs-Minicache gap in Table 4), the central 'without compromising generation quality' claim is unsupported in the preemptive setting. The paper should also explicitly disclose the value of gamma and state which prompt C was used in the original evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 8.1 states that for accuracy evaluation they 'split the last 128 tokens of the context to simulate the user’s input' and then apply KV-cache compression to the remaining context, but it never states which prompt C is passed to Algorithm 1 when selecting the compression strategy. If C is the full context including those 128 tokens, then the strategy is chosen with knowledge of the future user input, which is exactly what Challenge 1 (§2.3) says is unavailable in the real deployment. In that case, Table 4's near-baseline accuracy does not validate the preemptive compression strategy selector, and the central claim of 'without compromising generation quality' is untested in the intended setting. The I-R layer detection in Algorithm 1 lines 3-8 depends on attention directed to the first and last 10% of C; with the user input appended, deep layers may be classified as I-R because they attend to the newly added recent tokens, whereas the historical-only attention distribution could be quite different. No passage in §4 or §8 clarifies that attention weights are obtained from the historical prefix alone before the user input is appended; this ambiguity makes the reported accuracy evidence uninterpretable as support for the preemptive claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Krul, an LLM inference system for multi-turn conversations that restores historical KV caches faster and with less memory by dynamically selecting which layer pairs to compress, based on attention-weight similarity. The system has three main components: a preemptive compression strategy selector that identifies input-insensitive I-R layers and customizes compression per conversation, a token-wise heterogeneous attention similarity estimator that offloads prefill attention computation to the CPU while handling decoding attention on the GPU, and a bubble-free restoration scheduler that overlaps recomputation and loading. The authors report 1.5x-2.68x TTFT reduction and 1.33x-2.35x KV storage reduction versus state-of-the-art baselines, with accuracy evaluated on LongBench tasks. The core idea is that I-R layer attention is input-invariant, making those layers safe to compress, while non-I-R layers are preserved to avoid degrading future-turn quality.","tokens_in":20364,"tokens_out":4155,"duration_ms":45315,"significance":"If validated, the paper makes a useful systems contribution: it replaces fixed layer-pair compression with conversation-adaptive compression, addresses the memory cost of attention similarity computation, and introduces a scheduler that hides restoration latency. The preemptive selector is a plausible response to the real challenge that future user inputs are unavailable at compression time, and the multi-model TTFT and storage experiments are broad. However, the central claim that quality is not compromised rests on an accuracy evaluation whose experimental protocol is ambiguous and whose statistical support is thin; the current evidence is not sufficient to establish the headline claim.","major_comments":[{"comment":"The accuracy evaluation in §8.1 does not state whether the prompt C passed to Algorithm 1 includes the 128 tokens that simulate the user's input or only the historical prefix. If C is the full context including those 128 tokens, then the I-R layer detection (Algorithm 1 lines 3-8) and the layer-pair similarity computation are performed with access to the future user input, which is precisely the information that Challenge 1 (§2.3) says is unavailable in deployment. Table 4 would then validate an oracle compression selector, not the preemptive selector that the central claim depends on. The authors should specify C explicitly and, if C includes the simulated user input, rerun the accuracy evaluation with attention weights obtained from the historical prefix alone.","section":"§8.1 and Algorithm 1"},{"comment":"Accuracy is reported only for Qwen1.5-7B, with no variance, error bars, or significance tests. The average is only 0.24 points below the full-KV baseline (26.70 vs. 26.94), and several tasks drop below the baseline (e.g., gov_report 25.3 vs. 23.24, qasper 19.7 vs. 18.46, trec 65.5 vs. 64.0). Thus the abstract's claim of 'without compromising generation quality' is not established. Please report per-task variance and statistical significance, and include accuracy results for at least one additional model used in the TTFT experiments, since the TTFT claims are made for LLaMA-7B, LLaMA-30B, and Qwen2-72B.","section":"§8.5, Table 4"},{"comment":"The I-R layer pattern is validated through a controlled study with five synthetic queries on Qwen1.5-7B (Figure 5 and Table 2), and then extrapolated to DeepSeek-6.7B, LLaMA-13B, and Qwen2-72B solely by counting I-R layers (Table 3). The safety claim that I-R layers can be compressed without future-turn degradation is model-specific and attention-dependent; the absence of accuracy measurements on those models leaves the generalization claim unsupported. Please either add per-model accuracy checks or restrict the safety conclusions to Qwen1.5-7B.","section":"§4.1, Table 3"},{"comment":"The abstract reports a 1.5x-2.68x TTFT reduction, while the conclusion and §8.2 state 1.28x-2.68x. This is an internal inconsistency in the headline quantitative claim and must be resolved by using one consistent range throughout the paper, with the lower bound justified by the reported experiments.","section":"Abstract, §8.2, §10"}],"minor_comments":[{"comment":"Eq. (1) begins with 'ere' instead of 'Here', and the recomputation ratio r_c is not clearly defined before it appears in the optimization; please define r_c before using it in Eq. (1).","section":"§6.1, Eq. (1)"},{"comment":"The y-axis label in Figure 21 says 'Latency (s)' while the values (tens) and the surrounding text suggest milliseconds; please correct the units.","section":"Figure 21"},{"comment":"The overview contains a duplicated phrase: 'identifies model layers input-sensitive model layers' should be 'identifies input-sensitive model layers'.","section":"§3"},{"comment":"The expression for avg_weight_sum is notationally ambiguous; please clarify which entries of the attention matrix are summed and how heads and positions are averaged.","section":"Algorithm 1, line 6"},{"comment":"Reference [3] is cited as LongBench, but the arXiv identifier 2412.15204 corresponds to LongBench v2; please confirm which benchmark was actually used and cite accordingly.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The evaluation-leak concern in §8.1 is the main risk to the paper's central claim; as written, the ambiguity allows the accuracy result to be read as an oracle evaluation. If the authors clarify that C is the historical prefix only, the concern is resolved, but the paper should also address the lack of statistical significance and the missing per-model accuracy before I can recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Krul makes a genuine point: the best layers to share or compress KV caches differ per conversation, and a fixed policy like MiniCache's leaves accuracy on the table. The evidence for that (Table 1, three random strategies varying across tasks) is thin but enough to justify adaptive selection. The three-component system is thoughtful, and the CPU/GPU split for attention-similarity computation addresses a real overhead that dynamic selection creates. That part is worth reading.\n\nThe soft spots are mostly in evaluation, and they are addressable. Accuracy is reported for only one model (Qwen1.5-7B) in Table 4, with no variance or significance tests. For a claim of 'without compromising generation quality', that is too thin. The abstract quotes 1.5x-2.68x TTFT reduction while the conclusion says 1.28x-2.68x; pick one. KVSharer is the closest dynamic baseline and is only discussed, not compared. The code link is a placeholder, and gamma is unreported.\n\nThere is also a protocol ambiguity. Section 8.1 says the last 128 tokens of the context are used to simulate the user's input and compression is applied to the remaining context, but the paper does not state which prompt is passed to Algorithm 1 for strategy selection. If C is the full context, the strategy selection leaks the future user input, which is exactly the scenario the 'preemptive' claim excludes. If C is the remaining context, the evaluation is fine but the paper needs to say so. The stress-test note raises this, and on reading, I think the ambiguity is real but probably not a fatal flaw—the natural reading is that C is the remaining context—but the authors must state it explicitly.\n\nOverall, the core idea is plausible and the system is clearly described. It deserves a serious referee, but the revision must add accuracy results for more models with error bars, the missing KVSharer baseline, full hyperparameters, and a precise evaluation protocol.","headline":"Per-conversation KV compression layer selection is a real idea, but the accuracy evaluation is underpowered and the protocol needs to rule out future-input leakage.","tokens_in":20899,"tokens_out":3547,"would_cite":true,"duration_ms":35194,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Krul claims that conversation-adaptive KV cache compression, guided by attention similarity across layers, lets a serving system restore a paused multi-turn chat 1.5x–2.68x faster and with 1.33x–2.35x less cache storage than…","keywords":["KV cache compression","multi-turn conversation","state restoration","LLM inference","attention similarity","cross-layer KV sharing","time-to-first-token","recompute-load pipeline"],"falsifier":"Take a long multi-turn history with several facts placed in the middle of the context, ask a later turn about one of those middle facts, and compare Krul's compressed-restoration answer against the answer from full recompute on a model where the reported I-R layer ratio is lower than claimed; if the compressed response loses the fact while full recompute keeps it, the safety of I-R compression fails. A cheaper per-model check is to compute the average attention weight directed to the first and last 10% of tokens across many query-history pairs; if any layer that the selector compresses consistently falls below the threshold gamma on those pairs, the preemptive exclusion rule is violated.","tokens_in":19933,"feed_emoji":"⚡","tokens_out":4506,"duration_ms":51070,"temperature":0.7,"pith_summary":"This paper argues that resuming a paused multi-turn conversation by recomputing the full history or reloading an uncompressed key-value (KV) cache is wasteful, and that fixed cross-layer KV compression is the wrong cure because the best layer pairs to share differ from one conversation to another. Krul instead picks a compression strategy per conversation, using the similarity of attention patterns between model layers, then restores the conversation by overlapping compressed-cache loading with recomputation. The paper reports that this cuts time-to-first-token by 1.5x–2.68x and KV cache storage by 1.33x–2.35x against state-of-the-art baselines, with average accuracy loss under 1%. If this holds, serving systems that juggle many inactive conversations could keep far more history resident and restart turns much faster.","feed_headline":"Adaptive KV compression restarts LLM chats up to 2.68x faster","feed_subtitle":"Krul picks per-conversation layer pairs to share, cutting cache storage 1.33x-2.35x while keeping answer quality.","key_machinery":"The central object is the Euclidean distance between flattened attention-weight matrices of layer pairs, used as a similarity score for deciding which layers' KV caches can be shared. It is paired with the I-R (initial-recent) layer pattern, defined as layers whose attention concentrates on the first 10% and last 10% of tokens regardless of user input, which acts as a preemptive filter that excludes input-sensitive layers from compression. This filter is implemented in Algorithm 1: for each layer, the average attention weight directed to the initial and recent token subsets is compared against a threshold gamma, and layers below it are marked non-I-R. Among the remaining I-R layers, the algorithm picks the layer pairs with the smallest attention distance up to a compression ratio r_l, and it avoids using any layer in more than one shared pair. The supporting machinery is the token-wise heterogeneous estimator, which splits attention-similarity computation between CPU (prefill, offloaded asynchronously) and GPU (decoding, per token), and the bubble-free restoration scheduler, which balances recomputation and loading so that computation and I/O streams fully overlap.","core_discovery":"Krul's central claim is that a per-conversation compression strategy selected from attention-weight distances can safely share KV caches across layer pairs without degrading future answers, because a large fraction of transformer layers are input-insensitive. The paper identifies three attention patterns, calling the deep layers I-R layers: these layers consistently concentrate attention on the first and last 10% of tokens regardless of user input, so their KV caches can be compressed. Krul preemptively excludes non-I-R layers from compression, then greedily picks layer pairs with the smallest Euclidean distance between their attention-weight vectors, subject to a tunable compression ratio. To make this affordable, it offloads prefill-phase attention weights to the CPU for asynchronous similarity computation while decoding-phase attention is processed per token on the GPU, and it uses a bubble-free scheduler that balances recomputation and loading across layers. Empirical evaluation on LongBench and ShareGPT across LLaMA-7B/30B, Qwen1.5-7B, and Qwen2-72B shows TTFT reductions of 1.5x–2.68x and KV storage reductions of 1.33x–2.35x compared with full reload, fixed compression, and recompute-load baselines, with accuracy close to the uncompressed model.","pith_inferences":["Beyond the paper's results, the I-R layer property is likely architecture- and training-dependent, not universal; the paper validates it with five queries on Qwen1.5-7B and then extrapolates to other models, so future models with different attention biases may need an online calibration step rather than a fixed I-R assumption.","The paper does not claim that Euclidean distance between attention weights is the optimal proxy for compressibility; a testable extension is to compare it against KL-divergence, mutual information, or the actual task-loss increase from compressing each candidate layer pair.","Krul's approach is complementary to token-level eviction and KV quantization, so combining it with H2O-style heavy-hitter eviction or low-bit KV quantization could compound storage savings, though the paper does not explore this combination.","The CPU offload of prefill attention weights assumes spare CPU bandwidth; on memory-bandwidth-limited servers the asynchronous similarity computation could become a bottleneck, a system-level condition not directly measured in the paper."],"forward_implications":["Multi-turn conversation serving can restart a paused conversation without full prefill, cutting time-to-first-token by 1.5x–2.68x against state-of-the-art restoration methods.","KV cache storage for inactive conversations drops by 1.33x–2.35x, so a server can keep many more conversations resident in CPU memory before spilling to SSD.","Compression is not limited to the latter half of the model: the paper finds I-R layers in the former half (roughly layers 9–24 of LLaMA-7B), so dynamically chosen pairs outperform MiniCache's fixed latter-half adjustment-layer compression.","Accuracy stays close to the uncompressed baseline (average loss under 1% on LongBench), because input-sensitive layers are excluded from compression.","The recompute-vs-load ratio can be tuned per hardware configuration, allowing the restoration pipeline to adapt to different CPU-GPU bandwidth ratios."],"supporting_citations":[{"why":"MiniCache supplies the fixed cross-layer KV compression baseline whose sharing formula Krul adopts and whose layer-pair search space Krul extends from latter-half adjacent layers to all I-R layers.","marker":"[20]"},{"why":"KVSharer motivates layer-wise dissimilarity-based KV sharing but fixes one policy for all data; Krul contrasts its own per-conversation dynamic selection against this static policy.","marker":"[32]"},{"why":"Cake provides the recompute-and-load KV restoration pipeline baseline whose balancing problem Krul's bubble-free scheduler addresses.","marker":"[15]"},{"why":"Pensieve is a stateful serving system with a suffix cache compute-loading pipeline that Krul compares against as a restoration approach without compression.","marker":"[38]"},{"why":"CachedAttention (referred to as AttentionStore) is the full KV cache offload/load baseline for multi-turn serving that Krul aims to improve upon in both latency and storage.","marker":"[10]"},{"why":"Hcache stores hidden states across turns and recomputes KVs, giving Krul a comparison point for storage reduction without cross-layer compression.","marker":"[11]"},{"why":"LongBench is the real-world benchmark used to measure generation quality after KV cache compression in the paper's accuracy evaluation.","marker":"[3]"},{"why":"ShareGPT supplies the multi-turn conversation data used for the token-length analysis, throughput evaluation, and the compression-strategy statistics.","marker":"[5]"},{"why":"H2O represents the KV eviction baseline that drops tokens based on current attention, which Krul argues cannot anticipate future user inputs and therefore loses accuracy.","marker":"[43]"}],"fun_headline_variants":["Krul adapts KV cache sharing per conversation, cutting TTFT up to 2.68x","Dynamic layer-pair KV sharing restores LLM chat state faster","Per-conversation KV cache sharing yields 2.68x faster chat resumption","Krul's adaptive cross-layer KV sharing trims TTFT and cache storage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that layers whose attention concentrates on the first and last tenth of the tokens are input-insensitive in every model and conversation, so compressing their KV caches will not degrade answers to future turns; this is demonstrated with five queries on one 7B model and then assumed to hold across other models and long real-world conversations.","fun_headline_variants_meta":{"raw":{"variants":["Krul adapts KV cache sharing per conversation, cutting TTFT up to 2.68x","Dynamic layer-pair KV sharing restores LLM chat state faster","Per-conversation KV cache sharing yields 2.68x faster chat resumption","Krul's adaptive cross-layer KV sharing trims TTFT and cache storage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1432,"prompt_tokens":1102,"completion_tokens":330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":718,"completion_tokens_details":{"reasoning_tokens":242}},"tokens_in":718,"tokens_out":330,"duration_ms":3813,"temperature":1.0,"reasoning_tokens":242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:41:59.712349+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a long multi-turn history with several facts placed in the middle of the context, ask a later turn about one of those middle facts, and compare Krul's compressed-restoration answer against the answer from full recompute on a model where the reported I-R layer ratio is lower than claimed; if the compressed response loses the fact while full recompute keeps it, the safety of I-R compression fails. A cheaper per-model check is to compute the average attention weight directed to the first and last 10% of tokens across many query-history pairs; if any layer that the selector compresses consistently falls below the threshold gamma on those pairs, the preemptive exclusion rule is violated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MiniCache supplies the fixed cross-layer KV compression baseline whose sharing formula Krul adopts and whose layer-pair search space Krul extends from latter-half adjacent layers to all I-R layers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Pensieve is a stateful serving system with a suffix cache compute-loading pipeline that Krul compares against as a restoration approach without compression."},{"cited_title":"2024.{Cost-Efficient} large language model serving for multi-turn conversations with{CachedAttention}","cited_arxiv_id":null,"evidence_quote":"CachedAttention (referred to as AttentionStore) is the full KV cache offload/load baseline for multi-turn serving that Krul aims to improve upon in both latency and storage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Hcache stores hidden states across turns and recomputes KVs, giving Krul a comparison point for storage reduction without cross-layer compression."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ShareGPT supplies the multi-turn conversation data used for the token-length analysis, throughput evaluation, and the compression-strategy statistics."}],"review_version":1}