{"id":"a2177b41-04a2-44b2-9cb4-b164d6dd6dad","arxiv_id":"2412.16187","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"HashEvict uses locality-sensitive hashing to evict the KV-cache token least similar to the current query, compressing 30-70% with modest quality loss across several LLM benchmarks.","lead":"Researchers designed a shortcut for AI language models that decides which old words to forget without doing expensive math. It cuts memory use by 30-70% while keeping answers accurate on math, medical, retrieval, and summarization tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 30–70% compression claim is not supported by the paper's own tables at the 70% end: at a 30% cache budget, GSM8K GPT4-Judge drops 34% (3.845 to 2.520) and Rouge-L drops 15%.","rationale":"The central claim is a range claim: 30–70% compression with maintained quality. The most load-bearing condition is therefore that quality holds at the top of the range. The paper's own tables show a large drop at 30% cache budget: GSM8K GPT4-Judge falls 34%, Rouge-L falls 15%, and MedQA GPT4-Judge falls 25%. Since no error bars, seeds, or code are reported, the reader cannot determine whether this is a stable property of the method or a favorable run. The reader's weakest assumption flags the single-projection SimHash and normalization; the normalization part is not a real flaw because sgn(Rx) is invariant to positive scaling, so unnormalized vectors produce the same hash as normalized ones. The single-projection/no-seed part remains valid, and Appendix E's own correlation analysis (0.30 even with averaging and normalization) suggests the ranking signal is weak. These points reinforce the conditional verdict: the method is plausible and useful at moderate compression (approximately 30–50%), but the 70%-compression end of the headline range and the stochastic stability need to be demonstrated before the claim is relied upon. The reader's conditional accept already requires such validation, so no verdict adjustment is needed.","tokens_in":19001,"tokens_out":10829,"duration_ms":120786,"concrete_test":"Rerun the GSM8K experiment at a 30% cache budget (70% compression) 10 times with different random projection matrices (seeds) and report mean plus/minus standard deviation for GPT4-Judge and Rouge-L. If the mean replicates a quality drop of more than 20% relative to the full-cache baseline, the 70%-compression side of the headline claim should be revised to the largest budget for which all metrics meet a predefined degradation threshold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the headline claim, task quality must remain high across 30–70% compression. The paper's own tables contradict this at the upper end. At a 30% cache budget (70% compression), GSM8K GPT4-Judge falls from 3.845 (full cache) to 2.520 (Table 6), Rouge-L from 0.403 to 0.341 (Table 5), and MedQA GPT4-Judge from 3.337 to 2.511 (Table 9). A 34% drop in a 1–5 judge score is not 'maintaining high performance' under any standard reading. No error bars, seeds, or released code are provided, so it is unclear whether the 30–70% range is robust or an artifact of a favorable run. Additionally, Appendix E reports only a 0.30 average Pearson correlation between attention and the LSH ranking even when using normalized vectors and multiple averaged projections; the deployed single-projection variant is likely noisier still, further weakening the eviction rationale at high compression. The normalization mismatch in the reader's weakest assumption is not itself a real flaw because sgn(Rx) is invariant to positive scaling, so unnormalized vectors hash identically to normalized ones; the single-projection and missing-seed issue remains valid and is the part that matters.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"HashEvict proposes an attention-free KV cache eviction strategy for transformer LLMs. At each decoding step, the method projects the current query embedding and all cached key embeddings to c-bit binary codes via a random Gaussian SimHash matrix, computes Hamming distances, and evicts the key/value pair whose hash is farthest from the query hash, replacing it with the current token. The paper claims that this achieves 30-70% KV cache compression while maintaining high performance on GSM8K, MedQA, long-context retrieval, and summarization tasks, with up to 1.5-2x prefill speedups over H2O/Scissorhands and 17x prefill speedup over FastGen. The evaluation compares against L2-norm eviction, H2O, Scissorhands, and FastGen across cache budgets from 10% to 90%.","tokens_in":19282,"tokens_out":9145,"duration_ms":87364,"significance":"If the claims are fully supported, HashEvict would be a useful contribution to KV cache compression because it avoids computing attention scores for eviction decisions and maintains only a small binary hash table. The paper presents a broad evaluation across seven task types, a hash-dimension ablation, and an attention-loss-ratio analysis. However, the strongest claimed compression range is not backed by the paper's own tables, and the mechanistic validation of the LSH ranking is weak, so the significance is currently conditional on additional evidence.","major_comments":[{"comment":"The headline claim of 30-70% compression while maintaining high performance is not supported at the upper end of the range. At a 30% cache budget (70% compression), GSM8K GPT4-Judge similarity to ground truth falls from 3.845 (full cache) to 2.520 (Table 6), and Rouge-L falls from 0.403 to 0.341 (Table 5); MedQA GPT4-Judge similarity falls from 3.337 to 2.511 (Table 9). These are large degradations (about 34% and 15% on GSM8K) and are inconsistent with the paper's claim that the method 'maintains high performance' across the full 30-70% range. The authors should either restrict the claim to the range where degradation is small (e.g., 30-50% compression) or define an explicit tolerance criterion and show that the 70%-compression results meet it.","section":"Section 4.1, Tables 5, 6, and 9"},{"comment":"The theoretical justification in Eq. (3) is an expectation over the random projection matrix R for unit vectors, but the deployed Algorithm 1 uses a single random projection and makes a deterministic argmin decision each step without averaging. The unit-vector assumption is not the main issue because sgn(Rx) is invariant to positive scaling; the load-bearing gap is expectation-versus-single-draw. The paper reports no variance across random seeds or hash draws, so the stability of the eviction decision is unknown. Appendix E, which reports a mean Pearson correlation of only 0.2978 ± 0.1947 between attention scores and inverted average Hamming distance, uses normalized vectors and averages multiple projections, which are more favorable conditions than the deployed single-projection algorithm. Please provide a seed-variance analysis and an evaluation of the deployed single-projection setting in the correlation/ALR analysis.","section":"Section 3, Eq. (3), and Appendix E"},{"comment":"The ALR analysis in Appendix D is performed using a global ranking of tokens by average Hamming distance against later queries, which the paper acknowledges 'does not perfectly capture the ranking system' of HashEvict. Because the deployed method makes a fresh per-step argmin over the current cache, the reported ALR does not directly measure the attention loss of the actual eviction sequence. Table 3's ALR values therefore do not validate the dynamic behavior of Algorithm 1. Please clarify how the static ALR analysis maps to the algorithm's per-step decisions, or provide a dynamic ALR computed by simulating the actual eviction process.","section":"Appendix D and Table 3"}],"minor_comments":[{"comment":"The text contains the garbled fragment 'cwecwasdf'; this appears to be an editing artifact and should be removed.","section":"Section 2, after Eq. (3)"},{"comment":"The pseudocode does not indicate that the first 4 and last 10 tokens are always kept in the cache, although Section 4 states this; please align the pseudocode with the implementation.","section":"Algorithm 1 and Section 4"},{"comment":"The column 'Compression Ratio' is never defined, and the reported values (e.g., 0.8355 at 10% budget for L2) are not consistent with the reported cache memory numbers under any obvious definition such as 1 - cache_memory/full_memory. Please define the metric and verify the values.","section":"Table 2"},{"comment":"The metric labeled 'GPT4-Rouge' in the text appears to be an inconsistency; the tables use 'GPT4-Judge', and the appendix describes a similarity-to-ground-truth component. Please harmonize the terminology.","section":"Section 4, Metrics"},{"comment":"The 'Open-Source Implementation' contribution states that code will be released upon publication; for reproducibility of the reported numbers, including seed variance, it would be helpful to release the code and evaluation scripts with the submission.","section":"Contributions and Section 4.5"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper proposes a plausible low-overhead eviction method, but the headline 30-70% compression claim is not supported at the 70% end by the paper's own tables, and the mechanistic validation is weak (Pearson r ~ 0.30 even under favorable conditions). The authors should be asked to narrow the claim, report seed variance, and validate the deployed single-projection setting. The writing contains several typos and undefined terms. With these revisions the paper could be acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"HashEvict applies SimHash to KV cache eviction: at each decode step, binarize the query and cached keys, then evict the token with the largest Hamming distance. That is a clean, genuinely new twist on the pre-attention eviction line (L2 dropout, H2O, Scissorhands). The paper shows it is competitive with or better than L2 on GSM8K, MedQA, retrieval, and summarization across a range of budgets, and the hash-dimension ablation plus attention-loss-ratio analysis are useful. Credit where due: this is a real contribution, not a repackaging.\n\nThe headline claim, though, overstates what the tables show. At a 30% cache budget (70% compression), GSM8K similarity-to-ground-truth drops from 3.845 to 2.520, ROUGE-L from 0.403 to 0.341, and MedQA similarity from 3.337 to 2.511. That is not 'maintaining high performance' by any standard reading. The honest version is closer to '30–50% compression with modest loss; 70% with a clear quality cost.' The stress-test note is on target here.\n\nTwo technical soft spots. First, the theory (Eq. 3) is an expectation over random projections, but the deployed algorithm uses a single projection with c = 8–16 bits. The reader's unit-vector normalization worry is a red herring—sgn(Rx) is invariant to positive scaling—but single-projection variance is real. Appendix E uses normalized vectors and averages multiple projections to get a correlation of only about 0.30 between attention and Hamming distance; the deployed variant is noisier still. Second, there are no error bars, no reported seeds, and no released code. With a stochastic hash, that makes it hard to tell whether the numbers are a lucky draw. The paper's own Appendix E is honest about the weak correlation, which helps, but it also means the mechanism's efficacy is partly empirical and not well explained.\n\nI would send this to a serious referee. The idea is worth engaging with and the experimental breadth justifies reviewer time. The revision should either soften the compression claim or add seed/error-bar analysis, and ideally release the code. That is a moderate revision, not a rejection.","headline":"A genuinely new pre-attention eviction trick that deserves review, but the paper's own tables undercut the '30-70% compression' headline at the high end.","tokens_in":19863,"tokens_out":3451,"would_cite":false,"duration_ms":35520,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that HashEvict can shrink an LLM's KV cache by 30-70% while preserving task quality, by evicting the cached token whose binarized key is most Hamming-distant from the current query.","keywords":["KV cache compression","locality-sensitive hashing","token eviction","efficient LLM inference","Hamming distance","SimHash","pre-attention eviction","long-context understanding"],"falsifier":"Run HashEvict at a 30% cache budget on GSM8K with many random seeds for the projection matrix R, and also measure the attention mass of the tokens it evicts per head; if task scores show large seed-to-seed variance, or if evicted tokens carry as much attention as tokens chosen at random, then the Hamming ranking is not the mechanism behind the reported quality.","tokens_in":18802,"feed_emoji":"🧠","tokens_out":9866,"duration_ms":88809,"temperature":0.7,"pith_summary":"HashEvict asks whether a large language model's key-value cache can be compressed by deciding what to evict before attention is computed, using only cheap binary comparisons. The algorithm binarizes each cached key and the current query with locality-sensitive hashing, then evicts the cached token whose hash code is farthest in Hamming distance from the query hash, on the theory that Hamming distance tracks cosine dissimilarity and hence low attention. Across reasoning, multiple-choice, long-context retrieval, and long-document summarization, the authors report that 30%-70% cache compression keeps task quality close to the full cache, with a 1.5-2x prefill speedup over attention-accumulation baselines. A sympathetic reader would care because the eviction policy adds only a tiny binary hash table to GPU memory and makes no use of attention scores, which could make longer context windows and on-device inference cheaper.","feed_headline":"HashEvict cuts LLM KV cache 30-70% without computing attention","feed_subtitle":"Binary hashes pick eviction victims pre-attention, keeping quality on reasoning and retrieval while saving GPU memory.","key_machinery":"The machinery is the SimHash projection, which maps a d-dimensional key or query vector to a c-bit binary code by h(x) = sgn(Rx), where R is a c by d matrix of independent standard normal entries and sgn is the elementwise step at zero. The theoretical identity used is that for unit vectors x and y the expected Hamming distance between h(x) and h(y) equals the angle between x and y divided by pi, so Hamming distance approximates cosine distance. HashEvict keeps a binary hash table of the cached keys and, each decoding step, selects the index with the largest Hamming distance to the current query's hash as the eviction victim, then inserts the current token's key, value, and hash in its place. The hash table costs c bits per cached token and is stored on GPU, which is the entire extra memory overhead.","core_discovery":"HashEvict's central claim is that an attention-free eviction rule based on Hamming distances between binary hash codes can match or beat both attention-free and attention-accumulation compressors. At every decoding step it replaces the key and value of the cached token with the lowest LSH score, defined as the negative Hamming distance between the hash of the query and the hash of each cached key, thereby evicting the token estimated to be most cosine-dissimilar and least likely to receive attention. The paper argues that because the expectation of this Hamming distance is proportional to the angle between two unit vectors, low Hamming distance is a proxy for high dot-product attention, and it reports that the resulting evictions preserve medium-to-high attention tokens rather than only the highest, which the authors say explains the method's strength on free-response reasoning and summarization. The headline empirical result is 30%-70% KV cache compression with maintained performance over reasoning, multiple-choice, long-context retrieval, and summarization, plus faster prefill than attention-based baselines.","pith_inferences":["A natural extension is to use two or three independent hash projections instead of one; the theoretical identity is an expectation, and the paper's own appendix shows correlation with attention rises with projection length, so ensembling could stabilize eviction for little extra memory.","Since the method keeps a spread of medium-to-high attention tokens, a hybrid that reserves a small protected slot for tokens with accumulating high attention could close the remaining gap at very small cache budgets without paying full attention-accumulation cost.","The reported layer-wise correlation drop in later layers suggests an adaptive policy could increase hash dimension or switch to L2-norm eviction in those layers, which would be a cheap, testable improvement over a single global hash configuration."],"forward_implications":["The KV cache can be cut to 30%-70% of its full size on Llama-3-class models with only a small quality drop on reasoning, multiple-choice, long-context retrieval, and summarization tasks.","Eviction decisions become pre-attention and per-step, so the method avoids accumulating attention scores and is roughly 1.5-2x faster at prefill than H2O and Scissorhands and about 17x faster than FastGen, with competitive decoding speed.","The added GPU memory is only a c-bit hash per cached token; an 8-bit hash dimension works as well as 64-bit in the reported ablation, so the overhead stays near one byte per cache slot.","At a 50% cache budget, the attention mass lost by HashEvict's evictions is lower than the L2-norm method and comparable to Scissorhands, indicating the evicted tokens are genuinely low-attention."],"supporting_citations":[{"why":"Supplies the SimHash family that binarizes embeddings into c-bit codes.","marker":"Charikar, 2002"},{"why":"Provides the theorem that expected Hamming distance between hash codes equals the angle between unit vectors over pi, the theoretical basis of the eviction score.","marker":"Goemans and Williamson, 1995"},{"why":"The attention-free L2-norm eviction baseline that HashEvict must outperform.","marker":"Devoto et al., 2024"},{"why":"H2O, the attention-accumulation heavy-hitter baseline used for comparison.","marker":"Zhang et al., 2024a"},{"why":"Scissorhands, the persistence-of-importance attention baseline used for comparison.","marker":"Liu et al., 2024a"},{"why":"Supplies the long-context retrieval tasks, Needle-in-a-Haystack and Common Words, and their string-match metric.","marker":"Hsieh et al., 2024"},{"why":"Supplies the LongBench summarization tasks MultiNews and GovReport and Rouge-based evaluation.","marker":"Bai et al., 2023"},{"why":"Supplies GSM8K, the grade-school math reasoning dataset used in free-response and multiple-choice tests.","marker":"Cobbe et al., 2021"},{"why":"Supplies MedQA, the medical exam dataset used in free-response and multiple-choice tests.","marker":"Jin et al., 2021"}],"fun_headline_variants":["Pre-attention hashing evicts KV cache, saves 30-70% GPU memory","LSH eviction trims KV cache 30-70% before attention","Binary hashes pick cache victims pre-attention, cut 30-70%","Attention-free eviction cuts KV cache 30-70% via LSH","HashEvict uses LSH to shrink KV cache 30-70% pre-attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one random binarized projection of unnormalized keys and queries still ranks cached tokens by true cosine similarity, whereas the theory guarantees this only in expectation over many projections of unit vectors, so a single projection can mis-rank and evict a high-attention token.","fun_headline_variants_meta":{"raw":{"variants":["Pre-attention hashing evicts KV cache, saves 30-70% GPU memory","LSH eviction trims KV cache 30-70% before attention","Binary hashes pick cache victims pre-attention, cut 30-70%","Attention-free eviction cuts KV cache 30-70% via LSH","HashEvict uses LSH to shrink KV cache 30-70% pre-attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000617,"raw_usage":{"total_tokens":2880,"prompt_tokens":974,"completion_tokens":1906,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":1797}},"tokens_in":590,"tokens_out":1906,"duration_ms":14136,"temperature":1.0,"reasoning_tokens":1797,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:37:26.875280+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HashEvict at a 30% cache budget on GSM8K with many random seeds for the projection matrix R, and also measure the attention mass of the tokens it evicts per head; if task scores show large seed-to-seed variance, or if evicted tokens carry as much attention as tokens chosen at random, then the Hamming ranking is not the mechanism behind the reported quality.","supporting_citations":[{"cited_title":"Similarity estimation techniques from rounding algorithms","cited_arxiv_id":null,"evidence_quote":"Supplies the SimHash family that binarizes embeddings into c-bit codes."}],"review_version":1}