{"id":"b08e8792-cc84-4f93-934d-0831f717d204","arxiv_id":"2608.09412","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"KVDiagnosis is a diagnostic benchmark that pairs 59,800 compressed runs with FullCache controls and attaches cache, likelihood, attention, and decoding measurements to 12,520 failure rows, finding 63.2% of failures have low or partial evidence coverage and demonstrating selective repair of…","lead":"KV-cache compression failures in long-context language models are diagnosed with a new benchmark that pairs every compressed run to a full-cache control and attaches cache, likelihood, attention, and decoding measurements. On Qwen3-8B, 63.2% of 12,520 failures show low or partial evidence coverage, and all ten diagnostics separate failures from successes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evidence-span alignment is the load-bearing assumption; the paper records alignment success but reports no quantitative alignment validation, so the released benchmark should demonstrate span correctness before the headline percentages are taken as verified.","rationale":"The reader identified evidence alignment as the weakest assumption, and I agree. I considered two other candidates: the AUROC comparison could be confounded by compression setting, but the text says 'stratified' and 'mixed strata,' suggesting within-stratum comparison, and the C-to-C ledger is a matched control; the repair intervention is oracle-like, but the sham and C-to-C controls make it a legitimate validation of the access signature. The alignment issue is the one that, if wrong, invalidates the diagnostic denominations themselves. The paper handles N/A correctly but provides no evidence that the non-N/A alignments are correct; for a benchmark whose contribution is a reusable diagnostic resource, this is a release-blocking validation gap. I therefore recommend CONDITIONAL rather than unconditional ACCEPT: the headline numbers should be trusted after an independent alignment audit or a released alignment verification report, not before.","tokens_in":24752,"tokens_out":12896,"duration_ms":144049,"concrete_test":"Download the released prompt strings and evidence metadata; independently rebuild E_i token sets for all 400 Qasper/HotpotQA sources and 200 randomly sampled RULER sources using a separate character-offset-to-token aligner with the same Qwen3 tokenizer, resolving duplicates by exact span identity. Compare against the released E_i sets; then recompute ECov/ERR/EAR, the 63.2% low/partial coverage share, the QA category shares, and the 29.2%-vs-6.3% repair difference using only agreed-upon spans. Pre-register a tolerance (e.g., disagreement in more than 1% of sources, or a shift of more than 1 percentage point in the headline shares, or loss of McNemar significance at p < 0.05) as the criterion for the concern landing.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"All diagnostic quantities central to the paper's contribution—the 63.2% low/partial coverage share, per-method coverage profiles, the effectiveness of the low-EAR repair (29.2% vs 6.3%), and the coverage/attention AUROCs—are computed with respect to evidence spans E_i. Section 3.2 states that Qasper/HotpotQA support text and RULER generator spans are aligned after final-prompt tokenization and that alignment success is recorded, but the paper gives no alignment accuracy numbers, no per-source alignment concordance, and no description of how duplicate or near-duplicate support sentences in HotpotQA are resolved. Ambiguous mappings are relegated to N/A, so the reported rows are exactly the ones where alignment is asserted to be correct; without an independent check, a systematic tokenization or span-mapping error would silently miscalibrate ERR, ECov, EAM, and EAR. RULER rows dominate the corpus (11,366 of 12,520 C-to-W rows) and drive the repair cohort, so the risk is concentrated in the QA subsets (1,154 rows) and in the general claim that the resource provides reusable evidence-aligned diagnostics. The paper's N/A semantics handle missing spans but not incorrect ones.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces KVDiagnosis, a paired diagnostic benchmark for KV-cache compression failures. It fixes per-source FullCache controls, runs all supported method-setting cells over fixed source splits, selects FullCache-correct-to-compressed-wrong (C-to-W) rows separately for each method-setting, and attaches cache-retention, likelihood, attention, and decoding diagnostics with explicit N/A semantics. On Qwen3-8B over RULER-8K, RULER-16K, Qasper, and HotpotQA, it reports 59,800 supported compressed runs and 12,520 C-to-W rows; under predeclared diagnostic rules, 63.2% of failure rows show low or partial measured/projected coverage. Ten diagnostics separate C-to-W rows from a C-to-C control ledger (stratified AUROC 0.684-0.871), and a controlled 4x evidence-attention boost repairs 29.2% of 96 reproducible low-EAR failures versus 6.3% under a count-matched sham, with a matched C-to-C safety check. Cross-model results on Falcon3-7B and Mistral-Small-24B show that coverage trends transfer across architectures while quantization effects remain model-specific.","tokens_in":24903,"tokens_out":8012,"duration_ms":83867,"significance":"KVDiagnosis is potentially a valuable community resource. Its design choices address real gaps in compression evaluation: matched per-source FullCache controls, method-specific failure sets selected after the full run matrix is complete, explicit applicability states instead of silent imputation, a C-to-C control ledger, predeclared diagnostic rules, and a sham-controlled repair experiment with paired bootstrap and McNemar statistics. The PyramidKV adapter audit and the detailed run accounting in Tables 3 and 10 are unusually careful, and the released artifacts appear reproducible. If the evidence-span alignments are correct, the headline failure profiles, the AUROC separation, and the low-EAR repair result are well supported. The main open risk is that ERR, ECov, EAM, EAR, the diagnostic categories in Table 4, and the repair cohort all treat the evidence spans as ground truth, and the paper has not yet provided quantitative validation of those alignments.","major_comments":[{"comment":"The evidence-span alignment is the load-bearing ground truth for ERR, ECov, EAM, EAR, the diagnostic categories in Table 4, and the low-EAR repair cohort, yet the paper only states that Qasper/HotpotQA support text and RULER generator spans are aligned after final-prompt tokenization and that alignment success is recorded. It reports no per-source alignment concordance, no manual audit size or error rate, and no description of how duplicate or near-duplicate support sentences in HotpotQA are resolved. Because ambiguous mappings are relegated to N/A, the reported rows are exactly those asserted to be aligned, so a systematic tokenization or mapping error would silently miscalibrate every evidence-based diagnostic. Please add quantitative alignment validation, including exact-span match rates, position-offset distributions, a manual audit of a stratified sample, and released alignment maps, and describe the duplicate-resolution rule for HotpotQA.","section":"Section 3.2 and Appendix D"},{"comment":"The headline '63.2% low or partial measured/projected coverage' and the category counts in Table 4 are determined by hand-set cutoffs, specifically the ECov thresholds of 0.5 and 0.9, the Delta-NLL threshold of 1, and the EAR threshold of 0.5. The rules are predeclared, which rules out post hoc selection, but the numerical claims are still a direct function of these constants. Please report a sensitivity sweep over the cutoffs or provide a distribution-based justification, and state whether the main qualitative findings, namely that coverage loss dominates and that low-EAR repair is selective, are stable across reasonable threshold choices.","section":"Section 4 and Table 4"},{"comment":"The C-to-C control ledger is central to the claim that all ten diagnostics separate failures from successful compression, but the paper does not specify how the 4,936 C-to-C runs were selected or stratified beyond the phrase 'mixed strata,' and sources recur across rows. Since repeated sources can inflate apparent separation, please report the number of unique sources per stratum and use a source-clustered bootstrap or equivalent procedure for the AUROC intervals, and state explicitly whether the C-to-C controls are matched to the C-to-W rows on method, setting, and source.","section":"Section 5.3 and Figure 6"}],"minor_comments":[{"comment":"The abstract says 59,800 supported compressed runs, while Table 3 records 62,400 total records with 2,600 N/A runs; please state the total record count and the exclusion rule in the caption or a footnote so readers can reconcile the counts at a glance.","section":"Abstract and Table 3"},{"comment":"The '75/50/25' setting tier applies to retention and channel methods, while QuantizedCache uses 8b/4b/2b; the caption says this, but the shared axis labels in panels A and B could mislead readers, so consider adding a second axis label or a footnote.","section":"Figure 3 caption"},{"comment":"The 4x evidence boost is an oracle intervention that uses gold evidence spans; the word 'repairs' in the abstract and findings could be misread as a deployable repair method, so please add an explicit qualifier that the intervention is a controlled diagnostic probe, not a proposed inference-time method.","section":"Section 5.3"},{"comment":"The attention-based diagnostics and the intervention are Qwen-only, and this limitation is stated in Section 5.6, but Finding 4 should remind readers in the main text that cross-model validation covers coverage and likelihood diagnostics only, not attention or EAR-based repair.","section":"Section 5.5 and Section 5.6"},{"comment":"The PyramidKV adapter audit is well handled; consider moving one sentence about the exclusion into Section 5.1 so that main-text readers know the budget-allocation family is represented by AdaKV alone in the evaluated set.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the venue and the empirical design is unusually thorough for a benchmark paper. The requested evidence-span alignment validation is achievable within the manuscript's scope and should be the deciding factor: the central diagnostic quantities are all defined relative to the evidence spans, and without quantitative alignment support the released resource cannot be fully trusted. No concerns about citation patterns or novelty disclosure beyond what is stated in the report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the real thing, not a glossy benchmark. It ships the full paired run matrix with per-source FullCache controls, selects C-to-W rows separately per method-setting after the matrix is complete, and attaches diagnostics with explicit applicability states. The release includes versioned files, checksums, and an adapter audit (the PyramidKV exclusion is handled honestly). That alone puts it ahead of most benchmark papers.\n\nThe empirical claims are better supported than typical for the area. The C-to-C control ledger gives 4,936 successful runs; all ten diagnostics separate failures from successes with stratified AUROC 0.684-0.871. The low-EAR repair finding is the centerpiece: 29.2% repair with a 4x evidence-attention boost against 6.3% with a count-matched sham, with a paired bootstrap CI and McNemar p ~3e-6. That design is careful; matched C-to-C degradation of 3.3% is the right safety control.\n\nSoft spots, in proportion. The load-bearing assumption is evidence-span alignment for Qasper, HotpotQA, and RULER generator spans. The paper records alignment success and marks failures N/A, but it gives no alignment accuracy numbers, no concordance per source, and no description of how near-duplicate support sentences are resolved. A systematic span-mapping error would silently miscalibrate ERR, ECov, EAM, and EAR. The stress-test note is right about this. It is not fatal though: the QA subsets are small (200 each, only 59 and 148 FullCache-correct sources), so the pooled 63.2% coverage figure is RULER-dominated, where generator-derived spans are exact. And the repair cohort appears to be RULER-only because QA rows have no valid eager-attention traces, so the 29.2% vs 6.3% finding rides on RULER's deterministic spans, not on QA alignment. The hand-chosen thresholds (ECov 0.5, Delta-NLL 1.0, etc.) are predeclared but still arbitrary; that is a minor circularity, not a flaw. The QA scorer is a non-official containment metric, which is fine for a benchmark but should be read as such.\n\nI couldn't run the code or inspect the GitHub contents, so correctness risk is medium.\n\nWho this is for: anyone doing KV-cache compression evaluation or failure analysis. It gives the community something it didn't have: a reusable, method-specific failure corpus with paired controls and traces.\n\nRecommendation: yes, send to peer review. The requested revisions should be concrete: publish per-source alignment concordance and duplicate-resolution rules, add CIs on QA-level percentages, and pin a public commit hash. If they do that, the 63.2% and 29.2% numbers become something I'd trust and cite.","headline":"A genuinely useful diagnostic resource with a sound matched-control design; the headline numbers should be trusted only after the evidence-span alignment is made auditable, and I'd send it to review.","tokens_in":25541,"tokens_out":2993,"would_cite":true,"duration_ms":31053,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KVDiagnosis pairs every compressed run with a FullCache control, then attributes KV-cache compression failures to coverage, drift, attention, or decoding causes.","keywords":["KV-cache compression","long-context language models","diagnostic benchmark","evidence coverage","attention retention","failure analysis","FullCache control","likelihood drift"],"falsifier":"Independently re-annotate the evidence token positions in the final tokenized prompts for a random sample of the 12,520 C-to-W rows and recompute ERR, ECov, and EAR: if the remapped spans shift by more than a few tokens, the category shares (63.2% low or partial coverage) and the 29.2%-versus-6.3% low-EAR repair gap should shrink or disappear. A cheaper variant is to rerun the benchmark with evidence spans deliberately offset by several tokens and check whether the AUROC separations degrade.","tokens_in":24450,"feed_emoji":"🧠","tokens_out":9271,"duration_ms":90688,"temperature":0.7,"pith_summary":"KVDiagnosis is a benchmark and released dataset for diagnosing why KV-cache compression turns correct answers into wrong ones. Its central claim is that aggregate accuracy hides a method-specific failure anatomy: on Qwen3-8B, 59,800 supported compressed runs over 2,600 sources produced 12,520 runs that were correct under a per-source FullCache control but wrong after compression, and under fixed diagnostic rules 63.2% of those failures show low or partial measured/projected evidence coverage. The paper reports that all ten diagnostics—cache coverage, likelihood drift, attention retention, and decoding traces—separate failed from successful compression (stratified AUROC 0.684–0.871), and that a controlled 4x boost of attention to retained answer-supporting evidence repairs 29.2% of reproducible low-attention-retention failures versus 6.3% under a count-matched sham. If correct, the work provides a reusable format for attributing compression failures to specific mechanisms rather than reading them off task scores.","feed_headline":"63% of KV-cache compression failures trace to lost evidence","feed_subtitle":"A paired benchmark on Qwen3-8B shows attention boosts fix 29% of low-coverage failures vs 6% for sham.","key_machinery":"The load-bearing object is the paired run matrix: FullCache executed once per source, reused by every method-setting cell, with C-to-W failure rows selected only after all supported runs complete. On that matrix the benchmark defines ten diagnostics with explicit applicability states. Cache retention uses ERR (evidence retention rate) and ECov (evidence coverage), averaged over layer-KV-head slots with a threshold of $\\tau=0.5$ so a cross-head union cannot manufacture perfect coverage; likelihood drift uses $\\Delta\\mathrm{NLL}$, GPR, KL, Top-50 overlap, and gold-token rank shift; attention access uses EAM (evidence attention mass), EAR (evidence attention retention), and NEAE (normalized evidence attention enrichment); decoding records gold ranks and emitted output. The taxonomy of 25 methods into five mechanism families tells which diagnostics are valid for which compressed object—positions, head-wise slots, channels, precisions, or chunks—and the paper's validation machinery is the matched C-to-C control group plus the low-EAR intervention with a sham control.","core_discovery":"The core discovery is that a compressed-model answer error is attributable: it shows up as a specific, measurable gap between the compressed run and its FullCache partner. KVDiagnosis constructs that pairing by running FullCache once per source, running every supported method-setting cell over the same fixed split, and only then extracting C-to-W rows—sources correct under FullCache that become wrong under a specific compressor—so no compressor's failure set is defined by another method's failures. Across the four evidence-aware workloads, the 12,520 C-to-W rows separate into repeatedly observed signatures: low or partial measured/projected evidence coverage (63.2% of rows), structural position addressability with likelihood drift (17.0%), rare high-coverage drift (0.2%), low evidence-attention retention candidates (0.8%), and decoding/scoring sensitivity (3.2%). All ten diagnostics separate C-to-W failures from C-to-C successes on matched controls, and the low-EAR cohort responds selectively to an evidence-attention boost. The paper's terminology is careful: structural addressability is recorded separately from measured coverage, because retaining a position index does not by itself establish that the stored representation is intact.","pith_inferences":["Beyond the paper: if evidence coverage is the dominant failure mode, then long-context compressor comparisons should standardize an evidence-retention curve (coverage at each budget) as a reporting table, not only score-vs-budget points.","Beyond the paper: the low-EAR repair result suggests an inference-time controller that monitors EAR during generation and re-weights attention to retained evidence could serve as a runtime recovery mechanism; the paper's design stops at diagnosis, so that policy is an extension.","Beyond the paper: the AUROCs are computed with sources recurring across cells and described by the authors as descriptive; a held-out-sources split would test whether these same features predict failure rank on unseen prompts.","Beyond the paper: the paired C-to-W record format turns failure analysis into a supervised learning problem; one can train a classifier over cache, attention, and likelihood features to predict which method-settings will break a given source before decompression."],"forward_implications":["Two compressors with nearly equal mean scores can fail on almost disjoint source sets: SnapKV and TOVA agree on at most Jaccard 0.385 in 11 of 12 workload-setting cells, so method rankings must not be read as interchangeability.","Evidence coverage, not just score, is the dominant failure signature: 63.2% of C-to-W rows have low or partial measured/projected coverage, so budgets should be reported with evidence-retention curves.","Structural position addressability is not coverage: for ThinK and QuantizedCache, 2,126 rows show strong likelihood drift even though evidence positions remain addressable, which means tensor-fidelity diagnostics must accompany retention claims.","Access failures are repairable in principle: boosting attention to retained evidence positions repaired 29.2% of reproducible low-EAR failures versus 6.3% for a sham boost, with only 3.3% degradation on matched controls, so targeting the access component is a viable repair route.","The diagnostic protocol transfers across the three tested model families for coverage trends, while quantization-related drift remains model-specific; follow-up work should treat the procedure, not the absolute metric values, as the portable part."],"supporting_citations":[{"why":"Supplies the controlled retrieval, tracking, and extraction workloads whose generators provide exact answer-bearing evidence spans for coverage and attention diagnostics.","marker":"[20]"},{"why":"Supplies Qasper document-QA sources with evidence annotations that define the evidence token positions for the QA failure analyses.","marker":"[7]"},{"why":"Supplies HotpotQA multi-hop sources whose supporting-fact spans define the evidence positions and scoring targets.","marker":"[48]"},{"why":"Prior per-sample negative-sample benchmark that motivates the benchmark's FullCache-correct-to-compressed-wrong failure selection.","marker":"[13]"},{"why":"Prior failure-mode study that motivates moving beyond aggregate scores to instruction- and evidence-level diagnostics.","marker":"[5]"},{"why":"Establishes that evidence position in long contexts affects performance, motivating evidence-position-aware retention and attention metrics.","marker":"[30]"},{"why":"Defines Qwen3-8B, the primary model on which all 59,800 supported runs and 12,520 failure rows were produced.","marker":"[40]"},{"why":"Provides the ThinK channel-pruning implementation whose rows populate the structural position addressability category with nonexistent coverage diagnostics.","marker":"[46]"},{"why":"Provides the HQQ QuantizedCache implementation whose tensor-precision rows populate the structural position addressability category.","marker":"[21]"}],"fun_headline_variants":["KV-cache compression: 12,520 paired failures reveal why it fails","Diagnostic benchmark exposes 63% of KV-cache failures as coverage gaps","Attention boost fixes 29% of low-coverage KV-cache failures vs 6% sham","KV-cache compression errors are diagnosable: 63% low coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire diagnostic apparatus treats the evidence-span annotations as ground truth: if the support spans from Qasper, HotpotQA, and the RULER generators are misaligned after final-prompt tokenization, then ERR, ECov, EAM, EAR, and the low-EAR repair cohort are all miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["KV-cache compression: 12,520 paired failures reveal why it fails","Diagnostic benchmark exposes 63% of KV-cache failures as coverage gaps","Attention boost fixes 29% of low-coverage KV-cache failures vs 6% sham","KV-cache compression errors are diagnosable: 63% low coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001082,"raw_usage":{"total_tokens":4610,"prompt_tokens":1116,"completion_tokens":3494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":732,"completion_tokens_details":{"reasoning_tokens":3407}},"tokens_in":732,"tokens_out":3494,"duration_ms":29537,"temperature":1.0,"reasoning_tokens":3407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:44:49.124819+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-annotate the evidence token positions in the final tokenized prompts for a random sample of the 12,520 C-to-W rows and recompute ERR, ECov, and EAR: if the remapped spans shift by more than a few tokens, the category shares (63.2% low or partial coverage) and the 29.2%-versus-6.3% low-EAR repair gap should shrink or disappear. A cheaper variant is to rerun the benchmark with evidence spans deliberately offset by several tokens and check whether the AUROC separations degrade.","supporting_citations":[{"cited_title":"Smith, and Matt Gard- ner","cited_arxiv_id":null,"evidence_quote":"Supplies Qasper document-QA sources with evidence annotations that define the evidence token positions for the QA failure analyses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior per-sample negative-sample benchmark that motivates the benchmark's FullCache-correct-to-compressed-wrong failure selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior failure-mode study that motivates moving beyond aggregate scores to instruction- and evidence-level diagnostics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ThinK channel-pruning implementation whose rows populate the structural position addressability category with nonexistent coverage diagnostics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the HQQ QuantizedCache implementation whose tensor-precision rows populate the structural position addressability category."}],"review_version":1}