{"id":"5eed04e6-763f-4b95-b652-be04368c703b","arxiv_id":"2608.01631","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"KV cache compression can preserve final-answer accuracy while degrading the validity of reasoning chains that support those answers, creating an answer-evidence gap.","lead":"Large reasoning models can keep producing correct final answers under memory compression while the visible reasoning that supports those answers becomes invalid or fragile. This paper argues that judging compression quality by final-answer accuracy alone overstates competence and adds evidence-aware metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RWAC gap may be inflated if the LLM judge penalizes shorter compressed rationales; chain-support half of the claim needs a human-label check.","rationale":"The reader's weakest assumption identifies the LLM judge as the main point of vulnerability, and I agree. The paper's headline specifically claims that the validity of visible supporting rationales can diverge from accuracy. That validity is operationalized through RWAC, which is entirely judge-dependent. The human validation demonstrates inter-rater reliability, but it does not directly validate the method-specific gap: we do not know if the gap between Full-KV and compressed RWAC would shrink appreciably with human labels. The no-think baseline provides some evidence against a simple brevity bias, but compressed rationales are a different kind of text: they are attempted derivations with missing steps, rather than direct answers without reasoning. The perturbation-fidelity metric is objective and shows a robustness gap, but it does not measure rationale validity, so the 'chain support' component of the abstract remains judge-only. This concern does not overturn the qualitative conclusion, because the perturbation gap and the matched-answer analysis suggest a real phenomenon, but it does mean the specific numerical RWAC values and the strength of the 'chain support' claim should be treated as conditional on judge calibration. The proposed human-label recomputation would test whether the concern materially affects the reported gap. Since the reader already marked the paper CONDITIONAL and this concern is a refinement of that conditional assessment, the verdict is unchanged.","tokens_in":22137,"tokens_out":17862,"duration_ms":209205,"concrete_test":"Re-label the existing 400 stratified outputs (or a new balanced sample of Full-KV and compressed outputs) with human annotators using the same binary criterion, and compute RWAC/correct (and the Full-KV vs. compressed gap) using human labels instead of LLM labels. If the gap shrinks by more than 20% relative to LLM-judged values, the judge bias is material and the reported RWAC numbers should be re-reported with a correction or a bounding analysis. Additionally, perform a length-controlled comparison: bin outputs by number of rationale tokens and final correctness, and test whether compressed outputs are more likely to be judged 'not fully correct' than Full-KV outputs in the same bin.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that correct answers and supporting-rationale validity are preserved at different rates rests on the RWAC metric, which is LLM-judge-based (Section 3.1). Human agreement is high (κ=0.89, Appendix B), but this does not establish that the observed Full-KV vs. compressed gap is free of systematic judge bias. Compressed continuations are typically shorter and more fragmentary; if the judge labels omitted steps as 'not fully correct,' RWAC/correct for compressed methods is inflated. The no-think baseline (Table 3) shows the judge does not indiscriminately penalize brevity, but no-think outputs differ qualitatively from compressed sparse derivations. The objective perturbation-fidelity metric (Section 3.2) shows a real robustness gap, but it measures answer stability under injected claims, not the validity of the visible rationale, so it cannot independently support the 'chain support' half of the headline claim. Thus the quantitative magnitude of the answer-evidence gap for chain validity is only as strong as the judge's calibration against human judgments on compressed outputs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether KV cache compression methods that preserve final-answer accuracy also preserve the validity of the visible reasoning chains that support those answers. Using a fixed-trace replay protocol, the authors hold the reasoning prefix fixed and compress only its KV states, then measure three metrics: final-answer accuracy, answer–chain consistency (via an LLM judge with human validation), and perturbation-based faithfulness. They evaluate ten token-eviction methods and one quantization method (KIVI) on Qwen3-8B, DeepSeek-R1-Distill-Llama-8B, and Qwen3-30B-A3B across AIME, GPQA-Diamond, MedCalc, and RULER. The main empirical claim is an 'answer–evidence gap': token-eviction compression can preserve competitive final-answer accuracy while substantially increasing the rate of wrong-chain-correct answers and reducing perturbation fidelity, whereas KIVI is less affected. The paper argues that final-answer accuracy is an asymmetric diagnostic and that evidence-aware metrics should complement accuracy-only evaluation.","tokens_in":22394,"tokens_out":9767,"duration_ms":116985,"significance":"If the finding holds, it is an important contribution to the evaluation of KV cache compression for reasoning models: it identifies a failure mode that accuracy-only benchmarks miss and introduces a reusable fixed-trace protocol with matched-answer analysis. The study is unusually broad in coverage (11 methods, 3 models, 4 tasks, budget ablations) and includes a human-validated LLM judge (κ≈0.89 on 400 stratified outputs) as well as a second judge and second annotator. The paper also explicitly avoids over-strong mechanistic claims. The main risks are the judge-dependence of the chain-validity metric and some overstated interpretations of the quantization control, but these are addressable and do not undermine the core empirical pattern.","major_comments":[{"comment":"The fixed-trace protocol prefills q∥<think>r</think>, compresses the cache for r, and resumes after </think>. Since r is textually present, it is unclear what the LLM judge sees: the full output including r, or only the newly generated post-</think> continuation. If the judge sees r and r is correct, a correct final answer should almost always yield a correct chain, yet Table 3 shows large RWAC/correct increases for compressed methods (e.g., SnapKV 65.5 vs Full-KV 4.3 on AIME). Please specify the judge input and provide one concrete matched-answer example from Table 5 where the chain is judged not fully correct despite an identical correct r. Without this, the protocol's interpretation as 'same evidence, less usable support' is under-specified.","section":"§2, §3.1"},{"comment":"The RWAC gap depends on the LLM judge's classification of compressed outputs. Overall human–judge κ=0.89 is strong, but it does not rule out a systematic bias toward labeling shorter, more fragmentary compressed continuations as 'not fully correct'. The no-think baseline in Table 3 shows the judge does not penalize brevity indiscriminately, but no-think outputs are not compressed sparse derivations. Please report human–judge agreement separately for compressed vs Full-KV outputs and by continuation length, and state the direction of disagreements (e.g., whether the judge is stricter than humans on compressed outputs). This directly bears on the quantitative magnitude of the headline gap.","section":"§3.1, Appendix B"},{"comment":"The text says KIVI 'remains close to Full-KV across all metrics', but Table 3 does not fully support this. On GPQA-Diamond, KIVI-2bit has RWAC/correct=55.5 vs Full-KV 42.3, a 13-point degradation; on RULER, final accuracy is 64.3 vs Full-KV 78.9, a 14.6-point drop. KIVI is indeed less affected than the eviction methods, but it is not close to Full-KV across all metrics. Please revise the wording and report deltas explicitly; this tempers the mechanistic conclusion that coverage loss, rather than memory reduction itself, causes the answer–evidence gap.","section":"§4.1, Table 3"}],"minor_comments":[{"comment":"The claim that accuracy and reasoning quality 'move in opposite directions' on AIME/GPQA-Diamond is too strong for GPQA, where Spearman ρ=-0.20 is weak. Also, n=11 methods gives wide uncertainty; report p-values or confidence intervals, and qualify the GPQA interpretation.","section":"§4.3, Table 4"},{"comment":"The text says RULER is a case where 'accuracy itself collapses', but SnapKV and HeadKV retain accuracy 59.5 and 61.2 vs Full-KV 78.9. This is a substantial drop but not a collapse. Suggest softening the wording and clarifying the threshold used.","section":"§4.2, RULER"},{"comment":"For HeadKV and SnapKV end-to-end, RWAC/correct=0.0 is driven by a very small number of correct answers. The text explains this, but adding the correct-answer counts (denominators) in the table would prevent misinterpretation.","section":"Appendix A.3.2, Table 10"},{"comment":"Figures 2 and 3 are referenced frequently but lack detailed axis labels and definitions of 'chain consistency' and 'fidelity' in the captions. Adding a short legend describing the plotted quantities would improve readability. Minor typos: 'answer–evidencegap' in the abstract and 'tied less to' phrasing should be checked.","section":"Global"}],"recommendation":"minor_revision","confidential_remarks":"The central claim is empirically supported and the paper is a solid contribution. The main issues are overstatements and a need for more protocol transparency rather than fundamental flaws. I see no circularity or parameter-fitting problems. The paper should be publishable after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Should know: this is the first evaluation of KV cache compression through a CoT faithfulness lens, and the main result—the answer–evidence gap—looks real. With ten eviction methods plus a quantization control across three models and four tasks, they show that accuracy can survive compression while chain validity or perturbation robustness degrades. The fixed-trace replay protocol is a good idea: it holds the reasoning content fixed and isolates whether a compressed KV state preserves usable information from an already available trace. That's a cleaner component-level diagnostic than end-to-end comparison.\n\nWhat's well done: the RWAC metric is carefully validated. Human–Claude agreement is κ=0.89 on 400 stratified outputs, and cross-judge agreement with Gemini is nearly as high. The matched-answer analysis (Table 5) strengthens the causal reading: conditioning on Full-KV and the compressed method both producing the same correct answer, eviction methods degrade the chain far more often than the quantization control. The no-think baseline is a nice check that the judge isn't just penalizing brevity. The budget ablation in Section 4.5 shows the gap persists even as accuracy recovers. The paper is also honest about limitations: it frames the result as behavioral, not mechanistic, and explicitly acknowledges RWAC is judge-dependent.\n\nSoft spots, in proportion. The stress-test worry about judge bias is real but not fatal. High human agreement per label does not guarantee the gap magnitude is free of systematic bias toward shorter compressed continuations. The no-think baseline mitigates but doesn't fully settle it. I'd want a per-bucket judge calibration on compressed vs. Full-KV outputs, ideally human-labeled and matched on output length. That's a revision-level request, not a rejection-level flaw. Minor: the claim that KIVI stays 'close to Full-KV' across metrics is overstated on RULER, where accuracy drops from 78.9 to 64.3. Easy fix. Also, the perturbation-fidelity metric measures answer stability under injected claims, not validity of the visible rationale, so the chain-support half of the headline rests on the judge; that's fine given the validation, but it's worth stating explicitly.\n\nFor whom: people building or evaluating KV compression for reasoning models, and anyone who uses accuracy-only leaderboards in high-stakes settings. The citation pattern is fine; no sign of missing or self-serving citations. The paper deserves a serious referee. I'd send it to review, and I'd cite it. The main argument holds up; the weaknesses are an overclaim and a residual judge-calibration check.","headline":"First real test of whether KV compression preserves reasoning evidence, not just answers; the answer–evidence gap is a genuine finding that deserves peer review, with one overstated side claim and a residual judge-calibration caveat.","tokens_in":22827,"tokens_out":2536,"would_cite":true,"duration_ms":29581,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KV cache compression can keep final answers correct while degrading the reasoning evidence that supports them.","keywords":["KV cache compression","reasoning faithfulness","chain-of-thought validity","answer-evidence gap","token eviction","LLM-as-a-judge","fixed-trace replay","large reasoning models"],"falsifier":"Re-run the fixed-trace protocol with the reasoning judge blinded to compression status and with all rationales normalized to a uniform style and length, or judged by independent human experts on the same rubric. If the RWAC/correct differences between eviction methods and Full-KV collapse to near zero, the answer–evidence gap is a judge artifact rather than a genuine divergence between answer preservation and evidence preservation.","tokens_in":22063,"feed_emoji":"🧠","tokens_out":11500,"duration_ms":104317,"temperature":0.7,"pith_summary":"The paper tests a common assumption: that a compressed AI model that still gives the right answer has also preserved the reasoning behind that answer. Using a controlled fixed-trace replay protocol, it shows this assumption fails for large reasoning models under token-eviction KV cache compression: ten eviction methods kept competitive final-answer accuracy on math, science, and clinical-calculation tasks while substantially degrading the validity of the visible rationale or its robustness to injected errors. The authors call this divergence the answer–evidence gap. A quantization control that keeps full token coverage was far less affected, suggesting the damage comes from losing access to parts of the reasoning trace, not from memory reduction itself. The stakes are practical: in high-stakes settings, a correct-looking but unsupported answer can be more dangerous than a visibly wrong one, and accuracy-only evaluation would certify it.","feed_headline":"AI cache compression can keep answers right but wreck reasoning","feed_subtitle":"Eviction methods keep final accuracy while chain validity collapses; accuracy-only benchmarks mislead.","key_machinery":"The load-bearing mechanism is the controlled fixed-trace replay protocol: the uncompressed model generates a complete reasoning trace; each compression method replays the same trace with its own compressed KV cache, so all methods observe identical textual reasoning and differences isolate what the compressed representation retains. It is paired with three metrics: final-answer accuracy; answer–chain consistency, using a language-model judge to label wrong-chain correct answers (RWAC) where the answer is right but the visible chain is not fully correct; and perturbation faithfulness, measuring fidelity and bias rate after injecting a wrong answer at start, middle, or end of the trace. A matc","core_discovery":"The paper's central finding is an answer–evidence gap: under KV cache compression, the rate at which a model preserves the correct final answer and the rate at which it preserves valid evidence for that answer can diverge sharply. On AIME, GPQA-Diamond, and MedCalc-Bench, token-eviction compressors such as SnapKV and AdaKV at a 256-token budget retained much of Full-KV accuracy while wrong-chain correct answers (RWAC/correct) rose from single digits to over 60 percent on AIME and perturbation fidelity dropped by tens of points. Replaying identical reasoning traces isolates retention from rollout variation. Accuracy is an asymmetric diagnostic: collapse reveals damage, but preserved accuracy","pith_inferences":["The same divergence likely affects any downstream system that consumes compressed-model rationales for auditing or trust scoring; one testable extension is whether human readers make more wrong trust decisions when given compressed rationales than accuracy numbers predict.","A compressor designed to preserve dependency and provenance structure — protecting definitions, intermediate computations, and verification steps over answer-like conclusions — could be tested directly for whether it closes RWAC while keeping accuracy.","The fixed-trace replay protocol could double as a training or selection signal: compressors could be chosen by replaying teacher traces and optimizing chain validity, not just final-answer match.","Because judge-based RWAC may be sensitive to verbosity, a further test is to separate surface brevity from evidence presence by measuring the gap under controlled rationale-length normalization."],"forward_implications":["Accuracy-only leaderboards for KV compression can select methods that systematically produce correct-looking but unsupported answers; on AIME and GPQA-Diamond, accuracy and chain validity rank compressors in opposite directions.","More cache budget recovers final answers before it recovers evidence: chain validity remains degraded at all tested budgets on AIME, while fidelity recovers gradually.","Coverage-preserving quantization (KIVI-2bit) stays close to Full-KV across metrics, so the gap is tied to token eviction rather than KV memory reduction itself.","The gap is task-dependent: on answer-recoverable reasoning tasks it hides behind accuracy, while on evidence-constrained retrieval (RULER QA) compression damage shows up as accuracy collapse.","Deployment-style compressed generation also shows fragility in end-to-end checks, so the answer–evidence gap is not an artifact of the replay protocol."],"supporting_citations":[{"why":"Supplies the perturbation-based faithfulness measurement approach the paper adapts to compressed reasoning traces.","marker":"Lanham et al., 2023"},{"why":"Prior demonstration that chain-of-thought explanations can be unfaithful, motivating the concern that preserved answers may hide broken evidence.","marker":"Turpin et al., 2023"},{"why":"Background claim that chain-of-thought is not explainability, against which the answer–evidence gap is framed.","marker":"Barez et al., 2025"},{"why":"Provides the unlearning-style intervention method underlying the perturbation faithfulness probe.","marker":"Tutek et al., 2025"},{"why":"SnapKV is a representative attention-based eviction method whose accuracy/chain divergence instantiates the gap.","marker":"Li et al., 2024"},{"why":"StreamingLLM's sink/window retention is the source of the early-anchor perturbation failures.","marker":"Xiao et al., 2024"},{"why":"KIVI quantization serves as the coverage-preserving control showing the gap is tied to eviction rather than memory reduction.","marker":"Liu et al., 2024a"},{"why":"RULER QA supplies the evidence-constrained retrieval condition where accuracy collapse is the failure signature.","marker":"Hsieh et al., 2024"},{"why":"GPQA-Diamond supplies the multiple-choice scientific reasoning setting where correct guesses can hide unsupported rationales.","marker":"Rein et al., 2024"},{"why":"MedCalc-Bench supplies the clinical-calculation setting where robustness to injected errors is safety-relevant.","marker":"Khandekar et al., 2024"}],"fun_headline_variants":["Compression keeps answers right but unravels reasoning chains","Accuracy alone hides broken reasoning under cache compression","Cache eviction preserves answers while evidence collapses","Right answers, wrong rationale: cache compression's hidden gap","Correct answer, broken chain: KV compression's evidence gap"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The main results rest on a language-model judge's binary verdict that a reasoning chain is 'not fully correct'; if the judge systematically treats compressed, terser rationales as unsupported regardless of their actual validity, the answer–evidence gap would be overstated.","fun_headline_variants_meta":{"raw":{"variants":["Compression keeps answers right but unravels reasoning chains","Accuracy alone hides broken reasoning under cache compression","Cache eviction preserves answers while evidence collapses","Right answers, wrong rationale: cache compression's hidden gap","Correct answer, broken chain: KV compression's evidence gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00059,"raw_usage":{"total_tokens":2590,"prompt_tokens":718,"completion_tokens":1872,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":1798}},"tokens_in":462,"tokens_out":1872,"duration_ms":12839,"temperature":1.0,"reasoning_tokens":1798,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:50:19.822563+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the fixed-trace protocol with the reasoning judge blinded to compression status and with all rationales normalized to a uniform style and length, or judged by independent human experts on the same rubric. If the RWAC/correct differences between eviction methods and Full-KV collapse to near zero, the answer–evidence gap is a judge artifact rather than a genuine divergence between answer preservation and evidence preservation.","supporting_citations":[],"review_version":1}