{"id":"3c5bd7f3-7099-40de-a330-181f20bb95e5","arxiv_id":"2608.02691","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"OptR learns per-head orthogonal rotations that minimize post-WO attention-output error, lifting INT2 KV-cache quantization accuracy close to BF16 across reasoning, coding, and long-context benchmarks.","lead":"This paper introduces OptR, a calibration method that learns per-head output-aware rotations for INT2 KV-cache quantization, improving long-context LLM accuracy substantially over existing rotation methods at 2.32 bits per element. If it holds, INT2 KV-cache quantization becomes practical for serving long-context models, nearly matching BF16 accuracy while using a quarter of the memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Calibration and held-out rotation selection both use GPQA-Diamond, so the GPQA-Diamond column of Table 1 is test-set tuned; a non-GPQA calibration check is needed before the 'five benchmarks' claim is accepted.","rationale":"The central empirical claim is that OptR consistently improves INT2 KV-cache accuracy across models and benchmarks. The most direct threat to that claim is the calibration protocol rather than the sequential key/value optimization (which is an acknowledged approximation, supported by the Table 4 objective ablation) or the missing code. The Appendix states that both the optimization chunks and the held-out selection chunks come from the GPQA-Diamond evaluation pool. Consequently the GPQA-Diamond column in Table 1 is not an independent test: the rotations and their selection were fitted on that same benchmark. This matters because the paper's headline is 'consistently improves' across five benchmarks; if the GPQA column is set aside, the evidence rests on the remaining benchmarks. Those remaining results are encouraging and make rejection inappropriate, but a cross-domain calibration experiment is the minimal fix that would convert the CONDITIONAL verdict into a firm one. The reader's CONDITIONAL verdict already reflects this, so the verdict is unchanged.","tokens_in":19754,"tokens_out":15012,"duration_ms":185539,"concrete_test":"Recalibrate OptR exactly as in the paper but with QKV traces from a non-evaluation corpus (e.g., C4 or a reasoning benchmark not in Table 1), using the same held-out rotation-selection procedure, and re-report Table 1 and the RULER rows. If the GPQA-Diamond column falls back to near the non-calibrated baseline while other columns stay similar, the reported GPQA gains come from calibrating on the test set; if all gains persist, the rotations transfer and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"OptR's one-time calibration and its held-out rotation selection both draw from the GPQA-Diamond evaluation pool (Appendix, Calibration details: 'all 198 GPQA-Diamond prompts'; 'calibration and held-out chunks do not overlap, although their source prompts come from the GPQA-Diamond evaluation pool'). The per-head rotations and key-centering mean are therefore fit to the same distribution on which the GPQA-Diamond metrics in Table 1 are reported, and the held-out selection step tunes the rotations on that test distribution. The calibration set is also small (2,554 tokens for Qwen3-4B), making overfitting to GPQA prompts plausible. The GPQA column is not a valid generalization measurement: for example, QuaRot-INT2 on Qwen3-4B goes from 6.06 to 62.12. Because the central claim is that OptR 'consistently improves' across five benchmarks, the validity of this column is load-bearing. The other benchmarks (AIME24/25, MBPP+, LCB, RULER) are not calibration sources, so the method may still transfer; but a cross-domain calibration test is required to know whether the reported GPQA gains reflect the method or the calibration protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OptR, a calibration-based method for INT2 KV-cache quantization. Given an existing base rotation such as QuaRot or OSCAR, OptR first centers the post-RoPE keys with a per-channel mean and then learns per-head orthogonal correction rotations parameterized as exp(A-A^T), propagated through the full INT2 quantization and attention path, using a post-WO attention-output error objective. The exact error decomposition in Eq. (8) splits the post-WO output error into key-induced and value-induced terms; the method calibrates key rotations first with values in BF16 and then value rotations with the quantized-key attention fixed. Experiments on three models report large accuracy gains over both base rotations on AIME24/25, GPQA-Diamond, MBPP+, LiveCodeBench, and RULER-NIAH, with ablations showing that post-WO objectives outperform cache-reconstruction and pre-WO objectives, and with measurements indicating negligible inference overhead.","tokens_in":19948,"tokens_out":6520,"duration_ms":77514,"significance":"If the results hold, OptR is a practically valuable step for INT2 KV-cache serving. The paper provides an exact and useful error decomposition (Eq. 8), demonstrates large and consistent gains over two rotation initializations across several independent benchmarks, shows strong long-context retrieval gains at 64K and 128K, and preserves the standard paged KV-cache layout with negligible runtime overhead. The explicit confidence intervals, the objective ablations (Tables 3 and 4), and the calibration-size sensitivity study (Table B) are welcome strengths. The main caveat is that the GPQA-Diamond column of Table 1 is not an independent generalization measurement for this calibration scheme, because both calibration and held-out rotation selection use GPQA-Diamond prompts. In addition, the algorithm optimizes a sequential surrogate rather than directly minimizing the stated joint post-WO objective, so the paper overstates the objective it actually optimizes.","major_comments":[{"comment":"The calibration and held-out rotation-selection data both come from the GPQA-Diamond evaluation pool, as stated in the Appendix: 'We collect Q/K/V activation statistics from all 198 GPQA-Diamond prompts' and 'their source prompts come from the GPQA-Diamond evaluation pool.' The per-head rotations and the held-out selection step are therefore tuned on the same distribution as the GPQA-Diamond column of Table 1, so that column is not a valid held-out accuracy measurement. Because the central claim is that OptR 'consistently improves' across five benchmarks, the validity of this column is load-bearing. Please add a cross-domain calibration experiment, for example calibrating on AIME or MBPP traces and then evaluating all benchmarks, or explicitly relabel the GPQA-Diamond results as in-distribution performance rather than as evidence of generalization.","section":"Appendix, Calibration details; Table 1"},{"comment":"The paper states that OptR minimizes the post-WO attention-output error E_out (Eq. 10), but the algorithm does not directly optimize E_out. Key calibration (Eq. 14) uses a KL term plus lambda_K times the key-induced error with values in BF16, and value calibration (Eq. 16) then minimizes only the value-induced term with the quantized-key attention distribution fixed. This is a block-coordinate surrogate: the cross term between the key-induced and value-induced contributions in Eq. (8) is never evaluated, and the joint objective in Eq. (10) is not optimized. Please state explicitly that OptR minimizes a sequential approximation of E_out and justify or empirically test why this coordinate-wise ordering is appropriate, or revise the abstract and contribution claims from 'minimizes' to 'reduces via a sequential surrogate.'","section":"Eqs. (8), (10), (14), (16); Algorithm 1"}],"minor_comments":[{"comment":"The sentence 'Key-centering statistics are estimated from 28 full model-generated traces' does not state whether these traces are GPQA-Diamond traces or an independent corpus; please clarify the source and whether they overlap with the calibration or evaluation splits.","section":"Appendix, Calibration details"},{"comment":"The symbol e_K is used both for the effective keys (line 6) and for the key-induced error e_K(t,j) (line 8); renaming one of these quantities would remove a needless source of confusion.","section":"Algorithm 1"},{"comment":"The long-context results are strong, but the calibration traces appear to be short prefill chunks from GPQA prompts; please state the maximum input length in the calibration traces and explicitly note that the 128K RULER performance is an extrapolation beyond the calibration distribution, not an in-distribution measurement.","section":"Section 5.1, Table 2"},{"comment":"The caption defines 'Output error' and 'Hidden state error' informally; since the appendix gives precise normalization formulas for both metrics, please add a pointer to the appendix in the caption so that readers can interpret the logarithmic y-axes correctly.","section":"Figure 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and the experimental gains on the non-GPQA benchmarks are substantial. The calibration-distribution overlap with one of the five reported benchmarks is the main evaluation-protocol risk; it is disclosed in the appendix, which makes me treat it as a fixable issue rather than a sign of bad faith. I would be comfortable with acceptance after the authors add a cross-domain calibration study or clearly separate the in-distribution GPQA results, and after they tone down the 'minimizes E_out' claim to match the sequential optimization actually performed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper up front. First, the core idea is real and new: instead of picking rotations from fixed transforms or proxy reconstruction error, OptR optimizes per-head orthogonal corrections against the post-WO attention-output error, with an exact decomposition into key- and value-induced terms. Second, the experimental gains are large enough that you should care, even after discounting one benchmark column. On Qwen3-8B, QuaRot-INT2 goes from 17.33 to 66.67 on AIME25, and OSCAR+OptR reaches 66.00 against BF16's 68.00. That is not noise.\n\nWhat the paper does well: the error decomposition in Eq. 8 is clean, the skew-symmetric parameterization keeps the corrections orthogonal throughout training, and the key-centering reparameterization is attention-equivalent. The ablation in Table 4 is the right kind of evidence: it directly compares cache-reconstruction, pre-WO, and post-WO objectives and shows the post-WO objective wins. The long-context RULER results also hold up, with OptR pushing 64K and 128K retrieval well above the base rotations.\n\nWhere the soft spots are, in proportion:\n\nThe calibration/evaluation overlap on GPQA-Diamond is a real problem, and the paper admits it. Calibration and held-out rotation selection both draw from the 198 GPQA-Diamond prompts, and GPQA-Diamond is also a reported evaluation benchmark. So the GPQA column in Table 1 is partly test-set tuned. This does not kill the paper, because the AIME24/25, MBPP+, LCB, and RULER gains come from data the rotations never saw. But the \"five benchmarks\" claim is only four benchmarks if you are strict, and the authors need to run a calibration set from a different domain (e.g., general code or dialogue traces) and show the GPQA and AIME gains survive. That is a one-week experiment and should be a revision requirement.\n\nThe second weakness is minor: the stated objective is joint post-WO output error, but the implementation optimizes key and value corrections sequentially. The ablation suggests this is still the right direction, and the paper could just soften the claim to \"approximately minimizes\" or show a joint objective in the appendix.\n\nThird, no code is released yet despite a GitHub placeholder. The implementation details are specific enough that a determined reader could reproduce, but given the calibration issue, code would help the reviewers verify that the GPQA numbers are not an artifact of the selection procedure.\n\nMy overall take: the central mechanism is sound, the empirical evidence is strong on the non-calibration benchmarks, and the paper deserves a serious referee. I would send it out with a request for a non-GPQA calibration check and code, not a desk reject. The reader's conditional verdict is about right, though I would put a bit more weight on the AIME/RULER results when judging the method's transferability.","headline":"OptR is a genuinely new output-aware rotation objective for INT2 KV-cache quantization with large, consistent gains; the main caveat is that GPQA-Diamond is used for both calibration and evaluation.","tokens_in":20593,"tokens_out":1935,"would_cite":true,"duration_ms":26504,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the right objective for INT2 KV-cache quantization is the attention-output error after the output projection $W_O$, and that learning per-head rotations against it brings 2-bit cache accuracy close to BF16.","keywords":["KV cache quantization","INT2 quantization","rotation-based quantization","output-aware objective","orthogonal rotation refinement","attention-equivalent key centering","per-head calibration","long-context LLM inference"],"falsifier":"Recalibrate OptR on traces from a starkly different distribution — coding prompts or very long synthetic retrieval documents — and compare the learned rotations and downstream accuracy against the GPQA-calibrated ones on both AIME25 and a 128K retrieval task. If the rotations shift substantially and the GPQA-calibrated gains vanish on the held-out tasks, the objective is memorizing the calibration distribution rather than minimizing structural post-$W_O$ error; the paper's account predicts the objective transfers, since post-$W_O$ error is a property of the attention readout rather than of the task.","tokens_in":19496,"feed_emoji":"🧠","tokens_out":14580,"duration_ms":234060,"temperature":0.7,"pith_summary":"Compressing the KV cache of a large language model to 2 bits per element would shrink memory and bandwidth for long-context inference, but INT2 quantization severely degrades accuracy unless the cache is first rotated into a friendlier basis. Existing rotation-based methods pick that basis using cache statistics or the error in reconstructing the keys and values themselves, before the attention readout. This paper argues those are proxy objectives: the model actually feels the error in the attention output after the output projection $W_O$, the signal that enters the residual stream. OptR is the method built on that claim — it decomposes the post-$W_O$ error into a key-induced term and a value-induced term, then learns per-head orthogonal corrections to any base rotation by minimizing that error through the full quantized-attention path, after subtracting a per-channel key mean that leaves the softmax distribution unchanged. The result is INT2 KV-cache accuracy close to BF16 on reasoning and coding benchmarks and sharply better long-context retrieval, at 2.32 effective bits per element and negligible serving overhead.","feed_headline":"2-bit KV caches near BF16 accuracy with output-aware rotation","feed_subtitle":"Learning rotations on post-attention output error lifts Qwen3-8B AIME25 from 17% to 67% (BF16: 68%).","key_machinery":"The load-bearing identity is the exact decomposition of the post-$W_O$ attention-output error, $\\Delta y = \\left(\\sum_s \\Delta p_s\\, v_s\\right) W_O^\\top + \\left(\\sum_s \\tilde{p}_s\\, \\Delta v_s\\right) W_O^\\top$, which converts 'choose a good rotation' into a concrete output-space objective. Three mechanisms carry the argument: (1) per-head orthogonal corrections parameterized by skew-symmetric generators $S=A-A^\\top$ with $R=R_0\\exp(S)\\in O(d)$, which keep the corrected rotations orthogonal while remaining freely optimizable; (2) key centering, $k\\mapsto k-\\mu$, applied before rotation and quantization, which is attention-equivalent because subtracting the same mean from every key adds a query-dependent constant to all logits and thus leaves the softmax distribution unchanged while narrowing the dynamic range the INT2 grid must represent; and (3) a straight-through estimator for the non-differentiable INT2 rounding, so gradients flow through rotation, clipping, grouping, rounding, dequantization, and inverse rotation. The key rotation is optimized before the value rotation because the quantized keys determine the attention distribution that weights the value errors.","core_discovery":"The central claim is that the orthogonal rotation used for INT2 KV-cache quantization should be chosen to minimize the post-$W_O$ attention-output error $E_{\\mathrm{out}}=\\|\\tilde{y}-y\\|_2^2$ rather than the cache-reconstruction error $\\|K-\\tilde{K}\\|_F^2+\\|V-\\tilde{V}\\|_F^2$. The paper derives an exact decomposition of $E_{\\mathrm{out}}$ into a key-induced term, where quantized keys perturb the softmax attention distribution and the change is then weighted by values and mapped by $W_O$, and a value-induced term, where quantized values are aggregated by the quantized-key attention distribution and mapped by $W_O$. Because these terms weight cache errors by how much attention actually reads them, they can favor different rotations than reconstruction does — that is the objective mismatch the method exploits. OptR learns, per KV head and per layer, an orthogonal correction $R_0\\exp(S)$ with skew-symmetric generator $S=A-A^\\top$, first for keys (with a KL term that keeps the attention distribution close) and then for values under the fixed quantized-key path, backpropagating through the entire INT2 rotation-clipping-grouping-rounding path with a straight-through estimator. It also centers keys by a per-channel mean before rotation, a reparameterization that provably leaves softmax unchanged while shrinking the range INT2 must cover. On Qwen3-8B this lifts AIME25 accuracy from 17.33% (QuaRot) and 54.67% (OSCAR) to 66.67% and 66.00%, against 68.00% for BF16, at 2.32 effective bits per cache element.","pith_inferences":["The decomposition predicts a testable pattern the paper does not report: models with sharper attention distributions should gain more from key-rotation calibration, because key error only matters where softmax weight is large — one could rank heads by attention entropy and correlate with per-head OptR gains.","The output-aware principle generalizes beyond KV caches: the same recipe of measuring quantization error after the consuming operation could be applied to quantized activations or weights inside later layers, but the paper makes no such claim.","Key centering is softmax-equivalent only in full precision; under the four-level INT2 grid, centering changes which values fall into which quantization bins, so part of the gain may come from bin reassignment rather than pure range reduction — the paper does not isolate these two effects.","The GPQA-derived calibration is the main transfer risk: the reported 128K retrieval and coding gains are evidence of transfer but not a guarantee, and calibrating instead on coding or retrieval traces would test whether the learned rotations are task-agnostic."],"forward_implications":["The same 2.32-bit cache that collapses under existing rotations is largely recoverable: with OptR, Qwen3-8B AIME25 reaches 66.67% (from 17.33% with QuaRot) and 66.00% (from 54.67% with OSCAR), against 68.00% for BF16.","Output-space calibration transfers across rotation initializations, so the gain comes from the objective rather than from a particular rotation construction.","Long-context retrieval at 64K to 128K tokens, where plain rotated INT2 falls to near-zero accuracy, keeps most of its BF16-level performance with OptR (e.g., Qwen3-8B 64K RULER accuracy from 0.04-57.54% to 68.65-70.02%, against 84.22% for BF16).","The method stays deployment-compatible: it preserves the paged KV-cache layout, folds value-side rotations into the $W_V$ and $W_O$ weights, and adds no more than about 2% overhead in decode latency, throughput, or prefill time.","Calibration is cheap and stable: a few thousand tokens and 80 Adam steps per head suffice, with AIME25 accuracy roughly flat from 3.1K to 26K calibration tokens."],"supporting_citations":[{"why":"Supplies the OSCAR covariance-based base rotation, the official codebase, and the INT2 group-size/window configuration that OptR augments and benchmarks against.","marker":"Zhou et al. [2026]"},{"why":"Supplies the QuaRot Hadamard-rotation baseline and the rotated quantization pipeline that OptR improves as its second base initialization.","marker":"Ashkboos et al. [2024]"},{"why":"RotateKV, a prior rotation-based INT2 KV method with a different selection criterion, defining the family of approaches whose objective OptR replaces.","marker":"Su et al. [2025]"},{"why":"Defines the paged KV-cache memory layout that OptR must preserve for deployment compatibility.","marker":"Kwon et al. [2023]"},{"why":"The serving system whose paged and prefix caches OptR integrates with while measuring inference overhead.","marker":"Zheng et al. [2024]"},{"why":"Adam is the optimizer used for the 80-step per-head rotation calibration.","marker":"Kingma and Ba [2014]"},{"why":"RULER-NIAH long-context retrieval benchmark provides the 64K-128K evidence for the paper's long-context claims.","marker":"Hsieh et al. [2024]"},{"why":"GPQA-Diamond supplies the decoding traces from which the key means and rotations are calibrated.","marker":"Rein et al. [2024]"},{"why":"TurboQuant serves as the online vector-quantization baseline compared in the main results.","marker":"Zandieh et al. [2026]"}],"fun_headline_variants":["Output-aware rotation closes INT2 KV-cache gap to BF16","Rotate to match attention: INT2 KV cache hits BF16 level","Shrinking 2-bit KV cache error where attention looks","Post-attention rotation lifts INT2 cache to near-BF16","Learn rotation from output error, not cache copy error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The per-head rotations and the key-centering mean are estimated from a few thousand tokens of GPQA-Diamond decoding traces, and the method assumes those traces are representative enough of all downstream tasks and of context lengths up to 128K tokens that the same fixed rotations transfer without retraining.","fun_headline_variants_meta":{"raw":{"variants":["Output-aware rotation closes INT2 KV-cache gap to BF16","Rotate to match attention: INT2 KV cache hits BF16 level","Shrinking 2-bit KV cache error where attention looks","Post-attention rotation lifts INT2 cache to near-BF16","Learn rotation from output error, not cache copy error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000597,"raw_usage":{"total_tokens":2868,"prompt_tokens":1094,"completion_tokens":1774,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":710,"completion_tokens_details":{"reasoning_tokens":1686}},"tokens_in":710,"tokens_out":1774,"duration_ms":14219,"temperature":1.0,"reasoning_tokens":1686,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:54:10.859745+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recalibrate OptR on traces from a starkly different distribution — coding prompts or very long synthetic retrieval documents — and compare the learned rotations and downstream accuracy against the GPQA-calibrated ones on both AIME25 and a 128K retrieval task. If the rotations shift substantially and the GPQA-calibrated gains vanish on the held-out tasks, the objective is memorizing the calibration distribution rather than minimizing structural post-$W_O$ error; the paper's account predicts the objective transfers, since post-$W_O$ error is a property of the attention readout rather than of the task.","supporting_citations":[],"review_version":2}