{"id":"7d14caeb-3268-434c-911e-0a3ca0efa477","arxiv_id":"2412.14363","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"ResQ quantizes LLMs to 4-bit with a low-rank 8-bit residual subspace chosen by PCA, beating prior 4-bit methods on perplexity and accuracy.","lead":"ResQ is a new method for running large language models with 4-bit numbers, keeping only a small high-precision slice of each layer's data chosen by the statistics of the activations. It reports better accuracy than existing 4-bit quantization methods and faster inference than a 16-bit baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'provably optimal' claim rests on a false Frobenius-norm identity in Appendix A Eq. (12); with the correct Pythagorean relation the bound is not monotone in ||XP_h||_F, so PCA optimality is unsupported as written.","rationale":"The reader's weakest_assumption correctly isolates Eq. (12), and I agree that this is the load-bearing defect. This is not an external disagreement with a heuristic; it is an internal algebraic error in the only proof of the central theoretical claim. The paper's empirical evaluation is broad and credible: multiple model families, several benchmarks, ablations of each projection, comparisons against strong baselines, and a custom CUDA kernel. Those results do not depend on Eq. (12). However, the abstract and Section 4.2 advertise a proof of optimality, and Table 4 explicitly frames the PCA choice as theoretically optimal. With the proof invalid, that claim should be either corrected or removed. A conditional accept with mandatory proof repair or claim softening is proportionate; a straight reject would overstate the damage because the method's practical value is independently evidenced. The concrete test above would tell whether the optimality conclusion is true even if unproven.","tokens_in":29828,"tokens_out":6832,"duration_ms":60493,"concrete_test":"Re-derive Theorem 4.2 replacing Eq. (12) with the Pythagorean relation, and evaluate the corrected bound on the 512 Wikitext calibration activations for a representative Meta-Llama-3-8B layer, comparing three choices of high-precision subspace: PCA top-r eigenvectors, a random r-dimensional subspace, and the l-infinity outlier channels used by QUIK. If the corrected bound is not minimized by the PCA choice, the optimality claim in Theorem 4.2 fails; if it is minimized, the empirical optimality survives despite the flawed proof and only the proof needs repair.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix A derives Theorem 4.2 by substituting ||XP_l||_F = ||X||_F - ||XP_h||_F (Eq. 12). Because P_l and P_h have orthonormal columns and P_lP_l^T + P_hP_h^T = I, the exact relation is ||XP_l||_F^2 = ||X||_F^2 - ||XP_h||_F^2. The linear subtraction is false; the preceding line also appears to conflate ||XP_l||_F with ||tr(XP_lP_l^T X^T)||_F, where the trace is actually the squared norm. Under Lemma A.1, the corrected bound is a*E(sqrt(||X||^2 - ||XP_h||^2)) + b*E||XP_h||, with a much larger than b for 4-bit low / 8-bit high precision. The function f(t) = a*sqrt(C - t^2) + b*t is not decreasing in t; it has an interior maximum at t = b*sqrt(C)/sqrt(a^2 + b^2). Hence the inference 'to lower the upper bound, maximize ||XP_h||_F' does not follow, and the PCA choice of P_h is not established by this theorem. The empirical results may still be valid, but the advertised theoretical optimality guarantee is not supported. A secondary issue is Eq. (2): Xq = Q_L(XU_l) + Q_H(XU_h) is dimensionally inconsistent unless reconstruction by U_l^T and U_h^T is intended, which is not stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ResQ is a post-training quantization method for LLMs that targets W/A/KV = 4/4/4-bit inference. It projects activations, weights, and KV cache into an orthogonal basis, keeps the low-rank subspace (1/8 of hidden dimension) with highest activation variance at 8-bit precision, and quantizes the complementary subspace at 4-bit, applying random orthogonal rotations within each subspace to suppress outliers. The paper claims that PCA is a provably optimal choice for the high-precision subspace (Theorem 4.2), and supports this with an extended empirical evaluation on Llama 2/3/3.2, Qwen2.5, and Qwen2-VL models across language modeling, commonsense reasoning, MMLU, GSM8K, LongBench, and MMMU, reporting gains over SpinQuant and other baselines as well as CUDA-kernel speedups.","tokens_in":30204,"tokens_out":6130,"duration_ms":50748,"significance":"If the method and its theoretical justification were both sound, this would be a practically valuable contribution: it addresses a hard setting (4-bit weights, activations, and KV cache), it is training-free, and the experimental coverage is unusually broad, including several model families and both language-only and multimodal benchmarks. The paper also ships code and reports kernel-level speedups, which strengthens reproducibility. The central advertised claim, however, is the 'provably optimal' PCA choice for mixed precision; as written, the proof of Theorem 4.2 contains an algebraic error that invalidates the stated bound and the optimality conclusion. The empirical results may still stand on their own, but the paper's headline theoretical guarantee is not currently supported.","major_comments":[{"comment":"The proof of Theorem 4.2 uses the identity ||XP_l||_F = ||X||_F - ||XP_h||_F. This is false: since P_lP_l^T + P_hP_h^T = I and the columns of P_l and P_h are orthonormal, the correct relation is ||XP_l||_F^2 = ||X||_F^2 - ||XP_h||_F^2. The preceding line (Eq. 11) also conflates ||XP_l||_F with ||tr(XP_lP_l^T X^T)||_F, where the trace is the squared norm, not the norm itself. With the correct Pythagorean relation, the bound becomes a*E(sqrt(||X||^2 - ||XP_h||^2)) + b*E||XP_h||, which is not monotone in ||XP_h||_F; it has an interior maximum and decreases only on part of its domain. Consequently, the inference 'to lower the upper bound, maximize ||XP_h||_F' does not follow from the displayed equations, and the PCA optimality of Theorem 4.2 is not established by this argument.","section":"Appendix A, Eq. (12)"},{"comment":"There are two additional issues in the chain of inequalities leading to Theorem 4.2. First, Eq. (10) states an equality between E||X - Xq||_F and the sum of the two component errors; in general only an inequality (triangle inequality) holds, and the equality requires the squared norms and orthogonality of the subspaces. Second, the text says 'We know size(XP_l) = d - r and size(XP_h) = r', but X is an n by d matrix, so the number of elements in XP_l is n(d-r) and in XP_h is nr. The logarithmic factors in Eq. (7) should be log(n(d-r)) and log(nr), not log(d-r) and log(r). This changes the constants and the conditions under which the coefficient multiplying E||XP_h||_F is positive, so the theorem as stated needs correction beyond the Frobenius-norm identity.","section":"Appendix A, Eq. (11) and Eq. (10)"},{"comment":"Equation (2), Xq = Q_L(XU_l) + Q_H(XU_h), is dimensionally inconsistent as written. XU_l is n by (d-r) and XU_h is n by r, so the two matrices cannot be added. To reconstruct an n by d activation, the formula should include the right-multiplication by U_l^T and U_h^T, e.g., Xq = Q_L(XU_l)U_l^T + Q_H(XU_h)U_h^T. The same issue affects the interpretation of Eq. (4) and the discussion following it; although the shapes in Eq. (4) can be read consistently if U_l^T W is (d-r) by d, the presentation should state the reconstruction explicitly.","section":"Section 4.1, Eq. (2)"},{"comment":"The proof of Theorem 4.2 depends on Lemma 4.1, which is stated as a direct consequence of the Central Limit Theorem and a citation to prior work, and on Lemma A.1, which is imported from another paper. For the argument to apply, the entries of XU_l and XU_h must be approximately Gaussian. But P_l and P_h are data-dependent PCA projections estimated from calibration data, and R_l, R_h are random orthogonal matrices applied after those projections; it is not immediate that XU_l = XP_l R_l has Gaussian marginals for realistic n and d, especially with the heavy-tailed activation distributions observed in LLMs. The paper does not provide empirical diagnostics for this assumption on the actual calibration data. This is a correctness-risk issue for the key theorem, and a concrete test (e.g., measuring the Gaussianity of projected coefficients after the PCA step) would help assess whether the assumption holds in practice.","section":"Section 4.2, Lemma 4.1 and Theorem 4.2"}],"minor_comments":[{"comment":"There are typographical errors: 'upto' in the abstract and 'mlowers' in the Introduction should be corrected to 'up to' and 'lowers'.","section":"Abstract and Introduction"},{"comment":"The notation size(R) is used without defining it for a matrix; it should be stated explicitly as the number of elements in R, since the logarithmic factor in the bound is what makes the size dependence clear.","section":"Appendix A, Lemma A.1"},{"comment":"There is a missing closing parenthesis in the second term of Eq. (11): 'sqrt(log(size(XP_h)π)' should read 'sqrt(log(size(XP_h))π)'.","section":"Appendix A, Eq. (11)"},{"comment":"The text states that UA and UB projections are fused into weights with no additional inference cost, but UC and UD are computed at runtime; this contrast should be stated more explicitly so the reader does not infer that all projections are overhead-free.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional assessment aligns with my own reading. The empirical study is strong and likely publishable, but the paper's central theoretical claim, 'provably optimal mixed precision quantization', is not supported by the proof as written because of a false Frobenius-norm identity and related size/equation issues in Appendix A. These errors are load-bearing, but they appear correctable: the PCA conclusion may still hold under a corrected bound, and the empirical results are independent of the theorem. I recommend major revision rather than rejection, with the authors required either to supply a correct proof of the optimality claim or to remove the 'provably optimal' wording from the abstract and Section 1 and reframe Theorem 4.2 as a heuristic justification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ResQ is a genuinely useful empirical paper whose headline theory is broken. The method—PCA picks the top r eigendirections of the activation covariance for 8-bit, everything else 4-bit, with random rotations inside each group—is a clean combination that seems to work. The experiments are broad: Llama 2/3/3.2, Qwen2.5, Qwen2-VL, perplexity, commonsense, MMLU, GSM8K, LongBench, MMMU, with baseline numbers taken from official codebases. At W/A/KV=4/4/4, ResQ beats SpinQuant on most models without training, and the speedup story (single-decoder-block CUDA kernels, ~1.6–3× over FP16) is plausible, though it's measured on one block, not end-to-end.\n\nThe soft spot is real and central. The 'provably optimal' claim rests on Theorem 4.2, and the proof in Appendix A is not sound. Eq. (12) substitutes ||XP_l||_F = ||X||_F − ||XP_h||_F. Since P_l and P_h are orthogonal complements, the correct identity is ||XP_l||_F = sqrt(||X||_F^2 − ||XP_h||_F^2). With that, the upper bound becomes a·E(sqrt(||X||^2 − ||XP_h||^2)) + b·E||XP_h||, which is not monotone decreasing in ||XP_h|| over the whole range. So the inference 'maximize ||XP_h||_F, hence PCA' does not follow from the stated bound. There's also a smaller but real issue: Eq. (2) writes Xq = Q_L(XU_l) + Q_H(XU_h), which is dimensionally inconsistent; reconstruction via U_l^T and U_h^T is clearly intended, but not stated.\n\nEverything else is in much better shape. The Gaussianity assumption after random rotation is heuristic (they cite Lemma 4.1 from the CLT), and using Wikitext for both calibration and the headline perplexity is a mild in-distribution choice; the MMLU/GSM8K/LongBench/MMMU results mitigate that. The calibration-sample ablation shows saturation by 128–512 samples, which is reassuring.\n\nNet: the empirical method is solid, and this is a paper for people building 4-bit LLM inference stacks—they'll get a practical recipe. It deserves a serious referee. The theory section needs either a correct proof or a conspicuous softening of 'provably optimal' to something like 'empirically effective.' The speedup claim in the abstract should be labeled as single-block, and the code link should point to a pinned commit. I'd send it to review with major revision requested.","headline":"Solid empirical PTQ paper; the 'provably optimal' proof in Appendix A is wrong, and the paper needs a major revision before acceptance.","tokens_in":30746,"tokens_out":6092,"would_cite":true,"duration_ms":47890,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ResQ keeps a PCA-chosen 8-bit subspace and quantizes the rest of an LLM to 4-bit, claiming near-16-bit accuracy without any training.","keywords":["post-training quantization","mixed-precision quantization","large language models","principal component analysis","random rotation","KV cache quantization","low-rank residual","LLM inference acceleration"],"falsifier":"Recompute the bound in Theorem 4.2 using the exact relation $\\|XP_l\\|_F = \\sqrt{\\|X\\|_F^2 - \\|XP_h\\|_F^2}$ and check whether the inequality still implies that choosing the PCA subspace maximizes $\\|XP_h\\|_F$; alternatively, run ResQ with a random orthogonal $P_h$ of the same rank on the same models and compare Wikitext perplexity, since a random projection matching or beating PCA would contradict the claimed optimality.","tokens_in":29619,"feed_emoji":"⚡","tokens_out":6028,"duration_ms":51459,"temperature":0.7,"pith_summary":"ResQ is a post-training quantization method for large language models that aims to make 4-bit weight, activation, and KV-cache inference nearly as accurate as 16-bit inference. The core idea is to use PCA on calibration activations to find a low-rank subspace, typically one eighth of the hidden dimension, where variance is concentrated; coefficients in that subspace are kept at 8-bit while the complement is quantized to 4-bit, with random orthogonal rotations applied inside each subspace to suppress outliers. The paper claims this scheme provably minimizes quantization error, and reports that it outperforms prior uniform- and mixed-precision methods, including the rotation-based SpinQuant, on perplexity, reasoning, and multimodal benchmarks, while delivering up to 3x speedup over a 16-bit baseline on custom CUDA kernels. If the claim holds, it offers a parameter-free path to aggressive LLM compression that does not require gradient-based rotation learning.","feed_headline":"Top-variance subspace stays 8-bit; the rest of the LLM drops to 4-bit","feed_subtitle":"ResQ cuts memory and speeds up inference while beating SpinQuant on perplexity, with no training required.","key_machinery":"The central object is the orthogonal projection $U = PR$, where $P$ is the eigenbasis of the activation covariance $XX^{\\top}$ sorted by increasing eigenvalue and $R$ is a block-diagonal random orthogonal rotation. The last $r$ columns of $P$, corresponding to the highest-variance directions, form the high-precision subspace kept at 8-bit, while the first $d-r$ columns, rotated and quantized at 4-bit, form the low-precision residual. The machinery also includes four projection placements, $U_A$, $U_B$, $U_C$, and $U_D$, that push the projections into neighboring weights or, where necessary, apply fast Hadamard transforms at runtime so that the mixed-precision GEMMs dominate the cost.","core_discovery":"The central claim is that the right way to allocate mixed precision is to decompose the activation tensor along principal components of its empirical covariance, quantize the top-r high-variance directions at high precision, quantize the remaining low-variance directions at low precision, and then rotate each block by a random orthogonal matrix before quantization. The paper's Theorem 4.2 bounds the expected Frobenius quantization error in terms of the norm of the high-precision projection, showing that the bound is minimized when the high-precision subspace is spanned by the eigenvectors of $XX^{\\top}$ with the largest eigenvalues, i.e. the PCA subspace. Because the projection matrices can be fused into adjacent weights except where rotary embeddings or activation functions intervene, the scheme adds little runtime overhead, and the authors demonstrate consistent gains over SpinQuant and other baselines across Llama 2, Llama 3, Llama 3.2, Qwen2.5, and Qwen2-VL models at W/A/KV = 4/4/4 bits.","pith_inferences":["If the PCA-optimality argument can be repaired, the same spectral recipe would likely apply to any linear layer with heavy-tailed activation distributions, suggesting a general principle: choose quantization bases by the covariance of calibration activations rather than by per-channel outlier statistics.","The strong results on Qwen2.5, where Hadamard-rotation baselines degrade sharply, hint that architecture-specific activation geometry matters more than rotation alone; a testable extension is to apply ResQ to Mixture-of-Experts models or other architectures with different covariance structure.","The 512-sample calibration requirement could probably be reduced by estimating the activation covariance online or with a lightweight proxy, making the method usable when only a small calibration set is available.","Since rank tuning already traces a Pareto frontier, a natural extension is per-layer rank allocation driven by layerwise sensitivity, which the paper does not explore but its own ablation data would support."],"forward_implications":["ResQ can quantize weights, activations, and KV cache all to 4-bit while keeping one eighth of the hidden-dimension channels at 8-bit, closing much of the gap to 16-bit baselines on language modeling and reasoning tasks.","Because the projection matrices fuse into adjacent weights, the mixed-precision 4-bit/8-bit GEMMs run close to uniform INT4 speed, giving measured speedups of roughly 1.61x to 3.03x over 16-bit inference on a single decoder block.","The rank $r$ of the high-precision subspace provides a direct accuracy-efficiency trade-off, so practitioners can choose a Pareto-optimal operating point for a given hardware budget.","The method extends beyond language modeling to KV cache compression and to vision-language models, preserving a large fraction of MMMU accuracy when only the language model is quantized.","Combining ResQ with GPTQ for weight quantization maintains its advantage across model families from 0.5B to 72B parameters, suggesting the approach scales with model size."],"supporting_citations":[{"why":"Supplies the quantization-error bound for normally distributed tensors used as Lemma A.1 in the proof of Theorem 4.2.","marker":"(Li et al., 2024)"},{"why":"Provides Lemma 4.1, the central-limit-theorem argument that random orthogonal multiplication makes activation and weight distributions approximately Gaussian.","marker":"(Tseng et al., 2024)"},{"why":"Introduces invariant random rotation for outlier suppression and shows how to fuse rotation matrices into adjacent weights, a technique ResQ builds on.","marker":"(Ashkboos et al., 2024c)"},{"why":"SpinQuant is the strongest empirical baseline and the main comparison point for perplexity, accuracy, and training-free quantization.","marker":"(Liu et al., 2024b)"},{"why":"GPTQ is the weight-quantization method applied on top of ResQ's projections in all main experiments.","marker":"(Frantar et al., 2022)"},{"why":"Documents the extreme activation outliers, roughly 20x larger than typical values, that motivate the mixed-precision treatment.","marker":"(Dettmers et al., 2022)"},{"why":"QUIK is the closest mixed-precision baseline that keeps high-norm channels at 8-bit, and its comparison helps isolate the benefit of PCA-based subspace selection.","marker":"(Ashkboos et al., 2024b)"}],"fun_headline_variants":["ResQ: 8-bit for top variance, 4-bit for the rest, provably optimal","Keep the outliers rich: ResQ's 8-bit subspace, rest 4-bit","PCA finds where 8-bit matters; ResQ drops the rest to 4-bit","ResQ: Provably optimal mixed-precision via PCA and rotation","Mixed precision done right: ResQ beats SpinQuant with 8-bit subspace"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The optimality proof depends on treating rotated activation coefficients as Gaussian enough for the SVDQuant error bound in Lemma A.1 to apply, and on the identity $\\|XP_l\\|_F = \\|X\\|_F - \\|XP_h\\|_F$, which is not exact; the correct Pythagorean relation for orthogonal complements is $\\|XP_l\\|_F = \\sqrt{\\|X\\|_F^2 - \\|XP_h\\|_F^2}$, so as written the derived bound and the optimality conclusion are not established.","fun_headline_variants_meta":{"raw":{"variants":["ResQ: 8-bit for top variance, 4-bit for the rest, provably optimal","Keep the outliers rich: ResQ's 8-bit subspace, rest 4-bit","PCA finds where 8-bit matters; ResQ drops the rest to 4-bit","ResQ: Provably optimal mixed-precision via PCA and rotation","Mixed precision done right: ResQ beats SpinQuant with 8-bit subspace"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1730,"prompt_tokens":1010,"completion_tokens":720,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":612}},"tokens_in":626,"tokens_out":720,"duration_ms":5498,"temperature":1.0,"reasoning_tokens":612,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:18:09.268592+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the bound in Theorem 4.2 using the exact relation $\\|XP_l\\|_F = \\sqrt{\\|X\\|_F^2 - \\|XP_h\\|_F^2}$ and check whether the inequality still implies that choosing the PCA subspace maximizes $\\|XP_h\\|_F$; alternatively, run ResQ with a random orthogonal $P_h$ of the same rank on the same models and compare Wikitext perplexity, since a random projection matching or beating PCA would contradict the claimed optimality.","supporting_citations":[],"review_version":1}