{"id":"5d530d9d-3be9-4f5c-90d5-f7540b6d95bb","arxiv_id":"2412.07902","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A low-rank correction computed jointly with quantized weights recovers most of the original accuracy for 4-bit weight-and-activation LLM quantization.","lead":"This paper presents LRC, a post-training method that adds small full-precision low-rank matrices to correct errors from 4-bit weight and activation quantization in large language models. The method improves accuracy over current W4A4 quantization baselines on several LLMs, and with larger ranks it nearly matches the original models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main W4A4 comparison is not bitrate-matched: LRC at 10% rank is ~6.08 effective bits vs QuaRot's 4 bits, so the reported gains may reflect extra memory rather than the low-rank correction.","rationale":"The reader's weakest assumption is calibration representativeness. That concern has partial mitigation in Appendix C.1, which shows similar Phi-3 results with Alpaca and Wikitext2 calibration, and the calibration set is large relative to layer width. The bitrate confound is more load-bearing: it is explicitly admitted in Appendix C.2, it directly affects every headline comparison against QuaRot, and it can be settled by a single matched-bitrate experiment. If the matched-bitrate baseline matches LRC, the paper remains a useful recipe, but the core 'W4A4 beats SoTA' claim and the abstract's gap-reduction percentages would need to be reframed as a bit-budget trade-off rather than a pure W4A4 improvement. The authors do disclose the overhead, so this is a comparison-validity issue, not a claim of hidden results.","tokens_in":18483,"tokens_out":14590,"duration_ms":141890,"concrete_test":"Run QuaRot with the same Hadamard preprocessing and GPTQ, but quantize weights to ~6 bits (or int4 with a group size that yields ~6.08 effective bits) while keeping activations at W4A4, on Phi-3, Llama-2 7B/13B, Llama-3 8B, and Mixtral. If the 6-bit QuaRot baseline matches or exceeds LRC's average lm-eval accuracy at 10% rank, then LRC's reported advantage is attributable to extra memory, not to the low-rank joint optimization. For the 30% claim, repeat with a ~10-bit weight baseline. If LRC still wins at matched bitrate, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LRC outperforms SoTA W4A4 requires that its advantage comes from the joint low-rank optimization and not from spending additional bits. Appendix C.2 states that using 10% rank \"incurs an additional memory 13% of the original model\" and that LRC is \"effectively at 6.08 bits (4 + 0.13 * 16)\". Yet the main tables (Tables 1, 2) compare LRC with 4-bit QuaRot only. At 30% rank, Table 9 lists LRC sizes of 4.39 GB for Phi-3 against 1.69 GB for 4-bit QuaRot in Table 3, i.e. roughly 10 effective bits; \"closing the gap completely\" at that budget is unsurprising. No same-bitrate baseline (e.g., 6-bit QuaRot or GPTQ with 4-bit activations) is reported, so the W4A4 superiority claim and the associated abstract percentages are untested. This is a confound in the main comparison, not merely a missing baseline: the paper's own effective-bit calculation makes it concrete.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LRC, a post-training quantization method that augments a quantized weight matrix with low-rank full-precision matrices U and V acting on unquantized activations, with the goal of compensating for activation-quantization errors. The method alternates between a GPTQ-style quantization of an effective weight matrix and closed-form updates of U and V obtained from eigen-decompositions of calibration covariances. Experiments on Llama-2, Llama-3, Phi-3, and Mixtral at W4A4 report accuracy gains over QuaRot and over a fixed SVD-based low-rank baseline, and the abstract claims that 10% ranks reduce the accuracy gap by more than 50% and 30% ranks close it completely. The appendix reports calibration-dataset sensitivity, latency measurements, and proofs of the main propositions.","tokens_in":18702,"tokens_out":16011,"duration_ms":146071,"significance":"The central idea is useful: a low-rank correction that is optimized jointly with the quantized weights, rather than fixed by SVD, is a plausible and principled way to reduce quantization error. The derivations in Propositions 3.1, 3.3, and 3.4 appear mathematically correct under the stated full-rank assumptions, and Appendix D provides complete proofs. The paper also contains helpful ablations (rank, quantizer choice, calibration set) and, unusually, explicit latency measurements. However, the headline comparison is confounded by effective bitrate: by the authors' own calculation in Appendix C.2, 10% ranks correspond to 6.08 effective bits, and 30% ranks to roughly 10 effective bits, while the QuaRot baseline is 4-bit. The 'more than 50% gap reduction' claim is also not uniformly supported by the tables. These issues currently limit the strength of the central claims, though they are addressable with additional experiments and revised wording.","major_comments":[{"comment":"The main W4A4 comparison is not bitrate-matched. Appendix C.2 states that 10% ranks make LRC 'effectively at 6.08 bits (4 + 0.13 * 16),' and Table 3 gives LRC model sizes of 2.59 GB (Phi-3), 4.95 GB (Llama-3), and 32.1 GB (Mixtral), versus 1.69, 3.25, and 21.6 GB for 4-bit QuaRot. At 30% ranks (Table 9), the LRC sizes are 4.39, 8.35, and 53 GB, i.e. roughly 10 effective bits. Since the main tables compare LRC only against 4-bit QuaRot, the reported gains could be due to the additional bit budget rather than to the low-rank correction itself. The central claim that LRC 'outperforms all existing methods at W4A4' therefore needs a same-total-memory baseline, for example QuaRot with 6-bit weights and 4-bit activations, or an LRC variant with U and V quantized to lower precision.","section":"Appendix C.2 and Tables 1, 3, 9"},{"comment":"The claim that at 10% rank LRC 'reduces the accuracy gap with the original model by more than 50%' is not consistently supported by the reported numbers. Using the average lm-eval accuracy in Table 1, the gap reduction relative to QuaRot for LRC(1) is approximately 68% (Phi-3), 38% (Llama-3), 31% (Mixtral), 45% (Llama-2 7B), and 44% (Llama-2 13B). With group-scaling in Table 2, Mixtral still shows only 27% gap reduction. The abstract either needs to state the specific setting (e.g., with group-scaling, or a pooled metric) or the claim should be weakened; as written, a reader would reasonably expect the 50% figure to hold across the reported models.","section":"Abstract and Tables 1-2"},{"comment":"The method is a hybrid W4A4 plus FP16 low-rank pipeline rather than a pure 4-bit activation-compute scheme. In Eq. (2), the correction path U V^T X operates on the unquantized activations X and on full-precision U and V, so the full-precision activation sequence must be materialized at every layer in addition to the 4-bit path. The latency tables in Appendix C.2 show this has a noticeable cost (e.g., Table 6: 13.89 ms for the rank-0 int4 kernel versus 18.04 ms at rank 128). This should be stated prominently whenever 'W4A4' is used, and the effective-bitrate caveat should appear in the abstract, not only in an appendix.","section":"Section 3.1, Figure 1, and Appendix C.2"}],"minor_comments":[{"comment":"LRC(1) and LRC(5) report worse Wikitext-2 perplexity (8.05 and 7.94) than QuaRot (7.78) despite better lm-eval averages; a brief comment on this discrepancy would help readers interpret the perplexity metric.","section":"Table 1, Llama-3 rows"},{"comment":"The Limitations paragraph states that 'we have not studied the computational costs of adding low-rank computations to the forward pass,' but Appendix C.2 presents a latency experiment with detailed timings; the wording should be reconciled with the appendix.","section":"Section 5 and Appendix C.2"},{"comment":"The experiments report ranks as a percentage of the original weight-matrix size, but Algorithm 1 takes a single integer k and the paper does not specify how k is chosen for non-square layers; please state the per-layer rule explicitly.","section":"Algorithms 1 and 4, Section 3.2"},{"comment":"In the proposition statements, the notation suggests that Sigma_2 and Sigma_3 are constants, but in Proposition 3.3 they depend on the current cW^(t); adding the superscript to the displayed matrices in the statement would improve clarity.","section":"Propositions 3.3 and 3.4"},{"comment":"There are several typos and inconsistencies: 'LLama' (Section 4), 'orignal' (Tables 9 and 10), 'round-to-neatest' (Section 4.2), and 'experiements' (Section 4.1). A careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The effective-bitrate confound is the decisive issue. The paper's own calculation in Appendix C.2 shows that 10% ranks put LRC at 6.08 effective bits, and the main tables compare only against 4-bit QuaRot. I would require a same-total-memory baseline before the W4A4 superiority claim can stand. The mathematical contribution is sound enough that a major-revision path is appropriate, and the authors should also be asked to re-state the gap-reduction claim to match the reported numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is real: jointly optimizing quantized weights and full-precision low-rank corrections that act on unquantized activations, with closed-form updates using activation covariances. That is a genuine step beyond LQER, Ou et al., and Saha et al., and the derivations in Propositions 3.1, 3.3, and 3.4 look correct. The weight-only ablation is also honest: low-rank terms don't help when activations stay full precision, which supports the claim that the correction targets activation quantization specifically.\n\nBut the headline numbers are built on an apples-to-oranges comparison. The paper itself admits in Appendix C.2 that LRC at 10% rank runs at 6.08 effective bits (4 + 0.13*16), versus QuaRot's 4 bits. Tables 1 and 2 compare only against 4-bit QuaRot, with no same-bitrate baseline. So the reported gap reduction is partly bought with extra memory, not just with the low-rank correction. At 30% rank the effective bitrate is around 10 bits, so closing the gap completely is unsurprising. The abstract's \"more than 50%\" claim also does not hold for Phi-3 in Table 1 (gap reduction is ~32%), and the 30% rank results appear only for three models in the appendix.\n\nOther soft spots: the activation quantization scale c is a hyperparameter with no reported search range or values, and there are no error bars anywhere. The latency study in C.2 shows that even small ranks add meaningful overhead, which tempers the practical value. These are fixable, but together they push the paper's claims beyond what the evidence supports.\n\nWho is this for? Researchers working on low-bit inference and post-training quantization. The method is plausible and worth a serious look, but only after the authors reframe the claims and add same-bitrate baselines, like a 6-bit QuaRot or a memory-matched W4A4 variant. I would not cite it in its current form, but I would want to see a revised version.","headline":"Genuinely new algorithm for activation quantization, but the main comparison is not bitrate-matched, so the headline accuracy claims are overstated.","tokens_in":19257,"tokens_out":2872,"would_cite":false,"duration_ms":25551,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding full-precision low-rank matrices that read the unquantized activations reduces the W4A4 accuracy gap by more than half at 10% rank and closes it at 30%.","keywords":["post-training quantization","LLM quantization","W4A4 quantization","low-rank correction","activation quantization","weight quantization","GPTQ","large language models"],"falsifier":"Quantize an LLM with LRC at a 30% rank budget using a calibration set from one domain, then evaluate on tasks from a domain with very different activation statistics, such as code or non-English text; if the average accuracy gap to the FP16 model is substantially larger than the paper's near-lossless result, the calibration-representativeness assumption is falsified.","tokens_in":18245,"feed_emoji":"🧠","tokens_out":17317,"duration_ms":137157,"temperature":0.7,"pith_summary":"Post-training quantization of both weights and activations to 4 bits (W4A4) has been held back mainly by the errors introduced when activations are rounded. This paper introduces LRC (low-rank correction), a method that adds small full-precision low-rank weight matrices to the forward pass, placed so that they act on the unquantized activations while the quantized weights act on the quantized activations. The quantized weights and the low-rank matrices are solved jointly, per layer, from a calibration set. The paper's central claim is that at a rank budget of 10% of the original weight-matrix size, LRC cuts the accuracy gap against the original FP16 model by more than half on six standard tasks across four recent LLM families, and that at a 30% budget the gap closes completely. This matters because it offers a concrete path to run LLM inference at 4-bit weights and activations with near-full-precision quality.","feed_headline":"Low-rank correction halves the 4-bit LLM accuracy gap","feed_subtitle":"At 30% rank, 4-bit LLMs match full-precision accuracy on six tasks; at 10%, the gap drops by more than half.","key_machinery":"The load-bearing object is the pair of low-rank factors $(U,V)$ together with the decoupled forward computation $\\widehat{W}Q_a(X) + UV^\\top X$. The factors are chosen not from a residual SVD of the weight error but from the eigen-decomposition of $\\Sigma_{1}+\\Sigma_{2}-\\Sigma_{3}$, where $\\Sigma_{1}=WXX^\\top W^\\top$, $\\Sigma_{2}=\\widehat{W}YX^\\top(XX^\\top)^{-1}XY^\\top\\widehat{W}^\\top$, and $\\Sigma_{3}=\\widehat{W}YX^\\top W^\\top + WXY^\\top\\widehat{W}^\\top$, with $Y=Q_a(X)$; these are computed from calibration covariances $XX^\\top$, $YY^\\top$, and $XY^\\top$. The $\\widehat{W}$ update is equivalent to quantizing $\\widetilde{W}=(W-UV^\\top)XY^\\top(YY^\\top)^{-1}$ with any standard layer-wise quantizer. The covariance coupling is what lets the correction target the actual activation error rather than a generic low-rank approximation.","core_discovery":"The discovery is that activation-quantization error, not weight-quantization error, is the bottleneck at W4A4, and that this error has a low-rank structure that can be corrected by full-precision factors. For each layer the method minimizes $\\|W X - \\widehat{W} Q_a(X) - U V^\\top X\\|_2^2$, where $W$ is the original weight, $X$ the calibration activations, $Q_a(X)$ their 4-bit quantized version, $\\widehat{W}$ the quantized weight, and $U,V$ the low-rank correction applied to unquantized $X$. The weight update is re-expressed as a standard layer-wise quantization problem on a transformed target, so any existing weight quantizer can be reused; the low-rank update has a closed form given by the top-$k$ eigenvectors of a matrix built from input and output covariances. Running this alternating scheme once or a few times yields, at 10% rank, more than a 50% reduction in the accuracy gap to the FP16 model and, at 30% rank, a completely closed gap on the evaluated tasks.","pith_inferences":["If the calibration covariances can be estimated cheaply at deployment time, LRC could be adapted to domain-specific activation statistics, potentially reducing the rank budget needed for a given accuracy target.","The closed-form eigenvector update suggests the activation error concentrates in a low-dimensional subspace; an extension would allocate rank adaptively per layer based on the eigenvalue spectrum of $\\Sigma$, rather than a flat 10% or 30% of every matrix.","The paper's own timing numbers imply that the low-rank path must be fused with or run in parallel with the 4-bit GEMM for the accuracy gains to translate into serving speedups; a fused kernel is the natural next engineering step.","Because weights-only quantization is already near-lossless, the remaining performance ceiling for W4A4 is the activation quantizer itself; improving on-the-fly activation rounding may matter more than larger correction ranks."],"forward_implications":["At a 10% rank budget, LRC recovers more than half of the accuracy lost when an LLM is quantized to W4A4, improving on the W4A4 baseline by several percentage points on average over six tasks.","At a 30% rank budget, LRC reaches essentially the original FP16 model's average accuracy on the evaluated tasks, both with and without activation group scaling.","The weight-quantization step is modular: LRC improves any layer-wise quantizer it is paired with, and the gain is largest when the underlying quantizer is simple, such as round-to-nearest.","When activations are left in full precision (W4 only), low-rank correction provides no additional accuracy, confirming that the correction is specifically repairing activation-quantization error.","A 10% rank overhead in FP16 adds roughly 13% to the stored model size, putting the effective bit width near 6.08 bits; the accuracy gain is bought with extra memory and forward-pass latency."],"supporting_citations":[{"why":"Supplies the GPTQ layer-wise quantizer used in the weight update of LRC; the algorithm is built on this subroutine.","marker":"Frantar et al. (2022)"},{"why":"The rotation-based preprocessing and W4A4 baseline that LRC is built on top of and compared against; it defines the reference gap being reduced.","marker":"Ashkboos et al. (2024)"},{"why":"Prior low-rank quantization-error reconstruction for quantized weights and activations; the paper compares against its SVD variant and distinguishes LRC by using activation statistics.","marker":"Zhang et al. (2024)"},{"why":"Prior low-rank correction with PCA on output errors, used as a baseline; LRC extends the idea to activation quantization via joint optimization.","marker":"Ou et al. (2024)"},{"why":"Prior joint formulation of quantized weights and low-rank terms, but for weight quantization only; LRC adapts this joint view to activation-quantization error.","marker":"Saha et al. (2024)"},{"why":"Provides the integer-arithmetic-only inference scheme that motivates on-the-fly activation quantization, the setting LRC assumes.","marker":"Jacob et al. (2018)"},{"why":"The evaluation harness defines the six downstream tasks whose average accuracy is the paper's main performance metric.","marker":"Gao et al. (2024)"},{"why":"Defines the layer-wise quadratic reconstruction problem that the LRC objective extends to quantized activations and low-rank corrections.","marker":"Nagel et al. (2020)"}],"fun_headline_variants":["Low-rank correction closes 4-bit LLM accuracy gap at 30% rank","4-bit LLMs match full precision with 30% low-rank add-on","Activation low-rank fix halves 4-bit LLM gap at 10% rank","Full precision for 4-bit LLMs via low-rank activation correction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the 128 text sequences used to calibrate the activation statistics represent the inputs the model will see on every downstream task; if the calibration distribution is unrepresentative, the low-rank correction can overfit and the reported accuracy gains will shrink.","fun_headline_variants_meta":{"raw":{"variants":["Low-rank correction closes 4-bit LLM accuracy gap at 30% rank","4-bit LLMs match full precision with 30% low-rank add-on","Activation low-rank fix halves 4-bit LLM gap at 10% rank","Full precision for 4-bit LLMs via low-rank activation correction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001391,"raw_usage":{"total_tokens":5646,"prompt_tokens":981,"completion_tokens":4665,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":4581}},"tokens_in":597,"tokens_out":4665,"duration_ms":28858,"temperature":1.0,"reasoning_tokens":4581,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:26:47.842287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Quantize an LLM with LRC at a 30% rank budget using a calibration set from one domain, then evaluate on tasks from a domain with very different activation statistics, such as code or non-English text; if the average accuracy gap to the FP16 model is substantially larger than the paper's near-lossless result, the calibration-representativeness assumption is falsified.","supporting_citations":[],"review_version":1}