{"id":"a5b8397c-b632-439b-afd6-e70fca0a4e50","arxiv_id":"2506.01967","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Channel-wise scaling applied before Hadamard rotation reduces 4-bit layer-wise quantization error in LLaMA2-7B, especially on down-projection layers with massive activation outliers.","lead":"Large language models produce activation values with extreme outliers that wreck 4-bit quantization. This paper proposes smoothing those outliers first and then rotating the tensor, and shows this reduces layer-wise quantization error on LLaMA2-7B more than either step alone.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"In-sample smoothing scales are the main threat: Fig 4 reports quantization error on the same tokens used to compute s_j online, so the headline gains may not survive a held-out calibration split; Eq (9)'s d vs sqrt(d) normalization is a separate slip.","rationale":"The reader identified Eq (9)'s simplified model as the weakest assumption. I agree the derivation is fragile, and the d-vs-sqrt(d) normalization in Eq (9) is a concrete error that should be corrected; the phrase 'doubling the number of dimensions' does not follow from the token-level calculation. But I think the more load-bearing issue is that the empirical comparison in Fig 4 is in-sample with respect to the smoothing scales. Since smoothing is the only one of the four methods using data-dependent scales computed from the evaluated sample, the reported win over rotation could be an artifact of fitting the scale to the very massive outliers whose error is then reported. This is not an accusation of dishonesty: the paper discloses the online procedure and lists calibration dependence as a limitation, but the Sec VI conclusion drops that caveat. A held-out calibration/evaluation split would settle it. If the gain persists, the paper's central empirical claim stands and the Eq (9) issue is a fixable explanation bug; if it disappears, the conclusion should be weakened to 'in-sample layer-wise error reduction'. Since the reader already issued CONDITIONAL, that verdict remains appropriate: the method is plausible but needs held-out validation and a corrected equation before acceptance.","tokens_in":8580,"tokens_out":10861,"duration_ms":112502,"concrete_test":"Run the public code with a held-out calibration split: compute s_j on the first 64 tokens of the WikiText-2 sample, then quantize and measure layer-wise error on the remaining 64 tokens (and on an independent sample) for Original, Smooth, Rotation, and Smooth Rotation. If Smooth Rotation no longer gives the lowest error on down_proj 1/30/31, the Sec VI claim does not survive out-of-sample evaluation. In the same run, log the actual max(|~t|) for the maximum-magnitude token and compare it to Eq (9) with denominator d and with sqrt(d).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The Sec VI claim that Smooth Rotation 'significantly reduces quantization error' rests on Fig 4, but the scales s_j in Eq (4) are computed online from the exact sample being quantized (Sec III-C: 'calculating the scaling factor based on the current data'). For the massive-outlier token, this in-sample fit guarantees the smoothed outlier magnitude becomes sqrt(|o_i| max|W_i|), the best possible reduction; rotation-only has no such data-dependent adaptation, so the comparison is not method-vs-method on equal footing. The Sec V limitation 'depends on calibration' is therefore directly relevant, yet the Sec VI conclusion is stated without it. The mathematical explanation is not a safe fallback either: Eq (9) writes the smoothed-rotated maximum with denominator d, whereas Eq (8) for rotation alone has denominator sqrt(d); the correct smoothed-rotated max is sum_i sqrt(|o_i| max|W_i|)/sqrt(d) (plus noise). With the printed d, Eq (9) understates the maximum by a factor of sqrt(d) and cannot explain Fig 5(b); the 'doubling the number of dimensions' narrative is unsupported. Thus the mechanism is unverified and the empirical basis is in-sample.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies activation outliers in LLaMA2-7B and proposes \"Smooth Rotation,\" a hybrid preprocessing method that applies channel-wise scaling (SmoothQuant-style, alpha=0.5) before a Hadamard rotation. It introduces a \"quantization difficulty\" metric based on the standard deviation of channel magnitudes, reports layer-wise 4-bit quantization errors for original, smoothed, rotated, and smooth-rotated activations, and provides an analytical expression (Eq. (9)) intended to explain why smoothing before rotation reduces the maximum rotated activation magnitude. The main empirical claim is that Smooth Rotation gives the lowest layer-wise squared Frobenius quantization error, especially in down-projection layers that contain massive token-specific outliers.","tokens_in":8818,"tokens_out":6687,"duration_ms":69167,"significance":"If the claim holds, Smooth Rotation is a cheap and simple preprocessing step for W4A4 quantization, and the proposed quantization-difficulty metric could be a useful diagnostic tool. The paper also usefully documents the failure mode of rotation alone on massive-outlier layers. The strengths are the clear layer-wise experimental setup, the explicit connection to SmoothQuant and QuaRot, the promised code release, and the attempt to give a mathematical mechanism rather than only an empirical observation. However, the significance is currently limited by two load-bearing issues: the smoothing scales are computed in-sample, which makes the comparison to rotation unequal and not deployment-equivalent for pre-quantized weights, and Eq. (9) appears to be mis-normalized, undermining the theoretical narrative.","major_comments":[{"comment":"The normalization in Eq. (9) is inconsistent with Eqs. (7)-(8). For rotation alone, Eq. (8) gives max(|t_hat|) = (sum_{i in O} |o_i|)/sqrt(d) + |epsilon|. If smoothing replaces each |o_i| with sqrt(|o_i| max|W_i|), the same derivation yields max(|tilde t|) ≈ (sum_{i in O} sqrt(|o_i| max|W_i|))/sqrt(d), not the printed denominator d. As written, Eq. (9) is a factor of sqrt(d) too small, which appears to be the source of the \"doubling the number of dimensions\" claim. Please provide a corrected derivation or clarify why the denominator differs.","section":"Section IV-E, Eq. (9)"},{"comment":"The scaling factors s_j are computed online from the current data, whereas rotation is a fixed transform. Because the weight transform diag(s)W is tied to s, an online s_j would require re-quantizing W for every inference batch; the experiment therefore does not measure the method in the W4A4 deployment setting described in the Introduction. The paper should rerun the comparison with s_j fixed from a separate calibration set and report performance on distinct evaluation data, and should reconcile this with Section V's statement that the method \"depends on calibration.\"","section":"Section III-C and Fig. 4"},{"comment":"The derivation assumes outliers occupy a fixed small set O and all other coordinates are independent Gaussian noise, and it relies on the existence of a Hadamard column whose signs align with all outliers. Real activations exhibit correlated and token-varying outliers, and the first Sylvester-Hadamard column is not mean-zero, so the conditions for Eq. (8) should be verified empirically; for example, report how often an aligning column actually exists in the measured down-projection tokens. Without this check, the mathematical explanation is a hypothesis rather than a validated mechanism.","section":"Section IV-D, Eqs. (6)-(8)"}],"minor_comments":[{"comment":"The phrase \"By substituting the formula (3) with (4)\" is confusing; it should read \"substituting Eq. (4) into Eq. (3)\" or similar.","section":"Section IV-C"},{"comment":"The statement that the constructed Hadamard columns \"have a mean of 0\" is not correct for the first column of the Sylvester construction, which is all +1; please rephrase to \"all but the first column\" or explain why the first column does not affect the argument.","section":"Section III-D"},{"comment":"The reported correlation \"over 0.97\" is not accompanied by a numerical coefficient, a scatter plot, or a statement of the fitting procedure; please report the Pearson or Spearman r with and without the excluded layers and justify the exclusions.","section":"Section IV-B"},{"comment":"The limitations section appropriately restricts the practical recommendation to down-projection layers, but the Conclusion states the method \"consistently offers the lowest errors in other cases as well\" without repeating the calibration caveat; please align the two sections.","section":"Sections V and VI"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible central idea and a clean experimental skeleton, but the current evidence does not support the headline claim as stated. The in-sample computation of smoothing scales is the most serious issue because it breaks the equivalence to the W4A4 deployment scenario, and the Eq. (9) normalization error undermines the theoretical contribution. Both are fixable in a revision: a calibration-based evaluation and a corrected derivation could make the paper acceptable. The narrow scope (one model, one dataset, layer-wise error only) is acknowledged by the authors and is not by itself a reason for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core observation is simple and probably right: for the down-projection layers of LLaMA2-7B that have massive per-token outliers, applying SmoothQuant's channel scaling (alpha=0.5) before a Hadamard rotation gives lower 4-bit layer-wise squared-Frobenius error than either transform alone. That is a useful pointer for W4A4 quantization. The paper is also honestly written; Sec V tells you up front that it is one model, one WikiText-2 sample, no perplexity, and that the method depends on calibration.\n\nThe main soft spots are two. First, Eq (9) as printed has the wrong denominator: Eq (8) gives a sqrt(d) after rotation, and the smoothed version should inherit that sqrt(d) sum, not a bare d. So the mathematical explanation, including the 'doubling the dimensions' story, is not supported by the equations as written. It may be a typo, but the derivation is missing and the claim in Sec VI goes beyond what is established. Second, the scaling factor in Eq (4) is computed online from the exact sequence being quantized, so Fig 4 compares a method that adapts to the test sample against rotation with fixed statistics. That is an in-sample evaluation; the paper acknowledges the dependence on calibration in Sec V, but the conclusion states the benefit without that caveat. The 0.97 correlation is also computed after dropping four layers; the exclusions are explained, but it is fragile.\n\nThe bigger novelty problem is that the paper does not position itself against DuQuant, which already combines channel scaling with rotational transformations. The reader's uncertainty about prior-art overlap is justified. As it stands, the paper's contribution is a clean visualization and a specific recipe, not a demonstrably new mechanism.\n\nWho is this for? People working on activation quantization who want a quick empirical hint about ordering smoothing and rotation. It is not a systems paper and not a rigorous mechanism paper, but it is a legitimate small data point. I would send it to a serious referee, not desk reject it. The referee should demand: a corrected Eq (9) with derivation; a held-out calibration split or online-vs-calibrated comparison; a DuQuant comparison; and at least a perplexity number on a couple of models. With those, it could become a decent workshop or short-conference paper.","headline":"Scaling before Hadamard rotation is a sensible trick for 4-bit down-projection layers, but the in-sample evaluation and a normalization slip in Eq (9) leave the mechanism unproven, and the positioning against DuQuant is missing.","tokens_in":9371,"tokens_out":3636,"would_cite":false,"duration_ms":37296,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Applying channel-wise scaling before a Hadamard rotation (Smooth Rotation) lowers 4-bit layer-wise quantization error in LLaMA2-7B, with the largest gains on layers that have massive activation outliers.","keywords":["activation quantization","4-bit quantization","activation outliers","Hadamard rotation","channel-wise smoothing","quantization difficulty","LLM inference","weight-activation quantization"],"falsifier":"Record real activation tokens from a down-projection layer, identify channels whose values are far outside the noise floor, and compute the largest absolute value after smoothing plus Hadamard rotation for each token; if a substantial fraction of tokens have a maximum far above the Equation (9) estimate, then the paper's mathematical explanation is not what is driving the observed error reduction.","tokens_in":8343,"feed_emoji":"📉","tokens_out":8313,"duration_ms":82439,"temperature":0.7,"pith_summary":"The paper claims that a cheap preprocessing step—divide each activation channel by a scale computed from that channel's maximum weight magnitude, then apply a Hadamard rotation—makes LLaMA2-7B activations substantially easier to quantize to 4 bits without changing the model's learned weights. The authors call this Smooth Rotation and argue it gets the best of both existing fixes: channel-wise scaling flattens activations but makes weights harder to quantize, while rotation redistributes outliers but fails when a single token contains a few values far larger than the rest. They support the claim with a mathematical model of a token with massive outliers, plus layer-wise 4-bit quantization-error measurements across all modules. If the claim holds, it is a practical step toward W4A4 inference, where both weights and activations use 4-bit integers, with no additional training or codebook.","feed_headline":"Scale first, rotate second lowers 4-bit LLM quantization error","feed_subtitle":"Channel scaling plus a Hadamard rotation wins on layer-wise error, especially where massive activation outliers appear.","key_machinery":"The load-bearing object is the Smooth Rotation transform: first apply a diagonal scale $s_j = \\sqrt{\\max|X_j| / \\max|W_j|}$ to activations and its inverse to weights, then multiply both by the same Hadamard matrix $R$ whose entries are $\\pm 1/\\sqrt{d}$, so the linear map remains exactly unchanged. The argument runs through Equation (9): after smoothing, the largest entry of a rotated token that has massive outliers on a small set $\\mathcal{O}$ of channels is approximately $\\sum_{i \\in \\mathcal{O}} \\sqrt{|o_i|\\, \\max|W_i|} \\,/\\, d$, versus $\\sum_{i\\in\\mathcal{O}} |o_i| / \\sqrt{d}$ for rotation alone. It is that replacement of $|o_i|$ by $\\sqrt{|o_i|\\, \\max|W_i|}$ and the denominator changing from $\\sqrt{d}$ to $d$ that carries the paper's case that smoothing before rotation shrinks the quantization step size. A secondary device is the quantization difficulty metric, the standard deviation of channel magnitudes, which the paper uses to explain why rotation and smoothing behave differently on weights versus activations.","core_discovery":"On LLaMA2-7B with 4-bit symmetric round-to-nearest quantization, applying channel-wise scaling with smoothing strength $\\alpha = 0.5$ before a Hadamard rotation (Smooth Rotation) yields the lowest layer-wise squared Frobenius quantization error among the original model, scaling alone, rotation alone, and the hybrid, across nearly every module. The paper shows that rotation alone can be worse than no transformation on down-projection layers with massive outliers, because the rotated token's largest magnitude is approximately the sum of the outlier magnitudes divided by $\\sqrt{d}$. After smoothing, the largest magnitude becomes approximately $\\sum_{i\\in\\mathcal{O}} \\sqrt{|o_i|\\, \\max|W_i|} \\,/\\, d$, so the outlier contribution is both dampened by the weight magnitudes and spread over $d$ dimensions. The paper also introduces a quantization-difficulty metric—the standard deviation of channel magnitudes—and reports a correlation above 0.97 with layer-wise error once a few outlier-heavy layers are excluded. The authors are careful to claim Smooth Rotation is currently best justified on down-projection layers and recommend end-to-end perplexity tests for future work.","pith_inferences":["If Equation (9) is the real mechanism, the benefit of Smooth Rotation should be largest when each massive outlier $|o_i|$ is much larger than the corresponding $\\max|W_i|$; measuring the gain per layer against that ratio would test the explanation directly.","The paper measures only layer-wise error, not generative quality; a natural next experiment is W4A4 perplexity after Smooth Rotation, which the paper itself lists as future work.","The same smoothing-before-rotation pattern could plausibly help the value and query projections (which share the key projection's input tensor) or even weight-only quantization, though the paper does not claim this.","The sign-alignment assumption suggests that rotations with a deliberately balanced first column, or random orthogonal rotations, could behave differently from the Sylvester Hadamard matrix on the same layers; comparing rotations would reveal whether the benefit is specific to Hadamard alignment."],"forward_implications":["Smooth Rotation gives the lowest layer-wise quantization error among original, smoothing-only, rotation-only, and the hybrid across almost all LLaMA2-7B modules, with the largest margin on down-projection layers that contain massive outliers.","Because the transform is an exact reparameterization of each linear layer, it changes no model outputs before quantization; only a Hadamard application to activations and a rescaled weight matrix are added to inference.","The mechanism directly targets the quantization step size: after smoothing and rotation, the maximum absolute value that sets the 4-bit grid is roughly $\\sum \\sqrt{|o_i|\\max|W_i|}/d$ rather than $\\sum |o_i|/\\sqrt{d}$, shrinking the step in exactly the outlier-heavy layers where rotation alone fails.","The paper limits its current recommendation to down-projection layers; attention-output and gate projections may need a tuned $\\alpha$ above 0.5 before smoothing helps, as the experiments show for those modules.","The method uses fixed, online-computed scaling with $\\alpha = 0.5$ and no learned rotation, so the extra machinery is cheap and does not require quantizer training."],"supporting_citations":[{"why":"Supplies the LLaMA2-7B model whose activations are recorded for all layer-wise experiments.","marker":"[1]"},{"why":"Supplies the channel-wise scaling formula with smoothing strength 0.5 that the hybrid method builds on.","marker":"[9]"},{"why":"Identifies token-specific activation spikes in GLU-based LLMs, the massive-outlier phenomenon Smooth Rotation is designed to fix.","marker":"[10]"},{"why":"Supplies the Hadamard-rotation approach for outlier-free rotated LLM quantization that the paper compares against and extends.","marker":"[13]"},{"why":"Supplies the Hadamard matrix construction for the non-power-of-two dimension 11008 used by down-projection layers.","marker":"[18]"},{"why":"Supplies the flatness-based view of quantization difficulty that the paper adapts into its channel-magnitude standard deviation metric.","marker":"[21]"},{"why":"Supplies the WikiText-2 sample used to record activations for the experiments.","marker":"[22]"}],"fun_headline_variants":["Scale-then-rotate cuts 4-bit LLM quantization error","Smooth rotation: scaling before Hadamard wins for LLM quantization","Channel-magnitude spread predicts LLM quantization error","Scaling before rotation lowers LLM layer-wise quantization error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The derivation assumes that massive outliers sit in a small fixed set of channels while all other entries in a token are independent noise with one variance, and that some column of the Hadamard rotation has signs aligned with all of those outliers at once; if either fails on real activations, the stated proof of Smooth Rotation's benefit no longer goes through.","fun_headline_variants_meta":{"raw":{"variants":["Scale-then-rotate cuts 4-bit LLM quantization error","Smooth rotation: scaling before Hadamard wins for LLM quantization","Channel-magnitude spread predicts LLM quantization error","Scaling before rotation lowers LLM layer-wise quantization error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00091,"raw_usage":{"total_tokens":3879,"prompt_tokens":882,"completion_tokens":2997,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2926}},"tokens_in":498,"tokens_out":2997,"duration_ms":23761,"temperature":1.0,"reasoning_tokens":2926,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:26:55.268478+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record real activation tokens from a down-projection layer, identify channels whose values are far outside the noise floor, and compute the largest absolute value after smoothing plus Hadamard rotation for each token; if a substantial fraction of tokens have a maximum far above the Equation (9) estimate, then the paper's mathematical explanation is not what is driving the observed error reduction.","supporting_citations":[{"cited_title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaMA2-7B model whose activations are recorded for all layer-wise experiments."},{"cited_title":"SmoothQuant: Accurate and Efficient Post- Training Quantization for Large Language Models,","cited_arxiv_id":null,"evidence_quote":"Supplies the channel-wise scaling formula with smoothing strength 0.5 that the hybrid method builds on."},{"cited_title":"QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs","cited_arxiv_id":null,"evidence_quote":"Supplies the Hadamard-rotation approach for outlier-free rotated LLM quantization that the paper compares against and extends."},{"cited_title":"QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks,","cited_arxiv_id":null,"evidence_quote":"Supplies the Hadamard matrix construction for the non-power-of-two dimension 11008 used by down-projection layers."}],"review_version":1}