{"id":"073bf197-477e-49f0-941d-9c858f57109b","arxiv_id":"2505.06653","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"BOF4-S with outlier preservation computes Lloyd-optimal 4-bit block-wise codebooks and beats NF4 and AF4 on quantization error and perplexity.","lead":"This paper derives better 4-bit codes for compressing large language model weights, then adds a smarter scaling rule and a way to protect outlier weights. The result is a small but consistent accuracy improvement over the existing NF4 and AF4 methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (59) in Appendix B.2.2 omits a required truncated-CDF term, so the claimed MAE-optimality of the BOF4(-S) codebooks is not established even under the paper's Gaussian model.","rationale":"The paper's central claim is that BOF4(-S) codebooks are the correct optimal levels for block-wise absmax quantization under MAE and MSE. The MSE derivation appears internally consistent, and Table 8 demonstrates numerical agreement between the theoretical and Monte-Carlo centroids for MSE. The MAE derivation, however, makes a concrete algebraic error: substituting the truncated CDF into Eq. (57) leaves a residual -F_X(ξ(l-1)|M=m) term that Eq. (59) drops. This is not a matter of interpretation; for a uniform source and region [0,1], Eq. (59) predicts a median of -0.5 rather than 0.5. Consequently, the theoretical MAE solution is wrong, the codebooks in Table 6 are not proven MAE-optimal, and the claimed practical equivalence between the theoretical and data-driven MAE solutions is unsupported (Table 8 only covers MSE). The reader's concern about the Gaussian-i.i.d. assumption is valid but distinct: even under perfect Gaussian weights, the MAE theoretical derivation fails. The empirical MSE results, especially BOF4-S (MSE) + OPQ, may still stand, but the paper's strong 'correct optimal' language for MAE must be retracted or fixed. This supports the reader's CONDITIONAL verdict, with the added condition that the MAE derivation be corrected or the MAE-optimality claims be removed.","tokens_in":27611,"tokens_out":28142,"duration_ms":254667,"concrete_test":"Independently re-derive Eq. (59) from Eq. (57) using the standard truncated-CDF identity, then recompute the BOF4-S (MAE) codebook for block size I=64 with the corrected centroid condition ∫ m pM(m)[F_X(x̂|M=m)-F_X(ξ(l-1)|M=m) - (1/2)(F_X(ξ(l)|M=m)-F_X(ξ(l-1)|M=m))] dm = 0, and compare each level to Table 6. If any level shifts by more than 1e-3, the published MAE codebook is not the claimed MAE-optimal one. A minimal 2-level check on a uniform [-1,1] source with region [0,1] already shows Eq. (59) yielding -0.5 instead of the correct median 0.5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central theoretical novelty is that BOF4(-S) codebooks are the correct MAE/MSE-optimal levels for block-wise absmax quantization. The MAE derivation in Appendix B.2.2 contains an algebraic error. Eq. (57) correctly states the median condition ∫ m pM(m|X∈R_l) F_X(x̂|M=m,X∈R_l) dm = 1/2 ∫ m pM(m|X∈R_l) dm. Substituting the truncated CDF F_X(x̂|M=m,X∈R_l) = (F_X(x̂|M=m)-F_X(ξ(l-1)|M=m)) / (F_X(ξ(l)|M=m)-F_X(ξ(l-1)|M=m)) and pM(m|X∈R_l) = pM(m) A / P[X∈R_l], with A = F_X(ξ(l)|M=m)-F_X(ξ(l-1)|M=m), yields ∫ m pM(m) [F_X(x̂|M=m)-F_X(ξ(l-1)|M=m) - A/2] dm = 0. The paper's Eq. (59) drops the -F_X(ξ(l-1)|M=m) term. This is not a notational variant: for a uniform source on [-1,1] and region [0,1], Eq. (59) gives x̂ = -0.5 instead of the correct median 0.5. Consequently, the MAE-optimized codebooks in Table 6 are not proven to minimize MAE even under the paper's Gaussian model, and the claimed equivalence between the theoretical and data-driven MAE solutions (the latter being the correct weighted median, Eqs. 8/69) is unverified; Table 8 covers MSE only. The practical perplexity results mostly rely on MSE-optimized BOF4-S and may survive, but the theoretical 'correct MAE-optimal' claim is unsupported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies 4-bit block-wise absmax quantization of LLM weights, as used in QLoRA-style fine-tuning. It proposes BOF4 and BOF4-S, codebooks obtained by a Lloyd-style EM algorithm with centroid rules that minimize the end-to-end MSE or MAE of the unnormalized weights, rather than the error of the normalized weights. It also proposes signed absmax normalization (BOF4-S), an outlier-preserving mixed-precision scheme (OPQ), and reports experiments on Llama, Qwen, and Mistral models showing lower quantization error and perplexity than NF4 and AF4. A theoretical solution and a Monte-Carlo empirical solution are both presented, with a numerical equivalence check for the MSE case.","tokens_in":28012,"tokens_out":7095,"duration_ms":58384,"significance":"The MSE-based BOF4-S results, if correct, would be a useful, reproducible contribution: the modified centroid derivation is nontrivial, the codebooks are published, the MSE theoretical/empirical agreement at -56 dB is a strong sanity check, and the perplexity improvements over NF4/AF4, while small, are consistent across several models. The OPQ mechanism is a practical addition for handling outliers without abandoning the Gaussian weight assumption. However, the paper's central theoretical novelty is weakened by an algebraic error in the MAE centroid derivation, which is one of the two optimality results claimed in the abstract. Since the error is local and the practical results are dominated by MSE optimization, the contribution is defensible after a substantial revision.","major_comments":[{"comment":"The derivation of the MAE centroid condition is algebraically incorrect. Starting from Eq. (57) and substituting pM(m|X∈R_l) = pM(m) A / P[X∈R_l] with A = FX(ξ(l)|M=m) - FX(ξ(l-1)|M=m), and the truncated CDF FX(x̂|M=m,X∈R_l) = (FX(x̂|M=m) - FX(ξ(l-1)|M=m))/A, one obtains ∫ m pM(m)[FX(x̂|M=m) - FX(ξ(l-1)|M=m) - A/2] dm = 0. Equation (59) omits the term -FX(ξ(l-1)|M=m). This is not a notational variant: for a uniform source on [-1,1] and region [0,1], Eq. (59) gives x̂ = -0.5, whereas the correct MAE-optimal level is the median 0.5. Consequently, the MAE-optimized codebooks in Table 6 are not proven to minimize MAE even under the paper's Gaussian model, and the claimed equivalence between the theoretical and data-driven MAE solutions (Eq. (8) vs. Eq. (59)) is unverified; Table 8 covers MSE only. The theoretical MAE derivation must be corrected or the MAE-optimality claims must be withdrawn.","section":"Appendix B.2.2, Eq. (59) and Section 3.2, Eq. (7)"},{"comment":"The codebooks are described as 'information-theoretically optimal' and 'the correct, information-theoretically optimal codebook'. Appendix B.2 itself states that the modified centroid criterion establishes only local optimality of a Lloyd-algorithm fixed point. Lloyd's algorithm is not guaranteed to find a global optimum, so the unqualified 'information-theoretically optimal' overstates the result. Please replace with 'locally optimal under the stated necessary conditions' or equivalent wording.","section":"Abstract, Section 1, and Section 3.2"}],"minor_comments":[{"comment":"The bracket notation in Eq. (7) contains a typographical artifact ('FX(ˆx(ℓ)|M =m)]− 1/2'); the intended expression appears to be FX(ˆx(ℓ)|M =m) − 1/2 [FX(x|M =m)]ξ(ℓ)ξ(ℓ−1), matching Eq. (59). Please correct the typesetting.","section":"Section 3.2, Eq. (7)"},{"comment":"The reported perplexity differences between methods are small (e.g., 8.43 vs. 8.53 for Llama-3.1 8B in Table 1), and the fine-tuning experiments in Tables 3 and 4 involve stochastic training runs without variance or multiple seeds. Please report standard deviations or multiple seeds for the fine-tuning evaluations to support the ranking claims.","section":"Tables 1, 3, 4 and Appendix G"},{"comment":"The numerical equivalence between the theoretical and empirical centroid solutions is demonstrated only for MSE. Given the MAE derivation issue, please provide an analogous comparison for MAE or explicitly state that equivalence was verified only for MSE.","section":"Appendix C, Table 8"},{"comment":"Please fix the typos: 'Insitute' should be 'Institute' (title page), 'qunaitzation' should be 'quantization' (Appendix F), 'V oronoi' should be 'Voronoi' (Section 3.2), and reference [22] should list 'AAAI' instead of 'AIII'.","section":"Title page, Appendix F, References"},{"comment":"The limitation discussion of double quantization is useful. Please also state explicitly that the OPQ threshold q was tuned on a validation split (Appendix E.2), which introduces a small data-dependent element, and briefly discuss how this interacts with the 'data-free' characterization of the method.","section":"Section 6, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The MAE derivation error is the main technical barrier to publication; it is a genuine algebraic mistake that affects one of the two claimed optimality results. The MSE-based results and the OPQ experiments appear sound and reproducible, and the codebook release is a plus. I believe a major revision, in which the MAE derivation is corrected or the MAE-optimality claims are removed, can bring the paper to an acceptable standard. The authors should also moderate the 'information-theoretically optimal' language throughout."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: BOF4-S (MSE) with OPQ is a genuinely useful drop-in improvement over NF4 and AF4 for data-free block-wise 4-bit quantization, and the practical perplexity results are believable. The thing to be careful about: the theoretical MAE-optimality claim is not actually proven. The stress-test note is correct. In Appendix B.2.2, Eq. (58) is fine, but when they substitute the truncated CDF into Eq. (59) they drop the -F_X(ξ(l−1)|M=m) term. That is not a cosmetic slip; for a uniform source on [-1,1] with region [0,1], Eq. (59) gives x̂ = -0.5 instead of the correct weighted median 0.5. So the MAE-optimized codebooks in Table 6 are not established as MAE-optimal even under the paper's own Gaussian model, and the claimed equivalence between the theoretical and data-driven MAE solutions is unverified (Table 8 covers MSE only).\n\nWhat is genuinely new and well done: the signed absmax normalization (BOF4-S) is simple and clearly reduces quantization error; the weighted centroid rules (m² for MSE, m for MAE) are the right way to think about block-wise normalization; and the outlier-preserving quantization (OPQ) is a practical, low-overhead fix for the outlier mismatch. The paper ships code and codebooks, which is exactly the kind of thing that makes this reproducible. The empirical MSE codebooks match the theoretical ones to -56 dB, and the perplexity improvements over NF4/AF4, though small (0.1–0.3 PPL), are consistent across Llama, Qwen, and Mistral. The fine-tuning results, especially for code generation, also look credible.\n\nSoft spots, in proportion: the MAE derivation bug is the largest, because the paper explicitly sells \"correct MAE-optimal\" as a contribution. The practical case mostly rests on MSE-optimized BOF4-S, so I do not think the main result collapses, but the MAE claims need to be walked back or re-derived. Also, the abstract's \"information-theoretically optimal\" is too strong. What they prove, even in the MSE case, is local optimality from Lloyd's algorithm, not global information-theoretic optimality. Perplexity differences are reported without error bars; some are tiny, and the benchmark accuracies bounce around enough that one should read the rank-order claims cautiously. The OPQ hyperparameter q is tuned on a validation split and then reported as the main configuration, which is minor but should be disclosed as such.\n\nWho this is for: anyone working on QLoRA-style fine-tuning or memory-efficient LLM inference who wants a better NF4. It deserves a serious referee and, after the MAE issue is fixed or removed, probably publication. I would bring it to the reading group and I would cite BOF4-S (MSE) for the signed-normalization idea.","headline":"Useful and mostly sound incremental improvement for block-wise 4-bit LLM quantization; the MSE side is solid, but the MAE-optimality derivation in Appendix B.2.2 has a real missing term and the 'information-theoretically optimal' label oversells it.","tokens_in":28562,"tokens_out":1961,"would_cite":true,"duration_ms":18970,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows NF4 and AF4 optimize the wrong objective and derives BOF4, 4-bit block-wise codebooks that minimize the true weight quantization error; signed and outlier-preserving variants reach the best data-free 4-bit perplexity.","keywords":["BOF4","block-wise quantization","4-bit quantization","large language models","Lloyd's algorithm","outlier-preserving quantization","QLoRA","post-training quantization"],"falsifier":"Run the paper's Monte-Carlo centroid rule on the actual per-layer weights of a large pretrained LLM, quantize with the resulting per-layer codebooks, and compare MAE, MSE, and perplexity against the fixed Gaussian-derived BOF4-S codebook at the same block size. If the data-driven codebooks do not measurably beat the Gaussian-derived ones, the distributional assumption is adequate and the claimed optimality stands; if they do beat it, the claim of optimality is limited by that assumption.","tokens_in":27417,"feed_emoji":"⚖️","tokens_out":13322,"duration_ms":106500,"temperature":0.7,"pith_summary":"Existing 4-bit block-wise quantizers for large language model weights, NF4 and AF4, use codebooks that minimize the quantization error of the normalized weights rather than the error of the actual network weights. This paper shows the two objectives differ: each normalized value is scaled back up by its block maximum $m$, so the true end-to-end squared error is $m^2(x-\\hat{x})^2$, and it derives the codebooks that minimize that true error. The result is a family of 4-bit quantizers, BOF4 and BOF4-S, that reduce MAE and MSE relative to both baselines, with the signed-normalization variant freeing a reconstruction level by fixing only one endpoint. Adding a mixed-precision scheme that stores outlier weights in 16-bit precision (OPQ) yields the best perplexity among data-free 4-bit block-wise quantizers on the tested models. Because these quantizers need no calibration data, a better codebook translates directly to cheaper memory-efficient fine-tuning and inference.","feed_headline":"BOF4 minimizes the LLM quantization error NF4 ignores","feed_subtitle":"Derived from the true weight error, BOF4-S plus outlier preservation beats NF4 and AF4 on perplexity.","key_machinery":"The load-bearing object is a modified Lloyd's algorithm for block-wise absmax quantization. In each maximization step, the reconstruction level of a Voronoi region is updated not to the plain mean or median of its normalized samples but to a weighted version: for MSE, $\\hat{x}(\\ell)=\\sum_k w_k^2 x_k \\big/ \\sum_k w_k^2$, where $x_k$ are the normalized weights falling in the region and $w_k$ is the block maximum of the block containing each sample; for MAE, the weighted median with weights $w_k$. The paper proves these centroid conditions by deriving them from the end-to-end quantization error, and shows that the theoretical integral solution and the Monte-Carlo solution agree to about $-56\\,\\text{dB}$ of relative squared error. BOF4-S rests on the companion mechanism of signed absmax normalization: choosing as the block constant the sign-carrying maximum weight means only the $+1$ endpoint is a mandated reconstruction level, so the level that would otherwise be spent on $-1$ can be re-placed to lower the error.","core_discovery":"The paper claims that the optimality story behind NF4 is wrong on two counts: equal utilization of the 16 reconstruction levels is not a valid criterion for rate-distortion optimality, and the AF4 correction still minimizes the error of the normalized weights, not the error of the weights that are actually quantized. In block-wise absmax quantization a weight is reconstructed as $m\\hat{x}$ from its normalized value $x$ and the block maximum $m$, so the end-to-end squared error is $m^2(x-\\hat{x})^2$; the correct MSE centroid of a Voronoi region is therefore the mean of its normalized samples weighted by squared block maxima, and the correct MAE centroid is the median weighted by block maxima. Feeding these modified centroid updates into Lloyd's algorithm yields the BOF4 codebooks, and changing the normalization—using the signed absolute maximum so that only the $+1$ endpoint is a fixed reconstruction level—yields BOF4-S, which reduces the error further. The paper further claims that outlier weights violate the Gaussian assumption on which the analysis rests, and its outlier-preserving quantization (OPQ) responds by storing those outliers in bfloat16, replacing them with zero before normalization and thereby improving perplexity, especially at large block sizes.","pith_inferences":["The $m^2$- and $m$-weighted centroid rules are not specific to LLM weights: any block-scaled scalar quantizer, including block-wise optimizer-state quantization, suffers the same objective mismatch, so re-deriving its codebooks with the same argument is a direct testable transfer.","Because the Monte-Carlo and theoretical codebooks coincide, the natural next experiment is to run the centroid rule on real per-layer weight samples; if per-layer codebooks outperform the fixed Gaussian-derived one, the 'optimal' label moves from one global codebook to a data-dependent family.","The OPQ threshold $q$ is fixed against a theoretical distribution of block maxima; treating it as a per-layer or per-tensor knob rather than a global hyperparameter is a testable way to trade memory against perplexity, especially for small models where OPQ showed the largest gains."],"forward_implications":["Data-free 4-bit block-wise quantization of LLM weights can be made lower in MAE and MSE than NF4 and AF4 at zero extra memory cost, since only the codebook values change.","Signed absmax normalization (BOF4-S) is a stand-alone accuracy gain that transfers to any block-wise quantizer, not just the codebooks derived here.","MSE-optimized codebooks generally give lower perplexity than MAE-optimized ones, making the error metric of codebook optimization itself a design choice with measurable language-modeling consequences.","Outlier-preserving quantization keeps perplexity low at larger block sizes, reducing the memory used by quantization constants.","QLoRA-style fine-tuning inherits the ranking: instruction-following and code-generation models fine-tuned on BOF4-based weights match or exceed those fine-tuned on NF4 or AF4 weights."],"supporting_citations":[{"why":"Introduces NF4 and the QLoRA fine-tuning setup; NF4 is the primary baseline whose information-theoretic optimality claim the paper refutes.","marker":"[5]"},{"why":"Proposes AF4, the codebook obtained by minimizing MAE of normalized weights; the paper extends the critique of NF4 to AF4 and compares BOF4 against it.","marker":"[7]"},{"why":"Lloyd's algorithm, the quantizer-design algorithm whose centroid update rules the paper modifies for block-wise absmax quantization.","marker":"[8]"},{"why":"Introduces block-wise absmax normalization for quantizing optimizer states, the normalization scheme this paper analyzes and modifies.","marker":"[9]"},{"why":"Rate-distortion theory, the framing used to argue that equal utilization of reconstruction levels is not a valid optimality criterion.","marker":"[11]"}],"fun_headline_variants":["BOF4: Optimal block-wise quantization for LLMs","Weighted centroids fix NF4's suboptimal quantization","Outlier-preserving BOF4-S beats NF4 and AF4 on perplexity","Theoretical grounding for better 4-bit LLM quantization","BOF4 leverages signed maxima to reduce quantization error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The codebooks are derived under the assumption that model weights are independent draws from a single zero-mean symmetric distribution, taken as Gaussian; real LLM weights have layer-dependent and outlier structure that violates this, and the paper's own OPQ mechanism exists precisely because of that failure.","fun_headline_variants_meta":{"raw":{"variants":["BOF4: Optimal block-wise quantization for LLMs","Weighted centroids fix NF4's suboptimal quantization","Outlier-preserving BOF4-S beats NF4 and AF4 on perplexity","Theoretical grounding for better 4-bit LLM quantization","BOF4 leverages signed maxima to reduce quantization error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000269,"raw_usage":{"total_tokens":1689,"prompt_tokens":1083,"completion_tokens":606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":520}},"tokens_in":699,"tokens_out":606,"duration_ms":6066,"temperature":1.0,"reasoning_tokens":520,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:36:47.017733+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's Monte-Carlo centroid rule on the actual per-layer weights of a large pretrained LLM, quantize with the resulting per-layer codebooks, and compare MAE, MSE, and perplexity against the fixed Gaussian-derived BOF4-S codebook at the same block size. If the data-driven codebooks do not measurably beat the Gaussian-derived ones, the distributional assumption is adequate and the claimed optimality stands; if they do beat it, the claim of optimality is limited by that assumption.","supporting_citations":[{"cited_title":"QLORA: Efficient Finetuning of Quantized LLMs,","cited_arxiv_id":null,"evidence_quote":"Introduces NF4 and the QLoRA fine-tuning setup; NF4 is the primary baseline whose information-theoretic optimality claim the paper refutes."},{"cited_title":"NF4 Isn’t Information Theoretically Optimal (and That’s Good),","cited_arxiv_id":null,"evidence_quote":"Proposes AF4, the codebook obtained by minimizing MAE of normalized weights; the paper extends the critique of NF4 to AF4 and compares BOF4 against it."},{"cited_title":"Least Squares Quantization in PCM,","cited_arxiv_id":null,"evidence_quote":"Lloyd's algorithm, the quantizer-design algorithm whose centroid update rules the paper modifies for block-wise absmax quantization."},{"cited_title":"8-bit Optimizers via Block-wise Quantization,","cited_arxiv_id":null,"evidence_quote":"Introduces block-wise absmax normalization for quantizing optimizer states, the normalization scheme this paper analyzes and modifies."},{"cited_title":"Berger, Rate-Distortion Theory","cited_arxiv_id":null,"evidence_quote":"Rate-distortion theory, the framing used to argue that equal utilization of reconstruction levels is not a valid optimality criterion."}],"review_version":1}