{"id":"a234ee0d-45b7-49d0-879d-5f64878311e1","arxiv_id":"2506.05664","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"BAQ assigns per-column bitwidths to LLM weights by solving a convex quantization-loss problem, and reports large 2-bit perplexity gains over GPTQ.","lead":"This paper derives a closed-form bit allocation rule for quantizing large language model weights, using a Hessian-based sensitivity proxy, and packages it as a drop-in module for GPTQ. It reports large perplexity improvements at 2-bit precision, though the evaluation compares only to GPTQ and relies on single runs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (2)'s high-resolution distortion model is the load-bearing premise; without direct validation at 2 bits, the claimed optimality of the allocation rule is unestablished, and Table 1's unmatched average bit budgets confound the empirical gains.","rationale":"The paper's mathematical derivation is internally sound: given the loss model in (OP-A), water-filling yields (6), the equal-loss property, and the G/A ratio. I do not see an error in the algebra. The vulnerable step is the mapping from real quantization to that loss model. Eq. (2) is presented as a 'high-resolution approximation' of mean distortion, but the operating point is 2-bit weight-only quantization, where the step size is a large fraction of the weight range and the within-cell distribution is not uniform. The paper's only support is an unreported simulation claim; the heuristic is plausible because column-level averaging can restore some uniformity of fractional offsets, but that needs demonstration. If the true per-column distortion scales as A_j·2^{-βR} with β ≠ 2, or with A_j not proportional to range^2, the optimal allocation changes and the claimed provable optimality is for the wrong objective. The reader's report already flagged this as the weakest assumption; I agree. The experimental tables add a separate but related issue: BAQ's realized average bitwidth is not matched to GPTQ's 2.00 for several models, and the headline 56× figure is not recoverable from Table 1. Those are presentation and validation problems rather than flaws in the optimization itself, so they support a conditional rather than a rejection verdict.","tokens_in":13080,"tokens_out":5265,"duration_ms":57843,"concrete_test":"Re-run a controlled column-level MSE experiment on OPT-2.7B: for 20 layers, quantize each column at R=1,2,3,4 with the same range rule used in BAQ, measure actual MSE, and fit log2(MSE) = a - b·R. Accept Eq. (2) only if b is within 0.2 of 2 and the intercept is within 2× of log2(range^2/12). Separately, rerun Table 1 with BAQ's realized average bits forced to exactly 2.00 per model (by adjusting Lref and clipping), report mean±std over three independent calibration draws, and recompute the maximum ratio; if either check fails, the central claims need revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (2) is the load-bearing premise. The closed-form allocation (6), equal-loss property (7), and geometric/arithmetic gain ratio (9) all follow from minimizing Σ c_ij·2^{-2R_ij}, where c_ij incorporates the high-resolution uniform-quantizer distortion (range^2/12)·2^{-2R}. At 2 bits, and with OPT/LLaMA weight distributions that are concentrated near zero and heavy-tailed, the Δ^2/12 law is not automatically valid; it is an asymptotic result for uniform densities and small step sizes. The paper asserts in Sec. 3 that simulations show it 'remains relevant even when the bitwidth is intermediate or low,' but no such validation is shown, so the provable optimality is conditional on an unverified model. A secondary empirical issue compounds this: Table 1 lists BAQ average bits of 2.05, 2.08, 1.92, 2.07, and 1.95, not the 2.00 used for GPTQ, so part of the perplexity gain may reflect a larger bit budget; the abstract's 'up to 56×' is also not supported by Table 1, whose largest ratio is about 35.6. These are addressable issues, so the conditional verdict is appropriate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BAQ, a mixed-precision weight-quantization framework for large language models. It models per-weight quantization loss by combining the GPTQ/OBS Hessian weighting with a high-resolution uniform-quantizer distortion formula (Eq. 2), then solves a convex bit-allocation problem (OP-A) to obtain a closed-form allocation rule (Eq. 6), an equal-loss property (Eq. 7), and a geometric-vs-arithmetic-mean loss ratio (Eq. 9). The practical algorithm assigns a shared bitwidth per column with an adjustable reference loss (Algorithms 1–3) and is demonstrated as a plugin for GPTQ on OPT-125M to OPT-30B and LLaMA2-13B, reporting large perplexity reductions over GPTQ at 2-bit weight-only quantization. Appendices include proofs, LLaMA results, and a study of integration with QuIP-style transformations.","tokens_in":13318,"tokens_out":7233,"duration_ms":67430,"significance":"If the central cost model and the empirical claims hold, BAQ offers a simple, theoretically grounded, and low-overhead way to turn a fixed-bit PTQ method into a mixed-precision one, with a closed-form solution that can be plugged into existing pipelines. The derivation of the allocation rule is a correct application of standard rate-distortion optimization, and the equal-loss interpretation is a useful design principle. The paper also provides a falsifiable diagnostic (Ratio_C) that connects the dispersion of sensitivity coefficients to the expected benefit of bit allocation. However, the current evidence is not conclusive: the cost model is asserted to be valid at 2 bits without dedicated validation, the main experiments compare only against uniform-bit GPTQ without matched bit budgets, and the overhead claim is contradicted by the reported runtime. These are addressable with additional experiments and analysis, so the work is a solid candidate for major revision.","major_comments":[{"comment":"The high-resolution uniform-quantizer distortion formula Δ²/12 is the load-bearing premise for the entire optimization: the closed-form allocation (Eq. 6), the equal-loss property (Eq. 7), and the gain ratio (Eq. 9) all follow from minimizing Σ c_ij 2^{-2R_ij}. The paper asserts that \"as all our simulations have shown, this approximation remains relevant even when the bitwidth is intermediate or low\" (Section 3), but no simulation, figure, or table validating Eq. (2) at 2 bits is provided. At 2 bits, with weight distributions that are concentrated near zero and heavy-tailed, the asymptotic uniform-density justification of Δ²/12 need not hold. Please add a direct validation: compare the predicted per-column loss Σ_i c_ij 2^{-2R} with the actual quantization loss for representative GPTQ-quantized OPT layers at R=2, and report the distribution of weights or the approximation error.","section":"Section 3, Eq. (2)"},{"comment":"The comparison is not at the same bit budget: BAQ's reported average bits are 2.05, 2.08, 1.92, 2.07, and 1.95 for the OPT models, whereas GPTQ uses exactly 2.00. For several models BAQ uses more bits on average, so part of the perplexity gain may be due to a larger budget rather than to the allocation rule. Please report results with exactly matched total bits (or per-average-bitwidth curves) and specify what \"at the same bitwidth\" means in the abstract. In addition, the abstract's \"up to 56× lower perplexity\" cannot be verified from Table 1 because the table formatting is broken (entries such as \"500.729582521\" and \"33.6452.7170.84\" are concatenated without separators). Please fix the table and state which model/dataset gives the 56× figure.","section":"Table 1 and Abstract"},{"comment":"The main experimental comparison is only against GPTQ with uniform 2-bit quantization. Since BAQ is a mixed-precision method, the reported improvements may largely reflect the transition from fixed-bit to mixed-bit allocation rather than the specific optimality of the BAQ rule. To support the claim that BAQ is broadly effective, please compare against existing mixed-precision or low-bit PTQ baselines under the same average bit budget, such as AWQ, SqueezeLLM, SpQR, and QuIP (the paper already contains QuIP integration results in Appendix C, but those are not in the main table). At minimum, state clearly that the main claim is about improving GPTQ, not about outperforming the current state of the art.","section":"Section 5, Main results"},{"comment":"The paper states in Section 4.5 that BAQ brings \"negligible additional computational complexity\" and \"minimal overhead,\" but Table 2 shows BAQ is about 1.54× slower than GPTQ on OPT-2.7B (797.07 s vs 517.08 s) and similar ratios on other models. A 54% slowdown is not negligible in the context of one-shot PTQ. Please either revise the overhead claim to specify that the overhead is small relative to inference or calibration, or provide a more careful accounting of where the extra time comes from (e.g., the per-column GPTQ passes at mixed bitwidths may be less efficient than a single uniform pass).","section":"Section 4.5 and Table 2"},{"comment":"The theoretical optimality is derived for the per-weight relaxed problem (OP-A), but the implemented algorithm allocates a shared bitwidth per column and rounds to integer bitwidths. The paper does not analyze how much loss is incurred by these two restrictions: the column-wise structure and the rounding step. At 2-bit precision, rounding to the nearest integer can change the average bitwidth by up to 0.5 bits per column, which is substantial relative to the budget, and the equal-loss property is broken after rounding. Please provide an analysis or empirical study of the rounding and column-structure losses; otherwise the \"provably optimal\" claim applies only to the relaxed per-weight problem, not to the algorithm that is actually evaluated.","section":"Section 4.2 and Algorithm 1"}],"minor_comments":[{"comment":"The quantities w_max_ij and w_min_ij are referred to as the \"maximum and minimum bounds of the quantizer Q_ij,\" but it is not specified how these bounds are estimated per weight (e.g., min/max over a calibration set, or per-column range). Please define them precisely.","section":"Eq. (2)"},{"comment":"The statement that λ is a normalization factor proportional to the geometric mean of {c_ij} is imprecise; the exact expression is given in Appendix A.1. Use a consistent notation for λ and λ′ to avoid confusion.","section":"Section 4.1"},{"comment":"The sentence \"BAQ reduces WikiText2 perplexity from 254.44 to 54.44\" appears to compare the C4 column (253) with the WikiText2 column (54.44) in the table; please align the text with the column headers.","section":"Table 3, Appendix B"},{"comment":"Reference [19] is listed as \"Thomas M Cover. Elements of information theory\" but the standard citation includes the co-author Joy A. Thomas; please correct the reference.","section":"References"},{"comment":"Figures 3 and 4 are mentioned in the text but are not visible in the provided manuscript text (Figure 3 appears to be missing entirely). Please ensure all figures are included and properly referenced in the final version.","section":"Figures"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically sound in its derivation, but the empirical support is currently confounded by unmatched bit budgets and a single baseline. The broken table formatting in the submitted text is severe and should be fixed before any acceptance decision. I would encourage the authors to add a validation of Eq. (2) at 2 bits, a matched-budget comparison, and a comparison with at least one other low-bit PTQ method. There are no apparent ethical or novelty concerns beyond the missing baseline breadth."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The practical idea is real: BAQ reallocates bits per column inside GPTQ using a Hessian-based sensitivity coefficient, and it does lower perplexity relative to uniform 2-bit GPTQ across OPT and LLaMA2. The theory, as the authors concede by citing Gersho and Gray, is textbook water-filling for a sum-of-exponential distortion; the new part is the column-level adaptation and the equal-loss heuristic that turns the closed form into a practical algorithm. Code is released, and the QuIP integration experiment is a nice touch—it honestly shows that incoherence transforms eat most of the gain, which is consistent with the geometric-vs-arithmetic mean story.\n\nThe soft spots are real and mostly addressable. Equation (2), the high-resolution quantizer distortion model, is load-bearing: the entire optimality claim depends on loss going as range^2/(12*2^{2R}). The paper asserts in Section 3 that simulations show it remains relevant at low bitwidths, but does not show those simulations. For concentrated, heavy-tailed weight distributions at 2 bits, that is not a trivial claim. Second, Table 1 shows BAQ with average bits of 2.05–2.08 against GPTQ's 2.00; that is a budget mismatch that can explain some of the perplexity gap. The abstract's 'up to 56×' does not match the table's largest ratio (about 35×). Third, the experimental section compares only to GPTQ; there are no error bars and no comparison to other mixed-precision methods or simple sensitivity heuristics. These are fixable: add a direct validation of Eq. (2), redo the bit-budget matching, expand the baselines, and report variance.\n\nWho should read it: anyone working on weight-only LLM quantization will find the algorithm useful and the equal-loss insight worth stealing. It is not a deep theoretical contribution, but it is a carefully engineered plug-in that likely helps in practice.\n\nMy recommendation: send it to peer review. The central mechanism is sound and the paper is honest about its own limitations; the open questions are empirical, not fatal. I would not desk-reject it.","headline":"Textbook water-filling cleverly repackaged for LLM quantization, with real empirical promise but a load-bearing approximation that needs validation and a bit-budget mismatch in the main table.","tokens_in":13872,"tokens_out":2319,"would_cite":true,"duration_ms":21901,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that allocating bits by the closed-form rule $R^*_{ij}=\\frac{1}{2}\\log_2(c_{ij}/G)+R_{\\text{sum}}/(MN)$ provably minimizes loss under a Hessian-weighted model, and that per-column implementation in GPTQ lowers perplexity…","keywords":["post-training quantization","bit allocation","LLM compression","Hessian sensitivity","equal-loss principle","mixed-precision quantization","GPTQ","convex optimization"],"falsifier":"Take a pretrained OPT layer, quantize each column at 2 bits, and compare the measured contribution to the loss with the model's prediction $C_j2^{-2R_j}$. If the predicted ordering of columns disagrees with the measured losses at low bitwidths, the optimality result no longer explains the reported gains.","tokens_in":12861,"feed_emoji":"🎯","tokens_out":5902,"duration_ms":57746,"temperature":0.7,"pith_summary":"The paper tries to establish that the right way to spend a fixed quantization bit budget across a language model's weights is to allocate precision according to each weight's sensitivity, quantified through a Hessian proxy. It models per-weight quantization loss with the standard high-resolution uniform-quantizer distortion formula, turns bit allocation into a convex optimization problem, and solves it in closed form. The optimal solution obeys an equal-loss principle, and the paper packages it as a lightweight algorithm, BAQ, that drops into GPTQ-style pipelines by assigning one bitwidth per column. If the argument is right, it explains why uniform 2-bit quantization fails and gives a principled replacement that improves perplexity at the same bit budget.","feed_headline":"Closed-form bit rule cuts 2-bit LLM perplexity by up to 56x","feed_subtitle":"Hessian-informed allocation beats uniform GPTQ bitwidths across OPT models from 125M to 30B, at the same 2-bit budget.","key_machinery":"The load-bearing object is the sensitivity coefficient $c_{ij}=\\frac{(w^{\\max}_{ij}-w^{\\min}_{ij})^2}{12\\,[H^{-1}_F]_{n_{ij}n_{ij}}}$, which combines the quantizer's step-size range with the diagonal of the inverse Hessian proxy. The paper feeds these coefficients into the convex problem (OP-A), $\\min\\sum_{i,j}c_{ij}2^{-2R_{ij}}$ subject to $\\sum_{i,j}R_{ij}\\le R_{\\text{sum}}$, whose closed-form solution carries the entire argument: it produces the bit-allocation rule, the equal-loss identity, and the geometric-mean-over-arithmetic-mean loss ratio. The algorithm then approximates this per-weight rule at column granularity by defining column coefficients $C_j=\\sum_i c_{ij}$ and setting each column's bitwidth from $C_j2^{-2R_j}=L_{\\text{ref}}$.","core_discovery":"The central discovery is that, under a Hessian-weighted squared-error loss model, the optimal bitwidth for each weight is not uniform but is $R^*_{ij}=\\frac{1}{2}\\log_2(c_{ij}/G)+\\frac{R_{\\text{sum}}}{MN}$, where $c_{ij}$ is a sensitivity coefficient built from the weight range and the inverse-Hessian diagonal, and $G$ is the geometric mean of all $c_{ij}$. This solution makes every term $c_{ij}2^{-2R^*_{ij}}$ equal, which the paper calls the equal-loss principle, and it implies that the ratio of optimal-allocation loss to uniform-allocation loss is the geometric mean divided by the arithmetic mean of the $c_{ij}$ values. The paper therefore claims that the benefit of adaptive bit allocation grows exactly with the dispersion of Hessian-derived sensitivities, and it reports consistent perplexity and accuracy gains when BAQ replaces GPTQ's uniform bitwidths on OPT models from 125M to 30B parameters.","pith_inferences":["Editorial inference: the geometric-to-arithmetic ratio of column sensitivities could be used before quantization as a cheap predictor of whether mixed precision will pay off, a diagnostic the paper illustrates but does not turn into a standalone tool.","Editorial inference: the equal-loss property is proved for continuous bitwidths, while the algorithm rounds to integers; a dedicated analysis of rounding tie-breaking rules could show whether near-optimality is preserved and how much precision is lost.","Editorial inference: BAQ's gains should concentrate in layers where the Hessian's eigenvalue spread is largest, so per-layer Ratio_C could be used to choose which layers to quantize adaptively and which to leave at uniform precision."],"forward_implications":["GPTQ-type pipelines can replace fixed bitwidths with closed-form per-column bitwidths at negligible overhead, improving perplexity without changing the total bit budget.","The equal-loss principle gives a diagnostic: layers with a low geometric-to-arithmetic mean ratio of column sensitivities are exactly where adaptive bit allocation buys the most.","The gain over uniform allocation is predicted to grow with the spread of Hessian diagonal entries and weight ranges, so models with more heterogeneous sensitivities should benefit more.","Because the loss model is written as $c_{ij}2^{-2R_{ij}}$, the allocation rule generalizes to any compression method whose per-component loss can be expressed that way, including pruning or low-rank factor precision.","Transformations such as QuIP's incoherence processing homogenize the sensitivity coefficients, which shrinks the benefit of adaptive allocation; BAQ's role is therefore complementary to such preprocessing."],"supporting_citations":[{"why":"Supplies the GPTQ quantization pipeline and the inverse-Hessian diagonal that BAQ reuses for its sensitivity coefficients.","marker":"[3]"},{"why":"Documents that quantization noise affects some OPT weights more than others, motivating sensitivity-aware bit allocation.","marker":"[5]"},{"why":"Provides the second-order Optimal Brain Surgeon loss justification that motivates the Hessian-weighted per-weight loss in Eq. (1).","marker":"[13]"},{"why":"Gives the high-resolution uniform-quantizer distortion formula $\\Delta^2/12$ used in Eq. (2).","marker":"[18]"},{"why":"Supplies the information-theoretic background for the distortion-rate relation $2^{-2R}$ used in the optimization objective.","marker":"[19]"},{"why":"Cited to support the claim that the high-resolution distortion approximation remains relevant at intermediate and low bitrates.","marker":"[20]"},{"why":"Provides the classic convex bit-allocation result that yields the closed-form solution in Eq. (6).","marker":"[21]"},{"why":"Supplies the OPT model family used for all main experiments.","marker":"[22]"}],"fun_headline_variants":["Hessian-based bit allocation cuts LLM perplexity by 56x","BAQ: Closed-form bit rule for LLM quantization beats GPTQ","Adaptive bitwidths equalize loss, slashing LLM perplexity 56x","Equal-loss bit allocation gives 56x perplexity win over GPTQ","Hessian-driven bit allocation lowers LLM perplexity up to 56x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the standard shortcut that each weight's quantization error equals its quantization step squared divided by 12, an approximation the paper asserts but does not separately validate for 2-bit or non-uniform weight distributions.","fun_headline_variants_meta":{"raw":{"variants":["Hessian-based bit allocation cuts LLM perplexity by 56x","BAQ: Closed-form bit rule for LLM quantization beats GPTQ","Adaptive bitwidths equalize loss, slashing LLM perplexity 56x","Equal-loss bit allocation gives 56x perplexity win over GPTQ","Hessian-driven bit allocation lowers LLM perplexity up to 56x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00056,"raw_usage":{"total_tokens":2705,"prompt_tokens":1033,"completion_tokens":1672,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":1569}},"tokens_in":649,"tokens_out":1672,"duration_ms":10503,"temperature":1.0,"reasoning_tokens":1569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:13:44.611787+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a pretrained OPT layer, quantize each column at 2 bits, and compare the measured contribution to the loss with the model's prediction $C_j2^{-2R_j}$. If the predicted ordering of columns disagrees with the measured losses at low bitwidths, the optimality result no longer explains the reported gains.","supporting_citations":[{"cited_title":"QuIP: 2-bit quantization of large language models with guarantees","cited_arxiv_id":null,"evidence_quote":"Documents that quantization noise affects some OPT weights more than others, motivating sensitivity-aware bit allocation."},{"cited_title":"Optimal brain surgeon and general network pruning","cited_arxiv_id":null,"evidence_quote":"Provides the second-order Optimal Brain Surgeon loss justification that motivates the Hessian-weighted per-weight loss in Eq. (1)."},{"cited_title":"Gray and David L","cited_arxiv_id":null,"evidence_quote":"Gives the high-resolution uniform-quantizer distortion formula $\\Delta^2/12$ used in Eq. (2)."},{"cited_title":"Jpeg2000: Image compression fundamentals, standards and practice.Journal of Electronic Imaging, 11(2):286–287, 2002","cited_arxiv_id":null,"evidence_quote":"Cited to support the claim that the high-resolution distortion approximation remains relevant at intermediate and low bitrates."}],"review_version":1}