Pith. sign in

REVIEW 3 major objections 4 minor 14 references

Which Decisions Low-Bit Quantization Breaks, and How to Predict Them

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Low-bit quantization damages LLM decisions by shrinking margins proportionally, making decision flips predictable from a measured linear channel.

desk verdict A careful empirical case that quantization shrinks decision margins multiplicatively, with an honest boundary on where the instrument stops measuring; worth a serious referee, but the headline 3-bit numbers need re-checking on the anchored subset. read the letter →

arxiv 2608.06564 v2 pith:BHOC4LF2 submitted 2026-08-06 cs.LG cs.CL

Zekun Wu , Swati Dhiman , Adriano Koshiyama This is my paper · ORCID
classification cs.LGcs.CL
keywords post-trainingquantizationlow-bitLLMdecisionmarginsmarginshrinkagetoolcallingsafetyrefusalflippredictionmodelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that low-bit quantization damages a language model's decisions by shrinking the margin between the chosen option and its best alternative proportionally, not by adding roughly fixed noise. It reports that the surviving fraction of the margin, $c$ in $m' = c\cdot m + b + \varepsilon$, falls from a median 0.86 at 4 bits to 0.33 at 3 bits and 0.00 at 2 bits, and that this per-cell linear channel predicts held-out decision-flip rates to a median of 1.8 percentage points. The account explains why a compressed agent stops calling tools and loses safety refusals while benchmark scores barely move: individual margins cross zero item by item while averages hide the losses. If correct, the result turns decision-level damage from something measured after the fact into something computable from a small set of paired margins measured per model and bit-width.

What carries the argument

The central object is the paired decision margin: for each binary decision, the logit of the chosen first token minus the logit of the full-precision model's best alternative, measured before and after quantization. The law $m' = c\cdot m + b + \varepsilon$, with $c$ the surviving fraction (margin shrinkage), $b$ a decision-family-specific push, and $\sigma$ the leftover noise, carries the argument because its conditional reading forces the flip-probability formula $P(\text{flip}|m) = \Phi\left(-(c m + b)\cdot \operatorname{sign}(m)/\sigma\right)$, turning a curve fit into a predictive channel with no parameter fitted to flips. The comparison against additive competitors and a parameter-free bound on error accumulation establish that the collapse is shrinkage, not noise accumulation.

What would settle it

Measure the quantized-versus-full-precision margin pairs in a held-out damaged core family (whether-to-call or safety) at 3 bits; if the slope $c$ is statistically indistinguishable from 1 while the flip rate is substantial, or if a constant-noise additive account wins the BIC comparison by more than 10 points, the multiplicative-shrinkage law fails.

Watch

Extended reading notes

Core claim

The central discovery is that the effect of post-training quantization on a decision is a multiplicative contraction of the decision margin plus a decision-family-specific push, not an additive perturbation of roughly constant size. For each two-way decision, the margin is the logit difference between the chosen continuation and its best alternative; after quantization the margin obeys $m' = c\cdot m + b + \varepsilon$ with $c$ shrinking toward zero as bit-width drops, $b$ carrying the directional bias of a decision family, and $\varepsilon$ the residual noise. The paper shows this law fits damaged tool-calling and safety-decision cells across 16 models and three quantization methods, with the multiplicative account beating additive competitors in every damaged core-family cell (whether-to-call and the two safety instruments), while one non-core family, tool-result use, prefers a margin-scaled-noise account, a boundary the paper treats as a predicted failure. Read conditionally, the law yields a flip-probability formula that matches held-out flip rates to a median of 1.8 points with no flip used in the fit. The paper is explicit that this selects the best description among the stated alternatives, not a proof of the generative mechanism.

Load-bearing premise

The measurement assumes that the first-token logit margin between two designated continuations is the decision the model would actually make; the paper's own full-vocabulary checks show this anchor weakens from 85% agreement at 4 bits to 30% at 3 bits and zero at 2 bits.

Editorial extensions

If this is right

  • Aggregate benchmark scores understate decision damage: a model's flip rate can grow tenfold while benchmark scores drift by a few points, so evaluation must measure decisions, not only scores.
  • A small paired-margin calibration set measured per model and bit-width is sufficient to predict which decisions flip; borrowed constants from another model fail by 18\u201333 points at 3 bits, so the measurement cannot be skipped.
  • Importance-based weight protection and uniform rescaling (temperature, global bias) cannot repair the damage; no tested repair beats the baseline of spending one more bit, which recovers a median 0.305 of flipped decisions.
  • The invocation\u2013selection split means agents stop acting rather than acting wrongly at the collapse bit-width; system design should treat 'should I act' decisions as the fragile ones.
  • The fit-free bound rules out independent error accumulation: no member of that class can lose more than half the margin per bit, yet 107 of 183 measured steps violate the bound, so the collapse is not noise accumulation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editor's inference: if margin shrinkage is the right model, then any repair that raises the surviving fraction $c$ (rotations, activation-aware quantizers) buys roughly the same number of bits; the paper's measured two-bit gain from a Hadamard rotation suggests a general design rule: move the collapse below the deployment bit-width rather than trying to protect individual weights.
  • Editor's inference: the law could transfer to other two-way decisions beyond tools and safety, such as stopping versus continuing generation or choosing between two response formats, wherever a first-token margin is measurable; the paper does not test this, but its instrument is general.
  • Editor's inference: because the paper shows the margin at 2 bits is near-uniform over the vocabulary, per-decision predictions there describe conditional preference under constrained decoding, not free generation; users of grammar-constrained decoding may still get accurate flip predictions, but unconstrained deployments should not extrapolate the 2-bit row.
  • Editor's inference: the paper's law, if replicated, changes the target of quantization research from preserving logits to preserving margin order; one testable extension is whether gradual mixed-bit-width allocation can keep the slope $c$ near 1 for the decisions that matter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes that low-bit quantization damages LLM decisions through proportional margin shrinkage rather than additive noise. For each binary decision, the authors measure the first-token logit margin between the chosen continuation and its best alternative at full precision and after quantization, fitting the per-cell relation m' = c*m + b + ε. They report that c collapses with bit-width (median 0.86 at 4 bits, 0.33 at 3, 0.00 at 2), that a BIC comparison rejects additive accounts in the damaged tool-call and safety families, that a parameter-free bound rules out independent error accumulation in 107 of 183 adjacent-bit steps, and that the implied flip probability predicts held-out flip rates to a median of 1.8 percentage points. They also document a one-directional collapse of the whether-to-call decision, a split between invocation and selection, and a repair comparison in which no tested intervention beats spending one more bit. The paper is unusually explicit about its limitations, including the point that at 3 and 2 bits the two designated tokens are often no longer the model's top choices over the full vocabulary.

Significance. If the margin-shrinkage law holds for free model behavior, the paper is a substantial advance: it overturns the fixed-noise assumption underlying several prior accounts, explains why confident decisions are not safe, ties agentic failures to a decision-level quantity invisible to benchmarks, and provides a calibrated per-decision flip probability. The strengths include a large and systematic measurement matrix (16 models, three methods, bit-widths down to 2), a model comparison that is run per family rather than pooled, a genuinely parameter-free ceiling on independent error accumulation, explicit candid limitations, and a reproducibility statement with checked-in scripts and verified numeric literals. The significance is conditional: the central quantitative content at 3 bits rests on the validity of the forced-choice margin as a proxy for the model's free decision, and the paper's own full-vocabulary checks show that proxy fails for most tool items at that bit-width.

major comments (3)
  1. [§2, Limitations, Appendix Table 8] The 3-bit regime carries the paper's central quantitative claims: median c=0.33, the one-directional collapse of whether-to-call in Table 2, and the 1.8pp held-out forecast error. Yet at 3 bits the instrument's behavioral anchor fails for most tool items: the two designated first tokens are the quantized model's argmax in only 30% of tool items (median vocabulary rank about 3,500), and at 2 bits in none. The headline c values, flip rates, and forecast errors are computed on all items, not on the anchored subset. Appendix Table 8 shows the §3.2 model comparison is unanimous in the anchored strata, but it does not report c or forecast accuracy restricted to those cells. The paper's own free-generation check in §5.2 shows the margin can be slower to change than emitted behavior (lost refusals deflect rather than comply), so the 3-bit collapse may characterize the model's conditional preference under constrained decoding rather than its free decision. Please report the central quantities on the anchored subset at each bit-width, or explicitly reframe the 3-bit claims as applying to grammar-constrained decisions rather than to free generation.
  2. [§4.2, Eq. (2)] The held-out flip forecast uses per-cell constants (c, b, σ) fitted on one half of the same cell's margins, and Eq. (2) is derived from that fitted Gaussian channel. The median 1.8pp error is therefore a consistency check of the conditional-normal assumption, not evidence that flip rates can be predicted for an unmeasured condition. The manuscript acknowledges this in §4.2 ('this tests the law's shape, not whether its constants transfer'), but the abstract and Contribution 2 present it as prediction. The only transfer test, Appendix Figure 8, shows 18–33pp errors at 3 bits, which is the regime where prediction matters most. Please reword the predictive claim so the abstract distinguishes in-sample shape validation from cross-condition transfer, and make the transfer error the headline for practical use.
  3. [§3.3, Figure 2] The fit-free bound is a strong and valuable result, but the text compresses its domain: the ceiling 'cannot lose more than half the margin per bit' is derived for the specific class of independent, variance-accumulating, renormalized layer errors. The paper correctly notes that signal-correlated errors and bit-dependent scale parameters sit outside the class, but the main-text phrasing 'excludes the class without fitting any member of it' may overstate what is excluded, since the class is defined by the very independence assumption being tested. I recommend stating explicitly in §3.3 that the bound rules out only the independent-accumulation family, and that the 107-of-183 count is a rejection of that family rather than a proof of the specific multiplicative form.
minor comments (4)
  1. [§3.3] The displayed equation 'c(B) = (1+Aγ −B)−1/2' seems to have lost its superscript formatting; please render it as c(B) = (1 + A γ^{-B})^{-1/2} and define γ immediately before use.
  2. [Figure 2 caption] The caption says 'no error-accumulation model can lose more than half the margin per bit,' but the bound is on the log-derivative and implies at most a factor-of-two decrease per bit. The wording should say 'more than a factor of two per bit' to avoid a literal reading of 'half the margin' as exactly 50 percentage points.
  3. [§5.1] The sentence 'the two sub-2B models flip in inconsistent directions' is vague; please specify which two models and what direction each exhibits, since the invocation–selection asymmetry claim depends on where it does and does not appear.
  4. [Reproducibility statement] The statement says code, test sets, and the full result matrix 'will be released' but provides no link or artifact identifier; for a review process, please include a URL or an anonymized artifact link.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central derivation is a fitted conditional model tested on held-out margins, with transferability explicitly measured and reported as failing.

full rationale

The paper's core law, m' = c·m + b + ε, is estimated from paired before/after margins, and the flip-probability formula Eq. (2) is a mathematical corollary of the Gaussian conditional assumption, not a restatement of the data used to fit it. The held-out validation splits each cell's decisions and compares predicted flip rates against observed flips that were not used as fitting labels; this is a genuine test of the conditional model's shape, even though the constants are per-cell. The paper explicitly disclaims transferability of constants, and indeed reports that borrowed constants are wrong by 18–33 percentage points at 3 bits, which is evidence against dressing fitted values as universal predictions. No load-bearing self-citation appears; the cited prior work is external, and the rotation bit-shift prediction and the fit-free bound of §3.3 are parameter-free from measured quantities. The acknowledged failure of the two-token anchor at 3 and 2 bits is a measurement-validity limitation, disclosed candidly in the Limitations section, not a circular derivation. Overall, the derivation chain is self-contained and honestly bounded.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a per-cell fitted linear channel and on the first-token margin proxy. The Gaussian assumption is the main unproved input, and the paper itself measures where it fails. No new physical entities are introduced; the named concepts (margin shrinkage, core families) are summary labels for fitted quantities.

free parameters (6)
  • per-cell slope c = 0.86 at 4 bits, 0.33 at 3 bits, 0.00 at 2 bits (medians)
    Fitted by least squares in Eq. 1 for each (result file, decision family, bit-width) cell; the central shrinkage parameter that collapses with bit-width.
  • per-cell intercept b = varies; example -2.9 and -13.5 for Qwen3-4B whether-to-call
    Family-specific push that sets the direction of flips; estimated in the same cell-wise fit.
  • per-cell residual sigma = varies per cell; not tabulated globally
    Noise scale in Eq. 1; required for the flip probability formula Eq. 2.
  • damage cutoff = 0.70
    Threshold defining 'damaged' cells in the model comparison; the paper shows the verdict is robust from 0.50 to 0.90, so it is a reporting choice.
  • c-difference floor = 0.04
    Interpretability floor derived from replicate spread; used to judge when c differences are meaningful.
  • down_proj range quantile = 90th percentile
    Quantile of the MLP down-projection range used in the rotation bit-shift predictor; the paper states the median would predict 0.96 bits, so this choice is selected to match the mechanism.
assumptions (5)
  • domain assumption m' | m ~ N(c m + b, sigma^2) with c, b, sigma constant in m
    Assumed Gaussian conditional channel underlying Eq. 2; justified by central-limit intuition and partially tested; constant variance is rejected in tool-result use (93% of cells) and code tokens (58%).
  • domain assumption First-token margin between designated continuations proxies the decision
    Used in every measurement; the paper's full-vocabulary check shows argmax agreement 85% at 4 bits, 30% at 3 bits, 0% at 2 bits, so the proxy has a measured validity boundary.
  • domain assumption Independent per-layer error accumulation with renormalization for the bound
    The §3.3 bound assumes variance-additive independent errors with scale parameter gamma=4; signal-correlated errors or bit-dependent scale sit outside the class.
  • standard math BIC model comparison with conventional decisive threshold
    Used to select among A0, A1, A2, and M; BIC differences above 10 treated as decisive, a standard convention.
  • domain assumption Alternative fixed to the full-precision model's highest-scoring wrong option
    Defines the paired margin; the quantized model might have a different top alternative, so the fixed alternative may not be the actual runner-up after quantization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Which Decisions Low-Bit Quantization Breaks, and How to Predict Them." pith.science (2026). https://pith.science/paper/BHOC4LF2

@misc{pith2026260806564,
  author       = {Pith},
  title        = {Pith review of: Which Decisions Low-Bit Quantization Breaks, and How to Predict Them},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BHOC4LF2}},
  note         = {Machine review of arXiv:2608.06564}
}
read the original abstract

Quantization is how large language models are actually deployed, and below four bits it hurts. What nobody can say is which decisions change at a given bit-width -- which matters most where a model acts rather than answers, since a tool call it declines to make is a failure no score reports. A compressed agent stops calling its tools, then loses half its safety refusals, while benchmark scores barely move. Prior work assumes the added noise has a roughly fixed size, which would make confident decisions safe. We measure the decision instead: the margin between the option a model picks and its best alternative, before and after quantization, across 16 models, three methods, and 8 down to 2 bits. Kinds of decision do not break together -- at 3 bits the decision to call a tool collapses toward inaction while the choice of which tool is untouched -- and the damage is proportional rather than fixed, the margin multiplied by a factor that collapses with bit-width (median 0.86 at 4 bits, 0.33 at 3, 0.00 at 2). Fitted against additive competitors, including one whose noise grows with the margin, no account with an additive mean wins a damaged tool or safety cell; that is the best description among those stated, not a proof of generative form. Given a condition's own constants the relation predicts flip rates on its held-out decisions to a median of 1.8 points, with calibrated per-decision probabilities (calibration error 0.004 over 131,758 predictions), and no flip was used in the fit. Borrowed constants are wrong by 18-33 points at 3 bits, so a small paired margin set measured per model is the instrument, not a way to skip measuring. It is anchored to behaviour where used: at 4 bits the most likely token over the vocabulary is one of the two options in 85% of tool items, and the 2-bit floor is where the instrument stops measuring. Nothing repairs the damage more cheaply than one more bit.

Figures

Figures reproduced from arXiv: 2608.06564 by the authors.

Figure 1
Figure 1. Quantized against full-precision margins (Qwen3-4B, whether-to-call, round-to-nearest). Additive noise predicts a constant-width band around the identity; the data is a fan through the origin whose slope collapses with bit-width. 2 Measurement setup The margin: Each item presents the model with a choice between two continuations; the margin is the difference between the logits of the two choices’ first tokens. One r… view at source ↗
Figure 2
Figure 2. A parameter-free upper bound: no error￾accumulation model can lose more than half the margin per bit. 107 of 183 measured steps sit above it, none where damage is mild. right. Appendix [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. One model’s released GGUF builds, end to end: (a) benchmark scores barely move; (b) the decision margins underneath (lowest-bit build against the 8-bit reference); (c) the flipped share, the should-not-call side already losing 29% one-directionally, five times the benchmark drift. One bit lower, round-to-nearest shows the full collapse ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The invocation/selection split at 3 bits (selection untouched in 32 of 34 conditions while whether-to-call collapses) and its closure at 2, where selection collapses too. W8 A16 W4 A16 W8 A8 W4 A8 W8 A4 W4 A4 0.0 0.2 0.4 0.6 0.8 1.0 1.2 fraction of decision margin surv…
Figure 5
Figure 5. Figure 5: The activation-axis grid: 72 conditions, five models, both test sets; at four-bit activations the weight setting stops mattering; the activation axis dominates the weight axis. margin instrument sees the flip mass grow tenfold over the same span. Mac- and cluster-score…
Figure 6
Figure 6. Figure 6: The 25% of cells whose slope cannot be estimated are the most damaged ones, not the quiet ones: median flip rate 0.33 against 0.11, and 54% of them lose more than a quarter of their decisions against 37%. Dropping them would discard the worst damage in the matrix, whic…
Figure 7
Figure 7. Figure 7: Forecast accuracy across the full result matrix: 868 cells, every axis of damage, one parameterization; flip rates predicted on held-out items, never fitted on flips. Median error 1.8 points (in-sample error on these 868 cells is 1.2 points; the held-out version is plo…
Figure 8
Figure 8. Figure 8: The forecast on two models never used to build the table (circles: Gemma-3-4B; squares: Granite-3.3-8B; colour gives the bit-width; the five worst misses are labelled). At 4 bits it transfers (median error 0.5pp); at 3 bits and below, borrowed constants are wrong by 18…
Figure 9
Figure 9. Figure 9: The intervention: a block-Hadamard rotation around the activation quantizer moves the collapse about two bits lower (left), a shift derivable to 0.21 bits from the range compression of the worst tensor, and at equal measured damage changes nothing else: same c, same fl…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 8 linked inside Pith

  1. [1]

    Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. QuaRot : Outlier-free 4-bit inference in rotated LLMs . Advances in Neural Information Processing Systems

    2024

  2. [2]

    Vishnu Kabir Chhabra and Mohammad Mahdi Khalili. 2025. Towards understanding and improving refusal in compressed models via mechanistic interpretability. arXiv preprint arXiv:2504.04215

    arXiv 2025

  3. [3]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8() : 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems

    2022

  4. [4]

    Peijie Dong, Zhenheng Tang, Xiang Liu, Lujun Li, Xiaowen Chu, and Bo Li. 2025. Can compressed LLMs truly act? an empirical evaluation of agentic capabilities in LLM compression. arXiv preprint arXiv:2505.19433

    arXiv 2025

  5. [5]

    Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, and Martin Vechev. 2024. Exploiting LLM quantization. Advances in Neural Information Processing Systems. ArXiv:2405.18137

    arXiv 2024

  6. [6]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ : Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323

    arXiv 2022

  7. [7]

    Jiwon Jang, Kisu Yang, Heuiseok Lim, and Hyunwoo Park. 2026. Flat score, amplified failures: How the error budget masks damage in quantized LLM agents. arXiv preprint arXiv:2607.27275

    arXiv 2026

  8. [8]

    Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher R \'e , and Aditi Raghunathan

    Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher R \'e , and Aditi Raghunathan. 2025. Scaling laws for precision. In International Conference on Learning Representations. ArXiv:2411.04330

    arXiv 2025

Show all 14 references
  1. [9]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ : Activation-aware weight quantization for LLM compression and acceleration. In Proceedings of Machine Learning and Systems (MLSys)

    2024

  2. [10]

    Xu Ouyang, Tao Ge, Thomas Hartvigsen, Zhisong Zhang, Haitao Mi, and Dong Yu. 2025. Low-bit quantization favors undertrained LLMs : Scaling laws for quantized LLMs with 100t training tokens. In Proceedings of ACL. ArXiv:2411.17691

    2025 arXiv

  3. [11]

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. BBQ : A hand-built bias benchmark for question answering. In Findings of ACL

    2022

  4. [12]

    Samuel Salfati. 2026. Quantization dominates rank reduction for KV -cache compression. arXiv preprint arXiv:2604.11501

    2026 arXiv

  5. [13]

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems

    2023

  6. [14]

    Patil, Ion Stoica, and Joseph E

    Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function-calling leaderboard. https://gorilla.cs.berkeley.edu/leaderboard.html

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.