Pith. sign in

REVIEW 3 major objections 5 minor 39 references

Spending a fixed bit budget unevenly, not uniformly, lets language models hold more parameters in less storage and decode faster.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:19 UTC pith:7FZ45ZX2

load-bearing objection Solid train-time mixed-precision recipe with real bigger-but-smaller and bandwidth-driven speed results; the load-bearing transfer of the small-pilot layout is under-controlled but the paper is honest about scope. the 3 major comments →

arxiv 2607.02893 v1 pith:7FZ45ZX2 submitted 2026-07-03 cs.LG cs.CL

Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models

Hamish Ogilvy This is my paper
classification cs.LG cs.CL
keywords variable bit-width quantizationmixed-precision trainingper-group precisionlanguage modelsGumbel-Softmaxbigger-but-smallermemory-bandwidth decodequality-per-byte
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Language-model quantization almost always forces every weight to the same bit-width. This paper argues that precision should instead be a learnable, per-group resource. During training each contiguous block of 64 weights chooses its own resolution from 1, 2, 4 or 8 bits; the resulting allocation is strongly heterogeneous even inside a single projection type, with most groups collapsing to 1 bit while a few sensitive blocks keep more. That pattern freezes into a fixed, reusable recipe. With the recipe a larger model at roughly 1.8 mean bits can match or beat a smaller full-precision model while using several times less storage, and the same packed weights also accelerate memory-bandwidth-bound decoding. The practical claim is that layout of the bit budget, not merely its total size, decides quality per byte.

Core claim

Spending a fixed bit budget unevenly across groups of weights, with the allocation discovered during from-scratch training and then frozen into a fixed recipe, produces a bigger-but-smaller regime: models with more parameters but far lower mean bits match or beat smaller full-precision models on perplexity at several times less storage, while also decoding faster because autoregressive generation is memory-bandwidth bound.

What carries the argument

Variable Bit-width Quantization (VBQ): each group of 64 weights carries a categorical choice over {1,2,4,8} bits, relaxed by Gumbel-Softmax and trained by alternating weight and precision steps with gradient isolation so the bit logits receive a clean task-aligned signal; the discovered hierarchy is then distilled into a fixed recipe (4-bit first MLP, 2-bit elsewhere, frequency-tiered head).

Load-bearing premise

The heterogeneous allocation found on small models freezes into a fixed recipe that stays near-optimal and transferable without re-search when architecture, data, and scale grow, and that co-adaptation of weights to those bit-widths during from-scratch training is what recovers quality.

What would settle it

Train a matched pair of models at 1–2 B scale under identical token budgets, one with the frozen VBQ recipe and one with a strong uniform or post-training 4-bit baseline of equal storage; if the VBQ model no longer matches or beats the baseline on perplexity and quality-per-byte, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Precision becomes a structural design choice made before training rather than a post-training compression knob.
  • A fixed per-projection recipe plus a token-frequency table for the head can be reused across new architectures without re-running the search.
  • Because decode is bandwidth-bound, the same packed weights deliver wall-clock speedups that grow with model size, compounding the storage win.
  • Depth itself acts as an error-correcting code: later low-bit blocks progressively reduce the representational gap injected by early layers.
  • Quality-per-byte improves by roughly 4–8 imes relative to full-precision models of comparable or better perplexity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the recipe continues to transfer, mixed low-bit layouts discovered once could become the default pretraining format rather than an after-the-fact quantisation step.
  • The self-healing observation suggests future designs might deliberately under-allocate bits early and rely on depth for recovery, changing how residual budgets are spent.
  • Hardware that natively streams packed 1/2/4-bit groups without register expansion would amplify the measured decode speedups still further.
  • Post-training adaptation of already-trained full-precision checkpoints is unlikely to recover the same gains, because the win depends on weights co-adapting to their bit-widths from the start.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Variable Bit-width Quantization (VBQ): during from-scratch training of decoder-only LMs, each contiguous group of 64 weights learns a bit-width from {1,2,4,8} via Gumbel-Softmax, made trainable by gradient isolation of the candidate reconstructions and alternating weight/logit steps (Algorithm 1). On a ~58M TinyStories pilot the search yields a strongly heterogeneous allocation (69% 1-bit groups, LM head ~1.09 bits, first MLP ~2.5 bits) that is frozen into a fixed recipe (4-bit first MLP, 2-bit elsewhere, frequency-tiered head). With this recipe a 131M model at 1.82 mean bits reaches PPL 4.2, beating a 55M FP16 baseline (PPL 4.4) at 3.8 imes less storage; the same recipe lets a 1.46B model match a 593M FP16 control on FineWeb-Edu at ~3.7 imes less storage. Custom fused dequantize-and-multiply kernels convert the packed storage into decode speedups that grow with scale (parity at 131M, ~1.9× at 1B, 4.7× at 9B). A KL/argmax-flip depth probe shows later 2-bit blocks progressively reduce the error injected by the early 4-bit block. The authors delimit the result as a from-scratch phenomenon and flag economical scaling of the search past ~1.5B as open.

Significance. If the transfer of the frozen recipe is reliable, the work supplies a concrete, deployable path to the “bigger-but-smaller” regime: more parameters, substantially lower storage, and bandwidth-driven decode speedups that compound with scale, all from a single reusable bit layout rather than per-model search. The technical ingredients (gradient isolation + alternating optimization that actually moves the precision logits, the within-projection heterogeneity that per-layer methods cannot express, the self-healing depth analysis, and the same-weights fused-kernel measurements) are carefully engineered and honestly scoped. The quality-per-byte gains (3.9–8.4×) and the explicit mapping from recipe to packed storage are practically valuable. The main open question is whether the small-pilot recipe remains near-optimal once architecture, tokenizer, data, and scale change; answering that would make the contribution substantially stronger.

major comments (3)
  1. §5.2 / Table 1 and the FineWeb-Edu 1.46B result: the central “bigger-but-smaller” claim attributes the quality recovery to the specific heterogeneous layout distilled from the ~58M TinyStories pilot. At 1.46B the paper never re-runs the Gumbel search and never reports a matched from-scratch uniform ~1.8-bit (or pure 2-bit) control of the same architecture and token budget. Without that control it remains possible that parameter count plus any low-bit QAT regularization, rather than the particular 4-bit-first / 2-bit-elsewhere / frequency-tiered-head allocation, explains the parity with the 593M FP16 baseline. A same-scale uniform-low-bit ablation (or a short re-search) is load-bearing for the claim that “spending a fixed bit budget unevenly beats spending it uniformly.”
  2. §7 (1.0B hybrid gate and 9B KD attempt): the only larger-scale transfer probe is deliberately short (0.37 tokens/parameter) and still trails its matched FP16 control by 0.28 nats at the stopped checkpoint; the 9B run is correctly flagged as confounded (18% unquantized parameters, severe under-training). These do not yet close the transfer gap. Either a longer matched run at ≥1B with the recipe versus a uniform low-bit twin, or an explicit statement that the 1.46B FineWeb result is the sole quality evidence above ~130M, would make the scope of the claim precise.
  3. §5.1–5.2 and Experimental Setup: all quality numbers (TinyStories ladder, FineWeb-Edu match, KL/flip rates) appear to be single-run. Given the stochasticity of Gumbel selection (even if later frozen) and of LM training, multi-seed means and error bars—or at least a statement of observed run-to-run variation on the pilot—are needed before the 0.2 PPL margin (4.2 vs 4.4) and the exact 1.46B parity can be treated as robust.
minor comments (5)
  1. §3.2 / Algorithm 1: the stop-gradient notation “sg[·] (≡detach)” is introduced only in the prose after the algorithm; placing the detach explicitly inside the algorithm listing would remove ambiguity about which path is isolated.
  2. Figure 2b caption: the “running minimum” smoothing of validation curves is useful but should be stated in the main text as well, so readers do not misread the faint scatter as the primary signal.
  3. §6 / Table 2: the 9B row is correctly caveated as a speed-only measurement; a one-sentence reminder in the table caption that quality for that particular run is unresolved would prevent over-reading.
  4. Related Work: concurrent GSQ and dMX are discussed carefully; a short explicit sentence on why integer power-of-two bit-widths (rather than continuous MX formats or scalar grids) are the right target for the packed-kernel story would further clarify the design choice.
  5. Typos / polish: “sg[·]” vs “detach”, occasional missing spaces around “~”, and the arXiv date line “3 Jul 2026” look like residual draft artifacts; a final pass would clean them.

Circularity Check

0 steps flagged

No significant circularity: empirical discovery of bit allocation via task loss + penalty, then frozen recipe evaluated against independent FP16 controls.

full rationale

This is a standard empirical ML methods paper. The bit-width allocation is obtained by joint optimization of network weights and Gumbel-Softmax precision logits against cross-entropy plus an explicit bit-budget penalty (Algorithm 1, Section 3); the resulting heterogeneous pattern is observed (Section 5.1), then frozen into a fixed recipe that is applied from-scratch and measured by validation PPL, KL, flip rate, storage, and decode speed against matched FP16 baselines of different sizes (Tables 1–2, Figures 2–5). Nothing is defined so that a claimed prediction equals its input by construction; λ is a hyper-parameter chosen to target ~1.8 mean bits, not a fit that is later re-labeled as a forecast of the same quantity. The recipe’s transferability is an empirical claim (with acknowledged open problems on scale), not a tautology. Self-reference is limited to the ordinary practice of distilling a search result into a reusable artifact; there are no load-bearing uniqueness theorems, self-citations of unverified prior results, or ansatzes smuggled via author-overlapping citations that force the central bigger-but-smaller numbers. Quality-per-byte and speedups are external measurements, not rearrangements of the training objective. Score 1 only for the minor, non-load-bearing fact that the recipe originates from the same pilot it later replaces.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 2 invented entities

VBQ is an empirical training method. Its claims rest on standard quantization and Gumbel machinery plus several hand-chosen design knobs (group size, candidate set, λ, recipe cutoffs) that are fitted or selected so the mean lands near 1.8 bits and the hierarchy matches the adaptive pilot. No new physical entities; the invented objects are the method and the frozen recipe.

free parameters (6)
  • bit-budget penalty λ = chosen for ~1.8 mean bits
    Exchange rate that trades cross-entropy for mean bits; deliberately chosen so the mean lands at ~1.8 bits, below the 3.5–4-bit floor.
  • group size G = 64
    Fixed at 64; creates the local quality/precision trade-off (G=1 is degenerate).
  • candidate bit-widths B = {1,2,4,8}
    Discrete set {1,2,4,8} supplied to the Gumbel-Softmax.
  • Gumbel temperature schedule τ = 5.0 → 0.5
    Annealed from 5.0 to 0.5; controls softness of the discrete choice.
  • logit-update interval m
    Alternating optimization frequency (one logit step every m weight steps).
  • LM-head frequency-tier cutoffs = 0.5/1.5/10/88 percentiles
    Top 0.5 % at 8-bit, next 1.5 % at 4-bit, next 10 % at 2-bit, bottom 88 % at 1-bit; chosen once to match the adaptive head allocation.
axioms (4)
  • domain assumption Straight-through estimator allows gradients through round/clip of the absmax quantizer.
    Standard QAT assumption (Bengio et al.); invoked in §3.1.
  • domain assumption Gumbel-Softmax provides a usable continuous relaxation of categorical bit-width selection.
    Jang/Maddison; used for πg in §3.2.
  • domain assumption Autoregressive decode at batch-1 is memory-bandwidth bound, so fewer bytes per weight translate into wall-clock speedup when dequantize-and-multiply is fused.
    Stated in §6; underpins the inference claims.
  • ad hoc to paper Weights co-adapt to their assigned bit-widths only when trained from scratch; post-hoc search on frozen pretrained weights collapses to trivial uniform 4-bit.
    Empirical claim used to delimit scope in §7; not independently proven.
invented entities (2)
  • Variable Bit-width Quantization (VBQ) with gradient isolation + alternating optimization independent evidence
    purpose: Make per-group bit-width logits trainable against the task loss without being overwhelmed by the CE gradient.
    Core algorithmic contribution; independent evidence is the reported collapse of selection entropy and the heterogeneous hierarchy.
  • Fixed VBQ recipe (4-bit first MLP, 2-bit elsewhere, frequency-tiered head) independent evidence
    purpose: Distill the adaptive search into a reusable, search-free training prescription.
    Product of the method; validated by matching or beating the adaptive pilot on TinyStories and the 1.46 B FineWeb run.

pith-pipeline@v1.1.0-grok45 · 20467 in / 3666 out tokens · 33983 ms · 2026-07-12T06:19:13.974383+00:00 · methodology

0 comments
read the original abstract

Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width. We introduce Variable Bit-width Quantization (VBQ), a training-time method in which each contiguous group of 64 weights learns its own resolution from {1,2,4,8} bits via a Gumbel-Softmax relaxation, trained jointly by an alternating optimization that gives the precision logits a clean, task-aligned signal. VBQ discovers a consistent, strongly heterogeneous allocation within individual projection types, not merely across layers, impossible to express with per-layer methods: 69% of groups collapse to 1 bit, the LM head averages 1.09 bits, while the first MLP block keeps ~2.5 bits. This pattern is stable enough to freeze into a fixed recipe and reuse without further search. The recipe yields a "bigger-but-smaller" regime: a 131M model at 1.82 mean bits reaches perplexity 4.2 on TinyStories, beating a 55M FP16 model (PPL 4.4) at 3.8x less storage, and lets a 1.46B model on FineWeb-Edu match a 593M FP16 control at ~3.7x less storage with 2.5x more parameters. As quality-per-byte, VBQ is 3.9-8.4x more efficient than FP16. The recipe maps directly to packed low-bit storage, so it also accelerates inference: with custom fused dequantize-and-multiply kernels, memory-bandwidth-bound autoregressive decode is faster at equal output, and the speedup grows with scale (parity at 131M, 1.9x at 1.0B, 4.7x at 9B on Apple silicon). A distributional analysis (KL divergence and argmax-flip rate) reveals a striking mechanism: deeper layers progressively self-heal the quantization error injected by early layers. The win is a from-scratch, train-time phenomenon; scaling the search economically beyond 1.5B parameters remains open. VBQ reframes precision as a learnable, non-uniform resource and shows that spending a fixed bit budget unevenly beats spending it uniformly.

Figures

Figures reproduced from arXiv: 2607.02893 by Hamish Ogilvy.

Figure 1
Figure 1. Figure 1: VBQ learns a heterogeneous precision allocation. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Bigger-but-smaller. (a) Validation perplexity vs. packed storage on TinyStories: every VBQ model (crimson stars) sits on or below the FP16 baselines (blue circles); the 131M VBQ model at 29 MB beats the 55M FP16 model at 109 MB. (b) Validation perplexity over training for three fixed-architecture 35K-iteration runs. Faint markers are the raw per-eval points; solid lines trace the best perplexity reached so… view at source ↗
Figure 3
Figure 3. Figure 3: Quality-per-byte (1/PPL)/storage (GB). VBQ models (crimson) and FP16 baselines (blue) do not overlap; the iso-parameter 131M comparison is 8.4×. 5.3 Distributional analysis: depth self-heals quantization error Perplexity can hide behavioral change [9]. We therefore measure mean token-level KL divergence and argmax-flip rate of the 131M VBQ model against a 131M FP16 reference. The full model has mean KLD 0.… view at source ↗
Figure 4
Figure 4. Figure 4: Self-healing through depth. Cumulative KL divergence to FP16 (crimson) and argmax-flip rate (blue) measured by projecting each block’s hidden state through a shared head. The early 4-bit block injects the largest gap; later 2-bit blocks progressively correct it. 6 Inference: storage savings become speed A learned bit budget is only useful if it survives to deployment. Because autoregressive decode at batch… view at source ↗
Figure 5
Figure 5. Figure 5: The inference win compounds with scale. Measured single-stream decode speedup of the packed VBQ model over the same model dequantized to dense FP16 (Apple M4 Max, batch 1), plotted against parameter count on a log axis. At 131M the model is too small for weight traffic to dominate and VBQ runs at FP16 parity (0.95×); by 1.0B it is 1.86× and by 9B it is 4.68× faster. The dashed line is an illustrative log-l… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 15 linked inside Pith

  1. [1]

    Introducing apple’s on-device and server foundation models

    Apple. Introducing apple’s on-device and server foundation models. Apple Machine Learning Research, https://machinelearning.apple.com/research/introducing-apple-foundation-models,

  2. [2]

    On-device model compressed via mixed 2-/4-bit palettization averaging 3.7 bits-per-weight (3.5 bits-per-weight without significant quality loss)

  3. [3]

    Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432, 2013

    Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432, 2013

    Pith/arXiv arXiv 2013

  4. [4]

    Rethinking differentiable search for mixed-precision neural networks

    Zhaowei Cai and Nuno Vasconcelos. Rethinking differentiable search for mixed-precision neural networks. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    2020

  5. [5]

    Zeyu Cao, Cheng-Chang Kao, Chao Wang, Tianhua Xiao, Ling Zhu, Yibo Wang, Yiren Lin, and Nicholas D. Lane. Scaling laws for mixed quantization in large language models.arXiv preprint arXiv:2410.06722, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    GSQ: Highly-accurate low-precision scalar quantization for LLMs via gumbel-softmax sampling

    Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan, Michael Helcig, Eldar Kurtic, and Dan Alistarh. GSQ: Highly-accurate low-precision scalar quantization for LLMs via gumbel-softmax sampling. arXiv preprint arXiv:2604.18556, 2026

    Pith/arXiv arXiv 2026

  7. [7]

    The case for 4-bit precision: k-bit inference scaling laws

    Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. International Conference on Machine Learning (ICML), 2023

    2023

  8. [8]

    LLM.int8(): 8-bit matrix multiplication for transformers at scale

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  9. [9]

    Mahoney, and Kurt Keutzer

    Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. HAWQ: Hessian aware quantization of neural networks with mixed-precision. InIEEE/CVF International Conference on Computer Vision (ICCV), 2019

    2019

  10. [10]

    Accuracy is not all you need

    Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, and Ramachandran Ramjee. Accuracy is not all you need. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2407.09141

    Pith/arXiv arXiv 2024

  11. [11]

    Extreme compression of large language models via additive quantization.International Conference on Machine Learning (ICML), 2024

    Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization.International Conference on Machine Learning (ICML), 2024

    2024

  12. [12]

    TinyStories: How small can language models be and still speak coherent english?arXiv preprint arXiv:2305.07759, 2023

    Ronen Eldan and Yuanzhi Li. TinyStories: How small can language models be and still speak coherent english?arXiv preprint arXiv:2305.07759, 2023. 13

    Pith/arXiv arXiv 2023

  13. [13]

    dMX: Differentiable mixed-precision assignment for low-precision floating-point formats.arXiv preprint arXiv:2606.04115, 2026

    Giuseppe Franco, Ian Colbert, Pablo Monteagudo-Lago, Felix Marty, and Nicholas Fraser. dMX: Differentiable mixed-precision assignment for low-precision floating-point formats.arXiv preprint arXiv:2606.04115, 2026

    Pith/arXiv arXiv 2026

  14. [14]

    GPTQ: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  15. [15]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  16. [16]

    SliM-LLM: Salience-driven mixed-precision quantization for large language models

    Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi. SliM-LLM: Salience-driven mixed-precision quantization for large language models. arXiv preprint arXiv:2405.14917, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Categorical reparameterization with Gumbel-Softmax

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-Softmax. In International Conference on Learning Representations (ICLR), 2017

    2017

  18. [18]

    Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan

    Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision. International Conference on Learning Representations (ICLR), 2025

    2025

  19. [19]

    AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. InProceedings of Machine Learning and Systems (MLSys), 2024

    2024

  20. [20]

    DARTS: Differentiable architecture search

    Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations (ICLR), 2019

    2019

  21. [21]

    ParetoQ: Improving scaling laws in extremely low-bit LLM quantization

    Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, Lin Xiao, Yuandong Tian, Bilge Soran, Raghuraman Krishnamoorthi, Tijmen Blankevoort, and Vikas Chandra. ParetoQ: Improving scaling laws in extremely low-bit LLM quantization. InAdvances in Neural Information Processing Systems...

    arXiv 2025

  22. [22]

    The era of 1-bit LLMs: All large language models are in 1.58 bits.arXiv preprint arXiv:2402.17764, 2024

    Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit LLMs: All large language models are in 1.58 bits.arXiv preprint arXiv:2402.17764, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Maddison, Andriy Mnih, and Yee Whye Teh

    Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. InInternational Conference on Learning Representations (ICLR), 2017

    2017

  24. [24]

    Introducing NVFP4 for efficient and accurate low-precision in- ference

    NVIDIA. Introducing NVFP4 for efficient and accurate low-precision in- ference. NVIDIA Technical Blog, https://developer.nvidia.com/blog/ introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/ , 2025. 4- bit floating-point format (E2M1) with two-level micro-block scaling: a per-16-value FP8 (E4M3) scale plus a per-tensor FP32 scale; reports...

    2025

  25. [25]

    Pretraining large language models with NVFP4

    NVIDIA. Pretraining large language models with NVFP4. arXiv:2509.25149, 2025. 4-bit pretraining of a 12B hybrid Mamba-Transformer on 10T tokens; keeps a hand-chosen subset of layers (first two plus final eight) in BF16, i.e. a fixed mixed-precision layout rather than a learned per-group allocation

    arXiv 2025

  26. [26]

    The FineWeb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. InAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024. arXiv:2406.17557; introduces FineWeb-Edu. 14

    Pith/arXiv arXiv 2024

  27. [27]

    Bonsai: 1-bit and ternary (1.58-bit) language models for on-device inference.https: //prismml.com/news/ternary-bonsai, 2026

    PrismML. Bonsai: 1-bit and ternary (1.58-bit) language models for on-device inference.https: //prismml.com/news/ternary-bonsai, 2026. Open-weight 1-bit and 1.58-bit ternary models up to 8B parameters; whitepapers athttps://github.com/PrismML-Eng/Bonsai-demo

    2026

  28. [28]

    Qwen3 technical report

    Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Language models are unsupervised multitask learners.OpenAI Technical Report, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.OpenAI Technical Report, 2019

    2019

  30. [30]

    Neural hashing: The future of search

    Search.io. Neural hashing: The future of search. Search.io whitepaper, https: //cdn.prod.website-files.com/61830fa9d578e127186810ed/6238c27c15df074466448996_ neuralhashing-wp-searchio.pdf, 2022. Neural hashing compresses embedding vectors into compact binary hashes using a fraction of the storage while retaining up to 99% of the vector information (Search...

    2022

  31. [31]

    GLU variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. GLU variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Pith/arXiv arXiv 2002

  32. [32]

    RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 568, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 568, 2024

    2024

  33. [33]

    QuIP#: Even better LLM quantization with hadamard incoherence and lattice codebooks

    Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QuIP#: Even better LLM quantization with hadamard incoherence and lattice codebooks. InInternational Conference on Machine Learning (ICML), 2024

    2024

  34. [34]

    Unsloth dynamic GGUF quants

    Unsloth. Unsloth dynamic GGUF quants. Unsloth Documentation,https://unsloth.ai/docs/ models/glm-5.2, 2025. Post-training, per-layer dynamic mixed-precision GGUF quantization (UD-* series): salient layers are retained at higher precision via an importance heuristic while the rest are pushed to low bit-widths

    2025

  35. [35]

    BitNet: Scaling 1-bit transformers for large language models

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. BitNet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    HAQ: Hardware-aware automated quantization with mixed precision

    Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: Hardware-aware automated quantization with mixed precision. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  37. [37]

    Mixed precision quantization of ConvNets via differentiable neural architecture search.arXiv preprint arXiv:1812.00090, 2018

    Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of ConvNets via differentiable neural architecture search.arXiv preprint arXiv:1812.00090, 2018

    Pith/arXiv arXiv 2018

  38. [38]

    SmoothQuant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. InInternational Conference on Machine Learning (ICML), 2023

    2023

  39. [39]

    MixLLM: LLM quantization with global mixed- precision between output-features and highly-efficient system design

    Zhen Zheng, Xiaonan Song, and Chuanjie Liu. MixLLM: LLM quantization with global mixed- precision between output-features and highly-efficient system design. InProceedings of Machine Learning and Systems (MLSys), 2026. arXiv:2412.14590. 15

    Pith/arXiv arXiv 2026