Pith. sign in

REVIEW 4 major objections 5 minor 136 references

LLM accuracy at 4-bit precision can be largely preserved, without retraining, by giving each layer one 8-bit block for extreme-outlier columns, a 6-bit tier for mild outliers, and 4 bits for the rest at the same average bit budget.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 17:03 UTC pith:LUPQJ4YU

load-bearing objection Solid mixed-precision quantization scheme with plausible accuracy gains on MXINT hardware, but the paper overstates consistency and leaves a key hardware-validity question unanswered. the 4 major comments →

arxiv 2607.17733 v1 pith:LUPQJ4YU submitted 2026-07-20 cs.LG cs.AI

MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

Simla Burcu Harma , Danila Mishin , Zhengyuan Su , Ayan Chakraborty , Elizaveta Kostenok , Dongho Ha , Babak Falsafi , Martin Jaggi
show 2 more authors
Yunho Oh Amir Yazdanbakhsh
This is my paper
classification cs.LG cs.AI MSC 68T07
keywords LLM quantizationmixed-precision quantizationmicroscaling formats (MXINT)outlier-aware bitwidth allocationHessian-diagonal sensitivitypost-training quantizationW4A4KV4 inferencetriplet quantization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Quantizing large language models to 4-bit arithmetic damages their accuracy because a small number of extreme activation values — outliers — dominate the shared scale of each quantization block. This paper argues that the damage is repairable with no training at all: treat outlier severity as a spectrum, give each layer one 8-bit block to the few columns where extreme outliers concentrate, a 6-bit tier to the more numerous mild outliers, and 4 bits to everything else, with the 6/4 split scaled by each layer's measured sensitivity so the average bit budget stays near 4 bits. Using only a ~128-sample calibration pass, the authors report WikiText-2 perplexities of 3.77 on LLaMA-2-70B and 7.63 on LLaMA-3-8B under the W4A4KV4 setting, beating rotation-based and integer mixed-precision baselines at matched bitwidths while adding only a few percent overhead on hardware with native micro-scaling support. The net claim: fine-grained, sensitivity-guided precision — not tensor reshaping — is what makes accurate 4-bit LLM inference viable.

Core claim

The paper's central claim is that quantization sensitivity in LLMs is a continuum, not a binary outlier/normal split, and that this continuum can be exploited exactly where the hardware allows it. MXINT, a micro-scaling format in which 32 values share one power-of-two scale encoded in the hardware, fixes the mantissa width per 32-wide block; MXSens therefore measures a per-column sensitivity score from the diagonal of the empirical Hessian of the input activations (the sum of squared activation values in that column), plus a per-layer score from the final-output L2 error when only that layer is quantized. On that basis it assigns 8-bit mantissas to the top 32 sensitive columns of every layer

What carries the argument

Triplet quantization over MXINT blocks. MXINT is a hardware micro-scaling format: every block of 32 values shares one power-of-two scale (stored as a block exponent), and all values in the block use the same mantissa width, so bitwidth is a per-block decision. The allocation algorithm sorts each layer's columns by the sensitivity score s_j = Σ_k x_kj^2 accumulated over calibration activations; the top 32 columns receive an 8-bit mantissa (one MXINT block per layer), the next tier receives 6 bits, and the rest 4 bits, with the number of 6-bit columns in layer i proportional to that layer's normalized sensitivity and scaled by a global control parameter R (closed-form expression in the appendi

Load-bearing premise

The method's accuracy rides on the premise that in every layer the extreme-outlier columns number fewer than 32 and sit inside one 32-wide block, so a single 8-bit MXINT block per layer captures all of them; if any layer has more than 32 outlier-heavy columns, or they scatter across blocks, that block misses them and the reported gains should shrink.

What would settle it

Take a model not tuned by the paper — or Qwen1.5-7B, the case where its perplexity already trails baselines — compute per-column sensitivity scores from 128 calibration samples, and check each layer for two things: whether more than 32 columns show extreme-outlier-level scores, and whether the top-32 columns all fall inside one contiguous 32-column span. In any layer where either check fails, quantize with the single 8-bit block as prescribed and measure WikiText-2 perplexity; a visible jump over an unconstrained 8-bit assignment would settle the structural premise.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At the 4.03-bit budget, only about 1% of arithmetic operations require the higher mantissa widths; the authors estimate ~3% latency and under 1% memory overhead on hardware that natively executes MXINT, so the accuracy gains do not cancel out the speed of 4-bit serving.
  • The same recipe at 5.75 average bits brings LLaMA-2-7B to within 0.6 points of FP16 accuracy across seven zero-shot benchmarks at roughly one third of the bit cost — the paper's evidence that the method's benefit persists across the whole 3-6 bit budget range.
  • MXSens requires no retraining and no architecture changes, only a ~128-sample calibration pass, and it quantizes weights, activations, and KV cache together under W4A4KV4; it is therefore compatible with the serving configuration needed for long contexts.
  • The two-tier 4/8 scheme is consistently worse than the 4/6/8 triplet at equal average bitwidth, which the paper takes as evidence that mild outliers are a distinct population worth an intermediate precision tier rather than a second thought.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Boundary the paper leaves implicit: the single 8-bit block works only if each layer has fewer than 32 extreme-outlier columns that fall within one 32-wide span. The paper's appendix shows Qwen1.5-7B has a different sensitivity profile (middle layers most sensitive) and is the one model where MXSens's perplexity trails baselines — the earliest sign of where the structural premise might strain.
  • A testable extension the paper does not run: apply the 6-bit tier to the KV cache and to the layers left in FP16 (softmax outputs, embeddings, the final head). If mild outliers appear there, the same triplet logic could compress those components too and push the effective bitrate below today's 4.03.
  • The sensitivity ranking comes from 128 calibration samples, and the paper's own calibration-dataset ablation shifts results by only ~0.1 perplexity and ~0.1 accuracy, hinting the ranking is stable in-distribution; the open question is domain shift, so a stress test on corpora far from C4 and WikiText would establish how portable the 8/6/4 map is.
  • Because the output of MXSens is just a per-layer list of column indices, the method is a configuration artifact: a serving stack could cheaply re-run the 128-sample calibration on the model's actual traffic statistics and re-fit the precision map, turning a one-time quantization into a continuously tuned deployment knob.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MXSens, a training-free mixed-precision quantization scheme for LLMs that assigns 8-bit mantissas to the 32 most sensitive columns per linear layer and 6/4-bit mantissas to the remaining columns in a layer-sensitivity-weighted fashion, using the MXINT block format. Sensitivity is estimated from the Hessian diagonal of activations (column-wise) and from single-layer quantization output error (layer-wise). The method is evaluated on WikiText-2 perplexity and zero-shot commonsense tasks across LLaMA-2/3, Mistral, Qwen, and Gemma models, and the authors claim consistent improvement over QuaRot, RRS, Atom, and QUIK under W4A4KV4/W4A4KV16 settings.

Significance. If the MXINT implementation question is settled, MXSens is a meaningful contribution: it is training-free, has public code and clear pseudocode, shows large perplexity gains on LLaMA-3-8B (7.63 vs 8.11 for RRS), and includes useful ablations over bit budgets and calibration data. The core idea—fine-grained precision allocation driven by local sensitivity—is well motivated and distinct from rotation-based outlier handling. However, the paper's central claim that the 32 sensitive columns form one valid MXINT block is not supported, and the 'consistently outperforms' statement is contradicted by the paper's own tables. The method is promising but needs a major revision to establish MXINT compatibility and to scope the empirical claims.

major comments (4)
  1. [Section 4.1 / Algorithm 2] Algorithm 2 assigns 8-bit precision to I[0:32], the 32 highest-sensitivity columns. MXINT blocks are fixed contiguous groups of 32 elements; an arbitrary index set is not a block. The paper never specifies the tensor layout or permutation that would make these columns contiguous while keeping the forward pass valid for weights, activations, and KV cache. If the implementation instead emulates per-column mantissa bitwidths, the reported numbers are not for a valid MXINT format, and the 'one high-precision block per layer' claim loses its basis. This is load-bearing because the method's claimed advantage over per-column mixed precision is hardware compatibility. Please provide the exact permutation/layout used in the code, or validate on an MXINT simulator/hardware.
  2. [Section 4.1 / Figure 4] The paper states that 'the number of columns containing extreme outliers is typically smaller than the block size of MXINT (32)' and that the top 32 sensitive columns capture all extreme outliers. No evidence is provided for this per-layer count. Figure 4 is an ablation over the number of 8-bit columns on Llama-2-7B under idealized per-column selection; it does not measure outlier column counts or enforce block contiguity. If any evaluated layer has more than 32 outlier-heavy columns, or if the top-32 columns cannot be gathered into one block without harming other layers, the reported accuracy would not reflect the proposed single-8-bit-block design. Please add per-layer outlier column counts and repeat the Figure 4 ablation under the actual block layout for all principal models.
  3. [Abstract / Section 5.1 / Tables 2 and 5] The abstract and Section 5.1 claim MXSens 'consistently outperforms' SOTA methods, but the paper's own tables contradict this. In Table 2 (Mistral, W4A4KV16), MXSens is below RRS/QuaRot on OBQA (42.2 vs 43.4/44.4), BoolQ (81.5 vs 85.1/83.1), and ARC-C (49.2 vs 55.4/54.4). In Table 5, Qwen1.5-7B PPL is worse than RRS in both W4A4KV16 and W4A4KV4 (9.73/9.95 vs 9.17/9.37). Even in Table 1, MXSens at 4.27 bits is not better than Atom on PPL (5.32 vs 5.31) on LLaMA-2-13B. The conclusion should be scoped to 'most settings' and should identify the cases where the method falls behind.
  4. [Section 6 / Appendix A.3] The 3% latency overhead estimate is presented without a cycle-level or measured-throughput analysis. The claim that only 1% of arithmetic operations require higher precision is not derived from the bitwidth allocation: column fraction is not operation fraction, and the 6-bit operations are ignored. Additionally, Theorem A.1/Algorithm 2 compute R with real-valued N6, but Algorithm 2 applies floor, so the achieved average bitwidth may be below the target B. Please report achieved bitwidths and either remove or substantiate the hardware overhead claim.
minor comments (5)
  1. [Appendix A.3 / Theorem A.1] Typo: 'dimention' should be 'dimension'. The theorem should also state the assumed ranges of S_layer and the handling of the floor in N6bit.
  2. [Table 2] The table layout is confusing: model labels are interleaved with method labels, making it hard to know which rows belong to which model. Use a separate model column or group headers.
  3. [Section 5 / Experimental setup] Baseline numbers are quoted from other papers. Please state the exact average-bitwidth computation used for each baseline and confirm that the evaluation harness and calibration are matched, or re-run the baselines in the same environment.
  4. [Appendix A.2] The appendix says KV cache uses a '4-8 bitwidth configuration with average bitwidth fixed at 4.03'. It is unclear how this is consistent with the triplet 4/6/8 allocation described for weights and activations. Please clarify the exact KV-cache bitwidth assignment.
  5. [Figure 2] The figure axes and legend are not fully described in the caption; please add a complete caption explaining the two curves and the bitwidth configurations.

Circularity Check

0 steps flagged

No significant circularity: sensitivity metrics are inputs, bitwidth allocation is budget bookkeeping, and reported results are independently evaluated.

full rationale

The derivation chain is not circular. Column-wise sensitivity is defined from Hessian diagonals of calibration activations (Algorithm 1) and layer-wise sensitivity from isolated single-layer quantization error (Algorithm 3); both are inputs to Algorithm 2, not outputs of the evaluation. Theorem A.1 solves the control parameter R algebraically to enforce the average-bitwidth budget, which is bookkeeping rather than a fitted prediction. The choice of 32 8-bit columns is an ablated design decision supported by Figure 4, not a parameter fit to the reported benchmarks; calibration uses C4 while evaluation uses WikiText-2 and commonsense tasks, so no held-out metric is fed back into the allocation. The self-citations in Section 6 (Gil et al. 2025; Harma et al. 2022; Drumond et al. 2021) support hardware-emulation feasibility claims, but they are not load-bearing for the central accuracy-per-bit derivation and do not enter any equation used to produce the perplexity or accuracy numbers. The MXINT block-contiguity concern raised by the skeptic is a potential implementation-validity threat, but it is not a circularity: the reported numbers may not correspond to a valid MXINT layout, yet they are not equivalent to the method's inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The method rests on two empirical premises about LLM outlier structure (outliers are column-concentrated and fit within one MXINT block; layer errors compound predictably) plus the standard Hessian-diagonal sensitivity proxy. No new physical or mathematical entities are introduced; 'mild outliers' is a descriptive categorization, not an entity.

free parameters (4)
  • N_8bit (number of 8-bit columns per layer) = 32
    Fixed to the MXINT block size based on the observation/ablation that extreme outliers fit in one 32-wide block (Section 4.1, Fig. 4). This hand-chosen hyperparameter affects all results.
  • Triplet bitwidths = 4/6/8
    Assumed precision set; ablations only vary the average budget, not the choice of 6 as the intermediate level (Section 4, Table 3).
  • Average bitwidth target B = 4.02–4.03 (also 4.25/4.5 in comparisons)
    User-specified budget; the global scaling factor R is derived from it rather than fit, but the specific budget used in each comparison affects the reported numbers.
  • Calibration set = 128 C4 samples
    Used to estimate Hessians and layer errors; no stability analysis across seeds or set sizes (Section 5, A.9).
axioms (4)
  • domain assumption Column-wise sensitivity is proportional to the Hessian diagonal H_jj = Σ_k X_kj²
    Eq. (1) approximation from Lee et al. (2024) is used to rank columns for precision allocation. It equates sensitivity with activation column energy, ignoring weight error structure and cross-column interactions.
  • ad hoc to paper All extreme outliers in a layer fit within the 32 most sensitive columns
    Section 4.1: 'the number of columns containing extreme outliers is typically smaller than the block size of MXINT (32)'. This justifies reserving exactly one 8-bit block per layer; validated only by internal ablation (Fig. 4).
  • domain assumption Layer-wise sensitivity measured in isolation composes additively/independently across layers
    Algorithm 3 measures the L2 error at the final output when quantizing each layer alone, then allocates 6-bit columns proportionally to that error. Assumes error propagation is monotonic and roughly independent across layers (position-dependent amplification in Fig. 1).
  • domain assumption 128 C4 calibration samples give stable Hessian estimates
    Section 5: 'For sensitivity analysis, we use 128 randomly sampled examples from the C4'. No error bars or stability analysis are provided.

pith-pipeline@v1.3.0-alltime-deepseek · 20513 in / 12412 out tokens · 126957 ms · 2026-08-01T17:03:28.716639+00:00 · methodology

0 comments
read the original abstract

4-bit quantization enables efficient LLM inference, but suffers from significant accuracy degradation due to outliers. Prior work addresses this problem via data rotation or mixed-precision integer quantization, but often relies on software-managed scaling and frequent dequantization, incurring substantial overhead. Microscaling formats, such as MXINT, eliminate these inefficiencies by encoding scales in hardware, yet remain incompatible with rotation-based methods. Our analysis reveals that outliers vary in severity, from rare extremes to frequent mild deviations, and that quantization sensitivity is unevenly distributed across layers and columns. These insights motivate a fine-grained, sensitivity-guided approach. We introduce MXSens, a training-free method that assigns mixed mantissa bitwidths (4/6/8) based on column- and layer-wise sensitivity, naturally leveraging the block-wise structure of MXINT. MXSens outperforms state-of-the-art quantization methods across a range of models and tasks. Under the W4A4KV4 setting, MXSens achieves perplexities of 3.77 and 7.63 on LLaMA-2-70B and LLaMA-3-8B, respectively, substantially improving over existing baselines on WikiText-2. Our work establishes a new balance between accuracy and resource efficiency for LLM quantization.

Figures

Figures reproduced from arXiv: 2607.17733 by Amir Yazdanbakhsh, Ayan Chakraborty, Babak Falsafi, Danila Mishin, Dongho Ha, Elizaveta Kostenok, Martin Jaggi, Simla Burcu Harma, Yunho Oh, Zhengyuan Su.

Figure 1
Figure 1. Figure 1: MXINT4 error propagation in q-proj for single-layer quantization on LLaMA-2-7B [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Perplexity error dynamics when upgrading outlier columns from 6 to 8 bits across linear layers. [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Perplexity error dynamics when changing the number of 8-bit columns. [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Error dynamics for LLaMA-3-8B [PITH_FULL_IMAGE:figures/full_fig_p020_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Error dynamics for Qwen1.5-7B A.9 CALIBRATION DATA [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

136 extracted references · 4 canonical work pages · 3 internal anchors

  1. [1]

    arXiv:1804.01526 [cs, stat] , author =

    Training. arXiv:1804.01526 [cs, stat] , author =. 2018 , note =

    Pith/arXiv arXiv 2018

  2. [2]

    arXiv preprint arXiv:2304.09145 , year=

    Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling , author=. arXiv preprint arXiv:2304.09145 , year=

    Pith/arXiv arXiv

  3. [3]

    2021 , eprint=

    Measuring Massive Multitask Language Understanding , author=. 2021 , eprint=

    2021

  4. [4]

    arXiv preprint arXiv:2109.04404 , year=

    All bark and no bite: Rogue dimensions in transformer language models obscure representational quality , author=. arXiv preprint arXiv:2109.04404 , year=

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2109.12948 , year=

    Understanding and overcoming the challenges of efficient transformer quantization , author=. arXiv preprint arXiv:2109.12948 , year=

    Pith/arXiv arXiv

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    Outlier suppression: Pushing the limit of low-bit transformer language models , author=. Advances in Neural Information Processing Systems , volume=

  7. [7]

    arXiv preprint arXiv:2310.10537 , year=

    Microscaling data formats for deep learning , author=. arXiv preprint arXiv:2310.10537 , year=

    Pith/arXiv arXiv

  8. [8]

    Proceedings of the 36th ACM International Conference on Supercomputing , pages=

    Efficiently emulating high-bitwidth computation with low-bitwidth hardware , author=. Proceedings of the 36th ACM International Conference on Supercomputing , pages=

  9. [9]

    Proceedings of the 52nd Annual International Symposium on Computer Architecture , pages=

    Avant-Garde: Empowering GPUs with Scaled Numeric Formats , author=. Proceedings of the 52nd Annual International Symposium on Computer Architecture , pages=

  10. [10]

    MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture , pages=

    Equinox: Training (for free) on a custom inference accelerator , author=. MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture , pages=

  11. [11]

    arXiv preprint arXiv:2205.11380 , year=

    Outliers dimensions that disrupt transformers are driven by frequency , author=. arXiv preprint arXiv:2205.11380 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2206.09557 , year=

    Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models , author=. arXiv preprint arXiv:2206.09557 , year=

    Pith/arXiv arXiv

  13. [13]

    arXiv preprint arXiv:2306.07629 , year=

    Squeezellm: Dense-and-sparse quantization , author=. arXiv preprint arXiv:2306.07629 , year=

    Pith/arXiv arXiv

  14. [14]

    CoRR , volume =

    Tim Dettmers and Mike Lewis and Younes Belkada and Luke Zettlemoyer , title =. CoRR , volume =. 2022 , url =. doi:10.48550/ARXIV.2208.07339 , eprinttype =. 2208.07339 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2208.07339 2022

  15. [15]

    2018 , eprint=

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , author=. 2018 , eprint=

    2018

  16. [16]

    2019 , eprint=

    HellaSwag: Can a Machine Really Finish Your Sentence? , author=. 2019 , eprint=

    2019

  17. [17]

    Microscaling Data Formats for Deep Learning , journal =

    Bita Darvish Rouhani and Ritchie Zhao and Ankit More and Mathew Hall and Alireza Khodamoradi and Summer Deng and Dhruv Choudhary and Marius Cornea and Eric Dellinger and Kristof Denolf and Dusan Stosic and Venmugil Elango and Maximilian Golub and Alexander Heinecke and Phil James. Microscaling Data Formats for Deep Learning , journal =. 2023 , url =. doi:...

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.10537 2023

  18. [18]

    SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot , booktitle =

    Elias Frantar and Dan Alistarh , editor =. SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot , booktitle =. 2023 , url =

    2023

  19. [19]

    arXiv preprint arXiv:2405.16406 , year=

    Spinquant: Llm quantization with learned rotations , author=. arXiv preprint arXiv:2405.16406 , year=

    Pith/arXiv arXiv

  20. [20]

    Zico Kolter , title =

    Mingjie Sun and Zhuang Liu and Anna Bair and J. Zico Kolter , title =. CoRR , volume =. 2023 , url =. doi:10.48550/ARXIV.2306.11695 , eprinttype =. 2306.11695 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2306.11695 2023

  21. [21]

    Mishra and Jorge Albericio Latorre and Jeff Pool and Darko Stosic and Dusan Stosic and Ganesh Venkatesh and Chong Yu and Paulius Micikevicius , title =

    Asit K. Mishra and Jorge Albericio Latorre and Jeff Pool and Darko Stosic and Dusan Stosic and Ganesh Venkatesh and Chong Yu and Paulius Micikevicius , title =. CoRR , volume =. 2021 , url =. 2104.08378 , timestamp =

    Pith/arXiv arXiv 2021

  22. [22]

    Susan Zhang and Stephen Roller and Naman Goyal and Mikel Artetxe and Moya Chen and Shuohui Chen and Christopher Dewan and Mona T. Diab and Xian Li and Xi Victoria Lin and Todor Mihaylov and Myle Ott and Sam Shleifer and Kurt Shuster and Daniel Simig and Punit Singh Koura and Anjali Sridhar and Tianlu Wang and Luke Zettlemoyer , title =. CoRR , volume =. 2...

    Pith/arXiv arXiv doi:10.48550/arxiv.2205.01068 2022

  23. [23]

    Proceedings of Machine Learning and Systems , volume=

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration , author=. Proceedings of Machine Learning and Systems , volume=

  24. [24]

    Proceedings of the 50th Annual International Symposium on Computer Architecture , pages=

    Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization , author=. Proceedings of the 50th Annual International Symposium on Computer Architecture , pages=

  25. [25]

    SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models , booktitle =

    Guangxuan Xiao and Ji Lin and Micka. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models , booktitle =. 2023 , url =

    2023

  26. [26]

    5th International Conference on Learning Representations,

    Stephen Merity and Caiming Xiong and James Bradbury and Richard Socher , title =. 5th International Conference on Learning Representations,. 2017 , url =

    2017

  27. [27]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Deep residual learning for image recognition , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  28. [28]

    Wikimedia Downloads

    Wikimedia. Wikimedia Downloads

  29. [29]

    The IEEE International Conference on Computer Vision (ICCV) , month =

    Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books , author =. The IEEE International Conference on Computer Vision (ICCV) , month =

  30. [30]

    2009 IEEE conference on computer vision and pattern recognition , pages=

    Imagenet: A large-scale hierarchical image database , author=. 2009 IEEE conference on computer vision and pattern recognition , pages=. 2009 , organization=

    2009

  31. [31]

    Gomez and Lukasz Kaiser and Illia Polosukhin , editor =

    Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin , editor =. Attention is All you Need , booktitle =. 2017 , url =

    2017

  32. [32]

    Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,

    Jacob Devlin and Ming. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,. 2019 , url =. doi:10.18653/V1/N19-1423 , timestamp =

    doi:10.18653/v1/n19-1423 2019

  33. [33]

    9th International Conference on Learning Representations,

    Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby , title =. 9th International Conference on Learning Representations,. 2021 , url =

    2021

  34. [34]

    2019 , file =

    Google Cloud Blog , author =. 2019 , file =

    2019

  35. [35]

    2024 , eprint=

    Qwen2 Technical Report , author=. 2024 , eprint=

    2024

  36. [36]

    arXiv:1710.03740 [cs, stat] , author =

    Mixed. arXiv:1710.03740 [cs, stat] , author =. 2018 , note =

    Pith/arXiv arXiv 2018

  37. [37]

    Chung and Zhaoxia (Summer) Deng and Sam Naghshineh and Jongsoo Park and Maxim Naumov , editor =

    Bita Darvish Rouhani and Ritchie Zhao and Venmugil Elango and Rasoul Shafipour and Mathew Hall and Maral Mesmakhosroshahi and Ankit More and Levi Melnick and Maximilian Golub and Girish Varatkar and Lai Shao and Gaurav Kolhe and Dimitry Melts and Jasmine Klar and Renee L'Heureux and Matt Perry and Doug Burger and Eric S. Chung and Zhaoxia (Summer) Deng an...

    arXiv 2023

  38. [38]

    Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point , url =

    Darvish Rouhani, Bita and Lo, Daniel and Zhao, Ritchie and Liu, Ming and Fowers, Jeremy and Ovtcharov , Kalin and Vinogradsky, Anna and Massengill , Sarah and Yang, Lita and Bittner, Ray and Forin, Alessandro and Zhu, Haishan and Na, Taesik and Patel, Prerak and Che, Shuai and Chand Koppaka , Lok and SONG, XIA and Som, Subhojit and Das, Kaustav and T, Sau...

  39. [39]

    2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA) , pages=

    Fast: DNN training under variable precision block floating point with stochastic rounding , author=. 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA) , pages=. 2022 , organization=

    2022

  40. [40]

    Oberman and Mohammad Shoeybi and Michael Y

    Paulius Micikevicius and Dusan Stosic and Neil Burgess and Marius Cornea and Pradeep Dubey and Richard Grisenthwaite and Sangwon Ha and Alexander Heinecke and Patrick Judd and John Kamalu and Naveen Mellempudi and Stuart F. Oberman and Mohammad Shoeybi and Michael Y. Siu and Hao Wu , title =. CoRR , volume =. 2022 , url =. doi:10.48550/arXiv.2209.05433 , ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2209.05433 2022

  41. [41]

    Llama 2: Open Foundation and Fine-Tuned Chat Models , journal =

    Hugo Touvron and Louis Martin and Kevin Stone and Peter Albert and Amjad Almahairi and Yasmine Babaei and Nikolay Bashlykov and Soumya Batra and Prajjwal Bhargava and Shruti Bhosale and Dan Bikel and Lukas Blecher and Cristian Canton. Llama 2: Open Foundation and Fine-Tuned Chat Models , journal =. 2023 , url =. doi:10.48550/ARXIV.2307.09288 , eprinttype ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.09288 2023

  42. [42]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  43. [43]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    OPQ: Compressing deep neural networks with one-shot pruning-quantization , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  44. [44]

    ACM Transactions on Embedded Computing Systems (TECS) , volume=

    Quantized sparse training: a unified trainable framework for joint pruning and quantization in DNNs , author=. ACM Transactions on Embedded Computing Systems (TECS) , volume=. 2022 , publisher=

    2022

  45. [45]

    Advances in Neural Information Processing Systems , volume=

    Optimal brain compression: A framework for accurate post-training quantization and pruning , author=. Advances in Neural Information Processing Systems , volume=

  46. [46]

    ArXiv , year=

    Accelerating Sparse Deep Neural Networks , author=. ArXiv , year=

  47. [47]

    International Conference on machine learning , pages=

    Train large, then compress: Rethinking model size for efficient training and inference of transformers , author=. International Conference on machine learning , pages=. 2020 , organization=

    2020

  48. [48]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Boost vision transformer with gpu-friendly sparsity and quantization , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  49. [49]

    arXiv preprint arXiv:2010.01892 , year=

    Joint pruning & quantization for extremely sparse neural networks , author=. arXiv preprint arXiv:2010.01892 , year=

    Pith/arXiv arXiv 2010

  50. [50]

    Towards Optimal Compression: Joint Pruning and Quantization

    Ben Zandonati and Glenn Bucagu and Adrian Alan Pol and Maurizio Pierini and Olya Sirkin and Tal Kopetz , title =. CoRR , volume =. 2023 , url =. doi:10.48550/ARXIV.2302.07612 , eprinttype =. 2302.07612 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2302.07612 2023

  51. [51]

    CoRR , volume =

    Yujun Lin and Haotian Tang and Shang Yang and Zhekai Zhang and Guangxuan Xiao and Chuang Gan and Song Han , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2405.04532 , eprinttype =. 2405.04532 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.04532 2024

  52. [52]

    Deep Compression of Pre-trained Transformer Models , booktitle =

    Naigang Wang and Chi. Deep Compression of Pre-trained Transformer Models , booktitle =

  53. [53]

    arXiv preprint arXiv:2110.08271 , year=

    Training deep neural networks with joint quantization and pruning of weights and activations , author=. arXiv preprint arXiv:2110.08271 , year=

    Pith/arXiv arXiv

  54. [54]

    Frontiers in Artificial Intelligence , volume=

    Ps and qs: Quantization-aware pruning for efficient low latency neural network inference , author=. Frontiers in Artificial Intelligence , volume=. 2021 , publisher=

    2021

  55. [55]

    International Conference on Machine Learning , pages=

    Deja vu: Contextual sparsity for efficient llms at inference time , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  56. [56]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Apq: Joint search for network architecture, pruning and quantization policy , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  57. [57]

    Advances in neural information processing systems , volume=

    Learning both weights and connections for efficient neural network , author=. Advances in neural information processing systems , volume=

  58. [58]

    arXiv preprint arXiv:1510.00149 , year=

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding , author=. arXiv preprint arXiv:1510.00149 , year=

    Pith/arXiv arXiv

  59. [59]

    Advances in neural information processing systems , volume=

    Dynamic network surgery for efficient dnns , author=. Advances in neural information processing systems , volume=

  60. [60]

    Advances in neural information processing systems , volume=

    Learning structured sparsity in deep neural networks , author=. Advances in neural information processing systems , volume=

  61. [61]

    Advances in neural information processing systems , volume=

    Optimal brain damage , author=. Advances in neural information processing systems , volume=

  62. [62]

    IEEE international conference on neural networks , pages=

    Optimal brain surgeon and general network pruning , author=. IEEE international conference on neural networks , pages=. 1993 , organization=

    1993

  63. [63]

    Advances in neural information processing systems , volume=

    Movement pruning: Adaptive sparsity by fine-tuning , author=. Advances in neural information processing systems , volume=

  64. [64]

    2024 , eprint=

    The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits , author=. 2024 , eprint=

    2024

  65. [65]

    arXiv preprint arXiv:2203.07259 , year=

    The optimal bert surgeon: Scalable and accurate second-order pruning for large language models , author=. arXiv preprint arXiv:2203.07259 , year=

    Pith/arXiv arXiv

  66. [66]

    The Thirty-Third

    Zhuliang Yao and Shijie Cao and Wencong Xiao and Chen Zhang and Lanshun Nie , title =. The Thirty-Third. 2019 , url =. doi:10.1609/AAAI.V33I01.33015676 , timestamp =

    work page doi:10.1609/aaai.v33i01.33015676 2019

  67. [67]

    Accelerator-Aware Pruning for Convolutional Neural Networks , journal =

    Hyeong. Accelerator-Aware Pruning for Convolutional Neural Networks , journal =. 2020 , url =. doi:10.1109/TCSVT.2019.2911674 , timestamp =

    arXiv 2020

  68. [68]

    Channel Permutations for

    Jeff Pool and Chong Yu , editor =. Channel Permutations for. Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages =. 2021 , url =

    2021

  69. [69]

    9th International Conference on Learning Representations,

    Aojun Zhou and Yukun Ma and Junnan Zhu and Jianbo Liu and Zhijie Zhang and Kun Yuan and Wenxiu Sun and Hongsheng Li , title =. 9th International Conference on Learning Representations,. 2021 , url =

    2021

  70. [70]

    2021 , URL =

    Nvidia , title =. 2021 , URL =

    2021

  71. [71]

    2022 , URL =

    Nvidia , title =. 2022 , URL =

    2022

  72. [72]

    2020 , URL =

    Nvidia , title =. 2020 , URL =

    2020

  73. [73]

    Wei Sun and Aojun Zhou and Sander Stuijk and Rob G. J. Wijnhoven and Andrew Nelson and Hongsheng Li and Henk Corporaal , editor =. DominoSearch: Find layer-wise fine-grained. Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages =. 2021 , url =

    2021

  74. [74]

    Accelerated Sparse Neural Training:

    Itay Hubara and Brian Chmiel and Moshe Island and Ron Banner and Joseph Naor and Daniel Soudry , editor =. Accelerated Sparse Neural Training:. Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages =. 2021 , url =

    2021

  75. [75]

    International Conference on Machine Learning,

    Yucheng Lu and Shivani Agrawal and Suvinay Subramanian and Oleg Rybakov and Christopher De Sa and Amir Yazdanbakhsh , editor =. International Conference on Machine Learning,. 2023 , url =

    2023

  76. [76]

    arXiv preprint arXiv:2211.10737 , year=

    Accuracy Boosters: Epoch-Driven Mixed-Mantissa Block Floating-Point for DNN Training , author=. arXiv preprint arXiv:2211.10737 , year=

    Pith/arXiv arXiv

  77. [77]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Rigging the Lottery: Making All Tickets Winners , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , editor =

    2020

  78. [78]

    2023 , eprint=

    Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases , author=. 2023 , eprint=

    2023

  79. [79]

    Proceedings of Machine Learning and Systems , volume=

    Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference , author=. Proceedings of Machine Learning and Systems , volume=

  80. [80]

    Advances in neural information processing systems , volume=

    Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point , author=. Advances in neural information processing systems , volume=

Showing first 80 references.