Pith. sign in

REVIEW 3 major objections 123 references

Reliability of quantized LLMs peaks at 4 bits, even though raw accuracy keeps rising with total model bits.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 08:45 UTC pith:4ASEU4LS

load-bearing objection Solid empirical map of reliability vs total bits that cleanly shows a 4-bit peak; the envelope-plotting and incomplete efficiency axis are real but do not erase the main result. the 3 major comments →

arxiv 2607.10855 v1 pith:4ASEU4LS submitted 2026-07-12 cs.LG

Reliability Scaling Laws for Quantized Large Language Models

Sirine Ayadi , S\'andor Dar\'oczi , Stephan G\"unnemann , Bertrand Charpentier This is my paper
classification cs.LG
keywords quantized LLMsbit-level scaling lawsuncertainty quantificationcalibrationrobustnessnatural input perturbationspost-training quantizationreliability-efficiency trade-off
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether compressing large language models by reducing the bit-width of their weights makes them less trustworthy when inputs are noisy or slightly wrong. The authors evaluate many models (1B to 70B parameters) quantized to 2, 3, 4 and 8 bits with six modern methods, measuring not only accuracy but uncertainty quality, calibration, and robustness under 15 character- and word-level perturbations that preserve meaning (typos, slang, emojis, leetspeak, mild translations). They plot every metric against the single efficiency axis of total model bits (parameter count times bit-width). Accuracy improves steadily as total bits grow, but reliability metrics do not: they rise, peak for 4-bit models of moderate size, and often fall again for larger full-precision models. The same 4-bit sweet spot appears across LLaMA, OPT and Qwen families and across several tasks. Quantization can even improve robustness relative to the original 16-bit model. The practical claim is therefore that, for a fixed memory budget, a moderately sized 4-bit model is usually the most reliable choice.

Core claim

While downstream task performance of quantized LLMs scales monotonically with total model bits B = (#parameters) × (bitwidth), reliability metrics (entropy-based AUCROC, calibration scores, and accuracy under 15 natural input perturbations) are nonlinear and reach a clear peak for 4-bit quantized models; thus moderately sized 4-bit models give the best reliability-efficiency trade-off, and quantization can improve robustness over the full-precision base.

What carries the argument

Bit-level scaling: every model is located on the single horizontal axis of total bits B, and log-quadratic curves are fitted to accuracy and reliability metrics so that the 4-bit reliability peak becomes visible across families and methods.

Load-bearing premise

That total model bits alone is a fair enough efficiency axis that the observed reliability peak at 4 bits would survive if real serving costs such as activation precision, KV-cache size or hardware latency were measured instead.

What would settle it

Re-plot the same reliability metrics against measured inference latency or peak memory on a production GPU for matched total-bit budgets; if the 4-bit models no longer sit at the reliability maximum, the claimed sweet spot is an artifact of the bit-count axis.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For a fixed memory or storage budget, practitioners should prefer a larger 4-bit model over a smaller 8- or 16-bit model when reliability under natural noise matters.
  • Extreme 2-bit and 3-bit post-training quantization remains unreliable except when model scale is large or when quantization-aware training is used.
  • Natural, non-adversarial input noise (typos, slang, emojis) is a useful stress test that reveals reliability gains from quantization that standard clean benchmarks miss.
  • Pruning does not produce the same reliability peak that 4-bit quantization does, so the two compression families are not interchangeable for trustworthy deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the 4-bit peak is driven by a sweet-spot KL divergence to the full-precision teacher, then methods that further reduce that divergence (better calibration data, rotation, or QAT) should push the reliability optimum toward still lower bit-widths.
  • Serving systems that keep activations and KV cache at higher precision may erase part of the 4-bit advantage; the paper's bit-count ranking therefore needs re-validation under end-to-end memory footprints.
  • The same nonlinear reliability curve may appear for other compression axes (sparsity patterns, distillation) once they are plotted against an analogous resource unit.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper evaluates reliability of post-training quantized LLMs (uncertainty via token entropy and log-likelihood, calibration via Brier score, and robustness under 15 character- and word-level natural perturbations) across LLaMA-3/3.2, OPT, and Qwen3 families, six PTQ methods, and bitwidths 2–16. It reports bit-level scaling with total model bits B = (#parameters)×(bitwidth), fitting log-quadratic curves L(B)=a(log B)^2+b log B+c. Accuracy and perplexity improve roughly monotonically with B and favor 4-bit models for a fixed budget; reliability metrics (AUCROC of entropy, calibration, robust accuracy) are nonlinear and peak at 4-bit. A KL analysis of token-distribution shift relative to the full-precision base is offered as an explanation, and pruning is shown not to yield comparable reliability gains. The authors conclude that moderately sized 4-bit models give the best reliability–efficiency trade-off and that quantization can improve robustness to natural input noise.

Significance. If the 4-bit reliability peak is robust, the work supplies a practical, previously missing design rule for deploying trustworthy quantized LLMs under a fixed bit budget, and it expands the evaluation of compression beyond accuracy/perplexity to uncertainty, calibration, and natural (non-adversarial) robustness. Strengths include breadth (multiple families, six PTQ methods, 15 perturbations at two intensities, KL behavioral-shift analysis, pruning and limited QAT ablations) and the explicit separation of performance scaling from reliability scaling. The empirical regularities are descriptive rather than derived laws, but they are falsifiable and immediately actionable for practitioners choosing bitwidth versus model size.

major comments (3)
  1. Appendix A.1 and Tables 6–7 state that only the best-performing method is plotted for each bitwidth. Consequently the 4-bit curves are upper envelopes (AWQ/HQQ/GPTQ) while 2-/3-bit curves are lower envelopes of weaker methods. The claimed nonlinear reliability peak at 4 bits (Abstract, Fig. 1, §5.2) may therefore be an artifact of method selection rather than of bitwidth. Either plot all methods (or method-averaged curves with error bars) or demonstrate that the peak survives when the same method is held fixed across bitwidths.
  2. Section 4 and Table 1 define the efficiency axis solely as weight storage bits B. Activation precision, KV-cache cost, and hardware-supported latency/throughput are acknowledged in §C but not measured for the reliability rankings. Because real serving cost can reorder models of equal B, the claim that “moderately sized 4-bit models offer the best reliability-efficiency trade-off” is only partially supported. At minimum, report latency/throughput (or a proxy that includes activations) for the same model pairs used in the reliability plots, or qualify the claim as storage-bit efficiency only.
  3. Generation is truncated to 20 tokens and evaluation uses 1000 randomly sampled prompts (A.2). While Fig. 9 shows accuracy is stable under this subsample, reliability metrics (AUCROC of entropy, Brier) are more sensitive to sequence length and sample size; the limited ablation in Fig. 14 does not fully close the gap. Confirm that the 4-bit peak persists for longer generations and full-dataset evaluation on at least one primary benchmark.

Circularity Check

0 steps flagged

No circularity: purely observational empirical study; log-quadratic fits are descriptive summaries of measured points, not predictions derived from the same coefficients.

full rationale

The paper's central claims are empirical regularities measured on concrete models (LLaMA/OPT/Qwen families, six PTQ methods, 2–16 bits) under fixed reliability metrics (token entropy, Brier, AUCROC, accuracy under 15 natural perturbations). Section 4 explicitly states that the authors 'do not propose a formal predictive law in the classical sense' and adopt only the same empirical methodology used by prior bit-level scaling papers: they fit L(B)=a(log B)^2+b log B+c after the fact to summarize observed points. No coefficient fitted on one subset is then used to 'predict' a closely related quantity; no uniqueness theorem or self-citation is load-bearing for the 4-bit peak; the peak is simply the location of the highest measured reliability points among the evaluated configurations. Method selection (reporting only the best method per bit-width) and the incompleteness of the total-bits axis are methodological choices that may affect external validity, but they do not make any reported number equal to its own input by construction. The derivation chain therefore contains no self-definitional, fitted-as-prediction, or self-citation-forced steps.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claim rests on empirical measurements under a small set of modeling choices (token-averaged metrics, total-bit axis, chosen perturbation intensities, log-quadratic descriptive fits) rather than on free physical constants or invented particles. The free parameters are the fitted scaling coefficients; the axioms are standard domain assumptions about how to score generative reliability.

free parameters (3)
  • log-quadratic coefficients a,b,c per metric and bitwidth = per-curve least-squares (values not tabulated)
    Fitted to observed (B, L) points to visualize trends; not used for out-of-sample prediction.
  • perturbation intensity levels {4,16} = 4 and 16
    Hand-chosen discrete strengths at which character/word edits are applied; results are reported only at these two levels.
  • generation length cap (20 tokens) and temperature (0.7) = 20 tokens, T=0.7
    Fixed decoding hyperparameters that affect entropy and accuracy measurements; ablated only lightly.
axioms (4)
  • domain assumption Sequence-level reliability equals the average of token-level entropy / Brier / log-likelihood over the generated tokens.
    Stated in §3.1 Eq. (2); standard but not the only possible aggregation for open-ended generation.
  • domain assumption Total model bits B = (#parameters)×(weight bitwidth) is a sufficient efficiency axis for ranking reliability–efficiency trade-offs.
    Justified in §4 and Table 1 by correlation with storage and inference memory; latency/throughput relation is acknowledged as hardware-dependent.
  • domain assumption The 15 character- and word-level edits preserve semantics sufficiently to serve as realistic robustness tests.
    Motivated by digital-communication literature in §3.2; no human semantic-preservation study is reported.
  • domain assumption Post-training quantization methods can be compared fairly by loading public or library implementations without re-tuning calibration data.
    Implicit throughout §5; different methods use different calibration sets by design.
invented entities (1)
  • 15-type natural perturbation suite (emoji, leetspeak, slang, multi-language word translation, etc.) no independent evidence
    purpose: Provide a standardized, non-adversarial robustness stress test for quantized LLMs.
    Assembled from prior typo/adversarial literature plus novel emoji/slang/translation variants; no external validation that the suite is complete or calibrated to real user error rates.

pith-pipeline@v1.1.0-grok45 · 26361 in / 2942 out tokens · 45897 ms · 2026-07-14T08:45:02.924097+00:00 · methodology

0 comments
read the original abstract

Quantization is a powerful strategy to build capable and resource-efficient large language models (LLMs) by reducing the bitwidth of the parameters. While quantized LLMs achieve state-of-the-art performance on unperturbed inputs using standard predictive metrics, their performance on perturbed inputs, measured using reliability metrics, remains underexplored, despite its importance for reliable deployment. To address this gap, we first conduct a comprehensive reliability evaluation of quantized LLMs consisting of three key components: (1) Uncertainty: We assess the trustworthiness of LLMs quantized to 2, 3, 4, and 8 bits using six different quantization methods, employing established uncertainty metrics. (2) Calibration: We assess how well-calibrated the uncertainty estimates of quantized models are across model scales and bit precisions. (3) Robustness: We design character-level and word-level input perturbations to evaluate the reliability of quantized models under semantically-preserving variations in the inputs that arise in real-world applications. Second, we characterize how reliability scales with the total number of model bits. Our study reveals that while the performance scales monotonically with the total number of bits, the reliability scalings are nonlinear. A reliability peak occurs for 4-bit quantized models, indicating that quantizing moderately sized models offers the best reliability-efficiency trade-off. Additionally, our empirical findings reveal that quantization enhances the robustness of LLMs to natural input perturbations.

Figures

Figures reproduced from arXiv: 2607.10855 by Bertrand Charpentier, S\'andor Dar\'oczi, Sirine Ayadi, Stephan G\"unnemann.

Figure 1
Figure 1. Figure 1: Bit-level scaling trends of the accuracy and AUCROC(Entropy) on TriviaQA. We use four [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our perturbations. Illustrated is an example where perturbations with intensity level 1 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Radar plots of the accuracy (Top) and AUCROC (Entropy) (bottom) across all 15 character-level and word-level perturbations for two intensities. We evaluate the base LLaMa-3-8B model and five 4-bit quantization methods. Quantized models can provide more reliable uncertainty estimates under natural perturbations compared to their base counterparts, while maintaining a close performance. from the sequence. Th… view at source ↗
Figure 4
Figure 4. Figure 4: Scalings of the perplexity (top) and accuracy (bottom) for all quantized models and their corresponding full-precision models. The performance steadily improves with the total number of model bits. 4-bit models offer the best performance-efficiency trade-off given a fixed number of total model bits. Base and quantized LLMs. We consider four base pre-trained models (Grattafiori et al., 2024), including mode… view at source ↗
Figure 5
Figure 5. Figure 5: Bit-level scalings for all evaluated quantized models and their corresponding full-precision models [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Scaling trends of the KL￾Divergence. Why does 4-bit quantization offer a favorable reliability￾efficiency trade-off? The goal of model quantization is to create a more efficient model from a full-precision base model, while main￾taining as close a distance to its full-precision counterpart. While both perplexity and accuracy metrics are essential for evaluating the generalization capabilities of quantized … view at source ↗
Figure 7
Figure 7. Figure 7: Scaling trends of OPT and Qwen3 models across different benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Scaling trends of pruned LLaMA models using SparseGPT and Wanda. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Comparison of the accuracy when evaluating on the full datasets versus evaluating on 1000 randomly [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Bit-level scaling trends of Qwen3 models across four benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Bit-level scaling trends of LLaMA models across four benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Scaling behavior of the reliability metrics of the [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Bit-level inference scaling trends on TriviaQA for various temperature values for sampling. [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Bit-level inference scaling trends on TriviaQA for different numbers of output tokens. [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Bit-level scaling trends of quantized LLaMA models using EfficientQAT (Chen et al., 2025). [PITH_FULL_IMAGE:figures/full_fig_p027_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Bit-level inference scaling trends of LLaMA models under different 4-bit quantization methods. [PITH_FULL_IMAGE:figures/full_fig_p028_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Bit-level inference scaling trends of OPT models under different 4-bit quantization methods. [PITH_FULL_IMAGE:figures/full_fig_p028_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Bit-level inference scaling trends of Qwen3 models under different 4-bit quantization methods. [PITH_FULL_IMAGE:figures/full_fig_p029_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Additional KL-divergence analysis across LLaMA-3, OPT, and Qwen3 models on Wikitext (top) [PITH_FULL_IMAGE:figures/full_fig_p029_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

123 extracted references · 1 canonical work pages

  1. [2]

    Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated LLM s. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=dfqsW38v1X

    2024

  2. [3]

    Half- Quadratic Quantization of Large Machine Learning Models , November 2023

    Hicham Badri and Appu Shaji. Half- Quadratic Quantization of Large Machine Learning Models , November 2023. URL https://mobiusml.github.io/hqq_blog/

    2023

  3. [4]

    Towards 1-bit Machine Learning Models , March 2024

    Hicham Badri and Appu Shaji. Towards 1-bit Machine Learning Models , March 2024. URL https://mobiusml.github.io/1bit_blog/

    2024

  4. [5]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pp.\ 7432--7439, 2020

    2020

  5. [6]

    Verification of forecasts expressed in terms of probability

    Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78 0 (1): 0 1--3, 1950

    1950

  6. [7]

    Efficientqat: Efficient quantization-aware training for large language models

    Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 10081--10100, 2025

    2025

  7. [9]

    The case for 4-bit precision: k-bit inference scaling laws

    Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, pp.\ 7750--7774. PMLR, 2023

    2023

  8. [10]

    bitsandbytes: Efficient CUDA Primitives for 8-bit and 4-bit Neural Network Quantization

    Tim Dettmers, Artidoro Pagnoni, Alexander Borzunov, and Mike Lewis. bitsandbytes: Efficient CUDA Primitives for 8-bit and 4-bit Neural Network Quantization . https://github.com/bitsandbytes-foundation/bitsandbytes, 2022. Accessed: March 2025

    2022

  9. [13]

    Patterns of misspellings in l2 and l1 english: A view from the ets spelling corpus

    M Flor, Y Futagi, M Lopez, and M Mulholland. Patterns of misspellings in l2 and l1 english: A view from the ets spelling corpus. bergen language and linguistics studies, 6, 107--132, 2015

    2015

  10. [14]

    Unsupervised quality estimation for neural machine translation

    Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Fr \'e d \'e ric Blain, Francisco Guzm \'a n, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8: 0 539--555, 2020

    2020

  11. [15]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp.\ 10323--10337. PMLR, 2023

    2023

  12. [20]

    Interactions between text content and emoji types determine perceptions of both messages and senders

    Christopher J Hand, Kassandra Burd, Alex Oliver, and Christopher M Robus. Interactions between text content and emoji types determine perceptions of both messages and senders. Computers in human behavior reports, 8: 0 100242, 2022

    2022

  13. [26]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36: 0 62991--63010, 2023

    2023

  14. [28]

    How can we know when language models know? on the calibration of language models for question answering

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9: 0 962--977, 2021

    2021

  15. [32]

    Calibration-tuning: Teaching large language models to know what they don’t know

    Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. Calibration-tuning: Teaching large language models to know what they don’t know. In Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024), pp.\ 1--14, 2024

    2024

  16. [37]

    Less is more: Task-aware layer-wise distillation for language model compression

    Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, pp.\ 20852--20867. PMLR, 2023

    2023

  17. [38]

    Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics

    Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04), pp.\ 605--612, 2004

    2004

  18. [39]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 0 87--100, 2024 a

    2024

  19. [40]

    QServe : W4A8KV4 Quantization and System Co -design for Efficient LLM Serving , May 2024 b

    Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. QServe : W4A8KV4 Quantization and System Co -design for Efficient LLM Serving , May 2024 b . URL http://arxiv.org/abs/2405.04532. arXiv:2405.04532 [cs]

    Pith/arXiv arXiv 2024

  20. [43]

    Llm-qat: Data-free quantization aware training for large language models

    Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 467--484, 2024

    2024

  21. [46]

    PV -tuning: Beyond straight-through estimation for extreme LLM compression

    Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Pavlovich Burlachenko, Kai Yi, Dan Alistarh, and Peter Richt \'a rik. PV -tuning: Beyond straight-through estimation for extreme LLM compression. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=YvA8UF0I37

    2024

  22. [47]

    Building a large annotated corpus of english: The penn treebank

    Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19 0 (2): 0 313--330, 1993

    1993

  23. [50]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  24. [51]

    Coqa: A conversational question answering challenge

    Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7: 0 249--266, 2019

    2019

  25. [52]

    Beyond neural scaling laws: beating power law scaling via data pruning

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 0 19523--19536, 2022

    2022

  26. [53]

    Colloquail, slang and transformational language: comperative study

    Suhardianto Suhardianto et al. Colloquail, slang and transformational language: comperative study. Jurnal Basis, 6 0 (1): 0 105--118, 2019

    2019

  27. [57]

    Optimum quanto: A pytorch quantization backend for optimum

    Hugging Face Team. Optimum quanto: A pytorch quantization backend for optimum. https://huggingface.co/docs/optimum/quanto/index, 2023. Accessed: March 2025

    2023

  28. [59]

    Are large language models really robust to word-level perturbations? Submitted to Transactions on Machine Learning Research, 2024

    Haoyu Wang, Guozheng Ma, Cong Yu, Ning Gui, Linrui Zhang, Zhiqi Huang, Suwei Ma, Yongzhe Chang, Sen Zhang, Li Shen, Xueqian Wang, Peilin Zhao, and Dacheng Tao. Are large language models really robust to word-level perturbations? Submitted to Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=BMKJEGNMcZ. Rejected

    2024

  29. [62]

    Onebit: Towards extremely low-bit large language models

    Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. Advances in Neural Information Processing Systems, 37: 0 66357--66382, 2024 a

    2024

  30. [65]

    Benchmarking llms via uncertainty quantification

    Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking llms via uncertainty quantification. Advances in Neural Information Processing Systems, 37: 0 15356--15385, 2024

    2024

  31. [69]

    Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04) , pages=

    Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics , author=. Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04) , pages=

  32. [70]

    Text summarization branches out , pages=

    Rouge: A package for automatic evaluation of summaries , author=. Text summarization branches out , pages=

  33. [71]

    arXiv preprint arXiv:2403.15447 , year=

    Decoding compressed trust: Scrutinizing the trustworthiness of efficient llms under compression , author=. arXiv preprint arXiv:2403.15447 , year=

    Pith/arXiv arXiv

  34. [72]

    arXiv preprint arXiv:2407.04965 , year=

    Beyond perplexity: Multi-dimensional safety evaluation of llm compression , author=. arXiv preprint arXiv:2407.04965 , year=

    Pith/arXiv arXiv

  35. [73]

    arXiv preprint arXiv:2310.01382 , year=

    Compressing llms: The truth is rarely pure and never simple , author=. arXiv preprint arXiv:2310.01382 , year=

    Pith/arXiv arXiv

  36. [74]

    arXiv preprint arXiv:2407.09141 , year=

    Accuracy is not all you need , author=. arXiv preprint arXiv:2407.09141 , year=

    Pith/arXiv arXiv

  37. [75]

    Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages=

    Bleu: a method for automatic evaluation of machine translation , author=. Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages=

  38. [76]

    arXiv preprint arXiv:2002.07650 , year=

    Uncertainty estimation in autoregressive structured prediction , author=. arXiv preprint arXiv:2002.07650 , year=

    Pith/arXiv arXiv 2002

  39. [77]

    arXiv preprint arXiv:2305.19187 , year=

    Generating with confidence: Uncertainty quantification for black-box large language models , author=. arXiv preprint arXiv:2305.19187 , year=

    Pith/arXiv arXiv

  40. [78]

    arXiv preprint arXiv:1705.03551 , year=

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension , author=. arXiv preprint arXiv:1705.03551 , year=

    Pith/arXiv arXiv

  41. [79]

    Transactions of the Association for Computational Linguistics , volume=

    Coqa: A conversational question answering challenge , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher=

    2019

  42. [80]

    arXiv preprint arXiv:1609.07843 , year=

    Pointer sentinel mixture models , author=. arXiv preprint arXiv:1609.07843 , year=

    Pith/arXiv arXiv

  43. [81]

    Computational linguistics , volume=

    Building a large annotated corpus of English: The Penn Treebank , author=. Computational linguistics , volume=

  44. [82]

    Journal of machine learning research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=

  45. [83]

    Low-power computer vision , pages=

    A survey of quantization methods for efficient neural network inference , author=. Low-power computer vision , pages=. 2022 , publisher=

    2022

  46. [84]

    int8 (): 8-bit matrix multiplication for transformers at scale , author=

    Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale , author=. Advances in neural information processing systems , volume=

  47. [85]

    arXiv preprint arXiv:2210.17323 , year=

    Gptq: Accurate post-training quantization for generative pre-trained transformers , author=. arXiv preprint arXiv:2210.17323 , year=

    Pith/arXiv arXiv

  48. [86]

    Proceedings of Machine Learning and Systems , volume=

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration , author=. Proceedings of Machine Learning and Systems , volume=

  49. [87]

    arXiv preprint arXiv:2401.06118 , year=

    Extreme compression of large language models via additive quantization , author=. arXiv preprint arXiv:2401.06118 , year=

    Pith/arXiv arXiv

  50. [88]

    arXiv preprint arXiv:2306.03078 , year=

    Spqr: A sparse-quantized representation for near-lossless llm weight compression , author=. arXiv preprint arXiv:2306.03078 , year=

    Pith/arXiv arXiv

  51. [89]

    Advances in Neural Information Processing Systems , volume=

    Optimal brain compression: A framework for accurate post-training quantization and pruning , author=. Advances in Neural Information Processing Systems , volume=

  52. [90]

    arXiv preprint arXiv:2305.17888 , year=

    Llm-qat: Data-free quantization aware training for large language models , author=. arXiv preprint arXiv:2305.17888 , year=

    Pith/arXiv arXiv

  53. [91]

    arXiv preprint arXiv:2402.10631 , year=

    Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation , author=. arXiv preprint arXiv:2402.10631 , year=

    Pith/arXiv arXiv

  54. [92]

    arXiv preprint arXiv:2108.12237 , year=

    Evaluating the robustness of neural language models to input perturbations , author=. arXiv preprint arXiv:2108.12237 , year=

    Pith/arXiv arXiv

  55. [93]

    arXiv preprint arXiv:2302.09664 , year=

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation , author=. arXiv preprint arXiv:2302.09664 , year=

    Pith/arXiv arXiv

  56. [94]

    Transactions of the Association for Computational Linguistics , volume=

    Unsupervised quality estimation for neural machine translation , author=. Transactions of the Association for Computational Linguistics , volume=. 2020 , publisher=

    2020

  57. [95]

    Advances in Neural Information Processing Systems , volume=

    Benchmarking llms via uncertainty quantification , author=. Advances in Neural Information Processing Systems , volume=

  58. [96]

    Transactions of the Association for Computational Linguistics , volume=

    How can we know when language models know? on the calibration of language models for question answering , author=. Transactions of the Association for Computational Linguistics , volume=. 2021 , publisher=

    2021

  59. [97]

    Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024) , pages=

    Calibration-tuning: Teaching large language models to know what they don’t know , author=. Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024) , pages=

    2024

  60. [98]

    Monthly weather review , volume=

    Verification of forecasts expressed in terms of probability , author=. Monthly weather review , volume=

  61. [99]

    arXiv preprint arXiv:2407.11282 , year=

    Uncertainty is fragile: Manipulating uncertainty in large language models , author=. arXiv preprint arXiv:2407.11282 , year=

    Pith/arXiv arXiv

  62. [100]

    arXiv preprint arXiv:2207.05221 , year=

    Language models (mostly) know what they know , author=. arXiv preprint arXiv:2207.05221 , year=

    Pith/arXiv arXiv

  63. [101]

    arXiv preprint arXiv:2309.03882 , year=

    Large language models are not robust multiple choice selectors , author=. arXiv preprint arXiv:2309.03882 , year=

    Pith/arXiv arXiv

  64. [102]

    arXiv preprint arXiv:1704.04441 , year=

    How robust are character-based word embeddings in tagging and MT against wrod scramlbing or randdm nouse? , author=. arXiv preprint arXiv:1704.04441 , year=

    Pith/arXiv arXiv

  65. [103]

    ACM Transactions on Intelligent Systems and Technology (TIST) , volume=

    Adversarial attacks on deep-learning models in natural language processing: A survey , author=. ACM Transactions on Intelligent Systems and Technology (TIST) , volume=. 2020 , publisher=

    2020

  66. [104]

    arXiv preprint arXiv:2412.02904 , year=

    Enhancing Trust in Large Language Models with Uncertainty-Aware Fine-Tuning , author=. arXiv preprint arXiv:2412.02904 , year=

    Pith/arXiv arXiv

  67. [105]

    arXiv preprint arXiv:2502.16440 , year=

    Compression Scaling Laws: Unifying Sparsity and Quantization , author=. arXiv preprint arXiv:2502.16440 , year=

    Pith/arXiv arXiv

  68. [106]

    International Conference on Machine Learning , pages=

    The case for 4-bit precision: k-bit inference scaling laws , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  69. [107]

    A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios

    Ackerman, Samuel and Rabinovich, Ella and Farchi, Eitan and Anaby Tavor, Ateret. A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.158

    work page doi:10.18653/v1/2024.findings-emnlp.158 2024

  70. [108]

    Submitted to Transactions on Machine Learning Research , year=

    Are Large Language Models Really Robust to Word-Level Perturbations? , author=. Submitted to Transactions on Machine Learning Research , year=

  71. [109]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  72. [110]

    Badri, Hicham and Shaji, Appu , month =. Half-

  73. [111]

    Towards 1-bit

    Badri, Hicham and Shaji, Appu , month =. Towards 1-bit

  74. [112]

    GitHub repository , howpublished =

    Dettmers, Tim and Pagnoni, Artidoro and Borzunov, Alexander and Lewis, Mike , title =. GitHub repository , howpublished =. 2022 , publisher =

    2022

  75. [113]

    2023 , publisher =

    Optimum Quanto: A PyTorch Quantization Backend for Optimum , author =. 2023 , publisher =

    2023

  76. [114]

    doi:10.48550/arXiv.2405.04532 , urldate =

    Lin, Yujun and Tang, Haotian and Yang, Shang and Zhang, Zhekai and Xiao, Guangxuan and Gan, Chuang and Han, Song , month = may, year =. doi:10.48550/arXiv.2405.04532 , urldate =

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.04532

  77. [115]

    The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

    Vladimir Malinovskii and Denis Mazur and Ivan Ilin and Denis Kuznedelev and Konstantin Pavlovich Burlachenko and Kai Yi and Dan Alistarh and Peter Richt. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

  78. [116]

    Croci and Bo Li and Pashmina Cameron and Martin Jaggi and Dan Alistarh and Torsten Hoefler and James Hensman , booktitle=

    Saleh Ashkboos and Amirkeivan Mohtashami and Maximilian L. Croci and Bo Li and Pashmina Cameron and Martin Jaggi and Dan Alistarh and Torsten Hoefler and James Hensman , booktitle=. QuaRot: Outlier-Free 4-Bit Inference in Rotated. 2024 , url=

    2024

  79. [117]

    AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration , url =

    Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Chen, Wei-Ming and Wang, Wei-Chen and Xiao, Guangxuan and Dang, Xingyu and Gan, Chuang and Han, Song , booktitle =. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration , url =

  80. [118]

    doi:10.48550/arXiv.2210.17323 , urldate =

    Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan , month = mar, year =. doi:10.48550/arXiv.2210.17323 , urldate =

    Pith/arXiv arXiv doi:10.48550/arxiv.2210.17323

Showing first 80 references.