Pith. sign in

REVIEW 3 major objections 3 minor 2 cited by

The Uneven Impact of Post-Training Quantization in Machine Translation

T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that post-training quantization degrades machine translation most for low-resource languages—up to 17 COMET points at 2-bit—while 4-bit compression is nearly lossless for high-resource languages and large models.

desk verdict Useful empirical grid on PTQ for multilingual MT, but single-run COMET and misreported calibration numbers sink the fine-grained claims; the language-dependence cliff is probably real. read the letter →

arxiv 2508.20893 v1 pith:KGL7B2JD submitted 2025-08-28 cs.CL

Benjamin Marie , Atsushi Fujita This is my paper
classification cs.CL
keywords post-trainingquantizationmachinetranslationmultilingualLLMslow-resourcelanguagesCOMETGGUFweight-only2-bit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that post-training weight quantization does not degrade machine translation uniformly. Across five LLMs (1.7B to 70B), four quantization methods, and 55 languages, the authors find that 4-bit compression is generally harmless for high-resource languages and large models, while low-resource and typologically diverse languages—especially Bengali, Malayalam, and Zulu—can lose more than 10 COMET points at 2-bit. They also argue that the choice of quantization algorithm is load-bearing: GGUF keeps the most consistent quality across sizes and bitwidths, whereas BitsAndBytes works for small models but becomes the worst option at 70B. A reader should care because this means English-only or high-resource quantization benchmarks cannot predict deployment behavior for most of the world's languages, and it points to language-matched calibration as a partial remedy at extreme compression.

What carries the argument

The operative object is weight-only post-training quantization applied to each checkpoint, with the analysis carried by COMET-score deltas between full-precision and quantized versions across a grid of model sizes, quantization methods, and bitwidths. The empirical regularity that organizes the results is what the paper calls the inverse-performance law: the worse a language's baseline score, the larger its absolute quantization loss. On the mechanism side, the contrast among AWQ (activation-aware scale selection), BitsAndBytes NF4 (non-uniform 4-bit), GGUF K-quant with an importance matrix, and AutoRound (differentiable rounding) shows that group-wise scale assignment and calibration conten

What would settle it

Repeat the same quantized model and translation direction several times—for example Llama-3.1-8B with GGUF Q4_K_M translating Bengali into English—under different library versions and GPU kernels, and record the distribution of COMET scores. If the standard deviation exceeds the 0.2–3.1 point effects the paper attributes to temperature, top-p, and calibration language, those fine-grained conclusions are not distinguishable from noise; if the spread is small, the method rankings survive.

Watch

Extended reading notes

Core claim

The central claim is a language-dependent degradation law: the lower a language's full-precision translation score, the more quantization penalizes it. For example, Qwen3-8B quantized to 2-bit with GGUF loses about 2 COMET points on Japanese and French but 17 points on Bengali and Malayalam in English-to-X direction; Llama-3.1-8B at 4-bit loses under 2 points on high-resource pairs but 7.7–9.7 points on Bengali and Malayalam into English. The paper also establishes that algorithm choice and scale interact: BitsAndBytes ranks competitively at 8B and worst at 70B, while GGUF Q2_K remains strong even below 10B parameters. Finally, calibration in the target language recovers a few points only at

Load-bearing premise

The load-bearing premise is that one COMET score (a learned translation-quality metric) per configuration is a stable measurement; the paper itself concedes results are not bit-exact across library versions or GPU kernels, and the same setup appears in multiple tables with scores spread by as much as 4.1 points.

Editorial extensions

If this is right

  • For high-resource languages and models of 32B or larger, 4-bit weights are a safe deployment default; losses stay near one COMET point.
  • For low-resource or typologically distant languages, 2-bit quantization should be avoided or paired with larger models; losses can exceed 10 points.
  • GGUF Q2_K is the most robust 2-bit format tested; BitsAndBytes should be reserved for small models, not 70B-scale deployment.
  • Language-matched calibration is a cheap recovery mechanism that works only in low-bit regimes; at 4-bit it is irrelevant.
  • Quantization and decoding hyperparameters are approximately additive: temperature matters more than top-p, and neither reverses the ranking of weight formats.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step is repeated-run evaluation: the paper uses one COMET score per configuration, and nominally similar settings across tables differ by as much as 4.1 points, so conclusions about deltas under 3 points need variance estimates.
  • The inverse-performance law could be turned into a deploy-time fragility score: estimate a language pair's quantization headroom from its full-precision score or from data-resource proxies, and automatically choose higher bitwidth or better calibration for fragile pairs.
  • Because BitsAndBytes has no 2-bit mode and GGUF does, part of the method ranking may reflect bitwidth support rather than the quantization algorithm itself; a matched-bitwidth comparison of non-uniform formats would isolate algorithm quality.
  • The calibration result was obtained with generic Wikipedia and FineWeb text; using in-domain parallel or speech-domain calibration data may recover additional points at 2-bit, which the paper's own limitations section suggests.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper reports a large empirical study of post-training weight quantization for multilingual machine translation. It evaluates five LLMs (1.7B to 70B parameters) on WMT24++ with four PTQ methods (AWQ, BitsAndBytes, GGUF, AutoRound) at 4-bit and, for GGUF and AutoRound, 2-bit precision, scoring with COMET and reporting results for six representative languages. The central claims are that quantization damage is language-dependent—low-resource and typologically diverse languages such as Bengali, Malayalam, and Zulu degrade much more than Japanese and French—and that quantization algorithms differ in robustness, with GGUF most consistent and BitsAndBytes degrading at larger scales. A secondary set of claims concerns decoding temperature, nucleus sampling, and calibration-language effects, arguing that temperature dominates top-p, that language-matched calibration helps mainly at 2-bit, and that these factors interact additively with weight precision.

Significance. If the empirical claims hold, this is a useful and timely contribution: it is among the first broad multilingual PTQ-for-MT evaluations, and it addresses a practical deployment question. The paper's strengths are its breadth (five model sizes, four methods, 55-language benchmark), the use of an external standardized benchmark, and the public release of full translations and scores in Hugging Face repositories. The main limitation is measurement precision: nearly all conclusions are based on single COMET runs without confidence intervals, and the paper's own limitation section concedes that PTQ results are not bit-exact across library versions or GPU kernels. Several fine-grained findings—especially in Sections 5 and 6—rest on score differences of the same magnitude as the observed run-to-run and configuration-to-configuration variation. The large language-dependence cliffs (10–17 COMET points) are probably robust, but the exact magnitudes and the method-ranking claims need uncertainty quantification before the paper's practical recommendations can be accepted.

major comments (3)
  1. [§7, Tables 2/5/8] The paper's own limitation statement in Section 7 says results are 'not bit-exact across different library versions or GPU kernels,' and all tables report single-run COMET scores. Yet fine-grained claims in Sections 4–6 depend on differences of 0.2–3.1 points. For example, Llama-3.1-8B GGUF Q4_K_M bn->en is 68.0 in Table 2 and 67.5 in Table 5 for the same nominal setting; Table 8 shows 63.9 under an English-calibration run. This spread is comparable to the reported temperature effects (Table 5, e.g., ja_JP Original 79.2 at T=0 vs. 78.6 at T=0.6) and to many method comparisons. Please provide confidence intervals, repeated decoding runs, paired significance tests, or bootstrap intervals for the comparisons that support the temperature, top-p, calibration, and method-ranking claims, and explicitly identify which reported deltas exceed the noise floor.
  2. [§6, Tables 7–8] The calibration-language results are internally inconsistent. The text states that for Q4_K_M, calibrating on Bengali versus English 'yields negligible differences' and that 'changes are within 0.2 points' in both translation directions and for French and Bengali, but Table 8 shows bn->en scores of 63.9 (English calibration) vs. 65.8 (Bengali calibration), a 1.9-point difference. The text also states that for Q2_K, 'Bengali calibration leads to a 0.8 point gain over English calibration (71.5 vs. 72.3)'; the numbers show 72.3 (English) vs. 71.5 (Bengali), i.e., a 0.8-point loss, not a gain. The En->bn direction does show a +3.1 gain (Table 7), so the conclusion that language-matched calibration helps at 2-bit is supported only in one direction as reported. Please correct the text, re-analyze the data, and state the direction(s) for which the claim holds.
  3. [§4.4, Tables 1–4] The headline language-dependence is supported by large cliffs, e.g., Qwen3-8B GGUF 2-bit En->bn drops from 76.1 to 59.9 and En->ml from 57.9 to 40.6. However, several supporting generalizations are not uniformly supported by the tables. 'GGUF consistently yields the smallest loss' is contradicted by Table 1, Qwen3-1.7B zu_ZA (GGUF 26.0 vs. AutoRound 27.7) and Llama-3.1-8B ml_IN (GGUF 55.0 vs. BnB 55.9). The 'inverse-performance law' is asserted without any correlation test or confidence interval. Please qualify the method-ranking claims or restrict them to the settings where differences exceed plausible run-to-run noise, and add statistical support for the inverse-performance statement.
minor comments (3)
  1. [Table 5/6] The tables in Section 5 do not state which calibration corpus (WikiText vs. FineWeb English) was used for the GGUF imatrix. Since Section 6 shows that calibration language can change bn->en scores by several points, the reader cannot reconcile the 67.5 in Table 5 with the 68.0 in Table 2. Please state the calibration source in the table caption or in the experimental setup.
  2. [§4.1 / Appendix A] The paper reports results for six representative languages in the main text, while the abstract and title emphasize 55 languages. The external repositories are welcome, but for archival completeness, consider including a supplementary appendix with per-language results for all 55 languages or a summary table covering all directions.
  3. [§6] The sentence 'For translation into English, Bengali calibration leads to a 0.8 point gain over English calibration (71.5 vs. 72.3)' should be corrected to match the table; as written it is self-contradictory.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a direct measurement study against external checkpoints and an external benchmark; no fitted parameter is renamed as a prediction and no load-bearing self-citation is present.

full rationale

This is an empirical evaluation paper rather than a derivation. The central claims—quantization impact is language-dependent, GGUF is more consistent, BnB degrades at scale, calibration language helps mainly at 2-bit—are all grounded in COMET/chrF scores of translated outputs from externally supplied checkpoints (Qwen3, Llama-3.1, Llama-3.3) evaluated on the external WMT24++ benchmark. No quantity that the paper claims to predict is defined in terms of the paper's own outputs. The GGUF importance matrix is fitted to WikiText or FineWeb calibration text, and Section 6 explicitly varies the calibration language as an independent variable; the fitted object is the treatment being manipulated, not the outcome being concluded. Similarly, AutoRound calibration samples come from generic corpora, and the reported losses are measured rather than assumed. There are no self-citations of the present authors used as load-bearing authority, and no uniqueness theorem or prior-work-derived ansatz is invoked to force the conclusions. The paper's own limitation statement that results are 'not bit-exact across different library versions or GPU kernels' is a statistical robustness caveat, not evidence of circularity; likewise, the internal table inconsistencies flagged in review (e.g., the same GGUF Q4_K_M bn->en setting appearing as 68.0, 67.5, and 63.9 across tables) concern measurement precision and internal consistency, which are outside the circularity definition. Because every reported comparison is a direct measurement against external references, the derivation chain does not reduce to its inputs, and the appropriate finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No invented entities and no new theory. The listed free parameters are disclosed experimental settings rather than values fitted to the target claims, but every one of them shapes the numbers in the tables, and several (imatrix corpus, calibration tokens, language subset) are consequential for the main conclusions.

free parameters (6)
  • GGUF imatrix calibration corpus = 20k WikiText samples, context 512, batch 512
    Section 4.3: the choice of calibration data for the importance matrix affects which weights are quantized accurately; its language composition is exactly the variable Section 6 manipulates.
  • AutoRound configuration = 512 calibration samples, max length 4096, 512 iterations, group size 32 (2-bit) / 128 (4-bit)
    Section 4.3: calibration sample size, optimization iterations, and group size directly determine AutoRound's rounding optimization and thus the reported scores.
  • AWQ group size = 128, with zero points
    Section 4.3: a hyperparameter known to trade accuracy against overhead; chosen by hand, not justified against alternatives.
  • Representative evaluation languages = ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA
    Section 4.1: the 6-language subset displayed in all main tables; the 55-language claim rests on external data files not audited here.
  • Section 6 calibration tokens = 10k tokens per language from fineweb-2 (English/Bengali)
    Section 6: the size and source of calibration sets used for the language-matched calibration comparison.
  • Main-run decoding temperature = 0.0 (greedy)
    Section 7 states greedy decoding for all evaluations; Section 5 sweeps temperature only on one model. Greedy decoding is itself a choice that the authors note can obscure quantization artifacts.
assumptions (5)
  • domain assumption COMET (wmt22-comet-da) is a valid ordinal proxy for translation quality within a language pair
    Section 4.1: the paper itself notes absolute COMET scores are not interpretable and cannot be averaged across languages, yet all conclusions rest on single-run COMET deltas.
  • domain assumption Single-run evaluations represent model behavior with negligible variance
    Implicit in all tables; contradicted by the 4.1-point spread for the same nominal configuration (Tables 2, 5, 8) and by the Limitations section's non-bit-exactness admission.
  • domain assumption The fixed shared prompt is a fair instrument for comparing models
    Section 4.1, footnote 9: authors acknowledge prompt tuning or few-shot prompting could change absolute scores; the prompt text is never disclosed.
  • domain assumption WikiText and fineweb-2 snippets are adequate quantization calibration corpora
    Sections 4.3 and 6; the Limitations section admits this biases scale factors toward Latin-script, short, well-formed text, which directly bears on the low-resource language findings.
  • domain assumption WMT24++ reference data supports per-language comparisons
    Section 4.1: benchmark quality is taken as given, though the paper cites warnings about COMET failure modes (Perrella et al. 2024) and translationese in the reverse direction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Uneven Impact of Post-Training Quantization in Machine Translation." pith.science (2026). https://pith.science/paper/KGL7B2JD

@misc{pith2026250820893,
  author       = {Pith},
  title        = {Pith review of: The Uneven Impact of Post-Training Quantization in Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KGL7B2JD}},
  note         = {Machine review of arXiv:2508.20893}
}
read the original abstract

Quantization is essential for deploying large language models (LLMs) on resource-constrained hardware, but its implications for multilingual tasks remain underexplored. We conduct the first large-scale evaluation of post-training quantization (PTQ) on machine translation across 55 languages using five LLMs ranging from 1.7B to 70B parameters. Our analysis reveals that while 4-bit quantization often preserves translation quality for high-resource languages and large models, significant degradation occurs for low-resource and typologically diverse languages, particularly in 2-bit settings. We compare four quantization techniques (AWQ, BitsAndBytes, GGUF, and AutoRound), showing that algorithm choice and model size jointly determine robustness. GGUF variants provide the most consistent performance, even at 2-bit precision. Additionally, we quantify the interactions between quantization, decoding hyperparameters, and calibration languages, finding that language-matched calibration offers benefits primarily in low-bit scenarios. Our findings offer actionable insights for deploying multilingual LLMs for machine translation under quantization constraints, especially in low-resource settings.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Studying quantization trade-offs for efficient inference deployment in machine translation

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Quantized Hy-MT2 models stay accurate at long context, but quantized EuroLLM 9B/22B models collapse (up to ~60% chrF++ drop) while W4A8/W8A8 plus 200–400-token chunking improves serving throughput.

  2. Studying quantization trade-offs for efficient inference deployment in machine translation

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Combining 200-400 token document chunking with W4A8/W8A8 quantization improves MT serving efficiency, while long-context translation quality collapses for quantized EuroLLM but not for quantized Hy-MT2.

Reference graph

Works this paper leans on

35 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.282 The low-resource double bind: An empirical study of pruning for low-resource machine translation . In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3316--3333, Punta Cana, Dominican Republic. Association for Computati...

    work page doi:10.18653/v1/2021.findings-emnlp.282 2021

  2. [2]

    Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Zhen Stephen Gou, Phil Blunsom, Ahmet \"U st \"u n, and Sara Hooker. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/6c0ff499edc529c7d8c9f05c7c0ccb82-Paper-Conference.pdf Intriguing properties of quantization at scale . In Proceedings of the 37th International Conference on Neur...

    work page 2023

  3. [3]

    Sher Badshah and Hassan Sajjad. 2024. https://arxiv.org/abs/2405.03146 Quantifying the capabilities of LLMs across scale and precision . Preprint, arXiv:2405.03146

    arXiv 2024

  4. [4]

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/edbcb7583fd8921dad78adecfe06a99b-Paper-Conference.pdf Quantizable transformers: Removing outliers by helping attention heads do nothing . In Proceedings of the 37th International Conference on Neural Information Processing Systems, Ne...

    work page 2023

  5. [5]

    Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, Lv Kaokao, and Yi Liu. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.662 Optimize weight rounding via signed gradient descent for the quantization of LLM s . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11332--11350, Miami, Florida, USA. Association for...

    doi:10.18653/v1/2024.findings-emnlp.662 2024

  6. [6]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/c3ba4962c05c49636d4c6206a97e9c8a-Paper-Conference.pdf Llm.int8(): 8-bit matrix multiplication for transformers at scale . In Proceedings of the 36th International Conference on Neural Information Processing Systems, NeurIPS '22,...

    work page 2022

  7. [7]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/1feb87871436031bdc0f2beaa62a049b-Paper-Conference.pdf QLoRA : efficient finetuning of quantized LLMs . In Proceedings of the 37th International Conference on Neural Information Processing Systems, NeurIPS '23, Red Hook, NY, ...

    work page 2023

  8. [8]

    Tim Dettmers and Luke Zettlemoyer. 2023. https://proceedings.mlr.press/v202/dettmers23a The case for 4-bit precision: k-bit inference scaling laws . In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org

    work page 2023

Show all 35 references
  1. [9]

    Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, and Markus Freitag. 20...

    2025 doi

  2. [10]

    Harshita Diddee, Sandipan Dandapat, Monojit Choudhury, Tanuja Ganu, and Kalika Bali. 2022. https://aclanthology.org/2022.wmt-1.80/ Too brittle to touch: Comparing the stability of quantization and distillation towards developing low-resource MT models . In Proceedings of the S...

    2022

  3. [11]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. https://arxiv.org/abs/2210.17323 GPTQ : Accurate post-training quantization for generative pre-trained transformers . Preprint, arXiv:2210.17323

    2023 arXiv

  4. [12]

    Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. https://doi.org/10.1162/tacl_a_00437 Experts, errors, and context: A large-scale study of human evaluation for machine translation . Transactions of the Association for Comp...

    2021 doi

  5. [13]

    Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. https://openreview.net/forum?id=TQdd1VhWbe Continual pre-training for cross-lingual LLM adaptation: Enhancing Japanese langu...

    2024

  6. [14]

    Gemma Team , Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon,...

    2025 arXiv

  7. [15]

    Zhuocheng Gong, Jiahao Liu, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. 2024. https://doi.org/10.1609/aaai.v38i16.29765 What makes quantization for large language models hard? An empirical study from the lens of perturbation . In Proceedings of the Thirty-Eighth AAA...

    2024 doi

  8. [16]

    Yvette Graham and Timothy Baldwin. 2014. https://doi.org/10.3115/v1/D14-1020 Testing for significance of increased correlation with human judgment . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 172--176, Doha, Qata...

    2014 doi

  9. [17]

    Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014. https://doi.org/10.3115/v1/E14-1047 Is machine translation getting better over time? In Proceedings of the 14th Conference of the E uropean Chapter of the Association for Computational Linguistics , pages...

    2014 doi

  10. [18]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  11. [19]

    Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. 2021. https://arxiv.org/abs/1911.05248 What do compressed deep neural networks forget? Preprint, arXiv:1911.05248

    2021 arXiv

  12. [20]

    Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. 2020. https://arxiv.org/abs/2010.03058 Characterising bias in compressed models . Preprint, arXiv:2010.03058

    2020 arXiv

  13. [21]

    Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. 2024. https://doi.org/10.1007/s44267-024-00070-x An empirical study of LLaMA3 quantization: from LLMs to MLLMs . Visual Intelligence, 2

    2024 doi

  14. [22]

    Pegah Kharazmi, Zhewei Zhao, Clement Chung, and Samridhi Choudhary. 2023. https://doi.org/10.1109/ICASSP49357.2023.10095536 Distill-quantize-tune - leveraging large teachers for low-footprint efficient multilingual NLU on edge . In Proceedings of ICASSP 2023 - 2023 IEEE Intern...

    2023

  15. [23]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. https://proceedings.mlsys.org/paper_files/paper/2024/file/42a452cbafa9dd64e9ba4aa95cc1ef21-Paper-Conference.pdf AWQ : Activation-aware wei...

    2024

  16. [24]

    Kelly Marchisio, Saurabh Dash, Hongyu Chen, Dennis Aumiller, Ahmet \"U st \"u n, Sara Hooker, and Sebastian Ruder. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.935 How does quantization affect multilingual LLM s? In Findings of the Association for Computational Lingui...

    2024 doi

  17. [25]

    Kelechi Ogueji, Orevaoghene Ahia, Gbemileke Onilude, Sebastian Gehrmann, Sara Hooker, and Julia Kreutzer. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.619 Intriguing properties of compression on multilingual models . In Proceedings of the 2022 Conference on Empirical Meth...

    2022 doi

  18. [26]

    Davide Paglieri, Saurabh Dash, Tim Rocktäschel, and Jack Parker-Holder. 2024. https://arxiv.org/abs/2405.20835 Outliers and calibration sets have diminishing effect on quantization of modern LLMs . Preprint, arXiv:2405.20835

    2024 arXiv

  19. [27]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  20. [28]

    Stefano Perrella, Lorenzo Proietti, Alessandro Scir \`e , Edoardo Barba, and Roberto Navigli. 2024. https://doi.org/10.18653/v1/2024.acl-long.856 Guardians of the machine translation meta-evaluation: Sentinel metrics fall in! In Proceedings of the 62nd Annual Meeting of the As...

    2024 doi

  21. [29]

    Maja Popovi \'c . 2015. https://doi.org/10.18653/v1/W15-3049 chr F : character n-gram F -score for automatic MT evaluation . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392--395, Lisbon, Portugal. Association for Computational Linguistics

    2015 doi

  22. [30]

    Matt Post. 2018. https://www.aclweb.org/anthology/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Belgium, Brussels. Association for Computational Linguistics

    2018

  23. [31]

    Ricardo Rei, Jos \'e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.52/ COMET -22: Unbabel- IST 2022 submission for the metrics shared task . In Pr...

    2022

  24. [32]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.213 COMET : A neural framework for MT evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685--2702, ...

    2020 doi

  25. [33]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. https://proceedings.mlr.press/v202/xiao23c Smoothquant: accurate and efficient post-training quantization for large language models . In Proceedings of the 40th International Conference on Mach...

    2023

  26. [34]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://arxiv.org/abs/2505.09388 Qw...

    2025 arXiv

  27. [35]

    Vil \'e m Zouhar, Pinzhen Chen, Tsz Kin Lam, Nikita Moghe, and Barry Haddow. 2024. https://doi.org/10.18653/v1/2024.wmt-1.121 Pitfalls and outlooks in using COMET . In Proceedings of the Ninth Conference on Machine Translation, pages 1272--1288, Miami, Florida, USA. Associatio...

    2024 doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.