Pith. sign in

REVIEW 4 major objections 6 minor 36 references

PolyQ shows that per-channel mixed precision can run efficiently on ordinary CPUs by moving all layout irregularity into the compiler, so fractional bit budgets like 3.7 bits per weight become deployable design points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:33 UTC pith:42VJLAJM

load-bearing objection Real contribution, careful measurements, but unvalidated error proxy and missing artifact; deserves a serious referee. the 4 major comments →

arxiv 2607.14618 v1 pith:42VJLAJM submitted 2026-07-16 cs.LG cs.ARcs.OS

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

Hyunwoo Oh , Suyeon Jang , Hanning Chen , KyungIn Nam , Sanggeon Yun , Ryozo Masukawa , Mohsen Imani This is my paper
classification cs.LG cs.ARcs.OS
keywords LLM inferenceCPU quantizationmixed precisionpost-training quantizationactivation-aware quantizationedge deploymentcompiler optimizationquantization co-design
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that fine-grained, channel-wise bit allocation for large language models can be made practical on CPUs, where earlier mixed-precision methods were too irregular to run efficiently. It proposes assigning each weight channel its own bit-width from 2, 3, 4, 8, or 16 bits while keeping a user-chosen average budget, then moving the required channel reshuffling into the compile-time layout of weights and kernels. If correct, a single average-bit parameter would continuously trade memory footprint, output quality, latency, and energy, replacing coarse choices like '3-bit' or '4-bit' models with any fractional target such as 3.7 bits per weight. The measured claims are smooth quality scaling from 3 to 6 bits, perplexity gains of 2.4–32.1% over earlier methods at a 3-bit target, up to 70.8% less activation-reorder traffic, and under 2% energy overhead relative to an optimized low-bit backend.

Core claim

The central discovery is that per-channel mixed precision no longer has to be execution-hostile. The paper shows that if the bit-width of every input channel is chosen greedily by a waterfilling procedure that weights each channel's weight-quantization error by its activation energy, the resulting irregular maps can be clustered into bit-homogeneous blocks, matched to SIMD or table-lookup kernels, and stitched together by propagating compatible channel permutations across the model's operator graph at compile time. The authors report that the same average-bit budget then predicts both memory footprint and runtime with only small residual overhead, making fractional deployment a smooth design

What carries the argument

The load-bearing object is an activation-aware per-channel error proxy: for each channel, the squared reconstruction error between original and quantized weights, multiplied by that channel's calibration activation energy. A greedy waterfilling step repeatedly promotes the channel whose error reduction per extra bit is largest, under a fixed average bit budget, over the palette {2,3,4,8,16}. A compile-time compiler then permutes and clusters channels of equal bit-width, assigns a low-bit table-lookup or SIMD kernel to each block, and propagates compatible channel bases across graph edges so most activation shuffles never appear at runtime.

Load-bearing premise

The method assumes each channel's quantization error is fairly scored by its own weight error times its activation energy, independent of other channels, so spending extra bits on the highest-scoring channels is always the best allocation.

What would settle it

Take one of the three models and compare the greedy allocation against a near-exhaustive search over a few hundred random bit maps at the same average budget. If simple random or uniform maps match or beat the proxy-selected map on end-to-end perplexity, the proxy is not the reason for the reported gains.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The average-bit budget becomes a first-class design knob: setting it to, say, 3.4 or 3.9 bits yields corresponding memory, quality, latency, and energy points instead of only W2/W3/W4 choices.
  • Sub-3-bit operating points can be practical if chosen just past the model's collapse threshold, saving roughly 12–22% peak memory compared with the nearest uniform 3-bit deployment.
  • Compile-time layout regularization can remove most activation-reorder traffic (up to 70.8%) without changing the bit allocation itself.
  • Measured prefill latency and decode throughput scale nearly proportionally with the configured bit budget, so runtime can be predicted from the same parameter that controls memory and quality.
  • Fine-grained channel-wise precision on CPUs can be nearly energy-neutral, with energy per token within about 2% of a uniform low-bit backend.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the proxy's factorization holds broadly, the same waterfilling skeleton could be retargeted to other CPU or accelerator ISAs by swapping in the backend's block quanta, an abstraction the paper already sketches.
  • The collapse-to-recovery thresholds near 2.1–2.6 bits suggest a per-device memory-aware scheduler could automatically pick the smallest fractional budget that keeps a model usable, an extension beyond the paper's fixed-curve evaluation.
  • The method's quality claims rest on three models and one perplexity benchmark plus a small multitask set; a natural next test is whether the reported gains persist on a wider model family, especially models with correlated channels.
  • Because the compiler propagates channel bases through any basis-preserving operator, the layout machinery may also apply to attention-specific or non-linear structures beyond the transformer blocks evaluated.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes PolyQ, a quantization/compiler co-design for CPU-only LLM inference that assigns per-channel bit-widths from {2,3,4,8,16} under a user-specified fractional average-bit budget. The accuracy side uses an activation-aware per-channel error proxy (Eq. 2) to drive a greedy waterfilling allocation (Section 3.2), followed by ISA-aware quanta matching and bit-specific scaling. The compiler side permutes and clusters channels into bit-homogeneous blocks, generates LUT/SIMD kernels, and propagates compatible permutations across the operator DAG to avoid runtime reorders (Section 4). Experiments on Falcon-H1-3B, Llama2-13B, and Qwen3-32B report smooth quality scaling with budget, 2.4–32.1% perplexity improvements over prior methods at a 3-bit target, reduced modeled reorder traffic, and end-to-end CPU latency/throughput/energy measurements on three platforms.

Significance. If the claims hold, PolyQ is a meaningful step toward practical fractional-bit operation for CPU-only LLM inference, replacing coarse 2/3/4-bit points with a continuous budget-quality-latency design curve. The paper's strengths include end-to-end measurements on three CPU classes, a clearly specified waterfilling formulation, and a compiler design that explicitly treats layout regularization as a graph-labeling problem. The central direction is sound, but several load-bearing components—the error proxy, the DAG-labeling heuristic, and the calibration/evaluation split—require further validation before the headline claims can be fully credited.

major comments (4)
  1. [§5.1] The calibration/evaluation split is ambiguous. Calibration uses 128 randomly sampled WikiText-2 sequences; perplexity is then evaluated on WikiText-2. If the calibration and evaluation sets overlap, the reported perplexity improvements are optimistic and the comparison with baselines is unfair. State explicitly whether the calibration sequences are drawn from the training split and are strictly disjoint from the test set used for perplexity. If they are not disjoint, rerun the evaluation with a separated calibration set.
  2. [§3.1, Eq. (2)] The per-channel independent error proxy is load-bearing for the waterfilling allocation (Section 3.2–3.3) but is never validated. For a linear layer, the true expected squared output error is trace(ΔWᵀ C_x ΔW); Eq. (2) omits off-diagonal input covariance terms. Channel correlations induced by residual connections and LayerNorm may make these terms material. Provide a concrete validation: on at least one model, compare the proxy ranking and the resulting allocation against the full-covariance objective, or measure actual layer-wise output error after applying the proxy-based allocation.
  3. [§4.4] The DAG-labeling heuristic (topological propagation with local conflict resolution) is not proven to find near-optimal bases, and the headline reorder-traffic reduction (Figure 10) is a modeled quantity derived from this heuristic, not an end-to-end measured overhead. Add a comparison to an optimal/exhaustive labeling on small subgraphs, or report measured shuffle cost in the end-to-end runtime, to justify the 46.6–70.8% reduction claim.
  4. [§5.2, Figures 5–6] No error bars or seed variance are reported. The claims of 'stable quality scaling' and 'small local fluctuations' depend on the noise level, and the downstream accuracies in Figure 6 are computed once. Report mean and variance over at least 3 random seeds for calibration sampling and, where applicable, evaluation, especially at the 3-bit operating point that anchors the headline improvement.
minor comments (6)
  1. [Eq. (6)] The phrase 'using small residual corrections to preserve the total number of channels' is underspecified. Describe the rounding/correction procedure concretely (e.g., greedy adjustment by largest residual).
  2. [§4.4] The criterion 'keep the label with the largest downstream reuse' is qualitative. Define 'reuse' formally or give pseudocode so the heuristic is reproducible.
  3. [Table 2] The term 'W3 overhead' is computed as 3/B_min − 1; this is a memory ratio, not an overhead. Consider renaming to 'memory savings of B_min vs. uniform W3' or clarifying the definition.
  4. [Figure 10] The y-axis tick labels '0 9.0 18' are ambiguous. Label the axis as 'Average reorder traffic (MiB/token)' and use standard tick positions (e.g., 0, 5, 10, 15).
  5. [§5.5] The sentence 'prefill latency stays within 4.7% of AWQ(T-MAC) on Ryzen 9950X, within 5.8–7.3% on Ryzen 7840U' mixes the summary for the first platform with a range for the second; clarify whether the range is across bit budgets or across repeated runs.
  6. [§5.1] The claim that Slim-LLM, RPTQ, and Atom 'implement GPU-only execution backends and have no CPU inference path' is asserted without a supporting reference or implementation detail; please clarify how this was determined.

Circularity Check

0 steps flagged

No significant circularity: quality and runtime results are measured independently of the calibration proxy; the only author self-citation (T-SAR) is related work and not load-bearing.

full rationale

The central derivation chain is not circular. The per-channel allocation minimizes the activation-weighted reconstruction proxy in Eqs. (2)/(5), while the headline quality results are held-out WikiText-2 perplexity and downstream accuracy, which are not the objective being minimized. The claimed 2.4-32.1% improvements over AWQ/GPTQ/Slim-LLM/AMQ are therefore independent measurements of the fitted allocations, not fitted values renamed as predictions. The reported budget realization (Fig. 7) is a descriptive property of the quanta-matching stage (Eqs. 6-7), not a prediction derived from the same quantity; the small mismatch is expected because the algorithm explicitly rounds to backend-friendly quanta, but this is a design target, not circular validation. The compiler reorder-traffic and energy results are measured end-to-end or via a separate traffic model against prior policies, so they do not reduce to the paper's own assumptions. The only author self-citation is T-SAR [16], cited as one example of LUT-based sub-byte CPU kernels; it is not used to justify the bit-allocation or compiler claims, and is corroborated by T-MAC [25] and BitNet.cpp [23]. The Eq. (2) diagonal-independence assumption is a correctness/robustness risk (true layer error includes full input covariance), but that is an unvalidated modeling assumption, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 7 axioms · 0 invented entities

The central claims rest on a small number of domain assumptions and implementation choices rather than on heavy mathematical machinery; no new physical or mathematical entities are postulated. The main load-bearing choices are the activation-energy proxy, channel-wise error independence, and the compiler's permutation-planning heuristic.

free parameters (3)
  • per-layer bit-specific scaling exponent alpha_b = grid-searched per layer on calibration set; values not reported
    Section 3.4, Eq. 8: these exponents set per-channel scales used in final quantized weights and therefore influence all quality numbers.
  • channel quanta q_b^(t) = backend-dependent (e.g., multiples of 32 for AVX2 int8)
    Section 3.3, Eq. 6: quanta determine rounding of bit counts and realized budget; not data-fitted but chosen by backend packing format.
  • bit palette {2,3,4,8,16} = fixed set chosen by authors
    Chosen to match CPU LUT/SIMD efficiency; the central method is defined only for this palette, so it is an ad hoc design parameter rather than a derived consequence.
axioms (7)
  • domain assumption Activation energy E[x_j^2] is a valid saliency proxy
    Eq. 2 weights per-channel weight error by activation second moment; if activations are correlated across channels, the proxy is not tight.
  • domain assumption Quantization error is additive across input channels
    Eq. 2 treats each channel's contribution independently, ignoring off-diagonal terms in the output error covariance.
  • domain assumption Greedy waterfilling gives a near-optimal solution to Eq. 3
    Section 3.2: the algorithm assumes separable, monotone gains; no optimality proof is provided.
  • standard math Elementwise nonlinearities and residual adds preserve channel basis
    Section 4.3/4.4: ReLU/GELU act per element, so they commute with channel permutations; residual adds require equal labels on both inputs.
  • ad hoc to paper Topological propagation with local conflict resolution yields a valid low-cost labeling
    Section 4.4: the graph-labeling heuristic is not proven optimal; the claimed runtime benefit depends on it finding near-optimal permutation plans.
  • domain assumption T-MAC and OpenBLAS kernels provide an efficient baseline for LUT/SIMD paths
    Section 5.1: runtime and energy results rely on these libraries' behavior; no independent kernel ablations are provided.
  • domain assumption Peak-memory footprint model accounts for all live buffers
    Section 5.3: the footprint is modeled, not measured, and includes quantized storage, KV cache, activations, scratch, metadata, and one reorder buffer.

pith-pipeline@v1.3.0-alltime-deepseek · 17543 in / 14584 out tokens · 151672 ms · 2026-08-02T01:33:45.582046+00:00 · methodology

0 comments
read the original abstract

CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs. We present PolyQ, a CPU-oriented compiler/quantization co-design for activation-aware channel-wise bit allocation under a user-specified average-bit budget. PolyQ assigns per-channel bit-widths from $\{2,3,4,8,16\}$, then uses a compile-time model compiler to permute and cluster channels into bit-homogeneous blocks, generate SIMD- and LUT-compatible kernels, and merge compatible permutations across operators to keep layout regularization off the runtime path. This turns fine-grained budget fitting into a practical fractional-bit deployment method for CPU-only inference. Across Falcon-H1-3B, Llama2-13B, and Qwen3-32B on WikiText-2, PolyQ provides stable quality scaling from 3--6\,b and improves perplexity by 2.4--32.1\% over prior methods at a 3\,b target. End-to-end measurements on three representative CPUs -- workstation, laptop, and mobile -- show that compiler layout regularization reduces activation reorder traffic by up to 70.8\%, prefill latency and decode throughput scale nearly proportionally with the configured bit budget, and energy/token overhead stays below 2\% relative to an optimized LUT-based back-end. These results show that fractional-bit CPU deployment is practical, predictable, and energy-efficient across diverse edge targets.

Figures

Figures reproduced from arXiv: 2607.14618 by Hanning Chen, Hyunwoo Oh, KyungIn Nam, Mohsen Imani, Ryozo Masukawa, Sanggeon Yun, Suyeon Jang.

Figure 1
Figure 1. Figure 1: Budget-fit motivation for fine-grained quantization on CPUs, bench [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Kernel-side barrier for deployable fine-grained quantization on CPUs. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: PolyQ model compiler architecture. (a) A channel permutation fac [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Perplexity vs. target average bit budget for the three models. PolyQ [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Budget realization after ISA-aware quanta matching from 3.0 to [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Peak memory footprint on the three models under fixed batch [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative compiler layout example from the first transformer block of Llama2-13B at [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Modeled activation reorder traffic on the runtime path, aver [PITH_FULL_IMAGE:figures/full_fig_p008_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 9 linked inside Pith

  1. [1]

    Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. LLM in a Flash: Efficient Large Language Model Inference with Limited Memory. In62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL). 12562–12584

    2024

  2. [2]

    Feng Cheng, Cong Guo, Chiyue Wei, Junyao Zhang, Changchun Zhou, Edward Hanson, Jiaqi Zhang, Xiaoxiao Liu, Hai Li, and Yiran Chen. 2025. Ecco: Im- proving Memory Bandwidth and Capacity for LLMs via Entropy-Aware Cache Compression. In52nd Annual International Symposium on Computer Architecture (ISCA ’25). 793–807. doi:10.1145/3695053.3731024

    arXiv 2025

  3. [3]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answer- ing? Try ARC, the AI2 Reasoning Challenge.arXiv:1803.05457v1(2018)

    Pith/arXiv arXiv 2018

  4. [4]

    Hanebutte, Rahul Khanna, and Christian Le

    Howard David, Eugene Gorbatov, Ulf R. Hanebutte, Rahul Khanna, and Christian Le. 2010. RAPL: memory power estimation and capping. InProceedings of the 16th ACM/IEEE International Symposium on Low Power Electronics and Design (Austin, Texas, USA)(ISLPED ’10). Association for Computing Machinery, New York, NY, USA, 189–194. doi:10.1145/1840845.1840883

    arXiv 2010

  5. [5]

    Yucheng Ding, Chaoyue Niu, Fan Wu, Shaojie Tang, Chengfei Lyu, and Guihai Chen. 2024. Enhancing On-Device LLM Inference with Historical Cloud-Based LLM Interactions. In30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’24). Barcelona, Spain, 597–608. doi:10.1145/3637528.3671679

    arXiv 2024

  6. [6]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323 [cs.LG] https://arxiv.org/abs/2210.17323

    Pith/arXiv arXiv 2023

  7. [7]

    Ruihao Gong, Xianglong Liu, Yuhang Li, Yunqiang Fan, Xiuying Wei, and Jinyang Guo. 2025. Pushing the Limit of Post-Training Quantization .IEEE Transactions on Pattern Analysis & Machine Intelligence47, 07 (July 2025), 5556–5570. doi:10. 1109/TPAMI.2025.3554523

    arXiv 2025

  8. [8]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Under- standing.Proceedings of the International Conference on Learning Representations (ICLR)(2021)

    2021

  9. [9]

    Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi. 2024. SliM-LLM: Salience-Driven Mixed- Precision Quantization for Large Language Models. arXiv:2405.14917 [cs.LG]

    Pith/arXiv arXiv 2024

  10. [10]

    Mahoney, and Kurt Keutzer

    Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. 2024. SqueezeLLM: dense-and- sparse quantization. In41st International Conference on Machine Learning(Vienna, Austria)(ICML’24). Article 958, 23 pages

    2024

  11. [11]

    Sangjun Lee, Seung-taek Woo, Jun-gyu Jin, Changhun Lee, and Eunhyeok Park

  12. [12]

    Jindong Li, Tenglong Li, Ruiqi Chen, Guobin Shen, Dongcheng Zhao, Qian Zhang, and Yi Zeng. 2025. Hummingbird: A Smaller and Faster Large Language Model Accelerator on Embedded FPGA. In2025 IEEE/ACM International Conference On Computer Aided Design (ICCAD). 1–9. doi:10.1109/ICCAD66269.2025.11241002

    arXiv 2025

  13. [13]

    Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. 2024. Evaluating quantized large language models. In41st International Conference on Machine Learning (ICML’24). Vienna, Austria, Article 1144, 45 pages

    2024

  14. [14]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware Weight Quantization for on-Device LLM Compression and Acceleration. InAnnual Conference on Machine Learning and Systems (MLSys), Vol. 6. Santa Clara, CA, USA, 87–100

    2024

  15. [15]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer Sentinel Mixture Models. arXiv:1609.07843 [cs.CL]

    Pith/arXiv arXiv 2016

  16. [16]

    Hyunwoo Oh, KyungIn Nam, Rajat Bhattacharjya, Hanning Chen, Tamoghno Das, Sanggeon Yun, Suyeon Jang, Andrew Ding, Nikil Dutt, and Mohsen Imani

  17. [17]

    OpenBLAS Contributors. [n. d.]. OpenBLAS: An Optimized BLAS Library. https: //www.openblas.net/. Accessed: April 11, 2026. Hyunwoo Oh, Suyeon Jang, Hanning Chen, KyungIn Nam, Sanggeon Yun, Ryozo Masukawa, and Mohsen Imani

    2026

  18. [18]

    arXiv:2511.13676 [cs.AR] https://arxiv.org/ abs/2511.13676

    T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganization. arXiv:2511.13676 [cs.AR] https://arxiv.org/ abs/2511.13676

    arXiv

  19. [19]

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are Emergent Abilities of Large Language Models a Mirage?. InThirty-seventh Conference on Neural Information Processing Systems (NeurIPS). https://openreview.net/forum?id= ITw9edRDlD

    2023

  20. [20]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. WinoGrande: an adversarial winograd schema challenge at scale.Commun. ACM 64, 9 (Aug. 2021), 99–106. doi:10.1145/3474381

    doi:10.1145/3474381 2021

  21. [21]

    Qwen Team. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https: //arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  22. [22]

    Falcon-LLM Team. 2025. Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance. https://falcon-lm.github.io/blog/falcon- h1

    2025

  23. [23]

    Jinheng Wang, Hansong Zhou, Ting Song, Shijie Cao, Yan Xia, Ting Cao, Jianyu Wei, Shuming Ma, Hongyu Wang, and Furu Wei. 2025. Bitnet.cpp: Efficient Edge Inference for Ternary LLMs.arXiv preprint arXiv.2502.11880(2025)

    Pith/arXiv arXiv 2025

  24. [24]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models.arXiv preprint arXiv:2307.09288(2023)

    Pith/arXiv arXiv 2023

  25. [25]

    Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, and Mao Yang. 2025. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge. In21st European Conference on Computer Systems (EuroSys). Rotterdam, Netherlands, 278–292. doi:10.1145/3689031.3696099

    arXiv 2025

  26. [26]

    Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. 2019. HAQ: Hardware- Aware Automated Quantization With Mixed Precision. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Long Beach, CA, USA, 8604–8612. doi:10.1109/CVPR.2019.00881

    arXiv 2019

  27. [27]

    Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. 2025. Fast On-device LLM Inference with NPUs. In30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’25). Association for Computing Machinery, Rotterdam, Netherlands, 445–462. doi:10.1145/3669940.3707239

    arXiv 2025

  28. [28]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In40th International Conference on Machine Learning (ICML)

    2023

  29. [29]

    Zhongkai Yu, Shengwen Liang, Tianyun Ma, Yunke Cai, Ziyuan Nan, Di Huang, Xinkai Song, Yifan Hao, Jie Zhang, Tian Zhi, Yongwei Zhao, Zidong Du, Xing Hu, Qi Guo, and Tianshi Chen. 2024. Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM. In57th IEEE/ACM International Symposium on Microarchitecture (MICRO). 1474–1488. doi:...

    arXiv 2024

  30. [30]

    Chenhao Xue, Chen Zhang, Xun Jiang, Zhutianya Gao, Yibo Lin, and Guangyu Sun. 2024. Oltron: Algorithm-Hardware Co-design for Outlier-Aware Quanti- zation of LLMs with Inter-/Intra-Layer Adaptation. In61st ACM/IEEE Design Automation Conference (DAC). ACM, San Francisco CA USA, 1–6. doi:10.1145/ 3649329.3656221

    arXiv 2024

  31. [31]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence?. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Anna Ko- rhonen, David Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, Florence, Italy, 4791–4800. doi:10...

    doi:10.18653/v1/p19-1472 2019

  32. [32]

    Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. 2023. RPTQ: Reorder-based Post-training Quantization for Large Language Models. arXiv:2304.01089 [cs.CL] https://arxiv.org/abs/2304.01089

    Pith/arXiv arXiv 2023

  33. [33]

    Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low-bit Quantization for Efficient and Accurate LLM Serving.arXiv preprint arXiv:2310.19102(2024). https://arxiv.org/abs/2310.19102

    Pith/arXiv arXiv 2024

  34. [34]

    Bingyi Zhang, Rajgopal Kannan, Carl Busart, and Viktor Prasanna. 2024. GCV- Turbo: End-to-end Acceleration of GNN-based Computer Vision Tasks on FPGA. In2024 IEEE 32nd Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM). 66–77. doi:10.1109/FCCM60383.2024.00017

    arXiv 2024

  35. [36]

    Yue Zheng, Yuhao Chen, Bin Qian, Xiufang Shi, Yuanchao Shu, and Jiming Chen. 2025. A Review on Edge Large Language Models: Design, Execution, and Applications.ACM Comput. Surv.57, 8, Article 209 (March 2025), 35 pages. doi:10.1145/3719664

    doi:10.1145/3719664 2025

  36. [2025]

    InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

    Amq: Enabling automl for mixed-precision weight-only quantization of large language models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 35520–35538

    2025