ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM

Hanning Chen; Hyunwoo Oh; Mohsen Imani; Ryozo Masukawa; Sanggeon Yun; Suyeon Jang

REVIEW 3 major objections 4 minor 33 references

ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM

T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Low-bit GEMM on CPUs can be sped up 13.29x by adding only an in-register select network, provided support points are chosen per workload.

desk verdict ExaGEMM is a genuine step forward for CPU low-bit GEMM support selection, but the 13.29x speedup rests on an analytical surrogate whose argmin fidelity is not yet proven; send it for review with expectations of revision. read the letter →

arxiv 2607.14622 v1 pith:ROVYONXW submitted 2026-07-16 cs.AR cs.LGcs.OS

Hyunwoo Oh , Suyeon Jang , Hanning Chen , Sanggeon Yun , Ryozo Masukawa , Mohsen Imani This is my paper

classification cs.ARcs.LGcs.OS

keywords low-bitGEMMSIMDISAextensionin-registerlookuptablemixed-precisionquantizationCPUinferencedesign-spaceexplorationParetofrontier

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that very-low-bit matrix multiplication on ordinary CPUs has been held back not by the arithmetic, but by the lack of lightweight hardware support that fits the fragmented landscape of 1/2/4-bit weights. Its central claim is that existing SIMD add, dot-product, and accumulate circuits already cover table generation and accumulation; the only genuinely new hardware required is an in-register select/feed network whose cost can be modeled explicitly. On that basis, ExaGEMM treats CPU support selection as a design problem: it analytically explores a parameterized space of kernels, prunes more than 99% of candidates before simulation, and returns a frontier of non-dominated support points. Simulating the selected points on x86 and ARM vector CPUs gives 1.92–13.29x latency improvements over software baselines, with the largest gains on mixed-precision LLMs.

What carries the argument

The load-bearing mechanism is the parameterized execution space k=(e,b,r,c,u): e packing factor, b fan-in, r row tile, c column tile, and u instruction-level merge factor. These define the size of the register-resident table, the width of the select network, and the amount of aggregation work. The analysis uses two analytical models: C_exe, a per-layer execution-cost score combining arithmetic/control cycles with a calibrated memory-traffic penalty, and H_model, a hardware-overhead score summing per-primitive selector costs. Together they let the framework prune infeasible candidates and keep only non-dominated support points before any cycle-accurate simulation.

What would settle it

Pick a kernel that the analytical model prunes as infeasible or low-ranked, implement it on the simulated CPU, and measure its latency; if it beats a retained frontier point, the surrogate's ordering is wrong. Synthesize a support point with several primitives together and compare actual select-network area/power to the summed per-primitive estimates; a large discrepancy would invalidate the additive hardware model.

Watch

Extended reading notes

Core claim

The key discovery is that table-driven low-bit GEMM can be made CPU-native by keeping the lookup table in the vector register file and reusing the existing SIMD datapath for table construction and accumulation. The only added hardware is a select network that applies the register-resident table to staged weight blocks. Because the cost of that select network scales with precision, packing, fan-in, and tile shape, hardware overhead becomes a rankable design objective rather than a fixed cost. This turns support selection into a formal exploration problem: for each quantized layer, the framework chooses the kernel that minimizes a modeled execution cost under register-file constraints; for eac

Load-bearing premise

The pruning and frontier selection depend on the modeled execution cost C_exe ranking candidate kernels in the same order as real hardware, and on the hardware cost of a support point being the sum of its per-primitive selector costs; if either assumption fails in pruned regions, the selected frontier and speedups could change.

Editorial extensions

If this is right

A fixed aggressive SIMD extension is nearly sufficient for uniform low-bit workloads, such as the ViT case, so designers do not always need a suite of primitives.
For mixed-precision LLMs, workload-aware support selection yields points that are simultaneously faster and lower-cost than a one-size-fits-all aggressive design, so the design problem is real.
The added hardware overhead for selected support points is small (roughly 0.02–13.2% area and 0.04–20.8% power relative to the original SIMD datapath), making in-register table-driven GEMM affordable for edge CPUs.
Wider SIMD targets give much larger gains for mixed-precision LLMs than for ViTs, suggesting that vector width interacts strongly with quantization-regime diversity.
The exploration flow produces concrete ISA specifications and simulator patches, so a designer can go from model to validated support point without exhaustively simulating the whole space.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same analytical-surrogate approach could be applied to other table-driven or indirection-heavy kernels, such as attention or embedding lookups, since the key abstraction is modeling register-resident indirection cost.
The additive hardware-cost model is a conservative proxy; if real select networks share logic across primitives more than the model assumes, the true frontier could shift toward support points with more combined primitives, making the reported specific speedups either conservative or optimistic depending on sharing.
The 13.29x figure is relative to software-only SIMD baselines; a comparison against a CPU with a mature, well-tuned low-bit ISA extension would be a harder test, and the paper does not provide that comparison.
A natural testable extension is to run the same exploration on a RISC-V vector machine or on a CPU with different register-file pressure, to see whether the hotspot structure (one vs several) generalizes beyond the three targets studied.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

ExaGEMM is a genuine step forward for CPU low-bit GEMM support selection, but the 13.29x speedup rests on an analytical surrogate whose argmin fidelity is not yet proven. That's my take after reading it. The paper does something genuinely useful: it turns 'which lightweight CPU ISA feature should I build for low-bit GEMM' into a first-class design problem, and gives a concrete framework for answering it. The parameterized execution space (e,b,r,c,u) and the idea that the only new hardware is an in-register select network—resting on existing SIMD ADD/DOTP/ACC datapaths—are clearly articulated and plausible. The exploration flow, which prunes 99.2% of candidates analytically before gem5, is a sensible approach and the workload-aware frontier results are interesting: uniform DeiT-B stays near a single aggressive design, while mixed-precision Llama benefits from a richer frontier. That's a nice result if it holds up.

The paper also does some things well methodologically. The speedup numbers are gem5 measurements of materialized kernels, not just analytical estimates. The hardware cost model is calibrated against TSMC 28nm synthesis, and the MAPE numbers, while computed on the same data used for fitting, at least show the model is in the right ballpark. The citation pattern seems fair—T-MAC, Vec-LUT, T-SAR are positioned accurately.

The main soft spots, in order. First, no artifacts are shipped. That's a real limitation for a systems paper that claims reproducibility. Second, the analytical surrogate C_exe is used to prune and to select the argmin kernel for each layer, but fidelity is checked on only 30 sampled points per workload-target pair, with Spearman correlations down to 0.889. Spearman that high does not guarantee the top of the ranking is correct; the paper doesn't report argmin fidelity or top-k accuracy, and the sample is biased toward Pareto-retained points. If the surrogate misorders a candidate that was pruned, the measured speedup could shift. That's a load-bearing premise. Third, the hardware-cost model is validated on the same synthesis data used to fit alpha_sel, which is circularity in the validation, though the authors do admit the additive H_model is a conservative proxy.

These are addressable rather than fatal. The central speedup claim is not a pure analytical prediction—it comes from simulated execution of the selected kernels—so the paper's main result has independent grounding. The concern is about optimality of the selection, not the existence of those speedups.

Who is this for? CPU architects and ML-systems folks working on low-bit quantization. It deserves a serious referee. I'd send it to review with the expectation that the authors tighten the surrogate validation and release artifacts. I would not desk-reject it.

Referee Report

3 major / 4 minor

Summary. ExaGEMM formulates low-bit GEMM CPU acceleration as a support-selection problem. It introduces a parameterized kernel/ISA space k=(e,b,r,c,u), where table-driven LUT execution reuses existing SIMD ADD/DOTP/ACC datapaths and adds only an in-register select/feed network. It defines analytical cost models for register feasibility, instruction count, memory traffic, compute cost, and hardware overhead (Eqs. 4–16), uses them to prune 99.2% of the candidate space, selects layer-wise kernels by argmin C_exe (Eq. 17), and constructs non-dominated support frontiers under additive hardware cost (Eq. 19). Evaluated with gem5 on 128/256/512-bit x86/ARM targets for DeiT-B W1A8 and Llama2-7B W2A16/W{1,2,4}A16, it reports 1.92–13.29x latency/throughput improvements over T-MAC/llama.cpp baselines, with larger gains for mixed-precision LLMs.

Significance. If the claims hold, the paper makes a useful contribution: it turns support selection into a quantified co-design problem, provides a concrete lightweight ISA primitive, and demonstrates that mixed-precision LLM workloads benefit from workload-aware frontiers. Strengths include materializing selected points as gem5 ISA patches and kernels, grounding final speedups in gem5 simulation rather than only in the analytical model, and calibrating selector overhead with TSMC 28nm synthesis. The high Spearman correlations and large pruning rates are encouraging for screening. However, the framework's central selection step rests on an analytical surrogate whose argmin fidelity is not directly verified, and the hardware-cost model is validated in-sample; these points require additional validation before the speedup and frontier claims are fully supported.

major comments (3)

[§4.2.2, Eq. (17)] The headline speedups are measured for kernels chosen by Algorithm 1 from k*_l = argmin C_exe after 99.2% pruning. The fidelity check (Fig. 8) reports only Spearman correlations on 30 points per workload-target pair, sampled from retained Pareto points plus random feasible points. Spearman 0.889–0.982 does not guard against top-of-ranking inversions, and pruned regions are not sampled, so a surrogate misordering that removes the true best kernel would go undetected and would shift both k*_l and the reported frontier. Please add top-k/argmin agreement, sample from pruned candidates, or a noise-perturbation sensitivity analysis on the final speedups.
[§4.2.2, Eq. (16), Fig. 9] alpha_sel is fit to TSMC 28nm synthesis of selector-network RTL, and Fig. 9 reports MAPE=1.4%/6.3% against that same synthesis data. This is in-sample validation: it measures calibration fit, not predictive accuracy for unseen support points. Because H_sel directly ranks the frontier, the hardware-overhead claims need out-of-sample validation (e.g., leave-one-support-point-out cross-validation or synthesis of held-out points) before MAPE can support the frontier analysis.
[§3.2.2, Eq. (19)] H_model is assumed additive over supported primitive configurations. The paper itself calls this a 'conservative ranking proxy', but the non-dominated frontier and the comparison to the aggressive-fixed baseline (Fig. 11) are defined through this cost. If select logic is shared across primitives in a support point, the additive model systematically overestimates hardware overhead and can promote larger support sets onto the frontier. Please quantify sharing (e.g., synthesis of combined multi-primitive support points) or provide a sensitivity analysis showing the frontier is stable under a sharing discount.

minor comments (4)

[Abstract and §4.4] 'Improves latency by 13.29x' should be phrased as 'up to 13.29x' since Fig. 12 reports a range (2.43–13.29x prefill, 1.92–7.84x decode) that varies by platform.
[Fig. 12 caption] 'VX512-specific' appears to be a typo for 'AVX-512-specific'.
[§3.2.2, Eqs. (6)–(16)] The range 'Equation 6–Equation 16' should be formatted as 'Equations (6)–(16)' for consistency with journal style.
[General] No artifact availability statement is included. Since the framework generates ISA specs, gem5 patches, and kernels, releasing these artifacts would materially improve reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

Hardware-overhead validation is self-calibrated on its own synthesis fit, but the headline latency speedups are independently measured in gem5.

fitted input called prediction [Section 4.2.2 / Eq. 16 / Figure 9]
"We also calibrate the modeled hardware cost with ASIC synthesis results on the augmented SIMD datapath RTL for the Llama2-7B W{1,2,4}A16 Pareto support sets across the 128/256/512-bit targets. Figure 9 compares modeled costs with area and power overheads, showing near-linear correlation. The modeled cost, normalized to the smallest synthesized support point's area/power, tracks synthesis closely, with a mean absolute percentage error (MAPE) of 1.4% for area and 6.3% for power."

Eq. 16 defines H_sel(p) = alpha_sel * S_sel, where alpha_sel is 'derived from ASIC synthesis' of the selector-network and SIMD-datapath RTL. Figure 9 then reports the MAPE of this same modeled H_sel against the very same synthesis data used to fit alpha_sel. The reported 1.4%/6.3% error is therefore a goodness-of-fit statistic, not an independent validation of the hardware-cost model. Because H_model (Eq. 19) defines the Pareto frontier used for support-point selection, this is a partial self-calibration loop in the hardware-overhead component. It does not directly invalidate the gem5-measured latency speedups, but it does mean the hardware-cost axis of the reported frontiers is not independently confirmed.

full rationale

The central latency claim (Figure 12) is not circular: ExaGEMM materializes selected kernels as gem5 ISA patches and GEMM kernels, and the reported 2.43–13.29x prefill latency reductions and 1.92–7.84x decode throughput improvements are measured gem5 latencies against independent software baselines (T-MAC, llama.cpp). The analytical surrogate C_exe is used to prune and select candidates, but a surrogate error would weaken the optimality claim rather than make the measured speedup equal to the model input. The paper does not rely on a load-bearing self-citation chain: reference [16] (T-SAR) is presented as related work, not as a premise for ExaGEMM's derivation. The one clear circular step is the hardware-overhead validation in Section 4.2.2/Figure 9: alpha_sel is fitted to TSMC 28nm synthesis data, and the same data is then used to report MAPE for H_sel. This is a fitted parameter being presented as validated on its own fit, and since H_model defines the support-point Pareto frontier, it is a partial circuity in the co-design tradeoff analysis. The sparse 30-point fidelity check and Spearman 0.889–0.982 for C_exe are validation-density limitations, not circularity: they indicate possible argmin misordering risk, but the measured speedups are still real measurements of the selected kernels.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The evidence ledger shows the central claim depends on calibrated/unreported parameters (alpha_mem, alpha_sel, alpha_CTL, tau_*), a reuse-of-SIMD architectural bet, an additive hardware-cost model, and simulation fidelity. The speedup numbers themselves are not derived by fitting, but the selection process that produces them is guided by these assumptions.

free parameters (4)

alpha_mem (traffic-to-cost calibration coefficient, Eq. 15) = not stated; calibrated per target/process
Maps modeled memory traffic to execution-cost penalty; fitted against vectorized load/store microbenchmarks. Directly weights B_mem in C_exe, so it influences which kernels/support points are selected.
alpha_sel (selector-overhead coefficient, Eq. 16) = not stated; derived from TSMC 28nm synthesis
Maps select-network scaling term S_sel to normalized hardware overhead. Used in H_model and the Pareto frontier; validated on the same synthesis data used to fit it (Fig. 9), a circular component.
alpha_CTL (control-overhead per scheduled block, Eq. 10) = not stated
Platform-dependent cost per scheduled K-block; no calibration method or value is given, yet it is part of C_cmp and affects ILM factor u selection.
Per-instruction cycle costs tau_TBL, tau_GEMM, tau_ADD, tau_DOTP, tau_ACC (Eqs. 13-14) = not reported in text
Bind per-instruction arithmetic/control cost; tau_GEMM is a max over reused SIMD datapath costs. Values are target-dependent and not tabulated, making replication hard.

assumptions (5)

domain assumption Existing SIMD ADD/DOTP/ACC datapaths are sufficient for table generation, table-driven compute, and accumulation; only an in-register select/feed network is new hardware.
Section 3.1.3 and Figure 4; this is the architectural bet that keeps hardware overhead small. If false, area/power overheads and speedups are misestimated.
domain assumption The analytical execution cost C_exe (Eq. 15) is a consistent ranking of true gem5 performance across the candidate space.
Equations 6-15 plus Figure 8 (30 sampled points per pair); used to prune ~99% and to choose layer kernels. Not proven on all pruned candidates.
ad hoc to paper Hardware overhead for a support point is additive over supported primitive configurations (Eq. 19).
The paper calls it a conservative ranking proxy when configurations share hardware; non-additive sharing could reorder the Pareto frontier.
domain assumption gem5-AVX with the generated ISA patches faithfully models target CPU behavior (x86/ARM vector CPUs).
Section 4.1; all end-to-end latency/throughput claims come from simulation, not silicon.
domain assumption Inter-core coherence overhead is negligible because each core generates and consumes its own in-register table.
Section 4.1; real shared-memory cache/coherence effects for weights/outputs are not modeled in this claim.

invented entities (2)

SIMD-slice in-register select/feed network (select network)
purpose: Applies LUT values to staged weight blocks inside the vector register file; the proposed new hardware.
Only validated in gem5 extensions and in-house TSMC 28nm synthesis; no external or silicon evidence, so it is an invented component with no independent falsifiable handle.
TBL_r×be and GEMM_r×be×c ISA primitives
purpose: Software-visible instructions for table generation and table-driven GEMM.
Custom instructions defined by the framework; no external implementation or benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM." pith.science (2026). https://pith.science/paper/ROVYONXW

@misc{pith2026260714622,
  author       = {Pith},
  title        = {Pith review of: ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ROVYONXW}},
  note         = {Machine review of arXiv:2607.14622}
}

read the original abstract

Low-bit GEMM is increasingly central to efficient ML inference, yet very-low-bit execution remains a poor fit for conventional CPUs. Practical deployment spans fragmented regimes-from 1/2/4-bit weights to varying activation precision-whose feasibility, reuse opportunity, and support cost differ under fixed SIMD and register-file budgets, making lightweight CPU support selection a first-class design problem. We present ExaGEMM, a workload-aware codesign and exploration framework for CPU-native low-bit GEMM via register-resident LUT execution. The key insight is that existing SIMD datapaths already cover table generation and accumulation; the only new hardware is an in-register select/feed mechanism with explicitly modeled cost. ExaGEMM co-explores parameterized kernels and lightweight SIMD ISA support using analytical models of register feasibility, compute cost, memory traffic, and hardware overhead, pruning the candidate space by 99.2% before simulation. It then identifies non-dominated support points and generates ISA specs, gem5 patches, and GEMM kernels for validation. Across representative ML models and CPU targets, ExaGEMM improves latency by 13.29x over software-only baselines, while showing that workload-aware frontier selection is especially important for mixed-precision LLM workloads.

Figures

Figures reproduced from arXiv: 2607.14622 by the authors.

**Figure 2.** Framework overview of ExaGEMM. Given a quantized ML model and [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Design-space progression for in-register table-driven computation from GEMV to GEMM. Panels (a)–(d) expand the output dimensionality under [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Mapping ExaGEMM’s table-driven block computation onto a vector [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Effect of instruction-level merge (ILM) factor [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Explorer-driven search-space reduction. For DeiT-B W1A8 and [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 8.** Figure 8: Ranking fidelity of the analytical explorer. Each panel compares the [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

**Figure 7.** Figure 7: Mixed-precision support-set tradeoff space on the 256-bit target. Each [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 11.** Figure 11: Measured support-tradeoff frontier normalized to the aggressive [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]

**Figure 12.** Figure 12: End-to-end performance of ExaGEMM against external baselines [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 6 linked inside Pith

[1]

Arm Ltd. [n. d.]. A64 SIMD Vector Instructions. https://developer.arm.com/ documentation/100069/0609/A64-SIMD-Vector-Instructions. Accessed: March 26, 2025

2025
[2]

Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R

Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven K. Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R. Hower, Tushar Krishna, Somayeh Sardashti, Rathijit Sen, Korey Sewell, Muhammad Shoaib, Nilay Vaish, Mark D. Hill, and David A. Wood. 2011. The gem5 Simulator.ACM SIGARCH Comput. Archit. News39, 2 (Aug. 2011), 1–7. doi:10.1145/2024716.2024718

arXiv 2011
[3]

Razvan-Gabriel Dumitru, Vikas Yadav, Rishabh Maheshwary, Paul-Ioan Clotan, Sathwik Tejaswi Madhusudhan, and Mihai Surdeanu. 2024. Layer-Wise Quanti- zation: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels. arXiv:2406.17415 [cs.CL] https://arxiv.org/abs/2406.17415

arXiv 2024
[4]

Razvan-Gabriel Dumitru, Vikas Yadav, Rishabh Maheshwary, Paul Ioan Clotan, Sathwik Tejaswi Madhusudhan, and Mihai Surdeanu. 2025. Variable Layerwise Quantization: A Simple and Effective Approach to Quantize LLMs. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehv...

doi:10.18653/v1/2025 2025
[5]

Georgi Gerganov and ggml-org Contributors. 2026. llama.cpp: LLM inference in C/C++. https://github.com/ggml-org/llama.cpp. https://github.com/ggml- org/llama.cpp Accessed: 2026-01-18

2026
[6]

Mahoney, and Kurt Keutzer

Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. 2021. A Survey of Quantization Methods for Efficient Neural Network Inference. arXiv:2103.13630 [cs.CV] https://arxiv.org/abs/2103.13630

arXiv 2021
[7]

Yuanhong He, Peiyu Niu, Jun Chen, Chenchen Zhang, and Chao Yang
[8]

Huanqi Hu, Bowen Xiao, Shixuan Sun, Jianian Yin, Zhexi Zhang, Xiang Luo, Chengquan Jiang, Weiqi Xu, Xiaoying Jia, Xin Liu, and Minyi Guo. 2025. Liq- uidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving. InProceedings of the International Conference for High Performance Com- puting, Networking, Storage and Analysis (SC ’25). Associ...

arXiv 2025

Show all 33 references

[9]

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, An- drew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. InProceedings of the IEEE Conference on Computer Vi...

2018
[10]

Yongkweon Jeon, Chungman Lee, Eulrang Cho, and Yeonju Ro. 2022. Mr.BiQ: Post-Training Non-Uniform Quantization Based on Minimizing the Reconstruc- tion Error. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12329–12338

2022
[11]

Seungmin Lee, Youngsok Kim, Dukyun Nam, and Jong Kim. 2024. Gem5-AVX: Extension of the Gem5 Simulator to Support AVX Instruction Sets.IEEE Access 12 (2024), 20767–20778. doi:10.1109/ACCESS.2024.3359296

2024
[12]

Sangjun Lee, Seung-taek Woo, Jun-gyu Jin, Changhun Lee, and Eunhyeok Park
[13]

Ju-Hung Li, Jhih-Kuan Lin, Yung-Cheng Su, Chi-Wei Chu, Lai-Tak Kuok, Hung- Ming Lai, Chao-Lin Lee, and Jenq-Kuen Lee. 2023. SIMD Everywhere Op- timization from ARM NEON to RISC-V Vector Extensions.arXiv preprint arXiv:2309.16509(2023)

2023 arXiv
[14]

Xiangyu Li, Chengyu Yin, Weijun Wang, Jianyu Wei, Ting Cao, and Yunxin Liu
[15]

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. InMLSys

2024
[16]

Hyunwoo Oh, KyungIn Nam, Rajat Bhattacharjya, Hanning Chen, Tamoghno Das, Sanggeon Yun, Suyeon Jang, Andrew Ding, Nikil Dutt, and Mohsen Imani
[17]

arXiv:2512.06443 [cs.DC] https://arxiv.org/abs/2512.06443

Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices. arXiv:2512.06443 [cs.DC] https://arxiv.org/abs/2512.06443

arXiv
[18]

Fouda, Pramod Khargonekar, and Fadi Kurdahi

Mariam Rakka, Mohammed E. Fouda, Pramod Khargonekar, and Fadi Kurdahi
[19]

Mengshu Sun, Haoyu Ma, Guoliang Kang, Yifan Jiang, Tianlong Chen, Xiao- long Ma, Zhangyang Wang, and Yanzhi Wang. 2022. VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer. arXiv:2201.06618 [cs.LG] https://arxiv.org/abs/2201.06618

2022 arXiv
[20]

arXiv:2511.13676 [cs.AR] https://arxiv.org/ abs/2511.13676

T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganization. arXiv:2511.13676 [cs.AR] https://arxiv.org/ abs/2511.13676
[21]

ONNX Runtime Developers. 2026. ONNX Runtime Execution Providers. https://onnxruntime.ai/docs/execution-providers/. https://onnxruntime.ai/docs/ execution-providers/ Accessed: 2026-01-18

2026
[22]

Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, and Mao Yang. 2025. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge. InProceedings of the Twentieth European Conference on Computer Systems(Rotterdam, Netherlands)(EuroSys ’25). ...

2025
[23]

He Xiao, Qingyao Yang, Dirui Xie, Wendong Xu, Zunhai Su, Runming yang, Wenyong Zhou, Haobo Liu, Zhengwu Liu, and Ngai Wong. 2025. Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models. arXiv:2508.03332 [cs.LG] https://arxiv.org/...

2025
[24]

Tianyi Zhang, Jonah Yi, Bowen Yao, Zhaozhuo Xu, and Anshumali Shrivastava
[25]

Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low-Bit Quantization for Efficient and Accurate LLM Serving. InProceedings of Machine Learning and Systems, P. Gibbons, G. Pekhime...

2024
[26]

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. 2023. BitNet: Scaling 1-Bit Transformers for Large Language Models.arXiv preprint arXiv:2310.11453(2023)

2023 arXiv
[27]

Xubin Wang, Zhiqing Tang, Jianxiong Guo, Tianhui Meng, Chenhao Wang, Tian Wang, and Weijia Jia. 2025. Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models.ACM Comput. Surv.57, 9, Article 228 (April 2025), 39 pages. doi:10.1145/3724420

2025 doi
[31]

InProceedings of the 38th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’24)

NoMAD-attention: efficient LLM inference on CPUs through multiply- add-free attention. InProceedings of the 38th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’24). Curran Asso- ciates Inc., Red Hook, NY, USA, Article 3581, 25 pages
[33]

Yue Zheng, Yuhao Chen, Bin Qian, Xiufang Shi, Yuanchao Shu, and Jiming Chen. 2025. A Review on Edge Large Language Models: Design, Execution, and Applications.ACM Comput. Surv.57, 8, Article 209 (March 2025), 35 pages. doi:10.1145/3719664

2025 doi
[2024]

IEEE Transactions on Pattern Analysis and Machine Intelligence46, 12 (2024), 7793–

A Review of State-of-the-art Mixed-Precision Neural Network Frameworks. IEEE Transactions on Pattern Analysis and Machine Intelligence46, 12 (2024), 7793–

2024
[2025]

InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP)

AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP). 35520–35538

2025
[2026]

arXiv:2601.16536 [cs.DC] https://arxiv.org/abs/2601.16536

W4A16 Mixed-Precision Matrix Multiplication on Decoupled Archi- tecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs. arXiv:2601.16536 [cs.DC] https://arxiv.org/abs/2601.16536
[7812]

doi:10.1109/TPAMI.2024.3394390

2024

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Arm Ltd. [n. d.]. A64 SIMD Vector Instructions. https://developer.arm.com/ documentation/100069/0609/A64-SIMD-Vector-Instructions. Accessed: March 26, 2025

2025

[2] [2]

Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R

Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven K. Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R. Hower, Tushar Krishna, Somayeh Sardashti, Rathijit Sen, Korey Sewell, Muhammad Shoaib, Nilay Vaish, Mark D. Hill, and David A. Wood. 2011. The gem5 Simulator.ACM SIGARCH Comput. Archit. News39, 2 (Aug. 2011), 1–7. doi:10.1145/2024716.2024718

arXiv 2011

[3] [3]

Razvan-Gabriel Dumitru, Vikas Yadav, Rishabh Maheshwary, Paul-Ioan Clotan, Sathwik Tejaswi Madhusudhan, and Mihai Surdeanu. 2024. Layer-Wise Quanti- zation: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels. arXiv:2406.17415 [cs.CL] https://arxiv.org/abs/2406.17415

arXiv 2024

[4] [4]

Razvan-Gabriel Dumitru, Vikas Yadav, Rishabh Maheshwary, Paul Ioan Clotan, Sathwik Tejaswi Madhusudhan, and Mihai Surdeanu. 2025. Variable Layerwise Quantization: A Simple and Effective Approach to Quantize LLMs. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehv...

doi:10.18653/v1/2025 2025

[5] [5]

Georgi Gerganov and ggml-org Contributors. 2026. llama.cpp: LLM inference in C/C++. https://github.com/ggml-org/llama.cpp. https://github.com/ggml- org/llama.cpp Accessed: 2026-01-18

2026

[6] [6]

Mahoney, and Kurt Keutzer

Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. 2021. A Survey of Quantization Methods for Efficient Neural Network Inference. arXiv:2103.13630 [cs.CV] https://arxiv.org/abs/2103.13630

arXiv 2021

[7] [7]

Yuanhong He, Peiyu Niu, Jun Chen, Chenchen Zhang, and Chao Yang

[8] [8]

Huanqi Hu, Bowen Xiao, Shixuan Sun, Jianian Yin, Zhexi Zhang, Xiang Luo, Chengquan Jiang, Weiqi Xu, Xiaoying Jia, Xin Liu, and Minyi Guo. 2025. Liq- uidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving. InProceedings of the International Conference for High Performance Com- puting, Networking, Storage and Analysis (SC ’25). Associ...

arXiv 2025