FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices

David Brooks; Gu-Yeon Wei; Mujin Kwen; Yuji Chai

REVIEW 4 major objections 6 minor 34 references

FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FlexQuant claims that mixing module bit-widths from just two quantized models yields an elastic LLM ensemble with roughly 100 MB memory transitions at about a tenth of the storage cost of current elastic hosting.

desk verdict A clever module-substitution ensemble for elastic LLM serving that needs out-of-sample validation and code, but deserves a serious referee. read the letter →

arxiv 2501.07139 v1 pith:CADIJQYN submitted 2025-01-13 cs.AI cs.PF

Yuji Chai , Mujin Kwen , David Brooks , Gu-Yeon Wei This is my paper · ORCID

classification cs.AIcs.PF

keywords quantizationelasticservingedgedevicesLLMdeploymentmemoryfootprinttreesearchmodelpruningunified

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Edge devices with unified memory need language models that can shrink and grow as memory becomes available, but existing elastic hosting schemes either jump in large steps (AnyPrecision, with gaps over 1.5 GB) or demand 177 GB of storage for fine granularity (ExLlamaV2). FlexQuant claims that by mixing module weights from a high-bit and a low-bit quantized version of the same model, one can generate many intermediate hybrid models with memory steps of roughly 100 MB while storing only the endpoint models plus a few mid-precision ones. This yields 15x finer transition granularity at about 10x lower storage cost, with a memory–accuracy trade-off at or near the Pareto frontier. If correct, the approach makes locally hosted LLMs practical on phones, tablets, and laptops with shared memory.

What carries the argument

The load-bearing mechanism is the EQM ensemble: given two effective bit-widths $n_{\rm low}$ and $n_{\rm up}$, every hybrid model is a selection of modules drawn from $QM(n_{\rm up})$ and $QM(n_{\rm low})$ (plus optional mid-precision models), so any number of intermediate footprints is representable with only the parameters of those endpoint models. The one-way replacement policy (modules only move to lower bit-widths) bounds the transition cost and shrinks the search space; an MCTS-inspired tree search guided by single-module logit-distance sensitivity and calibration-set perplexity picks the sequences of swaps that form the ensemble's Pareto frontier, and module-usage ranking decides which mid-precision parameters survive pruning.

What would settle it

Quantize the same model with a different PTQ method (or a different model family) and measure perplexity after each single-module swap from 8-bit to 3-bit. FlexQuant's prediction fails if any individual swap causes a perplexity jump well above the reported <0.02, or if a greedy frontier found by sensitivity ranking is substantially worse (e.g., more than a few points worse) than an exhaustive search over all swap orders at the same footprint.

Watch

Extended reading notes

Core claim

The central claim is that the memory footprint of a quantized LLM can be made elastic without storing a full model per footprint. FlexQuant builds an ensemble of Elastic Quantization Models (EQMs) in which each hybrid is formed by replacing individual modules of a higher-bit model with their counterparts from a lower-bit model. Because both quantized models approximate the same FP16 target, the paper argues that swapping a module to a slightly lower bit-width causes negligible accuracy loss, so many small memory steps can be traversed. A tree search, pruned by per-module sensitivity and evaluated on a calibration set, selects which modules to swap in which order, and a usage-based pruning strategy lets designers trade storage against accuracy. On Llama 2 7B, the reported result is a ~100 MB transition granularity with 13–17 GB of storage, versus 177 GB for the ExLlamaV2 baseline at the same granularity, while matching or slightly beating the baseline's perplexity and downstream task accuracy.

Load-bearing premise

The method assumes that replacing any single module with a slightly lower-bit version costs almost no accuracy, and that these per-module costs add up monotonically, so that a greedy tree search through swaps finds good hybrids.

Editorial extensions

If this is right

Any quantization method that supports per-module mixed bit-widths can be turned into an elastic hosting scheme; the paper demonstrates this with ExLlamaV2 and AnyPrecision.
Edge devices can adjust the resident LLM size in ~100 MB steps, matching the typical memory footprint of mobile apps, so a model can yield memory as apps are opened and reclaim it when they close.
The same storage budget now buys either one fixed model or a whole continuum of footprint-accuracy trade-offs, changing the deployment calculus for on-device assistants.
A single set of quantized models can serve multiple storage limits via the pruning rate, letting one artifact support different devices with different storage budgets.
Because search doubles as a refinement step, hybrid models can match or slightly exceed the accuracy of a directly quantized model at the same footprint.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same swap-based ensemble idea could extend beyond bit-width to other per-module resource axes, such as sparsity or pruning ratios, generalizing elastic serving.
If the locality assumption transfers across tasks, the calibration-set search may under-protect skill-specific layers, so testing on factual-recall and multilingual benchmarks would be a cheap stress test.
The ensemble curve could feed a runtime memory-pressure predictor, allowing the device to prefetch the next smaller model before memory is actually needed, turning storage savings into transition-latency savings.
With activation or KV-cache quantization brought into the same module-swap view, transition steps could drop below 100 MB, matching even smaller memory events.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The gist: FlexQuant is a clever, genuinely novel mechanism for elastic LLM serving on edge devices. Instead of storing many separately quantized checkpoints or appending bits per parameter (AnyPrecision), it replaces whole modules from two or more quantized models (say 8-bit and 3-bit) to generate a continuum of hybrid models, with the search over which module to swap guided by a light Monte Carlo tree search plus a usage-based pruning rule. That is a real contribution, and the downstream numbers on ARC, HellaSwag, PIQA, and WinoGrande for Llama 2 7B are encouraging: at 17GB storage (vs 177GB for the ExLlamaV2 baseline ensemble) the hybrid models match or slightly beat the baseline at equal memory footprints. If that reproduces, it's an order-of-magnitude storage saving at no accuracy cost.

The main soft spot is evaluation hygiene. Figure 1, the headline perplexity curve, is measured on the same calibration set that the tree search optimizes, so that curve is in-sample; the paper's response ("we are presumably not overfitting") is not a check. The independent evidence is partial: the "downstream perplexity" in Figure 5 uses WikiText2 and C4, which are also part of the ExLlama calibration mixture; only PTB is disjoint, and the downstream task table (ARC, HellaSwag, PIQA, WinoGrande) is genuinely out-of-sample and positive. That gives me some confidence, but it's one model family and one bit-width gap. The load-bearing assumption that single-module swaps are near-lossless and that sensitivities combine additively is supported only by a single experiment on Llama 2 7B. No code, no hyperparameters (#stem, #branch), no error bars. The ELMS baseline is cited but not benchmarked, which weakens the "SoTA" claim.

None of these are fatal. The mechanism is coherent and the out-of-sample task accuracy is a real datapoint. But the paper needs a code release, a proper calibration-disjoint evaluation, and ablations showing the locality/additivity assumption holds across model families and bit-width gaps.

Recommendation: send to peer review. This deserves a serious referee; the reviewer should ask for code and hold-out validation, but the idea is worth the time. I'd bring it to a reading group to discuss evaluation pitfalls in system papers.

Referee Report

4 major / 6 minor

Summary. FlexQuant is an elastic-serving framework that generates an ensemble of quantized LLMs (EQMs) by starting from a high-bit-weight model and iteratively swapping in modules from lower-bit-weight quantized copies, using a tree search guided by a per-module logit-distance sensitivity ranking and calibration-set perplexity. The resulting hybrid models share parameters between endpoint quantizations, giving fine-grained memory transitions (~100 MB) with storage cost equal to the endpoints. The paper reports a 15x granularity improvement over ExLlamaV2-based elastic hosting, a 10x storage reduction, and retained or improved perplexity and downstream accuracy on Llama 1 7B, Llama 2 7B, and Llama 3 8B, with additional storage savings via module pruning.

Significance. The core idea is elegant and practically motivated: storing only two (or a few) quantized endpoint models and generating a continuum of hybrids by module replacement is a genuinely useful way to decouple transition granularity from storage cost. The paper gives credit-worthy independent evidence: Table I shows FQ-Ex and PFQ-Ex matching or exceeding Base-Ex downstream task accuracy (ARC, HellaSwag, PIQA, WinoGrande) at the same memory footprint, despite a 10x storage reduction. The experiments span three model families and two quantization methods. However, the headline Figure 1 comparison is in-sample with respect to the selection objective, and the load-bearing module-swap locality premise is supported only by a single model and single bit-width gap. These issues must be resolved before the central claims can be considered established.

major comments (4)

[Algorithm 1 / Section V-A] Figure 1, the headline evidence for the 15x granularity and 10x storage claims, plots perplexity on the calibration set, which is the same objective that Algorithm 1's calibrationEval uses to select top-#stem ensembles. The FQ-Ex and FQ-AP curves are therefore in-sample selection results, not independent predictions, and the statement in Section V-C that 'we are presumably not overfitting' does not substitute for a hold-out evaluation. Please re-plot Figure 1 on calibration-disjoint data (e.g., PTB plus a held-out split of C4/WikiText2) or otherwise demonstrate that the Pareto improvement over Base-Ex persists out-of-sample.
[Section III] The central premise that 'a quantized LLM does not experience significant loss of accuracy if a module is replaced with a slightly lower bit-width counterpart' is supported by exactly one experiment: per-module 8-bit-to-3-bit replacement on Llama 2 7B with perplexity drops under 0.02. The greedy tree search then assumes that these per-module effects accumulate monotonically and that the logit-distance sensitivity ranking is monotone with perplexity. No evidence is given for cumulative replacements (e.g., replacing 10, 20, or 50 modules), for other bit-width gaps, or for Llama 1 7B and Llama 3 8B. Please provide cumulative-replacement perplexity curves and a correlation analysis between logit-distance ranking and final perplexity for all three models; without this, the ensemble benefit is not robustly established.
[Section IV / Section V-C] There is a calibration/evaluation data overlap: Section IV states that the calibration set 'contains a mixture of data including C4, wikitext, code, and multilingual data,' while Section V-C evaluates 'downstream/non-calibration datasets (WikiText2, C4, and PTB).' Thus both C4 and WikiText2 appear on both sides of the evaluation, and Figure 5's perplexity results are partially in-sample. Only PTB is cleanly held out. Please report perplexity on a truly disjoint set, or explicitly exclude C4/WikiText2 from the calibration set in a separate run.
[Section V-B / Algorithm 1] The search involves several free parameters—#stem, #branch, pruning rate P, and the anchor bit-width set—but none of their values are reported, and no ablation or sensitivity analysis is given. The quantitative claims (15x granularity, 10x storage reduction, 40% pruning savings) depend on these choices. Additionally, the claim that 'performance takes a significant hit at around P=40%' (Section V-B) is based on single runs without error bars or repeated seeds. Please report the used hyperparameters and add variance estimates or at least state the number of repetitions.

minor comments (6)

[Abstract / Introduction] The '15x granularity improvement' and '10x storage reduction' are stated without precisely defining the baseline or the granularity metric; please give the exact definitions and the gap sizes for Base-Ex and Base-AP.
[Section V-B] The text says 'PFQ-Ex is implemented with four pruning rates, P=25%, 50%, and 75%' but the following sentence lists rates '0, 0.25, 0.5, ad 0.75' and Figure 3 appears to include P=0; please correct the inconsistency and the typo 'ad'.
[Section II-C] The target of ~100 MB granularity is motivated by app memory profiling, but the methodology and sample size of the month-long Android profiling are not described; please provide details so the target is reproducible.
[Section III-B] The pseudo-code in Algorithm 1 uses 'EQMlast' both as a single element and as a list of last EQMs across stems; the loop condition 'while EQMlast ≠ QM(nlow)' is ambiguous for multiple stems. Please clarify.
[Section III-B] The statement that the total design space 'could be as large as Q(m+1)*#module i=1 i' is unclear; the product notation and the claimed factorial size should be rewritten with explicit combinatorics for the one-way transition paths.
[Section V-C] The phrase 'we are presumably not overfitting' is informal and contradicts the requested rigor; replace it with a quantitative out-of-sample check once the hold-out evaluation is added.

Circularity Check

2 steps flagged · score 4.0 of 10

Headline Figure 1 is evaluated on the same calibration objective that Algorithm 1 optimizes, and the 'downstream/non-calibration' perplexity figure reuses C4 and WikiText2 from the calibration set; independent task benchmarks prevent the central claim from collapsing.

fitted input called prediction [Section V-A (Figure 1) vs Section III-B (Algorithm 1)]
"We compare these methods by measuring perplexity on the calibration set using Llama 2 7B. ... FlexQuant’s search method estimates the performance of each candidate EQM , before completing the entire tree traversal. Based on the evaluation results, top #stem EQMs are added into the ensemble and the search process moves onto the next iteration."

The tree search selects EQMs by ranking their perplexity on the calibration set (Algorithm 1), and Figure 1 then uses perplexity on that same calibration set as the headline evidence that FQ-Ex is 'almost always at the Pareto frontier' and retains LLM performance. This makes part of the reported advantage an in-sample selection effect rather than an independent prediction. The paper offers only the assertion 'we are presumably not overfitting' in Section V-C, without a holdout split of the calibration data. Independent downstream tasks in Table I mitigate the issue but do not remove the circular component.
fitted input called prediction [Section IV (calibration construction) vs Section V-C (Figure 5)]
"we generate our EQM ensemble using Exllama’s built-in calibration set containing a mixture of data including C4, wikitext, code, and multilingual data. ... We measure the hybrid model’s perplexity on C4 [28], WikiText2 [29], and PTB [30]. ... Figure 5 compares perplexity on downstream/non-calibration datasets (WikiText2, C4, and PTB)."

The calibration set used to select EQMs explicitly includes C4 and WikiText2, yet Figure 5 presents perplexity on C4 and WikiText2 as 'downstream/non-calibration' evidence that FlexQuant retains performance. For these two datasets, the evaluation data overlaps the data used for ensemble selection, so the reported 'downstream' perplexity partially re-measures the training objective. Only PTB and the Table I accuracy tasks are genuinely external, and no leakage or holdout analysis is provided.

full rationale

FlexQuant's core engineering claims—transition granularity and storage reduction—follow from its construction (reusing modules from a small set of quantized models) and are not circular. There are no load-bearing self-citations or imported uniqueness theorems. The load-bearing empirical assumption that a module can be replaced by a slightly lower-bit counterpart with negligible loss is under-supported (one model, one bit-width gap) but it is an empirical premise, not a circular derivation from the conclusion. The main circularity is evaluation-based: Algorithm 1 optimizes calibration-set perplexity, and the paper's headline Figure 1 reports calibration-set perplexity as evidence of superiority; additionally, the 'downstream/non-calibration' perplexity evaluation reuses C4 and WikiText2, which were part of the calibration set. These in-sample components inflate the apparent performance retention. However, Table I provides genuine external evidence on ARC, HellaSwag, PIQA, and WinoGrande, and PTB is a true holdout, so the central claim has independent support. Overall, the paper is partially circular in its headline evaluation but not fundamentally self-deriving; score 4.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

FlexQuant introduces no new physical or mathematical entities. Its central claim rests on an empirical locality assumption about cross-bitwidth module replacement, a monotonicity assumption for search, and a distributional assumption about the calibration set. The main free parameters are search and pruning controls whose values are not disclosed.

free parameters (4)

search stem count (#stem) = not stated
Controls how many candidate ensemble trajectories are kept during tree search; the value directly affects the final trade-off curve but is not reported.
branch filter count (#branch) = not stated
Limits how many successor EQMs are evaluated per stem after sensitivity analysis; not reported, so reproduction requires guessing.
pruning rate P = 0, 0.25, 0.5, 0.75
User-selected storage versus accuracy knob; results change substantially with P, especially above 50%.
anchor bitwidth set = 2.5, 4.0, 5.0, 7.5 bits per weight
The set of quantized checkpoints that define the ensemble is chosen by hand and affects the achievable memory-accuracy trade-off.

assumptions (4)

domain assumption Quantized parameters in different bit-widths share the same FP16 approximation target, so arbitrary cross-bitwidth module replacement is safe.
Invoked in Section III: 'representations of the layer in different bit-widths still share the same approximation target.' This is asserted, not proven generally.
domain assumption Single-module replacement sensitivity predicts the quality of multi-module hybrids.
Section III-B uses per-module logit Euclidean distance to filter candidates before calibration evaluation, assuming replacement errors are approximately separable and monotonic.
domain assumption One-way downward transition is sufficient and safe.
Section III-B forbids backward transitions to reduce memory IO cost; no experiment tests whether allowing mixed up and down transitions would improve the trade-off.
domain assumption The calibration set reflects the real deployment distribution.
The same calibration set is used both to guide the search and to produce the headline Figure 1 perplexity curve, which can overstate performance if deployment data differs.

invented entities (1)

Elastic Quantization Model (EQM) ensemble
purpose: A set of hybrid models that share parameters from two or more quantized checkpoints; this is the core artifact of the FlexQuant framework.
The EQM ensemble is a design construct, not an independently measured physical entity. Its value is only evidenced by the paper's own experiments, which are not yet independently reproducible.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices." pith.science (2026). https://pith.science/paper/CADIJQYN

@misc{pith2026250107139,
  author       = {Pith},
  title        = {Pith review of: FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CADIJQYN}},
  note         = {Machine review of arXiv:2501.07139}
}

read the original abstract

Deploying LLMs on edge devices presents serious technical challenges. Memory elasticity is crucial for edge devices with unified memory, where memory is shared and fluctuates dynamically. Existing solutions suffer from either poor transition granularity or high storage costs. We propose FlexQuant, a novel elasticity framework that generates an ensemble of quantized models, providing an elastic hosting solution with 15x granularity improvement and 10x storage reduction compared to SoTA methods. FlexQuant works with most quantization methods and creates a family of trade-off options under various storage limits through our pruning method. It brings great performance and flexibility to the edge deployment of LLMs.

Figures

Figures reproduced from arXiv: 2501.07139 by the authors.

**Figure 2.** An example of FlexQuant’s tree search for EQM ensemble. The shown [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Perplexity comparison between Base-Ex, Base-AP, FQ-Ex, and PFQ-Ex at different pruning rate. Results for AnyPrecision on Llama 3 8B is omitted [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Perplexity vs pruning rate at varying memory footprint bounds for FQ-Ex on Llama 1, Llama 2 and Llama 3 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Downstream perplexity comparison of quantized Llama models between Base-Ex FQ-Ex and PFQ-Ex at different pruning rate. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 14 canonical work pages

[1]

Introducing Copilot+ PCs - The Official Microsoft Blog,

Y . Mehdi, “Introducing Copilot+ PCs - The Official Microsoft Blog,” https://blogs.microsoft.com/blog/2024/05/20/introducing-copilot-pcs/

work page 2024
[2]

Introducing Apple Foundation Mod- els,

Apple, “Introducing Apple Foundation Mod- els,” https://machinelearning.apple.com/research/ introducing-apple-foundation-models, [Accessed 19-11-2024]

work page 2024
[3]

Google AI Edge — Google AI for Developers,

“Google AI Edge — Google AI for Developers,” https://ai.google.dev/ edge, [Accessed 19-11-2024]

work page 2024
[4]

How to bring your AI Model to Android devices,

K. Semenova, “How to bring your AI Model to Android devices,” https://android-developers.googleblog.com/2024/10/ bring-your-ai-model-to-android-devices.html, [Accessed 19-11-2024]

work page 2024
[5]

Gemini Nano is now available on Android via ex- perimental access,

T. Darra, “Gemini Nano is now available on Android via ex- perimental access,” https://android-developers.googleblog.com/2024/10/ gemini-nano-experimental-access-available-on-android.html, [Accessed 19-11-2024]

work page 2024
[6]

Snapdragon X Elite Product Brief,

“Snapdragon X Elite Product Brief,” https://docs.qualcomm.com/ bundle/publicresource/87-71417-1 REV F Snapdragon X Elite Product Brief.pdf, [Accessed 19-11-2024]

work page 2024
[7]

Retrieval- augmented generation for knowledge-intensive nlp tasks,

P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 2020

2020
[8]

NVIDIA Jetson AGX Orin — nvidia.com,

“NVIDIA Jetson AGX Orin — nvidia.com,” https://www.nvidia.com/ en-us/autonomous-machines/embedded-systems/jetson-orin/, [Accessed 20-11-2024]

work page 2024

Show all 34 references

[9]

Demystifying Unified Memory on Jetson — GTC Digital Spring 2022 — NVIDIA On-Demand — nvidia.com,

A. Brundyn, “Demystifying Unified Memory on Jetson — GTC Digital Spring 2022 — NVIDIA On-Demand — nvidia.com,” https: //www.nvidia.com/en-us/on-demand/session/gtcspring22-se2600/, [Ac- cessed 20-11-2024]

2022
[10]

Apple introduces M4 Pro and M4 Max,

“Apple introduces M4 Pro and M4 Max,” https://www.apple.com/ newsroom/2024/10/apple-introduces-m4-pro-and-m4-max/, [Accessed 19-11-2024]

2024
[11]

Gptq: Accurate post-training quantization for generative pre-trained transformers,

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323, 2022

2022 arXiv
[12]

Smoothquant: Accurate and efficient post-training quantization for large language models,

G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning . PMLR, 2023, pp. 38 087–38 099

2023
[13]

Any-precision LLM: Low-cost deployment of multiple, different-sized LLMs,

Y . Park, J. Hyun, S. Cho, B. Sim, and J. W. Lee, “Any-precision LLM: Low-cost deployment of multiple, different-sized LLMs,” in Proceedings of the 41st International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, R. Salakhutdinov, Z. Kolter, K...

2024
[14]

Loftq: Lora-fine-tuning-aware quantization for large language models,

Y . Li, Y . Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao, “Loftq: Lora-fine-tuning-aware quantization for large language models,” arXiv preprint arXiv:2310.08659 , 2023

2023 arXiv
[15]

A white paper on neural network quantization,

M. Nagel, M. Fournarakis, R. A. Amjad, Y . Bondarenko, M. Van Baalen, and T. Blankevoort, “A white paper on neural network quantization,” arXiv preprint arXiv:2106.08295 , 2021

2021 arXiv
[16]

Squeezellm: Dense-and-sparse quantization,

S. Kim, C. Hooper, A. Gholami, Z. Dong, X. Li, S. Shen, M. W. Mahoney, and K. Keutzer, “Squeezellm: Dense-and-sparse quantization,” arXiv preprint arXiv:2306.07629 , 2023

2023 arXiv
[17]

Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,

J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,” Proceedings of Machine Learning and Systems , vol. 6, pp. 87–100, 2024

2024
[18]

Zeroquant: Efficient and affordable post-training quantization for large- scale transformers,

Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y . He, “Zeroquant: Efficient and affordable post-training quantization for large- scale transformers,” Advances in Neural Information Processing Sys- tems, vol. 35, pp. 27 168–27 183, 2022

2022
[19]

Spqr: A sparse- quantized representation for near-lossless llm weight compression,

T. Dettmers, R. Svirschevski, V . Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh, “Spqr: A sparse- quantized representation for near-lossless llm weight compression,” arXiv preprint arXiv:2306.03078 , 2023

2023 arXiv
[20]

Up or down? adaptive rounding for post-training quantization,

M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort, “Up or down? adaptive rounding for post-training quantization,” in International Conference on Machine Learning . PMLR, 2020, pp. 7197–7206

2020
[21]

Elms: Elasticized large language models on mobile devices,

W. Yin, R. Yi, D. Xu, G. Huang, M. Xu, and X. Liu, “Elms: Elasticized large language models on mobile devices,” arXiv preprint arXiv:2409.09071, 2024

2024
[22]

Llm as a system service on mobile devices,

W. Yin, M. Xu, Y . Li, and X. Liu, “Llm as a system service on mobile devices,” arXiv preprint arXiv:2403.11805 , 2024

2024 arXiv
[23]

End the senseless killing: Improving memory management for mobile operating systems,

N. Lebeck, A. Krishnamurthy, H. M. Levy, and I. Zhang, “End the senseless killing: Improving memory management for mobile operating systems,” in 2020 USENIX Annual Technical Conference (USENIX ATC 20) . USENIX Association, Jul. 2020, pp. 873–887. [Online]. Available: https://w...

2020
[24]

Efficient selectivity and backup operators in monte-carlo tree search,

R. Coulom, “Efficient selectivity and backup operators in monte-carlo tree search,” in Computers and Games , 2006. [Online]. Available: https://api.semanticscholar.org/CorpusID:16724115

2006
[25]

Llama: Open and efficient foundation language models,

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

2023 arXiv
[26]

Llama 2: Open foundation and fine-tuned chat models,

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

2023 arXiv
[27]

The llama 3 herd of models,

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

2024 arXiv
[28]

Exploring the limits of transfer learning with a unified text-to-text transformer,

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020. [Online]. Available: ht...

2020
[29]

Pointer sentinel mixture models,

S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843 , 2016

2016 arXiv
[30]

Building a large annotated corpus of english: the penn treebank,

M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini, “Building a large annotated corpus of english: the penn treebank,” Comput. Linguist., vol. 19, no. 2, p. 313–330, Jun. 1993

1993
[31]

Think you have solved question answering? try arc, the ai2 reasoning challenge,

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” arXiv preprint arXiv:1803.05457 , 2018

2018 arXiv
[32]

Hel- laswag: Can a machine really finish your sentence?

R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hel- laswag: Can a machine really finish your sentence?” arXiv preprint arXiv:1905.07830, 2019

1905 arXiv
[33]

Piqa: Reasoning about physical commonsense in natural language,

Y . Bisk, R. Zellers, J. Gao, Y . Choi et al. , “Piqa: Reasoning about physical commonsense in natural language,” in Proceedings of the AAAI conference on artificial intelligence , vol. 34, no. 05, 2020, pp. 7432– 7439

2020
[34]

Winogrande: An adversarial winograd schema challenge at scale,

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,” Communications of the ACM , vol. 64, no. 9, pp. 99–106, 2021

2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Introducing Copilot+ PCs - The Official Microsoft Blog,

Y . Mehdi, “Introducing Copilot+ PCs - The Official Microsoft Blog,” https://blogs.microsoft.com/blog/2024/05/20/introducing-copilot-pcs/

work page 2024

[2] [2]

Introducing Apple Foundation Mod- els,

Apple, “Introducing Apple Foundation Mod- els,” https://machinelearning.apple.com/research/ introducing-apple-foundation-models, [Accessed 19-11-2024]

work page 2024

[3] [3]

Google AI Edge — Google AI for Developers,

“Google AI Edge — Google AI for Developers,” https://ai.google.dev/ edge, [Accessed 19-11-2024]

work page 2024

[4] [4]

How to bring your AI Model to Android devices,

K. Semenova, “How to bring your AI Model to Android devices,” https://android-developers.googleblog.com/2024/10/ bring-your-ai-model-to-android-devices.html, [Accessed 19-11-2024]

work page 2024

[5] [5]

Gemini Nano is now available on Android via ex- perimental access,

T. Darra, “Gemini Nano is now available on Android via ex- perimental access,” https://android-developers.googleblog.com/2024/10/ gemini-nano-experimental-access-available-on-android.html, [Accessed 19-11-2024]

work page 2024

[6] [6]

Snapdragon X Elite Product Brief,

“Snapdragon X Elite Product Brief,” https://docs.qualcomm.com/ bundle/publicresource/87-71417-1 REV F Snapdragon X Elite Product Brief.pdf, [Accessed 19-11-2024]

work page 2024

[7] [7]

Retrieval- augmented generation for knowledge-intensive nlp tasks,

P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 2020

2020

[8] [8]

NVIDIA Jetson AGX Orin — nvidia.com,

“NVIDIA Jetson AGX Orin — nvidia.com,” https://www.nvidia.com/ en-us/autonomous-machines/embedded-systems/jetson-orin/, [Accessed 20-11-2024]

work page 2024