REVIEW 6 cited by
Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency
read the original abstract
Deploying Large Language Models (LLMs) on edge devices presents significant challenges due to computational constraints, memory limitations, inference speed, and energy consumption. Model quantization has emerged as a key technique to enable efficient LLM inference by reducing model size and computational overhead. In this study, we conduct a comprehensive analysis of 28 quantized LLMs from the Ollama library, which applies by default Post-Training Quantization (PTQ) and weight-only quantization techniques, deployed on an edge device (Raspberry Pi 4 with 4GB RAM). We evaluate energy efficiency, inference performance, and output accuracy across multiple quantization levels and task types. Models are benchmarked on five standardized datasets (CommonsenseQA, BIG-Bench Hard, TruthfulQA, GSM8K, and HumanEval), and we employ a high-resolution, hardware-based energy measurement tool to capture real-world power consumption. Our findings reveal the trade-offs between energy efficiency, inference speed, and accuracy in different quantization settings, highlighting configurations that optimize LLM deployment for resource-constrained environments. By integrating hardware-level energy profiling with LLM benchmarking, this study provides actionable insights for sustainable AI, bridging a critical gap in existing research on energy-aware LLM deployment.
Forward citations
Cited by 6 Pith papers
-
Pimp My LLM: Leveraging Variability Modeling to Tune Inference Hyperparameters
Variability modeling from software engineering enables systematic sampling, measurement, and prediction of LLM inference configurations for energy, latency, and accuracy trade-offs.
-
Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
On bandwidth-bound edge hardware, MoE inference cost tracks total parameters rather than active ones, so sparse activation fails to deliver the expected throughput or energy gains.
-
Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations
vLLM attention kernel and prefix caching drive model- and task-dependent energy and latency effects, with no universal best config, and can unexpectedly shift measured accuracy.
-
The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
Quantized LLMs diverge from their base models at the decision level even when accuracy is preserved, with query and key attention projections showing the greatest structural distortion under low-bit compression.
-
Online LLM Selection via Constrained Bandits with Time-Varying Demand
Develops a constrained bandit algorithm for online LLM selection under packing and covering constraints with time-varying demand, claiming sublinear regret and constraint violations versus an offline full-information ...
-
Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
Empirical benchmarks show MoE inference cost on edge hardware tracks total parameters rather than active parameters, with OLMoE-1B-7B behind dense baselines especially on the Jetson device.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.