Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Understanding the Performance and Power of LLM Inferencing on Edge Accelerators

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 64GB Jetson Orin AGX edge accelerator can serve open LLMs up to 32.8B parameters at usable throughput, with sequence length and quantization driving systematic trade-offs in latency, energy, and accuracy.

desk verdict Useful, incremental benchmark for edge LLM serving, but memory-number inconsistencies and missing artifacts keep it from being fully trustworthy as-is. read the letter →

arxiv 2506.09554 v2 pith:52G7C5WU submitted 2025-06-11 cs.DC

Mayank Arya , Yogesh Simmhan This is my paper · ORCID
classification cs.DC
keywords LLMinferenceedgeacceleratorJetsonOrinAGXquantizationtokenthroughputlatencypowermodesenergyefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper establishes that the NVIDIA Jetson Orin AGX 64GB, a compact edge accelerator with shared CPU/GPU memory, can serve open-weights LLMs from 2.7B up to 32.8B parameters at usable token throughput for modest batch sizes. It measures how throughput, latency, memory, power, and energy respond to batch size, sequence length, quantization level, and power mode across four current models. The headline empirical finding is a trade-off: increasing sequence length always reduces token throughput because autoregressive decoding is memory-bound, and quantizing to INT8 shrinks memory but makes smaller models slower due to dequantization compute overhead. A moderate GPU-frequency reduction cuts power by about a quarter with only a modest latency penalty, while cutting memory frequency is catastrophic for latency. These measurements matter because they give application designers concrete, platform-specific guidance for hosting privacy- or latency-sensitive LLM workloads locally rather than in the cloud.

What carries the argument

The argument is carried by a systematic measurement sweep on a single hardware platform: the Jetson Orin AGX 64GB, an Ampere-GPU edge accelerator with 64GB of LPDDR5 RAM shared between the CPU and GPU. The study varies four controls — batch size (1–128), sequence length (128–1024 tokens, mostly output tokens), quantization precision (FP32/FP16/INT8/INT4 via the BitsAndBytes LLM.int8() library), and nine power modes that independently throttle GPU, CPU, and memory frequencies — and records throughput, latency, incremental memory, median power, and energy. The explanatory mechanism for the sequence-length trend is the autoregressive decode phase, which is assumed memory-bound because each generated token depends on all previous tokens.

What would settle it

Run the same four models at the same batch sizes and sequence lengths on the Orin AGX using a different INT8 engine such as TensorRT or llama.cpp; if Phi-2 and Llama 3.1 no longer run slower than FP16, the quantization slowdown is a runtime artifact. Also record memory-bandwidth utilization during the decode phase; if bandwidth is not saturated, the memory-bound explanation for shrinking throughput at longer sequences is wrong.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the Jetson Orin AGX 64GB is a viable platform for serving open LLMs up to DeepSeek-R1-Qwen-32B, but the efficiency story is governed by three measured trade-offs. First, token throughput rises with batch size (e.g., Llama 3.1 improves from 184 to 558 tok/s from batch 32 to 128) at the cost of latency and KV-cache memory. Second, longer sequences monotonically reduce throughput because generation is decode-dominated and memory-bound, e.g., Llama's throughput falls from 271 to 107 tok/s from sequence length 128 to 1024. Third, INT8 quantization reduces peak memory by roughly half but slows smaller models (Phi-2, Llama 3.1-8B) by about 60%, while the 24B Mistral model sees under a 2% latency penalty; perplexity degrades marginally from FP16 to INT8 and more sharply to INT4. Power-mode tuning shows GPU frequency reduction (mode A) delivers a ~28% power cut with ~26% latency increase, whereas cutting memory frequency (mode H) inflates latency by 370%.

Load-bearing premise

The paper's quantization and sequence-length conclusions assume that the BitsAndBytes INT8 runtime and the particular prompt/output splits tested are representative of how these models behave generally on the Orin's hardware, rather than properties of one software stack or one workload shape.

Editorial extensions

If this is right

  • Increasing batch size raises token throughput (Llama 3.1: 184 to 558 tok/s from batch 32 to 128) but inflates end-to-end latency and KV-cache memory usage toward the 64GB limit.
  • Longer sequences monotonically reduce throughput and increase latency; Llama 3.1 drops from 271 to 107 tok/s as sequence length grows from 128 to 1024 tokens.
  • INT8 quantization roughly halves peak memory for all models, but slows the two smaller models by about 60%, while the 24B Mistral model incurs under a 2% latency penalty.
  • Lowering GPU frequency to 800 MHz (power mode A) cuts power by about 28% at the cost of a 26% latency increase, the most favorable power-latency trade-off of the modes tested.
  • A Jetson Orin AGX 64GB can host and serve a 32.8B-parameter model in INT8, extending earlier Jetson LLM studies that were limited to models below 1.4B on the 32GB device.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The small-model INT8 slowdown is likely specific to the BitsAndBytes runtime; testing TensorRT or llama.cpp could show a speedup on the same GPU, which would change the paper's practical guidance.
  • The memory-bound decode assumption could be verified with hardware bandwidth counters; if the GPU's memory bus is not saturated during generation, the explanation for the sequence-length trend would need revision.
  • Combining power mode A with INT8 for large models and FP16 for small ones appears to be an energy-optimal configuration, a scheduling rule the paper does not explicitly propose.
  • At batch size 32, KV-cache growth implies that sequences beyond roughly 1024 tokens will exhaust the 64GB device, so serving long contexts would require chunking or streaming rather than a single pass.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents an empirical evaluation of LLM inference on an NVIDIA Jetson Orin AGX 64GB. It measures throughput, latency, memory, power, and energy for four open models (Phi-2, Llama-3.1-8B, Mistral-Small-24B, DeepSeek-R1-Distill-Qwen-32B) on WikiText2 and LongBench, varying batch size, sequence length, quantization precision (FP32/FP16/INT8/INT4 via BitsAndBytes LLM.int8()), and eight custom power modes. The main findings are that larger batch sizes increase throughput at the cost of latency and memory, longer sequences reduce throughput, quantization reduces memory but can slow smaller models, and power-mode tuning trades off latency and energy. The paper concludes that edge accelerators are viable for LLM serving but with remaining latency and energy challenges.

Significance. This is a useful and timely measurement study. Its strengths are the diverse model set, the two workloads, the perplexity checks, the power-mode analysis, and the detailed appendix tables. If the results hold, the study provides a practical reference for serving open LLMs on a 64GB edge accelerator and quantifies trade-offs that are often only discussed qualitatively. The power-mode results, especially the large latency penalty from reduced memory frequency, are a concrete contribution. However, the central quantitative conclusions are currently weakened by an unresolved inconsistency in the reported DeepSeek memory footprint and by an undefined default sequence length that lies outside the stated sweep; both issues are fixable in revision.

major comments (4)
  1. [Section 3.1, Table 1, Table 4] The memory numbers for DeepSeek-R1-Qwen-32B are mutually inconsistent: Section 3.1 states 'the model memory being 47.1 GB', Table 1 lists the INT8 peak memory as 34.3 GB, and Table 4 reports measured peak RAM of 34.82 GB at batch size 1 and 44.35 GB at batch size 128. Since the central viability claim is that a 64 GB Orin AGX can serve this model under INT8 with headroom for batching, the authors must state which value is measured and which is estimated and correct the inconsistency; if 47.1 GB were the true INT8 footprint, the batch-128 throughput of 146.83 tok/s in Table 4 would not be credible.
  2. [Section 2, Figures 1, 3, 5, Appendix Tables 4-7] The methodology defines the sequence-length sweep as 128-1024 tokens, but the default experiments in Figures 1, 3, and 5 and in Appendix Tables 4-7 use sequence length 96 (32 input + 64 output), and no default sequence length is stated in Section 2. This places the default configuration outside the stated sweep and makes the batch-size and quantization results not directly comparable with the sequence-length results; please state the default explicitly and justify or align it with the sweep.
  3. [Sections 3.2 and 3.3] The paper attributes the INT8 slowdown for smaller models to 'compute overhead for dequantization and quantization-aware processing' and states that the decode phase is 'memory bound' citing [11], but it provides no direct measurement of compute utilization, memory bandwidth, or a comparison with an alternative quantization runtime. Since the finding that 'quantization causes smaller LLMs to be slower' is a headline result, the authors should either restrict the claim to the BitsAndBytes/LLM.int8() software stack or test at least one other runtime (e.g., TensorRT or llama.cpp) and, if possible, profile memory bandwidth utilization to support the memory-bound explanation.
  4. [Section 2 and all results] Section 2 states that five runs are averaged for each configuration, but no variance, standard error, or confidence interval is reported in any figure or table. Quantitative comparisons such as 'INT8 ... is slower by 62%', 'within 2% of FP16 latency', and 'reduces power by approximately 28%' cannot be assessed for statistical significance; report at least standard deviations or min-max ranges for the averaged metrics.
minor comments (5)
  1. [Section 2] There is a typo in 'total number number of tokens' and another in 'memory shared between CPU and CPU', which should be 'CPU and GPU'.
  2. [Table 1] The caption says 'Estimates are shown in red', but it is not clear which entries are estimates, how they were computed, and how they relate to the measured values in Table 4 (e.g., Mistral is 47.33 GB at batch size 1 in Table 4 versus 47.1 GB in Table 1). Please clarify the provenance of each value.
  3. [Figure captions] Figure captions do not consistently state the workload; Section 3.1 says WikiText2 is reported 'for brevity', Section 3.2 does not name the dataset in the caption, and the appendix attributes Section 3.2 to LongBench. Please label each main-text figure with its dataset.
  4. [Appendix Figures 6-9] The x-axis labels in several appendix figures appear garbled (e.g., '148 16 32 64 128'), which should be corrected to '1 4 8 16 32 64 128'.
  5. [Reproducibility] The paper does not state whether code, configuration files, or raw measurements will be released; providing an artifact would substantially strengthen a benchmark study of this type.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a self-contained measurement study with no fitted parameters presented as predictions and no load-bearing self-citation.

full rationale

This paper is a purely observational measurement study of LLM inference on a Jetson Orin AGX. It reports measured latency, throughput, memory usage, perplexity, and power across batch sizes, sequence lengths, quantizations, and power modes. The central claims, such as 'increasing the sequence length causes a decrease in token throughput and quantization causes smaller LLMs to be slower,' are direct summaries of recorded measurements rather than outputs of a derived model. The throughput metric is defined operationally as TP = (sum of input and output tokens) / batch latency; this is a measurement definition, not a prediction that reduces to its own inputs. No parameter is fitted to a subset of the data and then used to predict a closely related quantity. The only self-citation, reference [7], is a prior poster used as related-work background to distinguish the present study; none of the paper's results are justified by or imported from that citation. The assertion that the decode phase is memory-bound is supported by reference [11], an external source, not by the authors' own prior work. The paper does not invoke a uniqueness theorem, does not smuggle in an ansatz via self-citation, and does not rename a known empirical pattern as a new organization. The skeptical concern about possibly inconsistent DeepSeek memory figures (47.1 GB in Section 3.1 versus 34.3 GB in Table 1 and 44.35 GB in Table 4) is a correctness, reproducibility, or internal-consistency issue, not a circularity issue: even if one number is erroneous, the reported claims are still measurements rather than quantities forced by construction. Accordingly, the circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities; it is an empirical study. Its conclusions depend on measurement assumptions (jtop accuracy, memory-bound decode) and on the representative nature of the BitsAndBytes quantization library. One hand-chosen default sequence length (96) is inconsistently used.

free parameters (1)
  • default_sequence_length = 96 tokens (32 input, 64 output)
    Hand-chosen default for the batch size and quantization sweeps; it is not part of the stated sequence length sweep (128-1024 tokens) and its choice is unexplained.
assumptions (3)
  • domain assumption The jtop power samples at 2-second intervals provide an accurate basis for energy integration over batches that last from a few seconds to minutes.
    Energy is computed via trapezoidal integration of power samples taken every 2s; this assumes power is roughly linear between samples.
  • domain assumption The decode phase of LLM inference is memory-bound on Orin AGX, making longer sequences reduce throughput.
    The paper explains throughput drop with sequence length by citing reference [11] (Splitwise) rather than profiling memory bandwidth on this device.
  • ad hoc to paper BitsAndBytes LLM.int8() is a representative quantization runtime for edge inference.
    The observed slowdown for smaller models under INT8/INT4 is attributed to quantization overhead, but no comparison is made with alternative runtimes such as TensorRT or llama.cpp; the conclusion may be software-specific.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding the Performance and Power of LLM Inferencing on Edge Accelerators." pith.science (2026). https://pith.science/paper/52G7C5WU

@misc{pith2026250609554,
  author       = {Pith},
  title        = {Pith review of: Understanding the Performance and Power of LLM Inferencing on Edge Accelerators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/52G7C5WU}},
  note         = {Machine review of arXiv:2506.09554}
}
read the original abstract

Large Language Models (LLMs) have demonstrated exceptional benefits to a wide range of domains, for tasks as diverse as code generation and robot navigation. While LLMs are usually served from cloud data centers, mission-critical and privacy-sensitive applications may require local hosting of open LLM models. Given the large GPU memory footprint needed for LLMs, edge accelerators such as Nvidia Jetson Orin AGX with 64GB of shared GPU-CPU RAM are a compelling choice. However, the feasibility and performance of LLM inference on edge accelerators is under-explored. This study presents a detailed evaluation of LLM inference on the NVIDIA Jetson Orin AGX, on four SOTA models ranging from 2.7B to 32.8B parameters, such as Meta Llama3.1, Microsoft-Phi2, Deepseek-R1-Qwen. We investigate the impact of varying batch sizes, sequence lengths, and quantization levels on latency, throughput, and perplexity, and also explore various custom power modes on the Orin AGX to perform power and energy consumption analysis. Our findings offer interesting insights on the trade-offs between efficiency, inference speed and resource use, e.g., increasing the sequence length causes a decrease in token throughput and quantization causes smaller LLMs to be slower. These results can help optimize LLM serving on edge accelerators for practical applications.

Figures

Figures reproduced from arXiv: 2506.09554 by the authors.

Figure 1
Figure 1. Varying batch size across models. MaxN with sl = 96 and FP16 (INT8 for DeepQ). 3 Results and Analysis 3.1 Impact of Batch Size [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Varying sequence length across models. MAXN with bs = 32 and FP16 (INT8 for DeepQ). Phi2 is OOM for sequence length > 256. 3.2 Impact of Sequence Length [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Impact of quantization across models. MAXN with bs = 32 and sl = 96. OOM stands for out-of-memory [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Power load and Energy use when varying batch sizes and quantization for Llama3.1. MAXN with sl = 96. further investigation of memory–latency–energy trade-offs due to quantization. 3.4 Impact of Power Modes We focus on Llama for the analysis of various power modes ( [P…
Figure 5
Figure 5. Figure 5: Varying power modes across models. bs = 32, sl = 96 and FP16 (INT8 for DeepQ). Bars are latency (left y-axis) and markers are energy use and power (right y-axes). 4 Conclusions In this study, we explore LLM inferencing on the Jetson Orin 64GB edge ac￾celerator using SO…
Figure 6
Figure 6. Figure 6: Impact of varying batch sizes on different models evaluated using the [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Impact of varying batch sizes on different models evaluated using the [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Impact of varying sequence lengths on different models evaluated [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Impact of varying sequence lengths on different models evaluated using [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Impact of varying quantization on Power and Energy Consumption, [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Impact of varying quantization on model throughput, end-to-end [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

    cs.PF 2026-06 accept novelty 6.5 of 10

    On bandwidth-bound edge hardware, MoE inference cost tracks total parameters rather than active ones, so sparse activation fails to deliver the expected throughput or energy gains.

  2. On the Sustainability of AI Inferences in the Edge

    cs.LG 2025-07 reject novelty 4.0 of 10

    A cross-platform measurement study of accuracy, speed, power, and memory for classical ML, deep learning, and LLM inference on four edge devices, with parameter tuning guidelines.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [11]

    Splitwise: Efficient generative llm inference using phase splitting,

    P. Patelet al., “Splitwise: Efficient generative llm inference using phase splitting,” arXiv, Tech. Rep. 2311.18677, 2024. 11 A Appendix These additional experiments and results complement the original results pub- lished in PAISE 2025. A.1 Impact of Varying Batch Size Figure 6 shows the impact of increasing batch sizes (§3.1) on the memory con- sumption,...

    arXiv 2024

  2. [1]

    Evaluating large language models trained on code,

    M. Chenet al., “Evaluating large language models trained on code,” arXiv, Tech. Rep. 2107.03374, 2021

    arXiv 2021

  3. [2]

    Plug in the safety chip: Enforcing constraints for llm-driven robot agents,

    Z. Yang, S. S. Raman, A. Shah, and S. Tellex, “Plug in the safety chip: Enforcing constraints for llm-driven robot agents,” inIEEE ICRA, 2024. 10

    work page 2024

  4. [3]

    A survey on efficient inference for large language models,

    Z. Zhouet al., “A survey on efficient inference for large language models,” arXiv, Tech. Rep. 2404.14294, 2024

    arXiv 2024

  5. [4]

    Llm-inference-bench,

    K. T. Chitty-Venkataet al., “Llm-inference-bench,” arXiv, Tech. Rep. 2411.00136, 2024

    arXiv 2024

  6. [5]

    Characterizing the performance of accelerated jetson edge devices for training dnns,

    S. K., Prashanthi, S. A. Kesanapalli, and Y. Simmhan, “Characterizing the performance of accelerated jetson edge devices for training dnns,” in SIGMETRICS, 2023

    work page 2023

  7. [6]

    Large Language Models on Small Resource-Constrained Systems: Performance Characterization, Analysis and Trade-offs

    L. Seymour, B. Kutukcu, and S. Baidya, “Large language models on small resource-constrained systems: Performance characterization, analysis and trade-offs,” arXiv, Tech. Rep. 2412.15352, 2024

    work page Pith review arXiv 2024

  8. [7]

    A preliminary performance analysis of llm inference on edge accelerators,

    M. Arya and Y. Simmhan, “A preliminary performance analysis of llm inference on edge accelerators,” inIEEE HiPCW, 2024

    work page 2024

Show all 15 references
  1. [8]

    Wikitext-2,

    “Wikitext-2,” 2016, https://paperswithcode.com/dataset/wikitext-2

    2016

  2. [9]

    Longbench,

    “Longbench,” 2023, https://github.com/THUDM/LongBench

    2023

  3. [10]

    Gpt3.int8(): 8-bit matrix multiplication for transformers at scale,

    T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Gpt3.int8(): 8-bit matrix multiplication for transformers at scale,” inNeurIPS, 2022

    2022

  4. [12]

    When compared with INT4, INT8’s power savings range between 20% and 43% (median 32%)

    Llama3-8b: •Power Consumption: INT8 shows up to 44% less power usage than FP16 (with a median reduction of 39%), though at peak batch sizes (e.g., batch size 16) savings drop to about 26%. When compared with INT4, INT8’s power savings range between 20% and 43% (median 32%). •E...

  5. [13]

    Against INT4, INT8 consistently yields over 27% power savings

    Mistral-Base: •Power Consumption: INT8 provides the lowest power draw, saving around 50% relative to FP16 for smaller batch sizes, though sav- ings reduce to roughly 26% for larger batches. Against INT4, INT8 consistently yields over 27% power savings. •Energy Consumption: In ...

  6. [14]

    •Energy Consumption: Similarly, INT8 achieves lower energy usage, with a median reduction of 24% compared to FP16 and 55% com- pared to INT4

    MS Phi-2: •Power Consumption: INT8 continues to deliver significant power re- ductions, with a median decrease of 39% compared to FP16 and 43% compared to INT4. •Energy Consumption: Similarly, INT8 achieves lower energy usage, with a median reduction of 24% compared to FP16 an...

  7. [15]

    Deepseek-Qwen: 17 •Due to memory constraints, FP16 could not be executed. •Power & Energy Consumption: Among the available configurations, INT8 records a median power reduction of 23% and exhibits a sub- stantial energy consumption advantage, with a median savings of 78% relat...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.