{"id":"b891d47e-3b27-4f1d-a7bc-90564f5e77dd","arxiv_id":"2506.09554","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An empirical benchmark of a 64GB Jetson Orin AGX shows that LLMs up to 32B parameters can run with INT8 quantization, but token throughput drops sharply as sequence length grows, and quantization slows smaller models.","lead":"How well do large language models run on a compact edge supercomputer? This paper measures inference speed, power, and memory on an NVIDIA Jetson Orin AGX across four LLMs from 2.7B to 32.8B parameters, under different batch sizes, sequence lengths, quantization levels, and power modes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Internally inconsistent DeepSeek memory numbers undermine the central viability claim for INT8 serving of the 32B model.","rationale":"The Reader's verdict is CONDITIONAL with high confidence, and the Reader's weakest_assumption focused on representativeness of the BitsAndBytes runtime and sequence-length defaults. As a stress-tester, I looked for an even more load-bearing concern: an internal numeric contradiction in the one model (DeepSeek 32B) that anchors the paper's strongest 'up to 32B' claim. The numbers appear in three places: Table 1 (34.3 GB), Section 3.1 (47.1 GB), and Table 4 (44.35 GB at bs=128). The provenance of each is unclear: Section 2 says memory is measured as incremental peak usage, while Table 1 labels only the FP32/FP16/INT8/INT4 values as 'estimates' in red text; the paper does not explain which rows are estimates vs measurements. If the true INT8 memory footprint is 47.1 GB, then the 64GB device has only ~17GB free for KV cache, prompts, and PyTorch overhead, making bs=128 (Table 4 reports 44.35 GB) or even bs=32 (40.87 GB) borderline. That would weaken the central viability claim that 'a 64GB Jetson Orin AGX can serve open LLMs up to 32B parameters with acceptable throughput for modest batch sizes.' It would also mean the DeepSeek throughput numbers are not reproducible because the memory plot (Fig. 6a) reaches ~57.2 GB at bs=128, which is consistent with 44.35 GB plus overhead, but not with 47.1 GB plus overhead. Because the inconsistency is internal to the paper and directly bears on the strongest factual claim, it is the most load-bearing concern. However, it does not by itself overturn the overall conclusion: the paper's broader findings on batch-size scaling, sequence-length effects, and power-mode trade-offs are supported by multiple consistent tables for the other three models, and the quantization-slowdown finding is qualitatively supported by [10]'s known behavior. So the correct verdict remains CONDITIONAL: the numeric inconsistency must be resolved before the DeepSeek-based claim can be trusted, but the paper's other conclusions can stand with artifact release and error bars. I agree with the Reader that the BitsAndBytes-specific quantization finding is a secondary concern; I am amplifying the memory inconsistency as the primary blocker. The concrete test is a direct re-measurement that settles which number is correct.","tokens_in":12768,"tokens_out":2358,"duration_ms":20976,"concrete_test":"Re-run the DeepSeek INT8 batch-size sweep on the Jetson Orin AGX 64GB with nvidia-smi or /proc/meminfo logging, and record (a) peak incremental memory at bs=1, (b) peak incremental memory at bs=128, and (c) whether bs=128 completes. Compare the measured values to 47.1 GB (Section 3.1), 34.3 GB (Table 1), and 44.35 GB (Table 4). If the measured bs=1 value is 34.3 GB and bs=128 fits, the inconsistency is cosmetic; if the measured value exceeds ~40 GB or OOMs at bs=128, the headline claim materially changes and the memory methodology must be corrected before any conclusions about DeepSeek viability are accepted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that a 64GB Jetson Orin AGX can serve models up to 32.8B parameters, with INT8 quantization enabling the DeepSeek model to run and become a usable edge LLM platform. That claim rests on the reported memory numbers for DeepSeek, which are contradictory. Table 1 lists peak memory for DeepSeek as 34.3 GB at INT8 and 18.7 GB at INT4. Section 3.1 states 'the model memory being 47.1 GB' for DeepSeek, which exactly matches Table 1's FP16 column for Mistral (47.1) and Table 1's FP16 estimate for DeepSeek is 62.0. Table 4 shows DeepSeek at 34.82 GB for batch size 1 at INT8, rising to 44.35 GB at batch 128, roughly consistent with a 34.3 GB weight footprint plus KV cache. If the actual INT8 footprint were 47.1 GB, batch sizes beyond 32 would likely OOM on the 64GB device, and the claimed throughput at bs=128 (146.83 tok/s) would be suspect. The Reader identified a three-way inconsistency (Section 3.1's 47.1 GB vs 34.3 GB in Table 1 vs 44.35 GB in Table 4). I flag the specific load-bearing consequence: if 34.3 GB is wrong and the correct value is closer to 47.1 GB, the central 'DeepSeek runs at INT8 with headroom for batching' result is unsupported. The paper must state which value is measured and which is estimated; the memory methodology says peak incremental usage is measured, so Table 1's 'estimates' of peak memory are not clearly separated from measured values. A single errant figure in such a small evaluation is exactly the kind of error that determines whether the headline conclusion survives.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical evaluation of LLM inference on an NVIDIA Jetson Orin AGX 64GB. It measures throughput, latency, memory, power, and energy for four open models (Phi-2, Llama-3.1-8B, Mistral-Small-24B, DeepSeek-R1-Distill-Qwen-32B) on WikiText2 and LongBench, varying batch size, sequence length, quantization precision (FP32/FP16/INT8/INT4 via BitsAndBytes LLM.int8()), and eight custom power modes. The main findings are that larger batch sizes increase throughput at the cost of latency and memory, longer sequences reduce throughput, quantization reduces memory but can slow smaller models, and power-mode tuning trades off latency and energy. The paper concludes that edge accelerators are viable for LLM serving but with remaining latency and energy challenges.","tokens_in":12999,"tokens_out":7490,"duration_ms":75933,"significance":"This is a useful and timely measurement study. Its strengths are the diverse model set, the two workloads, the perplexity checks, the power-mode analysis, and the detailed appendix tables. If the results hold, the study provides a practical reference for serving open LLMs on a 64GB edge accelerator and quantifies trade-offs that are often only discussed qualitatively. The power-mode results, especially the large latency penalty from reduced memory frequency, are a concrete contribution. However, the central quantitative conclusions are currently weakened by an unresolved inconsistency in the reported DeepSeek memory footprint and by an undefined default sequence length that lies outside the stated sweep; both issues are fixable in revision.","major_comments":[{"comment":"The memory numbers for DeepSeek-R1-Qwen-32B are mutually inconsistent: Section 3.1 states 'the model memory being 47.1 GB', Table 1 lists the INT8 peak memory as 34.3 GB, and Table 4 reports measured peak RAM of 34.82 GB at batch size 1 and 44.35 GB at batch size 128. Since the central viability claim is that a 64 GB Orin AGX can serve this model under INT8 with headroom for batching, the authors must state which value is measured and which is estimated and correct the inconsistency; if 47.1 GB were the true INT8 footprint, the batch-128 throughput of 146.83 tok/s in Table 4 would not be credible.","section":"Section 3.1, Table 1, Table 4"},{"comment":"The methodology defines the sequence-length sweep as 128-1024 tokens, but the default experiments in Figures 1, 3, and 5 and in Appendix Tables 4-7 use sequence length 96 (32 input + 64 output), and no default sequence length is stated in Section 2. This places the default configuration outside the stated sweep and makes the batch-size and quantization results not directly comparable with the sequence-length results; please state the default explicitly and justify or align it with the sweep.","section":"Section 2, Figures 1, 3, 5, Appendix Tables 4-7"},{"comment":"The paper attributes the INT8 slowdown for smaller models to 'compute overhead for dequantization and quantization-aware processing' and states that the decode phase is 'memory bound' citing [11], but it provides no direct measurement of compute utilization, memory bandwidth, or a comparison with an alternative quantization runtime. Since the finding that 'quantization causes smaller LLMs to be slower' is a headline result, the authors should either restrict the claim to the BitsAndBytes/LLM.int8() software stack or test at least one other runtime (e.g., TensorRT or llama.cpp) and, if possible, profile memory bandwidth utilization to support the memory-bound explanation.","section":"Sections 3.2 and 3.3"},{"comment":"Section 2 states that five runs are averaged for each configuration, but no variance, standard error, or confidence interval is reported in any figure or table. Quantitative comparisons such as 'INT8 ... is slower by 62%', 'within 2% of FP16 latency', and 'reduces power by approximately 28%' cannot be assessed for statistical significance; report at least standard deviations or min-max ranges for the averaged metrics.","section":"Section 2 and all results"}],"minor_comments":[{"comment":"There is a typo in 'total number number of tokens' and another in 'memory shared between CPU and CPU', which should be 'CPU and GPU'.","section":"Section 2"},{"comment":"The caption says 'Estimates are shown in red', but it is not clear which entries are estimates, how they were computed, and how they relate to the measured values in Table 4 (e.g., Mistral is 47.33 GB at batch size 1 in Table 4 versus 47.1 GB in Table 1). Please clarify the provenance of each value.","section":"Table 1"},{"comment":"Figure captions do not consistently state the workload; Section 3.1 says WikiText2 is reported 'for brevity', Section 3.2 does not name the dataset in the caption, and the appendix attributes Section 3.2 to LongBench. Please label each main-text figure with its dataset.","section":"Figure captions"},{"comment":"The x-axis labels in several appendix figures appear garbled (e.g., '148 16 32 64 128'), which should be corrected to '1 4 8 16 32 64 128'.","section":"Appendix Figures 6-9"},{"comment":"The paper does not state whether code, configuration files, or raw measurements will be released; providing an artifact would substantially strengthen a benchmark study of this type.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is a benchmark study with a clear practical scope; I see no circularity or citation-integrity problem, and the self-citation [7] is appropriately used as background. The blocking issue is the DeepSeek memory inconsistency, which must be resolved before the central viability claim can be accepted. I would also encourage requiring data/code availability, since the paper already provides detailed appendix tables and a release would improve reproducibility. The extended-appendix structure reads like a workshop paper plus supplementary material; the journal version should be self-contained and should not rely on appendix-only experiments for main-text claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on the Jetson LLM paper.\n\nWhat's new: they actually ran four modern models (2.7B–32.8B) on the 64GB Orin AGX and swept batch size, sequence length, quantization, and power modes. That's a wider net than the nearest prior work (which used the 32GB Orin and models up to 1.4B), and the power-mode data is something I haven't seen elsewhere. The qualitative findings—larger batches help throughput until memory saturates, longer outputs tank throughput, INT8 can slow down small models—are plausible and useful for anyone trying to run LLMs on Jetson hardware.\n\nWhat's good: the measurements look real, the perplexity table is a nice touch, and they're honest that the quantization results come from the BitsAndBytes stack. The paper is short and readable.\n\nWhere it's soft: First, the DeepSeek memory numbers are a mess. Section 3.1 says 47.1 GB, Table 1 says 34.3 GB, and Table 4 shows 34.82 GB at bs=1 and 44.35 GB at bs=128. The stress-test note says this undermines the viability claim. On balance, I think the 47.1 is a typo—it's exactly Mistral's FP16 number—and Tables 4 and 5, which are consistent, support the INT8 viability claim. Still, a benchmark paper cannot have this kind of internal contradiction. It must be resolved. Second, no code or raw data, and no error bars despite five runs. That caps the trust level. Third, the default sequence length is 96, which is outside the stated 128–1024 sweep. That inconsistency makes the batch-size and quantization configs hard to compare with the sequence-length sweep. Minor: the 'quantization slows smaller models' result is only checked with one quantization runtime; the hardware-level conclusion isn't isolated. They do acknowledge this in the conclusion, so it's a caveat, not a fatal flaw.\n\nBottom line: this is a solid-but-incremental empirical study. The core finding—that a 64GB Orin can serve a 32B model at INT8 with headroom—appears to hold once the numbers are corrected. I'd send it to peer review, but the authors need to fix the inconsistencies, add variance, and release the artifact. It's a useful data point for edge-LLM practitioners, not a game-changer.","headline":"Useful, incremental benchmark for edge LLM serving, but memory-number inconsistencies and missing artifacts keep it from being fully trustworthy as-is.","tokens_in":13663,"tokens_out":3231,"would_cite":true,"duration_ms":31547,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 64GB Jetson Orin AGX edge accelerator can serve open LLMs up to 32.8B parameters at usable throughput, with sequence length and quantization driving systematic trade-offs in latency, energy, and accuracy.","keywords":["LLM inference","edge accelerator","Jetson Orin AGX","quantization","token throughput","latency","power modes","energy efficiency"],"falsifier":"Run the same four models at the same batch sizes and sequence lengths on the Orin AGX using a different INT8 engine such as TensorRT or llama.cpp; if Phi-2 and Llama 3.1 no longer run slower than FP16, the quantization slowdown is a runtime artifact. Also record memory-bandwidth utilization during the decode phase; if bandwidth is not saturated, the memory-bound explanation for shrinking throughput at longer sequences is wrong.","tokens_in":12426,"feed_emoji":"🤖","tokens_out":9265,"duration_ms":80017,"temperature":0.7,"pith_summary":"This paper establishes that the NVIDIA Jetson Orin AGX 64GB, a compact edge accelerator with shared CPU/GPU memory, can serve open-weights LLMs from 2.7B up to 32.8B parameters at usable token throughput for modest batch sizes. It measures how throughput, latency, memory, power, and energy respond to batch size, sequence length, quantization level, and power mode across four current models. The headline empirical finding is a trade-off: increasing sequence length always reduces token throughput because autoregressive decoding is memory-bound, and quantizing to INT8 shrinks memory but makes smaller models slower due to dequantization compute overhead. A moderate GPU-frequency reduction cuts power by about a quarter with only a modest latency penalty, while cutting memory frequency is catastrophic for latency. These measurements matter because they give application designers concrete, platform-specific guidance for hosting privacy- or latency-sensitive LLM workloads locally rather than in the cloud.","feed_headline":"Edge GPU serves 32B LLM, but quantization slows small models","feed_subtitle":"Local hosting of open LLMs on a 64GB Jetson Orin is workable, with clear trade-offs in latency, energy, and accuracy.","key_machinery":"The argument is carried by a systematic measurement sweep on a single hardware platform: the Jetson Orin AGX 64GB, an Ampere-GPU edge accelerator with 64GB of LPDDR5 RAM shared between the CPU and GPU. The study varies four controls — batch size (1–128), sequence length (128–1024 tokens, mostly output tokens), quantization precision (FP32/FP16/INT8/INT4 via the BitsAndBytes LLM.int8() library), and nine power modes that independently throttle GPU, CPU, and memory frequencies — and records throughput, latency, incremental memory, median power, and energy. The explanatory mechanism for the sequence-length trend is the autoregressive decode phase, which is assumed memory-bound because each generated token depends on all previous tokens.","core_discovery":"The paper's central discovery is that the Jetson Orin AGX 64GB is a viable platform for serving open LLMs up to DeepSeek-R1-Qwen-32B, but the efficiency story is governed by three measured trade-offs. First, token throughput rises with batch size (e.g., Llama 3.1 improves from 184 to 558 tok/s from batch 32 to 128) at the cost of latency and KV-cache memory. Second, longer sequences monotonically reduce throughput because generation is decode-dominated and memory-bound, e.g., Llama's throughput falls from 271 to 107 tok/s from sequence length 128 to 1024. Third, INT8 quantization reduces peak memory by roughly half but slows smaller models (Phi-2, Llama 3.1-8B) by about 60%, while the 24B Mistral model sees under a 2% latency penalty; perplexity degrades marginally from FP16 to INT8 and more sharply to INT4. Power-mode tuning shows GPU frequency reduction (mode A) delivers a ~28% power cut with ~26% latency increase, whereas cutting memory frequency (mode H) inflates latency by 370%.","pith_inferences":["The small-model INT8 slowdown is likely specific to the BitsAndBytes runtime; testing TensorRT or llama.cpp could show a speedup on the same GPU, which would change the paper's practical guidance.","The memory-bound decode assumption could be verified with hardware bandwidth counters; if the GPU's memory bus is not saturated during generation, the explanation for the sequence-length trend would need revision.","Combining power mode A with INT8 for large models and FP16 for small ones appears to be an energy-optimal configuration, a scheduling rule the paper does not explicitly propose.","At batch size 32, KV-cache growth implies that sequences beyond roughly 1024 tokens will exhaust the 64GB device, so serving long contexts would require chunking or streaming rather than a single pass."],"forward_implications":["Increasing batch size raises token throughput (Llama 3.1: 184 to 558 tok/s from batch 32 to 128) but inflates end-to-end latency and KV-cache memory usage toward the 64GB limit.","Longer sequences monotonically reduce throughput and increase latency; Llama 3.1 drops from 271 to 107 tok/s as sequence length grows from 128 to 1024 tokens.","INT8 quantization roughly halves peak memory for all models, but slows the two smaller models by about 60%, while the 24B Mistral model incurs under a 2% latency penalty.","Lowering GPU frequency to 800 MHz (power mode A) cuts power by about 28% at the cost of a 26% latency increase, the most favorable power-latency trade-off of the modes tested.","A Jetson Orin AGX 64GB can host and serve a 32.8B-parameter model in INT8, extending earlier Jetson LLM studies that were limited to models below 1.4B on the 32GB device."],"supporting_citations":[{"why":"Supplies the INT8 quantization method (BitsAndBytes LLM.int8()) and the prior finding that quantized inference can run slower than FP16 for smaller models, which the paper confirms on the Orin.","marker":"[10]"},{"why":"Provides the claim that the autoregressive decode phase of LLM generation is memory-bound, which the paper uses to explain why longer sequences cut token throughput.","marker":"[11]"},{"why":"The closest prior study of LLM inference on a Jetson Orin (32GB, models up to 1.4B), which this paper extends to the 64GB device and larger models.","marker":"[6]"},{"why":"Benchmarks LLM inference on server GPUs such as A100 and H100, the comparison point for the paper's observation that quantization trends on edge accelerators differ from data-center GPUs.","marker":"[4]"}],"fun_headline_variants":["Jetson Orin runs 32B LLM, but INT8 slows small models","Batch size boosts LLM throughput, but memory and latency climb","Quantization hurts small LLMs on edge, big one barely flinches","Longer sequences slash LLM tokens per second on Jetson Orin"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's quantization and sequence-length conclusions assume that the BitsAndBytes INT8 runtime and the particular prompt/output splits tested are representative of how these models behave generally on the Orin's hardware, rather than properties of one software stack or one workload shape.","fun_headline_variants_meta":{"raw":{"variants":["Jetson Orin runs 32B LLM, but INT8 slows small models","Batch size boosts LLM throughput, but memory and latency climb","Quantization hurts small LLMs on edge, big one barely flinches","Longer sequences slash LLM tokens per second on Jetson Orin"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1809,"prompt_tokens":1050,"completion_tokens":759,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":678}},"tokens_in":666,"tokens_out":759,"duration_ms":8042,"temperature":1.0,"reasoning_tokens":678,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:45:12.708116+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four models at the same batch sizes and sequence lengths on the Orin AGX using a different INT8 engine such as TensorRT or llama.cpp; if Phi-2 and Llama 3.1 no longer run slower than FP16, the quantization slowdown is a runtime artifact. Also record memory-bandwidth utilization during the decode phase; if bandwidth is not saturated, the memory-bound explanation for shrinking throughput at longer sequences is wrong.","supporting_citations":[{"cited_title":"Gpt3.int8(): 8-bit matrix multiplication for transformers at scale,","cited_arxiv_id":null,"evidence_quote":"Supplies the INT8 quantization method (BitsAndBytes LLM.int8()) and the prior finding that quantized inference can run slower than FP16 for smaller models, which the paper confirms on the Orin."},{"cited_title":"Large Language Models on Small Resource-Constrained Systems: Performance Characterization, Analysis and Trade-offs","cited_arxiv_id":"2412.15352","evidence_quote":"The closest prior study of LLM inference on a Jetson Orin (32GB, models up to 1.4B), which this paper extends to the 64GB device and larger models."}],"review_version":1}