Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism

Jiangsu Du; Jiazhi Jiang; Jinhui Wei; Ye Huang; Yuhui Zhou; Yutong Lu

REVIEW 5 major objections 6 minor 41 references

Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Ghidorah claims up to 7.6x faster LLM decoding on edge devices by pairing speculative decoding with CPU-GPU parallel execution.

desk verdict The HCMP architecture and ARM SpMM kernels are genuinely interesting, and the acceptance-length measurements look honest, but the headline 7.6x speedup depends on an accounting assumption that the paper never verifies with absolute timing. read the letter →

arxiv 2505.23219 v2 pith:EA75K3XP submitted 2025-05-29 cs.DC

Jinhui Wei , Ye Huang , Yuhui Zhou , Jiazhi Jiang , Jiangsu Du , Yutong Lu This is my paper

classification cs.DC

keywords speculativedecodingedgeLLMinferenceunifiedmemoryarchitectureCPU-GPUheterogeneousparallelismmodelmulti-headself-draftingsparseattentionARMNEONmatrixmultiplication

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ghidorah is an inference system for running a single LLM request on edge devices where the CPU and GPU share one physical memory. Its claim is that the slowest part of generation, the decode phase, can be made much faster by speculative decoding (drafting several future tokens and verifying them at once) and by partitioning the resulting transformer work across the CPU and GPU. On a CPU-GPU edge device, the authors report up to 7.6x higher decode throughput than sequential decoding, decomposed into a 3.27x algorithmic gain from accepting more tokens per step and a 2.31x gain from parallel execution. If the system works as described, on-device LLM inference becomes several times more practical without changing the model or its output distribution.

What carries the argument

The load-bearing mechanism is the HCMP partition: column-splitting the weights of every linear layer, so each processing unit reads the same input and writes its own partial output to unified memory without an all-reduce between layers, and splitting each attention head into a dense and a sparse part assigned by compute affinity. The companion identity is online softmax, which allows each unit to run its own partial softmax over attention scores and fuse a scaling factor with the final reduction, removing the cross-unit softmax barrier. On top of this, the ARCA profiling pass constructs verification trees by multiplying per-head token accuracies to estimate sequence acceptance, refines them by local brute-force search, restricts verification widths to powers of two to match vectorization wave behavior, and tunes the CPU/GPU partition ratio against memory contention. A vectorized ARM sparse-matrix kernel with NEON instructions and register-blocked accumulation carries the CPU-side sparse attention.

What would settle it

On the same CPU-GPU edge device, measure per-step latency for sequential decoding, GPU-only speculative decoding, and the full CPU-GPU split while keeping the GPU at its stock clock rather than the locked low frequency. If the width-16 verification step is noticeably slower than the sequential step, the claimed 3.27x algorithmic speedup does not transfer, and the 7.6x throughput result would not reproduce outside the paper's frequency-locked configuration.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the memory-bandwidth-bound single-token decode step can be replaced by a wider verification step whose cost is nearly the same as one sequential step, because both are dominated by loading model weights. Ghidorah then splits that verification step between CPU and GPU: every linear layer is split by columns so each unit writes partial outputs and no all-reduce is needed, and every attention head is split into a dense part (multiplication against the KV cache) and a sparse part (multiplication against newly generated keys and values), with the sparse part sent to the CPU and the dense part to the GPU. Online softmax lets the two units compute partial attention results and reconcile them with a scaling factor at the end, avoiding a synchronization point. A profiling pass chooses the verification tree and width; the measured effect is acceptance length growing from 1 to 3.3-3.7 tokens at width 64 across the tested datasets, with best end-to-end throughput at width 16. The authors report up to 7.6x decoding speedup over sequential decoding, attributed to 3.27x algorithmic enhancement and 2.31x parallel speedup.

Load-bearing premise

The load-bearing premise is that the sequential decode step is so memory-bandwidth-bound that running speculative verification over many candidate tokens takes about the same time as generating one token, so extra accepted tokens are almost free.

Editorial extensions

If this is right

Decoding on the tested class of unified-memory edge devices can be accelerated several-fold relative to sequential decoding without changing output quality, because speculative verification accepts or rejects tokens using the exact model.
The verification width that maximizes throughput is not the one with the longest acceptance length; hardware vectorization and memory contention make an intermediate width (16 in the reported experiments) the best operating point.
Splitting attention by density makes low-parallelism CPU cores useful for the sparse portion of speculative verification, and the optimized sparse kernel outperforms both naive sparse and dense masked execution on the measured ARM CPU.
Dynamic partitioning of the attention module becomes more valuable as context grows, because attention's share of runtime rises with KV-cache length.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the 7.6x figure is anchored to a locked GPU frequency that makes the CPU relatively stronger; on stock clocks the parallel gain could shift, and the algorithmic gain would need to be measured separately.
Editorial inference: if the bandwidth-bound premise generalizes, the same column-split-plus-dense/sparse recipe could be applied to other unified-memory SoCs, though the optimal verification width and partition ratio would have to be re-profiled for each chip's vector width and memory system.
Editorial inference: the acceptance-length saturation seen above width 64 suggests that further decoding gains on such devices will have to come from reducing per-step memory traffic (weight compression, quantization, or KV-cache reduction) rather than from ever-wider verification.
Editorial inference: a natural stress test would be to move the tree-construction rule to a different self-drafting method whose attention sparsity pattern differs; the density boundary between CPU and GPU would need to be recomputed.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The systems contribution here is real. Splitting all linear layers by columns to avoid all-reduce on unified memory, splitting each attention head into dense and sparse parts between CPU and GPU with online softmax, and the custom ARM SpMM kernels form a sensible integration that isn't in Megatron, EdgeNN, or Medusa. The acceptance-length table is the strongest part: trees calibrated on MT-Bench and then evaluated on GSM8K, MBPP, and HumanEval, with the calibrated set also shown as a row. That's a mild circularity, but the generalization to held-out datasets mitigates it.

The problem is the speedup accounting. The paper decomposes 7.6x into 3.27x algorithmic gain and 2.31x parallel speedup, arguing that because sequential decoding is memory-bandwidth-bound, Medusa verification costs about the same per step as sequential decode. On the 204MHz Jetson NX with tensor cores, that premise is plausible for width up to maybe 16. But the paper never reports absolute per-step latencies, so we cannot check it. More troublesome: a 2.31x parallel speedup means Ghidorah's per-step time is about 0.43x the sequential step time. For a 7B FP16 model (~14GB) on a device with 51.2GB/s, loading the weights takes about 0.26 seconds. Sequential decode should be at that floor. Ghidorah also loads all weights each step—the column split doesn't reduce total bytes—so it cannot beat the floor unless there is a mechanism to cut memory traffic that isn't described. That makes the 7.6x number hard to swallow without absolute timings.

The other soft spots are smaller: GPU clock lock appears only in the setup, not the abstract; no error bars; no released code or verification trees; end-to-end results at context length 256 only. The ARM sparse optimization numbers (3.49x over naive sparse) are plausible and well explained.

Who is this for? Systems researchers working on on-device LLM inference. The architecture is worth refereeing, and the acceptance-length data are a useful reference. But the evaluation needs serious revision: report absolute throughput and per-step latency for every baseline, explain how Ghidorah's step time stays above the memory-load floor, and either defend or soften the 7.6x claim. I'd send it to peer review with those conditions. If the speedup doesn't survive contact with absolute numbers, the contribution can still stand—as an architecture study with good sparse kernels—with scaled-down claims.

Referee Report

5 major / 6 minor

Summary. This paper presents Ghidorah, an LLM inference system for edge devices with unified memory that combines Medusa-style speculative decoding with heterogeneous CPU-GPU model parallelism. The system has three main components: (i) the HCMP architecture, which partitions linear layers by columns to avoid all-reduce, splits attention into dense and sparse parts across CPU and GPU, uses online softmax, and includes custom ARM SpMM kernels; (ii) the ARCA profiling approach, which calibrates verification trees on a dataset and selects verification width and partition ratio; and (iii) an evaluation on Vicuna-7B on NVIDIA Jetson Xavier NX reporting acceptance lengths from 1.72 to 3.74 and up to 7.6x decoding speedup over sequential decoding, decomposed into 3.27x algorithmic and 2.31x parallel gains.

Significance. If the speedup is real, this is a valuable system contribution: it addresses an important gap in edge LLM inference by exploiting unified memory and heterogeneous cores, and the out-of-sample acceptance-length results on GSM8K, MBPP, and HumanEval provide credible evidence that the verification-tree construction generalizes beyond its calibration set. The ARM SpMM optimizations are evaluated with component-level baselines, and the design avoids the communication overhead of discrete-memory tensor parallelism. However, the headline decomposition rests on an unverified timing assumption and a throttled-clock configuration, so the current evidence does not yet establish the claimed 7.6x in realistic settings.

major comments (5)

[Section IV-C] The decomposition into a 3.27x algorithmic enhancement and a 2.31x parallel speedup relies on the premise that "the sequential decoding approach is memory-bandwidth-bound, its execution time is similar to that of Medusa." This premise is not demonstrated. The paper reports only normalized decoding throughput (Figure 9) and never gives absolute per-step latencies for Sequential, Medusa, Medusa+EM, or Ghidorah at the tested verification widths. At the locked 204MHz GPU clock, the compute for width-16 verification is substantially larger than for a single sequential token, so the assumption cannot be taken for granted. Please report wall-clock per-step latencies at widths 4-64 and at stock clocks, and compute the algorithmic speedup as acceptance_length x (T_seq / T_medusa_step) instead of booking the acceptance length directly as the algorithmic gain.
[Section IV-A] The experimental setup locks the GPU at 204MHz and the CPU at 1.9GHz to "simulate end-user devices with more balanced capabilities," but the abstract and Section IV-C present the 7.6x speedup without this qualification. Because the entire speedup argument depends on the compute/memory-bandwidth balance, results at stock clocks are essential for external validity. Please add stock-clock measurements for at least the width-16 configuration and report DRAM clock/bandwidth settings so readers can judge how representative the throttled configuration is.
[Section III-C.1, Table I, Figure 9] MT-Bench is used as the calibration dataset for verification-tree determination (Section III-C.1) and then appears as the first row of Table I and the first panel of Figure 9. The held-out datasets provide evidence of generalization, but the in-sample MT-Bench row should be labeled as calibration-evaluation. Moreover, the headline 7.6x speedup and the 3.27x acceptance length are not tied to a specific dataset and context length; Table I shows 3.27 only for MBPP at width 16. Please state the exact configuration (dataset, context length, clock settings) for the headline numbers and clearly separate calibration from evaluation results.
[Section III-B.1] The design description says the CPU and GPU "receive the same input, multiply it by different columns of weights, and then write the outputs to their designated memory regions without enforcing consistency between them. This allows the result to be used in the next operation without any additional data access." For a subsequent column-split linear layer, the input is the concatenation of both units' partial outputs, so each unit must read the other unit's partition from shared memory. This is not "no additional data access"; it is a different memory-access pattern than an all-reduce. Please provide a memory-traffic accounting comparing this scheme against Megatron-style row-split plus all-reduce, and describe the memory-consistency/synchronization protocol used.
[Section IV-C, Section IV-D] The claim that "the CPU can only maintain a similar execution time from 4 to 16 verification width" (Section IV-C) is not substantiated by any CPU-only timing data, and all end-to-end experiments use a context length of approximately 256. Figure 10(a) addresses dynamic partitioning for the attention module only, at verification length 64. Please add end-to-end decoding results at longer context lengths (e.g., 512-2048) and a per-unit (CPU/GPU) time breakdown to support the width-selection and dynamic-partitioning claims.

minor comments (6)

[Section IV-B] There are typos: "Huamn-eval" should be "Human-eval," and "CTranslated2" should be "CTranslate2."
[Figure 9] The y-axis label "Normalized Decoding Speed" should be explicitly defined (e.g., ratio of tokens/s relative to Sequential), and error bars or per-run variability should be reported so the reader can assess statistical significance.
[Section III-C.1] The verification-tree search is described only as "brute-force search based on the estimated tree"; please specify the search space, the number of Medusa heads used, the candidate tokens per head, and the calibration-set size for reproducibility.
[Section III-A] The ARCA contention-aware partitioning procedure is described only qualitatively as "gradual adjustments"; include algorithmic details such as initialization, step size, and stopping criteria.
[Section IV-D] The sentence "The execution time of optimized sparse implementation exceeds naive sparse and dense implementations by 3.49x and 1.90x respectively" should be rephrased as "is 3.49x and 1.90x faster," and the text should clarify that these are component-level, not end-to-end, improvements.
[Abstract] Since the 7.6x result is obtained with the GPU locked at 204MHz, the abstract should disclose this condition to avoid overstating the result.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central speedup claim rests on out-of-sample acceptance-length measurements and direct runtime comparisons, with only a transparently labeled in-sample calibration row.

full rationale

The derivation chain in Ghidorah is self-contained. HCMP is a runtime partitioning scheme evaluated by direct throughput comparison against external baselines (Sequential, Medusa, Medusa+EM). ARCA selects verification trees by per-head accuracy measured on MT-Bench; the acceptance lengths in Table I and speedups in Figure 9 are measured outcomes, not values derived from the fitted accuracy estimator. The headline 3.27x algorithmic gain corresponds to the out-of-sample MBPP acceptance length at width 16, and the paper explicitly validates the trees on GSM8K, MBPP, and HumanEval. The MT-Bench row is indeed in-sample, but it is transparently labeled as the calibration dataset and is not the load-bearing evidence for the headline claim. The statement that sequential and Medusa per-step times are similar because decoding is memory-bandwidth-bound is an empirical premise rather than a definitional or self-referential reduction; it may be fragile on stock clocks, but that is a correctness risk, not circularity. The only self-citations ([12], [36]) appear in related work and are not load-bearing. No step reduces, by construction or by self-citation, to its own inputs. Therefore the paper shows no significant circularity.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on four empirical hardware-behavior premises (bandwidth-bound decode, cheap unified-memory cross-unit access, stable dense/sparse attention split, power-of-two widths) and on per-head acceptance probabilities fit to a calibration set. HCMP and ARCA are software and methodology artifacts, not postulated physical entities, so no invented entities are recorded.

free parameters (4)

Per-head token acceptance probabilities (calibration-fitted) = not reported in paper
Section III-C.1 fits the accuracies of top predictions of each head on the MT-Bench calibration set and multiplies them to score candidate sequences; these numbers directly determine the verification trees used in all experiments.
Candidate verification width set = powers of two, 2 to 64
Section III-C.2 restricts the search to powers of two based on an observed alignment with vectorization and wave quantization; non-power-of-two widths are not explored.
GPU and CPU clock locks = GPU 204MHz, CPU 1.9GHz
Section IV-A locks the Jetson GPU at 204MHz to simulate balanced end-user devices; every reported speedup ratio, including the 7.6x headline, is measured under this throttle.
Per-unit partitioning ratio (ARCA contention-aware) = not reported
Section III-C.3 adjusts the partitioning ratio gradually based on measured execution times and contention; the final ratios for the reported experiments are not published, so the partition is fit to the test hardware.

assumptions (6)

domain assumption Decoding phase is memory-bandwidth-bound, so extra verification compute for speculative decoding is nearly free.
Section IV-C: 'Since the sequential decoding approach is memory-bandwidth-bound, its execution time is similar to that of Medusa.' This premise converts the 3.27x acceptance-length gain into an algorithmic speedup; if compute also matters at the locked 204MHz GPU, the decomposition fails.
domain assumption Unified memory makes cross-CPU/GPU access nearly free (page sync under 0.1ms), so column-split partitions need no all-reduce.
Section II-D states page synchronization between processing units takes less than 0.1ms on Jetson Xavier NX. HCMP's deferred-reduction column split (Section III-B.1) relies on cheap reads of the other unit's partial outputs.
domain assumption Attention in speculative decoding splits into a dense part (KV cache) and a sparse part (new tokens) with stable affinity to GPU and CPU.
Section III-B.2 divides each attention head into dense and sparse parts; the claimed 2.31x parallel speedup depends on this split being load-balanced and faster than GPU-only dense masked attention.
domain assumption Per-head prediction probabilities are independent, so candidate sequence acceptance is the product of token accuracies.
Section III-C.1: 'estimate the accuracy of a candidate sequence by multiplying the accuracies of its related tokens.' Medusa heads share context, so correlated errors would make the estimated tree suboptimal; the paper measures, rather than derives, the resulting acceptance length.
standard math Online softmax with per-unit partial attention outputs produces exactly the full softmax attention.
Section III-B.2 cites FlashAttention and Ring Attention for online softmax rescaling; this is standard numerics used to avoid cross-unit synchronization in the attention module.
ad hoc to paper Power-of-two verification widths are the efficient operating points due to wave quantization and vectorization.
Section III-C.2 states that setting candidate verification widths to powers of two aligns with the vectorization capabilities of these devices; this hand-chosen restriction bounds the ARCA search and may exclude better non-power-of-two widths.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism." pith.science (2026). https://pith.science/paper/EA75K3XP

@misc{pith2026250523219,
  author       = {Pith},
  title        = {Pith review of: Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EA75K3XP}},
  note         = {Machine review of arXiv:2505.23219}
}

read the original abstract

In-situ LLM inference on end-user devices has gained significant interest due to its privacy benefits and reduced dependency on external infrastructure. However, as the decoding process is memory-bandwidth-bound, the diverse processing units in modern end-user devices cannot be fully exploited, resulting in slow LLM inference. This paper presents Ghidorah, a LLM inference system for end-user devices with the unified memory architecture. The key idea of Ghidorah can be summarized in two steps: 1) leveraging speculative decoding approaches to enhance parallelism, and 2) ingeniously distributing workloads across multiple heterogeneous processing units to maximize computing power utilization. Ghidorah includes the hetero-core model parallelism (HCMP) architecture and the architecture-aware profiling (ARCA) approach. The HCMP architecture guides partitioning by leveraging the unified memory design of end-user devices and adapting to the hybrid computational demands of speculative decoding. The ARCA approach is used to determine the optimal speculative strategy and partitioning strategy, balancing acceptance rate with parallel capability to maximize the speedup. Additionally, we optimize sparse computation on ARM CPUs. Experimental results show that Ghidorah can achieve up to 7.6x speedup in the dominant LLM decoding phase compared to the sequential decoding approach in NVIDIA Jetson NX.

Figures

Figures reproduced from arXiv: 2505.23219 by the authors.

**Figure 2.** The partitioning [11] of Transformer model in the matrix form. initializes the KV cache. Since the prompt typically consists of many tokens, it has high parallelism. Next, with the KV cache, each step of the decode phase processes only one new token generated by the previous step and appends the new KV cache. In end-user scenarios, single-sample inference processes only a single token during each iteration of the de… view at source ↗

**Figure 4.** The unified memory architecture and the discrete architecture. [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗

Figures from the paper (6 more)

**Figure 3.** Figure 3: Sparsity visualization of the X = Q × KT for speculative decoding approaches. Light yellow indicates points that require computation, while dark blue represents points that do not need computation. X = QKT A = sof tmax(X) O = AV (1) [PITH_FULL_IMAGE:figures/full_fig_p…

**Figure 5.** Figure 5: The overview of Ghidorah. Linear Layer Attention Module Linear Layer Linear Layer Linear Layer GPU CPU cache_V V T K T cache_K Q Dram cache_V cache_K Online Softmax Reduce WB WC WD CPU Output GPU Output Split Weights by Column Split Each Head with Affinity Softmax Soft…

**Figure 6.** Figure 6: The illustration of Hetero-Core Model Parallelism. [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]

**Figure 8.** Figure 8: The verification tree determination with a verification width of 16. [PITH_FULL_IMAGE:figures/full_fig_p005_8.png]

**Figure 9.** Figure 9: The overall performance under different verification widths. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

**Figure 10.** Figure 10: Performance of Sparse Optimization find that utilizing heterogeneous processing unit yields a better acceleration effect than merely running Medusa on the GPU. Taking MBPP dataset as the instance, Ghidorah achieves an average speedup of 2.06× compared to running Medus…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 23 canonical work pages

[1]

Efficient memory management for large language model serving with pagedattention,

W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles , 2023, pp. 611–626

2023
[2]

Specinfer: Accelerating large language model serving with tree-based speculative inference and ver- ification,

X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y . Y . Wong, A. Zhu, L. Yang, X. Shi et al. , “Specinfer: Accelerating large language model serving with tree-based speculative inference and ver- ification,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volum...

work page 2024
[3]

Medusa: Simple llm inference acceleration framework with multiple decoding heads,

T. Cai, Y . Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple llm inference acceleration framework with multiple decoding heads,” arXiv preprint arXiv:2401.10774 , 2024

arXiv 2024
[4]

Break the sequential dependency of LLM inference using lookahead decoding,

Y . Fu, P. Bailis, I. Stoica, and H. Zhang, “Break the sequential dependency of LLM inference using lookahead decoding,” in Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. [Online]. Available: https://openreview.net/forum?id=eDjvSFOkXw

work page 2024
[5]

Codl: efficient CPU-GPU co-execution for deep learning inference on mobile devices,

F. Jia, D. Zhang, T. Cao, S. Jiang, Y . Liu, J. Ren, and Y . Zhang, “Codl: efficient CPU-GPU co-execution for deep learning inference on mobile devices,” in MobiSys ’22: The 20th Annual International Conference on Mobile Systems, Applications and Services, Portland, Oregon, 27 June 2022 - 1 July 2022 , N. Bulusu, E. Aryafar, A. Balasubramanian, and J. Son...

arXiv 2022
[6]

Edgenn: Efficient neural network inference for cpu-gpu integrated edge devices,

C. Zhang, F. Zhang, K. Chen, M. Chen, B. He, and X. Du, “Edgenn: Efficient neural network inference for cpu-gpu integrated edge devices,” in 2023 IEEE 39th International Conference on Data Engineering (ICDE). IEEE, 2023, pp. 1193–1207

work page 2023
[7]

High-throughput cnn inference on embedded arm big. little multicore processors,

S. Wang, G. Ananthanarayanan, Y . Zeng, N. Goel, A. Pathania, and T. Mitra, “High-throughput cnn inference on embedded arm big. little multicore processors,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , vol. 39, no. 10, pp. 2254–2267, 2019

work page 2019
[8]

Dopia: online parallelism management for integrated cpu/gpu archi- tectures,

Y . Cho, P. Jiyeon, N. Florian, J. Changyeon, T. R. Gross, and B. Egger, “Dopia: online parallelism management for integrated cpu/gpu archi- tectures,” in Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming , 2022, pp. 32–45

work page 2022

Show all 41 references

[9]

Apple m4,

“Apple m4,” https://en.wikipedia.org/wiki/Apple M4, 2024

2024
[10]

vllm github,

“vllm github,” https://github.com/vllm-project/vllm, 2024

2024
[11]

Megatron-lm: Training multi-billion parameter language models using model parallelism,

M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053 , 2019

1909 arXiv
[12]

Galaxy: A resource-efficient collaborative edge ai system for in-situ transformer inference,

S. Ye, J. Du, L. Zeng, W. Ou, X. Chu, Y . Lu, and X. Chen, “Galaxy: A resource-efficient collaborative edge ai system for in-situ transformer inference,” arXiv preprint arXiv:2405.17245 , 2024

2024 arXiv
[13]

Intel core ultra processor family,

“Intel core ultra processor family,” https://www.intel.com/content/www/ us/en/products/details/processors/core-ultra.html, 2024

2024
[14]

Meet jetson, the platform for ai at the edge,

“Meet jetson, the platform for ai at the edge,” https://developer.nvidia. com/embedded-computing, 2024

2024
[15]

Communication effi- cient distributed machine learning with the parameter server,

M. Li, D. G. Andersen, A. J. Smola, and K. Yu, “Communication effi- cient distributed machine learning with the parameter server,” Advances in Neural Information Processing Systems , vol. 27, 2014

2014
[16]

Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models,

S. A. Jacobs, M. Tanaka, C. Zhang, M. Zhang, S. L. Song, S. Rajbhandari, and Y . He, “Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models,”
[17]

Codl: efficient cpu-gpu co-execution for deep learning inference on mobile devices

F. Jia, D. Zhang, T. Cao, S. Jiang, Y . Liu, J. Ren, and Y . Zhang, “Codl: efficient cpu-gpu co-execution for deep learning inference on mobile devices.” in MobiSys, vol. 22, 2022, pp. 209–221

2022
[18]

Attention is all you need,

A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

2017
[19]

Distillspec: Improving speculative decoding via knowledge distillation,

Y . Zhou, K. Lyu, A. S. Rawat, A. K. Menon, A. Rostamizadeh, S. Ku- mar, J.-F. Kagy, and R. Agarwal, “Distillspec: Improving speculative decoding via knowledge distillation,” arXiv preprint arXiv:2310.08461 , 2023

2023 arXiv
[20]

EAGLE: speculative sampling requires rethinking feature uncertainty,

Y . Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE: speculative sampling requires rethinking feature uncertainty,” in Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. [Online]. Available: https://openrev...

2024
[21]

Apple a17,

“Apple a17,” https://en.wikipedia.org/wiki/Apple A17, 2024

2024
[22]

Amd reveals next-gen desktop processors for extreme pc gam- ing and creator performance,

“Amd reveals next-gen desktop processors for extreme pc gam- ing and creator performance,” https://www.amd.com/en/newsroom/ press-releases, 2024

2024
[23]

Qualcomm snapdragon,

“Qualcomm snapdragon,” https://en.wikipedia.org/wiki/Qualcomm Snapdragon, 2024

2024
[24]

Ring attention with blockwise transformers for near- infinite context,

H. Liu et al. , “Ring attention with blockwise transformers for near- infinite context,” arXiv preprint arXiv:2310.01889 , 2023

2023 arXiv
[25]

Flashattention: Fast and memory-efficient exact attention with io-awareness,

T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,”Advances in Neural Information Processing Systems , vol. 35, pp. 16 344–16 359, 2022

2022
[26]

Wave quantization,

“Wave quantization,” https://docs.nvidia.com/deeplearning/performance/ dl-performance-matrix-multiplication/index.html#wave-quant, 2024

2024
[27]

Nvidia fastertransformer: Transformer related optimization, including bert, gpt,

Nvidia, “Nvidia fastertransformer: Transformer related optimization, including bert, gpt,” in Github, 2023

2023
[28]

Ctranslate2,

“Ctranslate2,” https://github.com/OpenNMT/CTranslate2, 2024

2024
[29]

Judging llm-as-a-judge with mt-bench and chatbot arena,

L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, pp. 46 595–46 623, 2023

2023
[30]

Llama: Open and efficient foundation language models,

H. Touvron et al. , “Llama: Open and efficient foundation language models,” 2023. [Online]. Available: https://arxiv.org/abs/2302.13971

2023 arXiv
[31]

Training verifiers to solve math word problems,

K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al., “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168 , 2021

2021 arXiv
[32]

Evaluating large language models trained on code,

M. Chen, J. Tworek, et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

2021 arXiv
[33]

Program synthesis with large language models,

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

2021 arXiv
[34]

Edgenn: Efficient neural network inference for CPU-GPU integrated edge devices,

C. Zhang, F. Zhang, K. Chen, M. Chen, B. He, and X. Du, “Edgenn: Efficient neural network inference for CPU-GPU integrated edge devices,” in 39th IEEE International Conference on Data Engineering, ICDE 2023, Anaheim, CA, USA, April 3-7, 2023 . IEEE, 2023, pp. 1193–1207. [Onlin...

2023
[35]

Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve},

A. Agrawal et al., “Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve},” in 18th USENIX Symposium on Operat- ing Systems Design and Implementation (OSDI 24) , 2024, pp. 117–134

2024
[36]

Communication-efficient model parallelism for distributed in-situ trans- former inference,

Y . Wei, S. Ye, J. Jiang, X. Chen, D. Huang, J. Du, and Y . Lu, “Communication-efficient model parallelism for distributed in-situ trans- former inference,” in 2024 Design, Automation & Test in Europe Conference & Exhibition (DATE) . IEEE, 2024, pp. 1–6

2024
[37]

Petals: Collabo- rative inference and fine-tuning of large models,

A. Borzunov, D. Baranchuk, T. Dettmers, M. Ryabinin, Y . Belkada, A. Chumachenko, P. Samygin, and C. Raffel, “Petals: Collabo- rative inference and fine-tuning of large models,” arXiv preprint arXiv:2209.01188, 2022

2022 arXiv
[38]

Asymo: scalable and efficient deep-learning inference on asymmetric mobile cpus,

M. Wang, S. Ding, T. Cao, Y . Liu, and F. Xu, “Asymo: scalable and efficient deep-learning inference on asymmetric mobile cpus,” in ACM MobiCom ’21: The 27th Annual International Conference on Mobile Computing and Networking, New Orleans, Louisiana, USA, October 25-29, 2021 . ...

2021
[39]

Sequoia: Scalable, robust, and hardware-aware speculative decoding,

Z. Chen, A. May, R. Svirschevski, Y . Huang, M. Ryabinin, Z. Jia, and B. Chen, “Sequoia: Scalable, robust, and hardware-aware speculative decoding,” arXiv preprint arXiv:2402.12374 , 2024

2024 arXiv
[40]

Llmcad: Fast and scalable on-device large language model inference,

D. Xu, W. Yin, X. Jin, Y . Zhang, S. Wei, M. Xu, and X. Liu, “Llmcad: Fast and scalable on-device large language model inference,” CoRR, vol. abs/2309.04255, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2309.04255

2023 doi arXiv
[2023]

Available: https://arxiv.org/abs/2309.14509

[Online]. Available: https://arxiv.org/abs/2309.14509

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Efficient memory management for large language model serving with pagedattention,

W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles , 2023, pp. 611–626

2023

[2] [2]

Specinfer: Accelerating large language model serving with tree-based speculative inference and ver- ification,

X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y . Y . Wong, A. Zhu, L. Yang, X. Shi et al. , “Specinfer: Accelerating large language model serving with tree-based speculative inference and ver- ification,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volum...

work page 2024

[3] [3]

Medusa: Simple llm inference acceleration framework with multiple decoding heads,

T. Cai, Y . Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple llm inference acceleration framework with multiple decoding heads,” arXiv preprint arXiv:2401.10774 , 2024

arXiv 2024

[4] [4]

Break the sequential dependency of LLM inference using lookahead decoding,

Y . Fu, P. Bailis, I. Stoica, and H. Zhang, “Break the sequential dependency of LLM inference using lookahead decoding,” in Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. [Online]. Available: https://openreview.net/forum?id=eDjvSFOkXw

work page 2024

[5] [5]

Codl: efficient CPU-GPU co-execution for deep learning inference on mobile devices,

F. Jia, D. Zhang, T. Cao, S. Jiang, Y . Liu, J. Ren, and Y . Zhang, “Codl: efficient CPU-GPU co-execution for deep learning inference on mobile devices,” in MobiSys ’22: The 20th Annual International Conference on Mobile Systems, Applications and Services, Portland, Oregon, 27 June 2022 - 1 July 2022 , N. Bulusu, E. Aryafar, A. Balasubramanian, and J. Son...

arXiv 2022

[6] [6]

Edgenn: Efficient neural network inference for cpu-gpu integrated edge devices,

C. Zhang, F. Zhang, K. Chen, M. Chen, B. He, and X. Du, “Edgenn: Efficient neural network inference for cpu-gpu integrated edge devices,” in 2023 IEEE 39th International Conference on Data Engineering (ICDE). IEEE, 2023, pp. 1193–1207

work page 2023

[7] [7]

High-throughput cnn inference on embedded arm big. little multicore processors,

S. Wang, G. Ananthanarayanan, Y . Zeng, N. Goel, A. Pathania, and T. Mitra, “High-throughput cnn inference on embedded arm big. little multicore processors,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , vol. 39, no. 10, pp. 2254–2267, 2019

work page 2019

[8] [8]

Dopia: online parallelism management for integrated cpu/gpu archi- tectures,

Y . Cho, P. Jiyeon, N. Florian, J. Changyeon, T. R. Gross, and B. Egger, “Dopia: online parallelism management for integrated cpu/gpu archi- tectures,” in Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming , 2022, pp. 32–45

work page 2022