A custom Verilog module accelerates the token-acceptance step of speculative decoding by about 7x over GPUs, but the step is only a small fraction of total LLM inference.
LlamaF: An Efficient Llama2 Architecture Accelerator on Embedded FPGAs
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Large language models (LLMs) have demonstrated remarkable abilities in natural language processing. However, their deployment on resource-constrained embedded devices remains difficult due to memory and computational demands. In this paper, we present an FPGA-based accelerator designed to improve LLM inference performance on embedded FPGAs. We employ post-training quantization to reduce model size and optimize for off-chip memory bandwidth. Our design features asynchronous computation and a fully pipelined accelerator for matrix-vector multiplication. Experiments of the TinyLlama 1.1B model on a Xilinx ZCU102 platform show a 14.3-15.8x speedup and a 6.1x power efficiency improvement over running exclusively on ZCU102 processing system (PS).
citation-role summary
background 1
citation-polarity summary
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
HADES: Hardware Accelerated Decoding for Efficient Speculation in Large Language Models
A custom Verilog module accelerates the token-acceptance step of speculative decoding by about 7x over GPUs, but the step is only a small fraction of total LLM inference.