REVIEW 4 major objections 6 minor 20 references
TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAs
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Full ternary LLM inference runs on a 7-watt edge FPGA
desk verdict Real hardware, real measurements, and a clever architecture, but the missing output-quality evaluation is a load-bearing gap that needs fixing before the paper can claim practical deployment. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the table-lookup-based ternary matmul: weights in {-1,0,1} are packed into groups of G, each group encoded as a small index; on-chip LUTs are configured as a table of all $3^G$ signed sums for a group of activations, so a multiply-accumulate becomes an indexed add or subtract. Index vectors access multiple tables in parallel, and the table setup is reused across all output dimensions. The second mechanism is reversed attention scheduling for prefill, which iterates query tokens from the end of the sequence and fuses score, softmax, and value aggregation in one pass, eliminating masked computations that naive and dense schedules incur. Together these mechanisms reduce LUT pressure and DRAM traffic enough to fit both prefill and decoding on a 108K-LUT FPGA.
What would settle it
Run the exact 0.7B ternary model deployed on TeLLMe on a standard language-generation or perplexity benchmark against its full-precision counterpart; if quality degrades sharply, the speed and energy numbers do not translate into a usable deployment. Alternatively, measure time-to-first-token and decode throughput on an HBM-enabled FPGA: if decode speed does not scale with memory bandwidth, the bandwidth-bound claim would be contradicted.
Extended reading notes
Core claim
TeLLMe is, to the authors' knowledge, the first FPGA accelerator for a 1.58-bit ternary LLM that supports both prefill and autoregressive decoding on the same edge device. Using BitNet-1.58-style weights and 8-bit activations, it implements a table-lookup matmul engine that precomputes all add and subtract combinations for groups of ternary weights and reuses grouped activations across projection and feedforward layers. A fused attention unit with reversed reordering processes queries from the end of the sequence, avoiding invalid causal-mask computation and keeping bandwidth near constant. The decoding-phase attention hardware is reused for the language-model head because both are memory-bound matrix-vector operations. On AMD KV260 at 250 MHz, TeLLMe reports up to 9.51 tokens per second decoding, prefill latencies of 0.55 to 1.15 seconds for 64 to 128-token prompts, and under 7 W total power.
Load-bearing premise
The system assumes the 0.7B BitNet-style ternary model used in the experiments retains usable output quality after 1.58-bit quantization; the paper reports no perplexity, benchmark, or sample-quality evaluation, so high throughput and low power could coexist with unusable text.
Editorial extensions
If this is right
- Full on-device LLM inference becomes possible on low-cost FPGAs, removing the need to offload prefill to a host and the associated privacy and latency costs.
- The 9.51 tokens per second decode rate and roughly 0.55 second prefill for short prompts make interactive edge applications, such as chatbots and on-device summarization, plausible at 7 W.
- Table-lookup matmul provides a hardware template for other ultra-low-bit formats, including pure binary and 2-bit weights, by enlarging the lookup table accordingly.
- Because decoding is memory-bandwidth bound, moving the same design to a higher-bandwidth platform should raise tokens per second nearly proportionally, an architectural rather than algorithmic ceiling.
- Prior FPGA accelerators that only decode would need to add a prefill path; TeLLMe shows the resource overhead of doing so is affordable at roughly 108K LUTs.
Reading between the lines
- If 1.58-bit model quality holds up in evaluation, the same table-lookup and reversed-attention machinery could extend to 1-bit or 2-bit quantization with only lookup-table size changes, broadening the design's applicability.
- The reversed reordering schedule is independent of ternary quantization and could speed up any causal attention on bandwidth-limited FPGAs or other low-power accelerators.
- The paper's comparisons to mobile SoCs use different process nodes and memory systems; a normalized energy-per-token or quality-adjusted throughput metric would be needed to place the 7 W advantage on a fair footing.
- The absence of model-quality results suggests an immediate follow-up: benchmark the actual 0.7B ternary model on standard tasks to confirm that the generated tokens are useful, not merely fast.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents TeLLMe, an FPGA accelerator for ternary (1.58-bit) LLM inference on the AMD KV260 edge platform. The design combines a table-lookup-based ternary matrix multiplication engine, a fused attention unit with a reverse-reordering schedule for prefill, and hardware reuse of the decoding attention unit for the LM head. The authors report up to 9.51 tokens/s decoding throughput with 1024-token contexts, prefill latencies of 0.55–1.15 s for 64–128 token prompts, and power below 7 W. The paper claims this is the first edge FPGA accelerator to support both prefill and decoding for ternary LLMs end-to-end.
Significance. If the reported results are validated, TeLLMe would be a meaningful advance: it demonstrates that a complete LLM inference pipeline, including the compute-heavy prefill stage, can run on a low-power edge FPGA with ternary weights and 8-bit activations. The paper's strengths are its concrete hardware implementation (HLS/Vivado bitstream on real hardware), detailed resource breakdowns (Tables I and IV), and an explicit algorithmic contribution in the reverse attention scheduling that avoids causal-mask waste (Section III.B). The measured performance against prior FPGA works (Section IV.C) and mobile SoCs (Section IV.D) gives the paper practical relevance. The central limitation is that no output-quality evaluation of the 0.7B ternary model is provided, and the measurement methodology for latency, throughput, and power is not described; these gaps prevent the headline claims from being fully verified as usable LLM inference.
major comments (4)
- [Section IV, Table V and Figure 9] The paper reports no evaluation of the output quality of the 0.7B BitNet-style ternary model. Table V lists only the model size in MB, and Figure 9 reports throughput and prefill latency, but no perplexity, zero-shot benchmark scores, or generated-sample quality is given anywhere in Section IV. The deployment claim that TeLLMe is a 'practical viability' for real-world applications depends on the 1.58-bit quantized model retaining acceptable language quality. Since ternary weights are typically obtained by training from scratch (as in BitNet-1.58) or by carefully designed quantization, the manuscript must state the checkpoint provenance and quantization procedure and report at least one standard quality metric (e.g., perplexity on a held-out set, or a small set of zero-shot tasks). Without this, the reported 9.51 tokens/s and 0.55 s prefill are hardware figures for a system whose end-to-end functionality is unverified.
- [Section IV.A, Figure 9, Table III] The measurement methodology is not described. There is no statement of how latency, throughput, and power were measured (e.g., board-level power monitor vs. on-chip sensors, number of repeated runs, whether power is total system power or FPGA-only), and no error bars or variance information is reported for any of the headline numbers. Given that the power budget of 7 W is a central part of the contribution (abstract, Section I, Conclusion), the authors should specify the measurement setup and report at least the range or standard deviation across runs for the throughput and prefill latency figures.
- [Section IV.C, Table III] The comparison in Table III is not apples-to-apples and the 'up to 16.4x improvement' claim is overstated as stated. TeLLMe runs a 0.7B BitNet W1.58 model, whereas Li et al. run LLaMA2-7B W4, LlamaF runs TinyLLaMA W8, and SECDA runs TinyLLaMA W4; the models differ in parameter count, arithmetic intensity, and memory footprint, so raw tokens/s are not directly comparable. The authors should either normalize throughput by model size or compute/bandwidth requirements, or clearly frame the comparison as 'system-level demonstration' rather than architectural superiority. They should also clarify whether the KV260 bandwidth entry (19.2 GB/s) is the same for all KV260-based designs and how the BW column was obtained for the other works.
- [Section III.B, Table II] The bandwidth and iteration-count formulas in Table II are not derived in the text, and the units are unclear. For example, the reverse scheduling row gives 'N^2/(2p) + N' for both Data Block and Iteration Count, but these quantities have different dimensions (data volume versus number of iterations). The authors should define each column formally and show how the expressions follow from the schedule described in Section III.B.2, particularly the claim that reverse attention reduces both computation and data movement relative to dense scheduling while maintaining constant bandwidth.
minor comments (6)
- [Abstract and Section I] The abstract reports 'up to 9 tokens/s' while Section IV.C reports '9.51 tokens/s'; these numbers should be made consistent throughout the paper.
- [Section I] There is a typo in the introduction: 'Tenary' should be 'Ternary'.
- [Section II.B, references [3] and [8]] Reference [8] (DeepSeek-R1) duplicates reference [3], and the description of DeepSeek-R1 as applying 'hybrid quantization strategy' to MoE layers is not accurate for the R1 reasoning model; the authors likely mean DeepSeek-V3 or another work, and should correct the citation and description.
- [Section III.B, Equation (3)] The superscript notation in Equation (3) is confusing: the text uses both p(1) and p(2) for intermediate max values, but the display mixes m, ℓ, and o with superscripts without a clear convention. Please define each variable and its superscript explicitly.
- [Section IV.B, Table IV] The resource utilization percentages in Table IV are given as (71%), (28%), (66%), (93%), (75%) but the corresponding device capacities are not stated in the table caption; adding the XCK26 resource totals would help readers interpret the utilization.
- [Section IV.D, Table V] The source of the Snapdragon 8 Gen 3 numbers is not cited; please provide a reference or measurement details so readers can assess how those baselines were obtained.
Circularity Check
No significant circularity: TeLLMe's quantitative claims are measured hardware results against external baselines; no fitted parameter is renamed as a prediction.
full rationale
The paper's central claims (9.51 tokens/s decoding, 0.55–1.15 s prefill, 6.72 W power, resource usage) are presented as on-board measurements of a synthesized KV260 bitstream in Section IV, not as quantities derived from fitted parameters. The architectural parameters G=3, T=32, Q=16, and p=4 are implementation choices evaluated by resource and scheduling comparisons, and there is no indication that they were selected to force the reported benchmark numbers. The only self-citations, references [4] and [5], support the generic background statement that low-bit quantization reduces model cost; that premise is independently supported by the BitNet, BitNet-1.58, and DeepSeek-R1 citations, so the self-citations are not load-bearing. Equations (1)–(3) are standard bit-decomposition and FlashAttention-style identities; the reverse attention reordering is a scheduling strategy whose benefit is demonstrated by measurement rather than by definitional equivalence. The reviewer concern that no perplexity or output-quality evaluation of the 0.7B ternary checkpoint is provided is a validation gap, not a circularity: it does not make any reported result equal to its input by construction. The derivation chain from ternary matmul, attention scheduling, and hardware fusion to the measured throughput and latency is self-contained and empirically grounded.
Assumptions & free parameters
free parameters (5)
- G (ternary group size) =
3
- T (number of parallel TL tables) =
32
- Q (parallel index vectors) =
16
- p (reverse-attention parallelism) =
4
- Clock frequency =
250 MHz
assumptions (5)
- domain assumption Causal attention masking is assumed for all prefill decoding steps
- domain assumption BitNet/DeepSeek-style 1.58-bit quantization preserves acceptable model quality
- standard math Online softmax rescaling in the style of FlashAttention V2 is mathematically exact
- standard math The bit-decomposed matrix multiplication identity in Eq. 1 is valid for ternary weights
- domain assumption KV260 resource capacities and 19.2 GB/s DRAM bandwidth are representative of the measured device
Cite this review
Pith. "Pith review of TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAs." pith.science (2026). https://pith.science/paper/RG3KACTM
@misc{pith2026250416266,
author = {Pith},
title = {Pith review of: TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAs},
year = {2026},
howpublished = {\url{https://pith.science/paper/RG3KACTM}},
note = {Machine review of arXiv:2504.16266}
}
read the original abstract
Deploying large language models (LLMs) on edge platforms is challenged by their high computational and memory demands. Although recent low-bit quantization methods (e.g., BitNet, DeepSeek) compress weights to as little as 1.58 bits with minimal accuracy loss, edge deployment is still constrained by limited on-chip resources, power budgets, and the often-neglected latency of the prefill phase. We present TeLLMe, the first ternary LLM accelerator for low-power FPGAs (e.g., AMD KV260) that fully supports both prefill and autoregressive decoding using 1.58-bit weights and 8-bit activations. Our contributions include: (1) a table-lookup matrix engine for ternary matmul that merges grouped activations with online precomputation to minimize resource use; (2) a fused, bandwidth-efficient attention module featuring a reversed reordering scheme to accelerate prefill; and (3) a tightly integrated normalization and quantization--dequantization unit optimized for ultra-low-bit inference. Under a 7W power budget, TeLLMe delivers up to 9 tokens/s throughput over 1,024-token contexts and prefill latencies of 0.55--1.15 s for 64--128 token prompts, marking a significant energy-efficiency advance and establishing a new edge FPGA benchmark for generative AI.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Language mod- els are few-shot learners,
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020
1901
-
[2]
Llama: Open and efficient foundation language models,
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023
arXiv 2023
-
[4]
A two-stage efficient 3- d cnn framework for eeg based emotion recognition,
Y . Qiao, M. Alnemari, and N. Bagherzadeh, “A two-stage efficient 3- d cnn framework for eeg based emotion recognition,” in 2022 IEEE International Conference on Industrial Technology (ICIT). IEEE, 2022, pp. 1–8
work page 2022
-
[5]
Bnn an ideal architecture for acceleration with resistive in memory computation,
A. Ding, Y . Qiao, and N. Bagherzadeh, “Bnn an ideal architecture for acceleration with resistive in memory computation,” IEEE Transactions on Emerging Topics in Computing , vol. 11, no. 2, pp. 281–291, 2023
work page 2023
-
[6]
Bitnet: Scaling 1-bit transformers for large language models,
H. Wang, S. Ma, L. Dong, and et al., “Bitnet: Scaling 1-bit transformers for large language models,” arXiv preprint arXiv:2310.11453 , 2023
arXiv 2023
-
[7]
The era of 1-bit llms: All large language models are in 1.58 bits,
S. Ma, H. Wang, L. Ma, and et al., “The era of 1-bit llms: All large language models are in 1.58 bits,” arXiv preprint arXiv:2402.17764 , 2024
arXiv 2024
-
[8]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,
D. Guo, D. Yang, H. Zhang, and et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025
arXiv 2025
-
[9]
J. Li, T. Li, G. Shen, D. Zhao, Q. Zhang, and Y . Zeng, “Pushing up to the limit of memory bandwidth and capacity utilization for efficient llm decoding on embedded fpga,” arXiv preprint arXiv:2502.10659 , 2025
arXiv 2025
Show all 20 references
-
[10]
Understanding the potential of fpga-based spatial accel- eration for large language model inference,
H. Chen, J. Zhang, Y . Du, S. Xiang, Z. Yue, N. Zhang, Y . Cai, and Z. Zhang, “Understanding the potential of fpga-based spatial accel- eration for large language model inference,” ACM Transactions on Reconfigurable Technology and Systems, vol. 18, no. 1, pp. 1–29, 2024
2024
-
[11]
Fbi-llm: Scaling up fully bina- rized llms from scratch via autoregressive distillation,
L. Ma, M. Sun, and Z. Shen, “Fbi-llm: Scaling up fully bina- rized llms from scratch via autoregressive distillation,” arXiv preprint arXiv:2407.07093, 2024
2024 arXiv
-
[12]
Onebit: Towards extremely low-bit large language models,
Y . Xu, X. Han, Z. Yang, and et al., “Onebit: Towards extremely low-bit large language models,” in Advances in Neural Information Processing Systems (NeurIPS), 2024
2024
-
[13]
Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation,
D. Du, Y . Zhang, S. Cao, and et al., “Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , 2024
2024
-
[14]
Quip: 2-bit quan- tization of large language models with guarantees,
J. Chee, Y . Cai, V . Kuleshov, and C. De Sa, “Quip: 2-bit quan- tization of large language models with guarantees,” arXiv preprint arXiv:2307.13304, 2023
2023 arXiv
-
[15]
T-mac: Cpu renaissance via table lookup for low-bit llm deployment on edge,
J. Wei, S. Cao, T. Cao, and et al., “T-mac: Cpu renaissance via table lookup for low-bit llm deployment on edge,” arXiv preprint arXiv:2407.00088, 2024
2024 arXiv
-
[16]
Llamaf: An efficient llama2 architecture accelerator on embedded fpgas,
H. Xu, Y . Li, and S. Ji, “Llamaf: An efficient llama2 architecture accelerator on embedded fpgas,”arXiv preprint arXiv:2409.11424, 2024
2024 arXiv
-
[17]
Edge-moe: Memory- efficient multi-task vision transformer architecture with task-level spar- sity via mixture-of-experts,
R. Sarkar, H. Liang, Z. Fan, Z. Wang, and C. Hao, “Edge-moe: Memory- efficient multi-task vision transformer architecture with task-level spar- sity via mixture-of-experts,” in IEEE/ACM International Conference on Computer-Aided Design (ICCAD) , 2023, pp. 1–9
2023
-
[18]
Designing efficient llm accelerators for edge devices,
J. Haris, R. Saha, W. Hu, and J. Cano, “Designing efficient llm accelerators for edge devices,” arXiv preprint arXiv:2408.00462 , 2024, accessed: 2025-04-20. [Online]. Available: https://arxiv.org/abs/2408. 00462
2024 arXiv
-
[19]
Table- lookup mac: Scalable processing of quantised neural networks in fpga soft logic,
D. Gerlinghoff, B. Choong, R. Goh, W.-F. Wong, and T. Luo, “Table- lookup mac: Scalable processing of quantised neural networks in fpga soft logic,” 04 2024, pp. 235–245
2024
-
[20]
Uram storage bind,
AMD, “Uram storage bind,” 2024. [Online]. Available: https://docs. amd.com/r/2024.2-English/ug1399-vitis-hls/pragma-HLS-bind storage
2024
-
[21]
Flashattention: Fast and memory-efficient exact attention with io-awareness,
T. Dao, D. Y . Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” 2022. [Online]. Available: https://arxiv.org/abs/2205.14135
2022 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.