Pith. sign in

REVIEW 5 major objections 5 minor 21 references

VEDA: Efficient LLM Generation Through Voting-based KV Cache Eviction and Dataflow-flexible Accelerator

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A voting-based KV cache eviction policy keeps Llama-2 7B accurate at 10% cache size and, paired with a dataflow-flexible accelerator, runs it on a 375 mW edge chip.

desk verdict A credible algorithm-hardware co-design whose voting-based KV eviction idea is worth a serious look, but whose single-benchmark accuracy claim needs more support before it convinces. read the letter →

arxiv 2507.00797 v1 pith:5JZFJSV3 submitted 2025-07-01 cs.AR

Zhican Wang , Hongxiang Fan , Haroon Waris , Gang Wang , Zhenyu Li , Jianfei Jiang , Yanan Sun , Guanghui He This is my paper · ORCID
classification cs.AR
keywords KVcacheevictionvoting-basedalgorithmLLMgenerationedgeacceleratorflexibledataflowreconfigurablePEarrayelement-serialschedulingattentionsparsity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to make LLM text generation cheap enough for edge devices by attacking the memory bottleneck and the compute architecture together. It proposes a voting-based key-value (KV) cache eviction rule: each generated token votes against cache entries it barely attends to, votes accumulate across heads and time steps, and the most-voted entry is evicted when the cache is full. The authors argue that this stays accurate even at 10 percent of the original cache size, outperforming two established eviction schemes, while remaining simple enough for hardware. On the architecture side, the paper introduces a runtime-reconfigurable PE array that switches between inner-product and outer-product dataflow, so sequence-length variation becomes a scheduling matter rather than a utilization loss, and an element-serial scheme that overlaps softmax and layer normalization with matrix-vector work and cuts special-function-unit count from $O(N)$ to $O(1)$. If these claims hold, a single 28 nm chip can run Llama-2 7B at 18.6 tokens per second on 375 milliwatts.

What carries the argument

The load-bearing object is the voting engine coupled to the runtime-reconfigurable PE array. The voting engine keeps a per-position vote count $C(i)$; each generated token $i$ votes against position $j$ when the attention score $s'(i,j)$ falls below the adaptive threshold $T(i)=a\cdot \mathrm{mean}(s'(i))-b\cdot \mathrm{std}(s'(i))$, with the first 32 positions reserved and ties resolved toward the earliest position. This converts sparse attention into a thresholding operation that simple counters can track in hardware. The PE array is an $8\times 8\times 2$ grid of reconfigurable processing elements; in inner-product mode partial sums flow through a two-level adder tree to compute $qK^{T}=s$, and in outer-product mode each element accumulates a broadcast scalar times its local weight to compute $s'V=o$. The same element-serial principle feeds the output of the adder tree element-by-element into the softmax reduction unit and feeds normalized scores element-by-element into the outer-product array, so a single small special-function unit suffices to absorb the nonlinear-operator latency.

What would settle it

Run voting eviction at a 0.1 cache ratio on a long-context task with a hard topic shift, such as a document whose first half is irrelevant to questions in the second half, and compare against H2O and StreamingLLM; if the entries that carry the answer were evicted early because past votes ignored them and accuracy drops below H2O's, the persistence-of-importance premise fails and the claimed advantage does not hold on that workload.

Watch

Extended reading notes

Core claim

The paper's central claim is that the three obstacles to edge LLM generation—growing KV cache, dimension-varying matrix-vector multiplication, and serial nonlinear operators—can be reworked so that a modest hardware budget handles all three. The voting-based eviction algorithm identifies unimportant KV vectors by letting each token vote: for each new attention-score vector, positions below a threshold derived from its own mean and standard deviation get one vote, and the entry with the most votes is evicted, with a reserved prefix protected as an attention sink. Because votes are binary instead of summed raw scores, the algorithm claims to escape the item-count, criteria, and outlier biases that hurt accumulated-score methods such as H2O. The flexible-product dataflow computes $qK^{T}$ in inner-product mode and $s'V$ in outer-product mode on the same reconfigurable PE array, mapping the varying sequence length onto computation cycles rather than leaving hardware idle, and element-serial scheduling overlaps the reduction and normalization stages of softmax and layer normalization with the PE-array computation. The demonstrated consequence is that one VEDA chip can run Llama-2 7B at 18.6 tokens per second using 375.26 mW at 1 GHz in 28 nm, with higher energy efficiency than the compared platforms.

Load-bearing premise

The whole speedup rests on the assumption that attention preferences persist: a KV entry that receives few votes now will keep receiving few votes later, so votes from past tokens reliably pick the right entry to drop, and the fixed hand-tuned threshold values $a=1$ and $b=0.2$ transfer to workloads beyond the one tested.

Editorial extensions

If this is right

  • At a 0.1 cache ratio on PG-19, voting eviction keeps perplexity close to the full-cache baseline while H2O and StreamingLLM degrade, so aggressive KV compression becomes usable for language modeling.
  • Flexible-product dataflow alone cuts attention latency by over 25 percent against a conventional adder-tree design with equal peak throughput, and adding element-serial scheduling brings the total reduction to roughly 60 percent.
  • Because the KV cache stays at the target size instead of growing every step, voting eviction gives a 2.3x to 10.0x speedup over no eviction as the generated sequence extends from 128 to 1024 tokens.
  • One VEDA chip sustains 18.6 tokens/s for Llama-2 7B at 375.26 mW in 28 nm, and the special-function unit consumes under 3 percent of chip power because element-serial scheduling drops the SFU count from $O(N)$ to $O(1)$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The voting rule is really a family of eviction policies parameterized by a and b; if the fixed defaults fail on other tasks, per-model calibration is cheap and could preserve the accuracy advantage.
  • The element-serial principle likely extends beyond softmax and layer normalization to other sequence-level reductions, such as streaming normalizers or recurrent state updates that also process one time step at a time.
  • Nothing in the design binds the cache target to a fixed compression ratio, so a runtime scheduler could shrink or grow the cache based on measured attention uncertainty, trading speed for accuracy mid-generation.
  • The binary-vote design suggests a direct behavioral test: on prompts with a few dominant attention peaks, voting should evict those peak positions less often than accumulated-score eviction does.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents VEDA, an algorithm-hardware-dataflow co-design for LLM generation on edge devices. The algorithmic contribution is a voting-based KV-cache eviction policy that marks attention scores below an adaptive threshold T(i) = a*Mean - b*sigma as votes for eviction, evicting the most-voted KV position once the cache is full. The hardware contributions are a runtime-reconfigurable PE array that switches between inner- and outer-product GEMV dataflows, and an element-serial scheduling scheme that overlaps softmax/layernorm reduction and normalization with GEMV computation using a single SFU. The paper reports perplexity on PG-19, attention latency ablations, speedups of 2.3-10.0x from eviction, and a VEDA implementation in TSMC 28nm with 1.058 mm2 area, 375.26 mW power, and 18.6 tokens/s for Llama-2 7B.

Significance. If the results are confirmed, the work makes a useful contribution: it integrates KV-cache eviction with dataflow flexibility rather than treating algorithm and hardware separately; the element-serial scheduling idea is elegant and the reported SFU overhead (<3%) is impressive; the authors provide concrete RTL synthesis and cross-validated cycle-accurate modeling, and the paper discusses the three biases of accumulation-based eviction in a clear way. The central accuracy claim, however, is currently supported by a single unrepeated experiment with hand-set thresholds, and the hardware comparisons lack methodological details, so the strength of the claims is disproportionate to the evidence.

major comments (5)
  1. [Section VI, Figure 8 (left)] The central algorithmic claim that voting-based eviction 'consistently outperforms' H2O and StreamingLLM is supported by exactly one configuration: 1000 PG-19 samples with Llama-2-7B, no error bars, and no full-cache (no-eviction) baseline in the plot. The threshold hyperparameters a=1 and b=0.2 are stated in Section III to 'generally prove effective' but are not swept, and no held-out calibration is described. Because the right panel's speedup claims presume that accuracy is retained at 0.1 cache size, the reader cannot distinguish a genuine algorithmic advantage from threshold tuning on a single benchmark. Please add at least one additional model and dataset, report variance across multiple runs/seeds, show the no-eviction baseline, and include a sensitivity analysis of a,b.
  2. [Table II and Section VI] The end-to-end comparison with the NVIDIA 4090 is missing essential setup information: software stack, precision, batch size, prompt/generation lengths, power measurement methodology, and whether the GPU numbers include HBM power. The claims of 38.8x energy efficiency and 2.86x throughput for 8-VEDA cannot be reconstructed or reproduced from the text. The technology-scaling comparison with Sanger and Spatten mentions DeepScaleTool [13] but Table II reports only un-scaled numbers. Please provide a complete methodology table and, if possible, the scaled figures.
  3. [Section V, Voting Engine] The voting engine stores and accesses the vote-count vector off-chip ('vote count also needs to be stored and accessed from off-chip'). The speedup evaluation in Figure 8 (right) compares VEDA with eviction against VEDA without eviction, but the text does not state whether the off-chip vote-count traffic and the voting engine's serial comparison/update cost are included in the cycle-accurate latency model. If excluded, the 2.3-10.0x speedup is optimistic. Please state explicitly what the model includes and report the overhead of voting as a fraction of total cycles.
  4. [Section IV-B] The abstract and Section IV-B claim a reduction in hardware complexity 'from O(N) to O(1)' for nonlinear operations. The baseline for this complexity is not defined: a conventional architecture can use one SFU per output column, one per head, or one global unit with pipeline stalls, and these have different O(N) meanings. Moreover, the ablation in Figure 8 (center) gives the baseline 'the same number of SFUs as VEDA,' which does not test the O(N) to O(1) claim directly. Please define the baseline organization and provide an experiment that varies SFU count to support the complexity statement.
  5. [Section III] The voting algorithm relies on the persistence-of-importance hypothesis [8], but the distinguishing element is the adaptive threshold on binarized scores. The paper does not show vote-count distributions, head-wise or layer-wise consistency, or behavior over long generation horizons. Since votes are aggregated across all heads and averaged (Section V), an unjustified aggregation could erase the signal that H2O captures through heavy-hitter scores. Please provide per-head/per-layer vote statistics or an ablation on aggregation strategy.
minor comments (5)
  1. [Figure 3] The pseudocode and notation list are hard to follow: C(i) is used both for the cumulative vote vector and for the current vote count, and the condition 'if (i<=P) break;' is unclear in the generation phase. Please clarify variable names and control flow.
  2. [Figure 8 (left)] The x-axis shows cache sizes 128, 256, 512, 1024, 2048, and 4096, but the text claims the cache is 'reduced to 0.1 of the original length (4096)'; a cache size of about 409 is not shown. Align the claim with the plotted cache sizes or add the missing point.
  3. [Table II] The entry '8−V EDA' appears to contain a formatting artifact. Also, the table caption should state that area and power are post-synthesis estimates rather than measured silicon results.
  4. [Abstract] The phrase 'hardware complexity, from O(N) to O(1)' is too broad; the paper actually reduces the number of SFUs. Rephrase to avoid implying that total hardware complexity is constant.
  5. [References] Reference [17] is the authors' own COSA work, and the paper cites it but does not depend on it for the current claims; consider removing or citing it only where the prefilling-phase dataflow background is discussed.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the voting-based eviction, flexible dataflow, and element-serial scheduling claims are empirically evaluated rather than derived from their own conclusions.

full rationale

The paper's central claims are supported by external evaluation and self-contained hardware simulation, not by a derivation that reduces to its own inputs. The voting-based KV cache eviction algorithm is specified explicitly in Figure 3 and its perplexity is measured on the external PG-19 benchmark against H2O and StreamingLLM; the reported accuracy is an empirical result, not a quantity forced by the algorithm's definition. The hyper-parameters a=1 and b=0.2 are described as 'generally prove effective' but are not fitted to a held-out subset in a way that makes the reported perplexity numbers true by construction; this is a calibration and robustness concern, not circular reasoning. The hardware speedups come from cycle-accurate models cross-validated with RTL simulation and from ablations against a conventional adder-tree baseline, and the flexible-product dataflow and element-serial scheduling are justified by architectural analysis rather than by presupposing the claimed speedups. The 'persistence of importance' assumption is cited from external prior work [8] (Scissorhands) and is shared with existing eviction methods; it is an assumption, not a circular input. The only self-citation, [17] (COSA), appears as background in Section IV-A regarding prefilling-phase dataflow and is not load-bearing for VEDA's central algorithm or hardware contributions. No step could be identified where an equation or fitted parameter is renamed as a prediction, so no circular step is exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the persistence-of-importance assumption for the eviction algorithm, on the accuracy of the simulation environment for the hardware results, and on two hand-set hyper-parameters. No new physical entities are introduced.

free parameters (3)
  • a (threshold coefficient for mean) = 1
    Hand-set default for the voting threshold T(i) = a*Mean - b*sigma; no sensitivity analysis is provided, and the accuracy results are reported with this value.
  • b (threshold coefficient for standard deviation) = 0.2
    Hand-set default alongside a; the paper says these can be fine-tuned through model-specific calibration, which makes them fitted parameters.
  • R (reserved length before voting) = 32
    Reserved length set to establish a lower bound for the KV cache, aligned with the attention sink phenomenon; chosen by hand.
assumptions (4)
  • domain assumption Pivotal KV vectors are likely to remain pivotal, and vice versa (persistence of importance).
    Section III attributes this to [8] Scissorhands and uses it as the basis for the voting eviction algorithm.
  • standard math The mean of a softmax vector of length l is 1/l.
    The algorithm computes Mean = 1/l(i) for the attention score vector, which is a property of softmax summation.
  • domain assumption The cycle-accurate performance model and Ramulator HBM model accurately represent the real hardware.
    Section VI states latency measurements come from a cycle-accurate model cross-validated with RTL, but no validation details are given.
  • domain assumption The baseline accelerators A3, Sanger, and Spatten are configured at equal peak throughput and equal SFU count for fair comparison.
    Section VI ablation says identical peak throughput and same number of SFUs, but the configuration specifics are not detailed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VEDA: Efficient LLM Generation Through Voting-based KV Cache Eviction and Dataflow-flexible Accelerator." pith.science (2026). https://pith.science/paper/5JZFJSV3

@misc{pith2026250700797,
  author       = {Pith},
  title        = {Pith review of: VEDA: Efficient LLM Generation Through Voting-based KV Cache Eviction and Dataflow-flexible Accelerator},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5JZFJSV3}},
  note         = {Machine review of arXiv:2507.00797}
}
read the original abstract

Large Language Models (LLMs) excel in natural language processing tasks but pose significant computational and memory challenges for edge deployment due to their intensive resource demands. This work addresses the efficiency of LLM inference by algorithm-hardware-dataflow tri-optimizations. We propose a novel voting-based KV cache eviction algorithm, balancing hardware efficiency and algorithm accuracy by adaptively identifying unimportant kv vectors. From a dataflow perspective, we introduce a flexible-product dataflow and a runtime reconfigurable PE array for matrix-vector multiplication. The proposed approach effectively handles the diverse dimensional requirements and solves the challenges of incrementally varying sequence lengths. Additionally, an element-serial scheduling scheme is proposed for nonlinear operations, such as softmax and layer normalization (layernorm). Results demonstrate a substantial reduction in latency, accompanied by a significant decrease in hardware complexity, from O(N) to O(1). The proposed solution is realized in a custom-designed accelerator, VEDA, which outperforms existing hardware platforms. This research represents a significant advancement in LLM inference on resource-constrained edge devices, facilitating real-time processing, enhancing data privacy, and enabling model customization.

Figures

Figures reproduced from arXiv: 2507.00797 by the authors.

Figure 1
Figure 1. Detailed LLM structure with sizes denoted. In this paper, matrix-form [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Voting-based KV cache eviction algorithm with notations on the right. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Two mathematical interpretations of GEMV. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Runtime reconfigurable hardware architecture. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Element-serial scheduling. Scheduling. We summarize both softmax and layernorm operations into two stages: reduction and normalization (norm). In the reduction stage, elements are condensed into two numerical characteristics: maximum and exponent summation (exp sum) fo…
Figure 7
Figure 7. Figure 7: Hardware architecture of VEDA, with blue, green, and red denote input weight and output. product implemented GEMV before reduction, the sum of element and the sum of element square is computed simultaneously through the element-serial output to obtain mean and variance…
Figure 8
Figure 8. Figure 8: Experiments. Left: Language modeling evaluation. Center: Ablation study of dataflow optimization. Right: Speedup by voting-based cache eviction. [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 9 canonical work pages

  1. [13]

    Deepscaletool: A tool for the accurate esti- mation of technology scaling in the deep-submicron era,

    S. Sarangi and B. Baas, “Deepscaletool: A tool for the accurate esti- mation of technology scaling in the deep-submicron era,” in 2021 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 2021, pp. 1–5

    work page 2021

  2. [8]

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time,

    Z. Liu, A. Desai, F. Liao, W. Wang, V . Xie, Z. Xu, A. Kyrillidis, and A. Shrivastava, “Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time,” Advances in Neural Information Processing Systems , vol. 36, 2024

    work page 2024

  3. [1]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  4. [2]

    Keyformer: Kv cache reduction through key tokens selection for efficient generative inference,

    M. Adnan, A. Arunkumar, G. Jain, P. Nair, I. Soloveychik, and P. Ka- math, “Keyformer: Kv cache reduction through key tokens selection for efficient generative inference,” Proceedings of Machine Learning and Systems, vol. 6, pp. 114–127, 2024

    work page 2024

  5. [3]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” Advances in Neural Information Processing Systems , vol. 35, pp. 16 344–16 359, 2022

    work page 2022

  6. [4]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019. [Online]. Available: https://arxiv.org/abs/1810.04805

    arXiv 2019

  7. [5]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” 2021. [Online]. Available: https://arxiv.org/abs/2010.11929

    arXiv 2021

  8. [6]

    Aˆ 3: Accelerating attention mechanisms in neural networks with approximation,

    T. J. Ham, S. J. Jung, S. Kim, Y . H. Oh, Y . Park, Y . Song, J.-H. Park, S. Lee, K. Park, J. W. Lee et al. , “Aˆ 3: Accelerating attention mechanisms in neural networks with approximation,” in 2020 IEEE International Symposium on High Performance Computer Architecture . IEEE, 2020, pp. 328–341

    2020

Show all 21 references
  1. [7]

    Ramulator: A fast and extensible dram simulator,

    Y . Kim, W. Yang, and O. Mutlu, “Ramulator: A fast and extensible dram simulator,” IEEE Computer architecture letters, vol. 15, no. 1, pp. 45–49, 2015

    2015

  2. [9]

    Sanger: A co-design framework for enabling sparse attention using reconfigurable architecture,

    L. Lu, Y . Jin, H. Bi, Z. Luo, P. Li, T. Wang, and Y . Liang, “Sanger: A co-design framework for enabling sparse attention using reconfigurable architecture,” in MICRO-54: 54th Annual IEEE/ACM International Sym- posium on Microarchitecture, 2021, pp. 977–991

    2021

  3. [10]

    Online normalizer calculation for softmax,

    M. Milakov and N. Gimelshein, “Online normalizer calculation for softmax,” arXiv preprint arXiv:1805.02867 , 2018

    2018 arXiv

  4. [11]

    Cacti 6.0: A tool to model large caches,

    N. Muralimanohar, R. Balasubramonian, and N. P. Jouppi, “Cacti 6.0: A tool to model large caches,” HP laboratories, vol. 27, p. 28, 2009

    2009

  5. [12]

    Com- pressive transformers for long-range sequence modelling,

    J. W. Rae, A. Potapenko, S. M. Jayakumar, and T. P. Lillicrap, “Com- pressive transformers for long-range sequence modelling,”arXiv preprint arXiv:1911.05507, 2019

    1911 arXiv

  6. [14]

    Softermax: Hardware/software co-design of an efficient softmax for transformers,

    J. R. Stevens, R. Venkatesan, S. Dai, B. Khailany, and A. Raghunathan, “Softermax: Hardware/software co-design of an efficient softmax for transformers,” in 2021 58th ACM/IEEE Design Automation Conference (DAC), 2021, pp. 469–474

    2021

  7. [15]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Ho...

    2023 arXiv

  8. [16]

    Spatten: Efficient sparse attention architecture with cascade token and head pruning,

    H. Wang, Z. Zhang, and S. Han, “Spatten: Efficient sparse attention architecture with cascade token and head pruning,” in 2021 IEEE International Symposium on High-Performance Computer Architecture . IEEE, 2021, pp. 97–110

    2021

  9. [17]

    Cosa: Co-operative systolic arrays for multi-head attention mechanism in neural network using hybrid data reuse and fusion methodologies,

    Z. Wang, G. Wang, H. Jiang, N. Xu, and G. He, “Cosa: Co-operative systolic arrays for multi-head attention mechanism in neural network using hybrid data reuse and fusion methodologies,” in 2023 60th ACM/IEEE Design Automation Conference (DAC) . IEEE, 2023, pp. 1–6

    2023

  10. [18]

    Efficient streaming language models with attention sinks,

    G. Xiao, Y . Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  11. [19]

    Orca: A distributed serving system for {Transformer-Based} generative models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for {Transformer-Based} generative models,” in 16th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 22) , 2022, pp. 521–538

    2022

  12. [20]

    Nn-lut: Neural approximation of non-linear operations for efficient transformer inference,

    J. Yu, J. Park, S. Park, M. Kim, S. Lee, D. H. Lee, and J. Choi, “Nn-lut: Neural approximation of non-linear operations for efficient transformer inference,” 2021. [Online]. Available: https://arxiv.org/abs/2112.02191

    2021 arXiv

  13. [21]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models,

    Z. Zhang, Y . Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y . Tian, C. R´e, C. Barrett et al., “H2o: Heavy-hitter oracle for efficient generative inference of large language models,” Advances in Neural Information Processing Systems , vol. 36, pp. 34 661–34 710, 2023

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.