Pith. sign in

REVIEW 3 major objections 5 minor 5 cited by

DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DiffKV compresses the KV cache by 2.7x to 5.7x with near-lossless accuracy on complex reasoning workloads, and turns those savings into 1.9x to 5.4x higher serving throughput.

desk verdict A well-engineered KV cache compression system with a genuinely new memory manager and broad evaluation; the near-lossless numbers are credible but come with tuned thresholds and one internal inconsistency. read the letter →

arxiv 2412.03131 v3 pith:A44N37R3 submitted 2024-12-04 cs.LG cs.DC

Yanqi Zhang , Yuwei Hu , Runyuan Zhao , John C.S. Lui , Haibo Chen This is my paper · ORCID
classification cs.LGcs.DC
keywords KVcachecompressionLLMinferencedifferentiatedquantizationattentionsparsityGPUmemorymanagementparallelprefixsumchain-of-thoughtreasoningtokenpruning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the key-value cache that dominates LLM serving memory can be compressed far more aggressively than current methods allow by treating keys and values differently, grading tokens by importance, and letting each attention head spend memory according to its own sparsity pattern. On top of this differentiated compression policy, it contributes an on-GPU memory manager that packs the resulting irregular free pages into contiguous regions with parallel prefix sums, so the memory savings actually become throughput. Evaluated on open LLMs and on reasoning models that emit long chains of thought, the system reports $2.7\times$ to $5.7\times$ cache compression with accuracy within about 0.3 percent of the FP16 baseline, and $1.9\times$ to $5.4\times$ throughput gains over prior systems. The paper's central claim is that these three axes of differentiation, plus the memory manager that makes them scalable, yield a better accuracy-memory trade-off than uniform quantization or uniform pruning.

What carries the argument

The load-bearing mechanism is parallel KV compaction: after each attention head independently decides how many high- and low-precision pages it needs, a parallel prefix sum over the circular free page list assigns every head a disjoint contiguous region of page IDs, so allocation and recycling happen on the GPU in one coordinated step. The policy's significance score is the average attention score a token receives from subsequent tokens, max-aggregated across GQA heads, compared against the theoretical average $1/N$; tokens above $\alpha_h/N$ are kept at high precision, tokens between $\alpha_l/N$ and $\alpha_h/N$ at low precision, and tokens below $\alpha_l/N$ are pruned.

What would settle it

Run DiffKV on a long chain-of-thought task where early reasoning tokens receive high attention but are later abandoned, such as a multi-step proof with a misleading first path; if accuracy degrades sharply at the same compression ratios, the attention-score proxy is not preserving the tokens that matter. More directly, measure generation accuracy while varying sequence length at a fixed memory budget; the claim that long sequences can be compressed more aggressively predicts accuracy should stay flat, while a failure would show degradation growing with chain-of-thought length.

Watch

Extended reading notes

Core claim

DiffKV's central discovery is that the KV cache is not uniformly important, and the savings come from exploiting three asymmetries: keys matter more than values because keys shape the softmax scores for every token while each value only weights its own token; tokens differ sharply in attention score, spanning orders of magnitude; and each head and request has its own critical-token count, so per-head dynamic allocation beats static budgets. The framework converts these observations into a policy that stores important tokens at K8V4 precision (keys in 8 bits, values in 4 bits), moderately important tokens at K4V2, and prunes the rest, adjusting thresholds with sequence length. To make the resulting fragmented memory affordable, it uses a GPU-resident manager with unified pages, a circular free page list, and a bidirectional page table, compacting free pages in parallel; this keeps memory management under 0.9 percent of generation-step latency. The evidence for the claim is the reported near-lossless accuracy at 19.3 to 36.7 percent memory use on standard benchmarks and at about 27 percent memory use on thinking models.

Load-bearing premise

The load-bearing premise is that a token's average past attention score reliably predicts which cache entries future generations will need; if that proxy fails in long reasoning traces, where compression errors accumulate, the near-lossless claim collapses.

Editorial extensions

If this is right

  • Longer chain-of-thought generations become more affordable: at 16K-token generations on a 32B reasoning model, DiffKV sustains a batch of about 15.9 requests versus 2.7 for the baseline, a $5.4\times$ throughput gain.
  • Because keys carry more precision than values, mirror configurations that give values more bits collapse in accuracy (near zero on Qwen2.5-7B), so any future compression policy should preserve key precision first.
  • Per-head dynamic allocation makes memory use adapt to workload information density, using less memory on 5-shot MMLU than on 0-shot HumanEval+.
  • The memory manager keeps overhead below 1 percent of step latency, so compression benefits are not eaten by management costs even at large batch sizes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper does not take is making the precision thresholds adaptive at runtime; since token importance is measured online, thresholds could in principle be tightened for long or error-sensitive generations, which would test whether the offline-calibrated parameters are the limiting factor.
  • The same differentiated-precision idea could transfer to other memory-heavy structures, such as activations in deep fusion models or expert caches in mixture-of-experts serving, wherever a score-like signal ranks what must be kept exactly; the paper evaluates only Transformer KV caches.
  • The attention-score proxy is backward-looking, so workloads with abrupt topic shifts may need a query-aware or recency-adjusted importance signal; a controlled experiment on multi-document summarization with switched subjects would reveal whether DiffKV's near-lossless margin survives topic changes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. DiffKV is a KV-cache compression and memory-management framework for LLM serving. It assigns each cached token to one of three tiers—K8V4 high precision, K4V2 low precision, or pruned—based on an attention-derived significance score, uses higher precision for keys than for values, and adapts its decisions per request and per attention head. To make the resulting irregular memory layout efficient, the paper introduces parallel KV compaction on the GPU, built on unified pages, a circular free page list, and a bidirectional page table. The evaluation covers Llama3-8B/70B, Qwen2.5-7B/32B, and the thinking models QwQ-32B, R1-Distill-Qwen-14B, and R1-Distill-Llama-8B on a range of general, mathematical, code, and long-context benchmarks, reporting 2.7–5.7x KV cache compression with near-lossless accuracy and 1.9–5.4x throughput improvements relative to vLLM and several KV compression baselines.

Significance. The contribution is timely and, if the results are robust, significant. The paper is, to my knowledge, the first KV cache compression study evaluated on thinking models with long chain-of-thought generation, and it addresses a realistic serving bottleneck. The evaluation is unusually broad: seven models, multiple benchmark families, six baselines, and repeated runs with the reported averages. The memory-management contribution—parallel KV compaction on the GPU—is a genuine systems answer to the fragmentation introduced by the differentiated policy, and the source code is released. The main risk is the reliance on online attention-based importance scoring in long chain-of-thought settings, combined with per-model threshold calibration. The reported experiments mitigate this risk but do not fully quantify it. I also note that the stress-test arithmetic suggesting that about 90% of QwQ-32B tokens are stored at K4V2 is not supported by the paper's numbers: with K8V4/K4V2, a 27.4% memory fraction implies roughly 46% high-precision and 54% low-precision tokens, not 90% low-precision.

major comments (3)
  1. [Section 7.2, Parameter Calibration] The description of the Qwen2.5-7B configuration is internally inconsistent. The text states that the authors 'disable low-precision quantization for Qwen2.5-7B' when selecting alpha_h, but then reports alpha_l = 0.04 for Qwen2.5-7B and includes a Qwen2.5-7B panel for alpha_l in Figure 10. The memory numbers for Qwen2.5-7B in Table 1 (26.8% to 36.7%) also appear incompatible with a policy that has low precision disabled and only prunes tokens. Please state exactly which precision levels and thresholds are used for each model and correct the prose and figures accordingly; without this, the Qwen2.5-7B rows cannot be reproduced.
  2. [Section 4, Algorithm 1] The definition of the significance score and its use in the threshold formulas is ambiguous and appears inconsistent between phases. In the prompt phase, the score of token i is said to be the average of the N-i attention scores it receives from subsequent tokens, but the threshold is compared with 'the theoretical average 1/i' and is written ambiguously as 'alpha_h i' (presumably alpha_h / i). In the generation phase, Algorithm 1 uses thresholds alpha_h/N and alpha_l/N, and Score(t_c) is never explicitly defined for a token that has just left the recent window—in particular, which generated tokens' attention is included in the average. Please give the exact formula for Score(t) in both phases and reconcile the threshold formulas; the current presentation is not sufficient to reimplement the policy.
  3. [Section 7.2, Evaluating Differentiated Compression Policy] The paper states that in long chain-of-thought generation, 'errors introduced by compression are accumulated and propagated,' but it does not measure this accumulation or directly validate the central online importance proxy in that setting. Because a token is irreversibly demoted or pruned when it leaves the recent window, tokens whose relevance emerges only later in a long reasoning trace could be lost. Given that the near-lossless claim for QwQ-32B depends on the high-precision selection being correct, please add an experiment that (i) quantifies, using an FP16 reference run, the fraction of tokens whose final attention mass is high but whose online score at demotion time was below the low-precision threshold, and (ii) compares DiffKV's quality against an oracle policy that selects high-precision tokens using post-hoc final attention scores, or against a random-selection baseline at the same memory budget. This would directly test the load-bearing assumption behind the reported long-CoT results.
minor comments (5)
  1. [Section 1] The phrase 'finger-grained savings' should be 'finer-grained savings'.
  2. [Section 5.2] There is a missing space in 'free page listto efficiently', and the terminology for the 'circular free page list' should be used consistently throughout the section.
  3. [Table 2] The table heading uses 'Llama3.1-8B' while the rest of the paper and Table 1 use 'Llama3-8B'; please clarify which model was actually evaluated.
  4. [Algorithm 1] The recent-window size W is mentioned in the text but is not part of the algorithm's input list; please add it and specify how the most recent W tokens are handled relative to candidate token t_c in the generation phase.
  5. [Figure 10] The legends mix configurations 'K8V4-K4V2', 'K8V4-K4V2-Pr', and 'K8V4-Prune' without definitions; please define these labels in the caption or in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the compression policy is a transparently calibrated heuristic, and the reported accuracy and throughput are measured against external benchmarks rather than derived from DiffKV's own inputs.

full rationale

I walked the derivation chain from the attention-score significance measure (Section 4) through the threshold policy (Algorithm 1), the memory manager (Section 5), and the evaluation (Sections 7.2 and 7.3). The central claim—near-lossless accuracy at 2.7x–5.7x compression—rests on an empirical heuristic: tokens are stored at high or low precision, or pruned, according to average incoming attention. This is an assumption about token importance, and the online/causal limitations noted by the skeptic (a token can become critical only later, and compression errors accumulate over long chain-of-thought generations) are genuine robustness risks, but they are not circularity: the paper does not define accuracy in terms of significance scores, and the reported accuracies come from external benchmarks (GSM8K, MATH, MMLU, HumanEval+, LongBench, AIME24, GPQA) compared against an FP16 baseline. No equation in the paper makes the reported accuracy equal to the calibration objective. The thresholds alpha_h and alpha_l are fit on the MATH training split (Section 7.2), which makes MATH a within-distribution sanity check rather than an independent prediction, but the paper also evaluates on many other benchmarks; this is ordinary hyperparameter selection, not a fitted input renamed as a prediction. The self-citations ([28], [79]) support general serving and architecture statements and are not load-bearing. The Equation 2 decomposition of attention is an algebraic identity used only to motivate higher key precision; no result is imported from prior work by the same authors as a uniqueness theorem or ansatz. I therefore find no circular step that meets the evidentiary bar of the review rules.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The system components (unified pages, circular free page list, bidirectional page table) are implementation artifacts with shipped code, not speculative physical entities, so no entries are listed as invented entities. The central claim relies on four fitted or hand-chosen parameters and four domain assumptions, the most fragile of which is the attention-score significance proxy.

free parameters (4)
  • alpha_h (high-precision threshold) = 1 for Llama3-8B/70B, 3 for Qwen2.5-32B and QwQ-32B, inconsistent for Qwen2.5-7B
    Fitted on the MATH training split to maximize accuracy; controls which tokens get K8V4 precision. Section 7.2 Parameter Calibration.
  • alpha_l (low-precision threshold) = 0.02 (Llama3-8B), 0.04 (Qwen2.5-7B), 0 (others)
    Fitted on the MATH training split to set the boundary between low-precision K4V2 and pruning. Section 7.2 Parameter Calibration.
  • W (recent window size) = 64 tokens
    Hand-chosen constant: the most recent 64 tokens are always stored at high precision to avoid premature compression. Section 4.
  • Precision-level scheme K8V4-K4V2 = Two levels: K8V4 for high-precision tokens, K4V2 for low-precision tokens
    Selected by comparing K8V4, K4V8, K8V2, K4V2, K2V4, K4V1 in Section 7.2; the choice shapes both compression ratio and accuracy.
assumptions (4)
  • standard math Standard causal multi-head attention as in Eq. 1
    Used to define the KV cache and the attention-score significance signal.
  • domain assumption The L2-norm decomposition in Eq. 2 and the empirical CDF comparison justify the claim that keys matter more than values
    Section 3.1 uses the observation that attention scores span more orders of magnitude than value norms to motivate higher key precision; this is an empirical heuristic, not a proof.
  • domain assumption A token's importance is measured by the average attention score it receives from subsequent tokens, with max-aggregation for GQA
    Used in Section 4 and Algorithm 1 to decide high precision, low precision, or pruning. If this proxy is weak, the whole policy is undermined.
  • domain assumption Thresholds calibrated on the MATH training split transfer to other benchmarks and models
    Section 7.2 chooses alpha_h and alpha_l on MATH and then applies them to all reported benchmarks; the paper explicitly calls them fitted parameters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction." pith.science (2026). https://pith.science/paper/A44N37R3

@misc{pith2026241203131,
  author       = {Pith},
  title        = {Pith review of: DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A44N37R3}},
  note         = {Machine review of arXiv:2412.03131}
}
abstract

Large language models (LLMs) demonstrate remarkable capabilities but face substantial serving costs due to their high memory demands, with the key-value (KV) cache being a primary bottleneck. State-of-the-art KV cache compression techniques, such as quantization and pruning, apply uniform treatment to both keys and values, and discard unimportant tokens entirely, overlooking the fine-grained distinctions in the significance of individual KV cache components. To address such limitations, we introduce \textit{DiffKV}, a novel framework for efficient KV cache compression that exploits three levels of differentiation in the KV cache: (1) the differing impact of keys and values on attention computation, (2) the varying importance of tokens, and (3) the diverse dynamic sparsity patterns across attention heads. These levels of differentiation introduce irregular memory usage patterns across different requests and attention heads, posing significant scalability challenges for memory management. To address these challenges, DiffKV proposes an on-GPU memory manager that compacts fragmented free memory list into contiguous regions in parallel, effectively translating sparsity in the KV cache into performance gains. We evaluate DiffKV on several mainstream LLMs, including the emerging thinking models that generate extended chains of thought. DiffKV is able to compress the KV cache by $2.7\times$ to $5.7\times$ with near-lossless accuracy on complex workloads requiring sophisticated reasoning and long-generation capabilities, and enhances throughput by $1.9\times$ to $5.4\times$. Source codes of DiffKV are available at https://github.com/zyqCSL/DiffKV.

Figures

Figures reproduced from arXiv: 2412.03131 by the authors.

Figure 1
Figure 1. KV cache memory allocation patterns for (a) prun [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Distribution of attention score and value vector [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. shows the per-token attention scores in the 8 th layer, selected as a representative layer of Llama3-8B [20], on a sequence randomly sampled from the Wikitext dataset [49]. 10 −6 10 −4 10 −2 10 0 Numerical Value 0.0 0.2 0.4 0.6 0.8 1.0 CDF(%) Attention score vs. Value Norm in Llama3-8B score-layer-0 v-norm-layer-0 score-layer-15 v-norm-layer-15 score-layer-31 v-norm-layer-31 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Number of critical tokens per layer in Llama3-8B [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Number of critical tokens per KV head in Llama3- [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Memory management flow in the prompt phase. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: DiffKV architecture. 6.1 DiffKV Architecture The architecture of DiffKV is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Accuracy of differentiated KV quantization. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Accuracy of dynamic vs. static sparsity. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Calibrating the high- and low-precision thresholds [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: KV cache memory normalized to vLLM vs. bench [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Fraction of tokens pruned, quantized to low preci [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: Latency comparison between parallel KV com [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 14
Figure 14. Figure 14: further illustrates the latency breakdown of Diff￾KV during one inference step. The memory management overhead is remarkably low thanks to DiffKV’s on-GPU par￾allel KV compaction, contributing less than 0.2% of the total latency in the prompt phase and under 0.9% in t…
Figure 16
Figure 16. Figure 16: Comparison of average latency between DiffKV [PITH_FULL_IMAGE:figures/full_fig_p014_16.png]
Figure 17
Figure 17. Figure 17: Throughput and achieved batch size. sparsity. At the core of DiffKV is the parallel KV compaction technique that efficiently handles irregular memory require￾ments across requests and attention heads, effectively trans￾lating memory savings into performance gains. Our…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

    cs.DC 2026-07 conditional novelty 6.5 of 10

    Balancing only first-turn agent requests while keeping follow-ups cache-aware raises LLM cluster TPS 10-16% (colocation) and prefill TPS 2-34% (disaggregation) over SOTA schedulers on production agent traces.

  2. PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    PagedWeight pages quantized MoE expert weights on and off the GPU at runtime, releasing memory to the KV cache while using sensitivity, routing, and prompt signals to choose which experts to shrink.

  3. CaliDrop: KV Cache Compression with Calibration

    cs.CL 2025-07 conditional novelty 6.0 of 10

    CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.

  4. DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration

    cs.CL 2025-06 conditional novelty 4.0 of 10

    DAM derives per-layer and per-head attention masks from a calibration dataset and extrapolates them to long inputs, matching full-attention retrieval accuracy while reducing memory and compute.

  5. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

Reference graph

Works this paper leans on

79 extracted references · 40 canonical work pages · cited by 5 Pith papers

  1. [1]

    https://github.com/NVIDIA/ FasterTransformer

    FasterTransformer. https://github.com/NVIDIA/ FasterTransformer

    work page

  2. [2]

    https: //github.com/flashinfer-ai/flashinfer

    FlashInfer: Kernel Library for LLM Serving. https: //github.com/flashinfer-ai/flashinfer

    work page

  3. [3]

    https://huggingface.co/datasets/ AI-MO/aimo-validation-aime , 2024

    Aime 2024. https://huggingface.co/datasets/ AI-MO/aimo-validation-aime , 2024

    work page 2024

  4. [4]

    L., A LMEIDA , D., A L- TENSCHMIDT , J., A LTMAN , S., A NADKAT, S., ET AL

    ACHIAM , J., A DLER , S., A GARWAL , S., A HMAD , L., AKKAYA, I., A LEMAN , F. L., A LMEIDA , D., A L- TENSCHMIDT , J., A LTMAN , S., A NADKAT, S., ET AL . Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  5. [5]

    J., SOLOVEYCHIK , I., AND KAMATH , P

    ADNAN , M., A RUNKUMAR , A., J AIN , G., N AIR , P. J., SOLOVEYCHIK , I., AND KAMATH , P. Keyformer: Kv cache reduction through key tokens selection for effi- cient generative inference. Proceedings of Machine Learning and Systems 6 (2024), 114–127

    work page 2024

  6. [6]

    Gqa: Training generalized multi-query transformer mod- els from multi-head checkpoints

    AINSLIE , J., L EE-T HORP, J., DE JONG , M., Z EMLYAN- SKIY, Y., L EBRÓN , F., AND SANGHAI , S. Gqa: Training generalized multi-query transformer mod- els from multi-head checkpoints. arXiv preprint arXiv:2305.13245 (2023)

    arXiv 2023

  7. [7]

    Y., R AJBHANDARI , S., A WAN, A

    AMINABADI , R. Y., R AJBHANDARI , S., A WAN, A. A., LI, C., L I, D., Z HENG , E., R UWASE , O., S MITH , S., ZHANG , M., R ASLEY , J., ET AL . Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis (2022), IEEE, pp. 1–15

    work page 2022

  8. [8]

    Program synthe- sis with large language models

    AUSTIN , J., O DENA , A., N YE, M., B OSMA , M., MICHALEWSKI , H., D OHAN , D., J IANG , E., C AI, C., T ERRY, M., L E, Q., ET AL . Program synthe- sis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

Show all 79 references
  1. [9]

    Longbench: A bilingual, multitask bench- mark for long context understanding

    BAI, Y., L V, X., Z HANG , J., L YU, H., T ANG , J., HUANG , Z., D U, Z., L IU, X., Z ENG , A., H OU, L., ET AL . Longbench: A bilingual, multitask bench- mark for long context understanding. arXiv preprint arXiv:2308.14508 (2023)

    2023 arXiv

  2. [10]

    BROWN , T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)

    2020 arXiv

  3. [11]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling

    CAI, Z., ZHANG , Y., GAO, B., LIU, T., LU, K., XIONG , W., D ONG , Y., C HANG , B., H U, J., AND XIAO, W. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069 (2024)

    2024 arXiv

  4. [12]

    CHEN , M., T WOREK , J., J UN, H., Y UAN, Q., P INTO , H. P. D. O., K APLAN , J., E DWARDS , H., B URDA , Y., JOSEPH , N., B ROCKMAN , G., ET AL . Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  5. [13]

    E., S TOICA , I., AND XING , E

    CHIANG , W.-L., L I, Z., L IN, Z., S HENG , Y., W U, Z., ZHANG , H., Z HENG , L., Z HUANG , S., Z HUANG , Y., GONZALEZ , J. E., S TOICA , I., AND XING , E. P. Vi- cuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023

    2023

  6. [14]

    N., L I, T., L I, D., Z HANG , H., ZHU, B., J ORDAN , M., G ONZALEZ , J

    CHIANG , W.-L., Z HENG , L., S HENG , Y., A N- GELOPOULOS , A. N., L I, T., L I, D., Z HANG , H., ZHU, B., J ORDAN , M., G ONZALEZ , J. E., ET AL . Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132 (2024)

    2024 arXiv

  7. [15]

    Training ver- ifiers to solve math word problems

    COBBE , K., K OSARAJU , V., B AVARIAN , M., C HEN , M., J UN, H., K AISER , L., P LAPPERT , M., T WOREK , J., H ILTON , J., N AKANO , R., ET AL . Training ver- ifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)

    2021 arXiv

  8. [16]

    Flashattention-2: Faster attention with bet- ter parallelism and work partitioning

    DAO, T. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691 (2023)

    2023 arXiv

  9. [17]

    Flashattention: Fast and memory-efficient exact atten- tion with io-awareness

    DAO, T., F U, D., E RMON , S., R UDRA , A., AND RÉ, C. Flashattention: Fast and memory-efficient exact atten- tion with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359

    2022

  10. [18]

    DETTMERS , T., L EWIS , M., B ELKADA , Y., AND ZETTLEMOYER , L. Gpt3. int8 (): 8-bit matrix multipli- cation for transformers at scale. Advances in Neural In- formation Processing Systems 35 (2022), 30318–30332

    2022

  11. [19]

    Qaq: Quality adaptive quantization for llm kv cache

    DONG , S., C HENG , W., Q IN, J., AND WANG , W. Qaq: Quality adaptive quantization for llm kv cache. arXiv preprint arXiv:2403.04643 (2024)

    2024 arXiv

  12. [20]

    The llama 3 herd of models

    DUBEY, A., J AUHRI , A., P ANDEY , A., K ADIAN , A., AL-DAHLE , A., L ETMAN , A., M ATHUR , A., S CHEL - TEN , A., Y ANG , A., F AN, A., ET AL . The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  13. [21]

    Turbotrans- formers: an efficient gpu serving system for transformer models

    FANG , J., YU, Y., ZHAO, C., AND ZHOU , J. Turbotrans- formers: an efficient gpu serving system for transformer models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Pro- gramming (2021), pp. 389–402

    2021

  14. [22]

    Low latency rnn inference with cellular batching

    GAO, P., Y U, L., W U, Y., AND LI, J. Low latency rnn inference with cellular batching. In Proceedings of the Thirteenth EuroSys Conference (2018), pp. 1–15

    2018

  15. [23]

    Model tells you what to discard: Adap- tive kv cache compression for llms

    GE, S., Z HANG , Y., L IU, L., Z HANG , M., H AN, J., AND GAO, J. Model tells you what to discard: Adap- tive kv cache compression for llms. arXiv preprint arXiv:2310.01801 (2023)

    2023 arXiv

  16. [24]

    Github copilot

    GITHUB . Github copilot. https://github.com/ features/copilot, 2023

    2023

  17. [25]

    Olive: Ac- celerating large language models via hardware-friendly outlier-victim pair quantization

    GUO, C., T ANG , J., H U, W., L ENG , J., Z HANG , C., YANG , F., L IU, Y., G UO, M., AND ZHU, Y. Olive: Ac- celerating large language models via hardware-friendly outlier-victim pair quantization. In Proceedings of the 50th Annual International Symposium on Computer Ar- chite...

    2023

  18. [26]

    Deepseek-r1: Incentivizing reasoning capabil- ity in llms via reinforcement learning

    GUO, D., Y ANG , D., Z HANG , H., S ONG , J., Z HANG , R., X U, R., Z HU, Q., M A, S., W ANG , P., B I, X., ET AL . Deepseek-r1: Incentivizing reasoning capabil- ity in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  19. [27]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence

    GUO, D., Z HU, Q., Y ANG , D., X IE, Z., D ONG , K., ZHANG , W., C HEN , G., B I, X., W U, Y., LI, Y., ET AL . Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196 (2024)

    2024 arXiv

  20. [28]

    Microsecond-scale preemption for concurrent {GPU- accelerated}{DNN} inferences

    HAN, M., Z HANG , H., C HEN , R., AND CHEN , H. Microsecond-scale preemption for concurrent {GPU- accelerated}{DNN} inferences. In 16th USENIX Sym- posium on Operating Systems Design and Implementa- tion (OSDI 22) (2022), pp. 539–558

    2022

  21. [29]

    HARRIS , M., S ENGUPTA , S., AND OWENS , J. D. Paral- lel prefix sum (scan) with cuda. GPU gems 3, 39 (2007), 851–876

    2007

  22. [30]

    Zipcache: Accurate and efficient kv cache quantization with salient token identification

    HE, Y., ZHANG , L., W U, W., L IU, J., Z HOU , H., AND ZHUANG , B. Zipcache: Accurate and efficient kv cache quantization with salient token identification. Advances in Neural Information Processing Systems 37 (2024), 68287–68307

    2024

  23. [31]

    Mea- suring massive multitask language understanding

    HENDRYCKS , D., B URNS , C., B ASART , S., Z OU, A., MAZEIKA , M., S ONG , D., AND STEINHARDT , J. Mea- suring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020)

    2020 arXiv

  24. [32]

    W., SHAO, Y

    HOOPER , C., K IM, S., M OHAMMADZADEH , H., M A- HONEY , M. W., SHAO, Y. S., KEUTZER , K., AND GHO- LAMI , A. Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079 (2024)

    2024 arXiv

  25. [33]

    V., W U, Y., ET AL

    HUANG , Y., CHENG , Y., BAPNA , A., FIRAT, O., CHEN , D., C HEN , M., L EE, H., N GIAM , J., L E, Q. V., W U, Y., ET AL . Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems 32 (2019)

    2019

  26. [34]

    HUI, B., Y ANG , J., C UI, Z., Y ANG , J., L IU, D., ZHANG , L., L IU, T., Z HANG , J., Y U, B., L U, K., ET AL . Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 (2024)

    2024 arXiv

  27. [35]

    Openai o1 system card

    JAECH , A., K ALAI , A., L ERER , A., R ICHARDSON , A., EL-K ISHKY , A., L OW, A., H ELYAR, A., M ADRY, A., BEUTEL , A., C ARNEY , A., ET AL . Openai o1 system card. arXiv preprint arXiv:2412.16720 (2024)

    2024 arXiv

  28. [36]

    Q., S ABLAYROLLES , A., R OUX , A., M EN- SCH , A., S AVARY, B., B AMFORD , C., C HAPLOT , D

    JIANG , A. Q., S ABLAYROLLES , A., R OUX , A., M EN- SCH , A., S AVARY, B., B AMFORD , C., C HAPLOT , D. S., CASAS , D. D. L., H ANNA , E. B., B RESSAND , F., ET AL . Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

    2024 arXiv

  29. [37]

    P., Y OUNG , C., P ATIL, N., P ATTERSON , D., A GRAWAL , G., B AJWA, R., B ATES, S., B HATIA , S., B ODEN , N., B ORCHERS , A., ET AL

    JOUPPI , N. P., Y OUNG , C., P ATIL, N., P ATTERSON , D., A GRAWAL , G., B AJWA, R., B ATES, S., B HATIA , S., B ODEN , N., B ORCHERS , A., ET AL . In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on comp...

    2017

  30. [38]

    Flat: An opti- mized dataflow for mitigating attention bottlenecks

    KAO, S.-C., S UBRAMANIAN , S., A GRAWAL, G., YAZ- DANBAKHSH , A., AND KRISHNA , T. Flat: An opti- mized dataflow for mitigating attention bottlenecks. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Syste...

    2023

  31. [39]

    H., G ONZALEZ , J., Z HANG , H., AND STO- ICA , I

    KWON , W., L I, Z., Z HUANG , S., S HENG , Y., Z HENG , L., Y U, C. H., G ONZALEZ , J., Z HANG , H., AND STO- ICA , I. Efficient memory management for large lan- guage model serving with pagedattention. In Proceed- ings of the 29th Symposium on Operating Systems Prin- ciples (...

    2023

  32. [40]

    {InfiniGen}: Efficient generative inference of large language mod- els with dynamic {KV} cache management

    LEE, W., L EE, J., S EO, J., AND SIM, J. {InfiniGen}: Efficient generative inference of large language mod- els with dynamic {KV} cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 155–172

    2024

  33. [41]

    Solving quantitative reasoning problems with language models

    LEWKOWYCZ , A., A NDREASSEN , A., D OHAN , D., DYER , E., M ICHALEWSKI , H., R AMASESH , V., SLONE , A., A NIL , C., S CHLAG , I., G UTMAN -S OLO , T., ET AL . Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems 35 (2...

    2022

  34. [42]

    Snapkv: Llm knows what you are looking for before generation

    LI, Y., H UANG , Y., YANG , B., V ENKITESH , B., L O- CATELLI , A., Y E, H., C AI, T., L EWIS , P., AND CHEN , D. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469 (2024)

    2024 arXiv

  35. [43]

    E., ET AL

    LI, Z., Z HENG , L., Z HONG , Y., L IU, V., S HENG , Y., JIN, X., H UANG , Y., C HEN , Z., Z HANG , H., G ONZA - LEZ , J. E., ET AL . {AlpaServe}: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design an...

    2023

  36. [44]

    Qserve: W4a8kv4 quantization and system co-design for efficient llm serving

    LIN, Y., TANG , H., Y ANG , S., Z HANG , Z., X IAO, G., GAN, C., AND HAN, S. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532 (2024)

    2024 arXiv

  37. [45]

    Deepseek-v2: A strong, economical, and effi- cient mixture-of-experts language model

    LIU, A., F ENG , B., W ANG , B., W ANG , B., L IU, B., ZHAO, C., D ENGR , C., R UAN, C., D AI, D., G UO, D., ET AL . Deepseek-v2: A strong, economical, and effi- cient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 (2024)

    2024 arXiv

  38. [46]

    S., W ANG , Y., AND ZHANG , L

    LIU, J., X IA, C. S., W ANG , Y., AND ZHANG , L. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems (2023)

    2023

  39. [47]

    Scis- sorhands: Exploiting the persistence of importance hy- pothesis for llm kv cache compression at test time

    LIU, Z., D ESAI , A., L IAO, F., W ANG , W., X IE, V., XU, Z., K YRILLIDIS , A., AND SHRIVASTAVA, A. Scis- sorhands: Exploiting the persistence of importance hy- pothesis for llm kv cache compression at test time. Ad- vances in Neural Information Processing Systems 36 (2024)

    2024

  40. [48]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache

    LIU, Z., Y UAN, J., J IN, H., Z HONG , S., X U, Z., BRAVERMAN , V., C HEN , B., AND HU, X. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750 (2024)

    2024 arXiv

  41. [49]

    Pointer sentinel mixture models

    MERITY , S., X IONG , C., B RADBURY , J., AND SOCHER , R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 (2016)

    2016 arXiv

  42. [50]

    A., BONDARENKO , Y., V AN BAALEN , M., AND BLANKEVOORT , T

    NAGEL , M., F OURNARAKIS , M., A MJAD , R. A., BONDARENKO , Y., V AN BAALEN , M., AND BLANKEVOORT , T. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295 (2021)

    2021 arXiv

  43. [51]

    An optimal parallel prefix-sums algorithm on the memory machine models for gpus

    NAKANO , K. An optimal parallel prefix-sums algorithm on the memory machine models for gpus. In Interna- tional Conference on Algorithms and Architectures for Parallel Processing(2012), Springer, pp. 99–113

    2012

  44. [52]

    R., G ANGER , G

    NARAYANAN , D., H ARLAP , A., P HANISHAYEE , A., SESHADRI , V., D EVANUR , N. R., G ANGER , G. R., GIBBONS , P. B., AND ZAHARIA , M. Pipedream: Gen- eralized pipeline parallelism for dnn training. In Pro- ceedings of the 27th ACM symposium on operating sys- tems principles (20...

    2019

  45. [53]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    NARAYANAN , D., S HOEYBI , M., C ASPER , J., LEGRESLEY , P., P ATWARY, M., K ORTHIKANTI , V., VAINBRAND , D., K ASHINKUNTI , P., B ERNAUER , J., CATANZARO , B., ET AL . Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the inter...

    2021

  46. [54]

    Nvidia L40 datasheet

    NVIDIA . Nvidia L40 datasheet. https://www. nvidia.com/content/dam/en-zz/Solutions/ design-visualization/support-guide/ NVIDIA-L40-Datasheet-January-2023.pdf , 2023

    2023

  47. [55]

    Training lan- guage models to follow instructions with human feed- back

    OUYANG , L., W U, J., J IANG , X., A LMEIDA , D., WAINWRIGHT , C., M ISHKIN , P., Z HANG , C., A GAR - WAL, S., S LAMA , K., R AY, A., ET AL . Training lan- guage models to follow instructions with human feed- back. Advances in neural information processing sys- tems 35 (2022)...

    2022

  48. [56]

    Efficiently scaling transformer inference

    POPE , R., D OUGLAS , S., C HOWDHERY , A., D EVLIN , J., B RADBURY , J., H EEK , J., X IAO, K., A GRAWAL, S., AND DEAN , J. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems 5(2023), 606–624

    2023

  49. [57]

    L., S TICKLAND , A

    REIN , D., H OU, B. L., S TICKLAND , A. C., P ETTY, J., PANG , R. Y., D IRANI , J., M ICHAEL , J., AND BOW- MAN , S. R. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling (2024)

    2024

  50. [58]

    Capabilities of gemini models in medicine

    SAAB , K., T U, T., W ENG , W.-H., T ANNO , R., S TUTZ , D., W ULCZYN , E., Z HANG , F., S TROTHER , T., PARK , C., V EDADI , E., ET AL . Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416 (2024)

    2024 arXiv

  51. [59]

    Deepseek- math: Pushing the limits of mathematical reason- ing in open language models

    SHAO, Z., W ANG , P., Z HU, Q., X U, R., S ONG , J., ZHANG , M., L I, Y., W U, Y., AND GUO, D. Deepseek- math: Pushing the limits of mathematical reason- ing in open language models. arXiv preprint arXiv:2402.03300 (2024)

    2024 arXiv

  52. [60]

    Fast transformer decoding: One write- head is all you need

    SHAZEER , N. Fast transformer decoding: One write- head is all you need. arXiv preprint arXiv:1911.02150 (2019)

    2019 arXiv

  53. [61]

    Flexgen: High-throughput generative in- ference of large language models with a single gpu

    SHENG , Y., Z HENG , L., Y UAN, B., L I, Z., R YABININ , M., C HEN , B., L IANG , P., R É, C., S TOICA , I., AND ZHANG , C. Flexgen: High-throughput generative in- ference of large language models with a single gpu. In International Conference on Machine Learning (2023), PMLR,...

    2023

  54. [62]

    Welder: Scheduling deep learning memory access via tile-graph

    SHI, Y., YANG , Z., X UE, J., M A, L., X IA, Y., M IAO, Z., G UO, Y., Y ANG , F., AND ZHOU , L. Welder: Scheduling deep learning memory access via tile-graph. In 17th USENIX Symposium on Operating Systems De- sign and Implementation (OSDI 23) (2023), pp. 701– 718

    2023

  55. [63]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    SHOEYBI , M., P ATWARY, M., P URI , R., L EGRESLEY , P., CASPER , J., AND CATANZARO , B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)

    2019 arXiv

  56. [64]

    A., K HANI , A., AND KHASAHMADI , A

    TAGHANAKI , S. A., K HANI , A., AND KHASAHMADI , A. Mmlu-pro+: Evaluating higher-order reason- ing and shortcut learning in llms. arXiv preprint arXiv:2409.02257 (2024)

    2024 arXiv

  57. [65]

    Quest: Query-aware sparsity for efficient long-context llm inference

    TANG , J., Z HAO, Y., Z HU, K., X IAO, G., K ASIKCI , B., AND HAN, S. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774 (2024)

    2024 arXiv

  58. [66]

    M., H AUTH , A., ET AL

    TEAM , G., A NIL , R., B ORGEAUD , S., W U, Y., ALAYRAC , J.-B., Y U, J., S ORICUT , R., S CHALKWYK , J., D AI, A. M., H AUTH , A., ET AL . Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  59. [67]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    THE QWEN TEAM. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

  60. [68]

    Llama 2: Open foundation and fine-tuned chat models

    TOUVRON , H., M ARTIN , L., S TONE , K., A LBERT , P., ALMAHAIRI , A., B ABAEI , Y., BASHLYKOV , N., B A- TRA , S., B HARGAVA , P., B HOSALE , S., ET AL . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  61. [69]

    Attention is all you need

    VASWANI , A. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  62. [70]

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads

    XIAO, G., T ANG , J., Z UO, J., G UO, J., Y ANG , S., TANG , H., FU, Y., AND HAN, S. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.10819 (2024)

    2024 arXiv

  63. [71]

    Efficient streaming language models with attention sinks

    XIAO, G., T IAN , Y., CHEN , B., H AN, S., AND LEWIS , M. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 (2023)

    2023 arXiv

  64. [72]

    A preliminary study of o1 in medicine: Are we closer to an ai doctor? arXiv preprint arXiv:2409.15277 (2024)

    XIE, Y., W U, J., T U, H., Y ANG , S., Z HAO, B., Z ONG , Y., J IN, Q., X IE, C., AND ZHOU , Y. A preliminary study of o1 in medicine: Are we closer to an ai doctor? arXiv preprint arXiv:2409.15277 (2024)

    2024 arXiv

  65. [73]

    Infless: a native serverless system for low-latency, high-throughput inference

    YANG , Y., ZHAO, L., L I, Y., ZHANG , H., L I, J., Z HAO, M., C HEN , X., AND LI, K. Infless: a native serverless system for low-latency, high-throughput inference. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and O...

    2022

  66. [74]

    S., K IM, G.-W., K IM, S., AND CHUN , B.-G

    YU, G.-I., J EONG , J. S., K IM, G.-W., K IM, S., AND CHUN , B.-G. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) (2022), pp. 521–538

    2022

  67. [75]

    {SHEPHERD}: Serving {DNNs} in the wild

    ZHANG , H., T ANG , Y., K HANDELWAL , A., AND STO- ICA , I. {SHEPHERD}: Serving {DNNs} in the wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) (2023), pp. 787–808

    2023

  68. [76]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    ZHANG , Z., S HENG , Y., ZHOU , T., C HEN , T., Z HENG , L., C AI, R., SONG , Z., TIAN , Y., RÉ, C., BARRETT , C., ET AL . H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems 36 (2024)

    2024

  69. [77]

    Atom: Low-bit quantization for efficient and accurate llm serving

    ZHAO, Y., L IN, C.-Y., Z HU, K., Y E, Z., C HEN , L., ZHENG , S., C EZE , L., K RISHNAMURTHY , A., C HEN , T., AND KASIKCI , B. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Ma- chine Learning and Systems 6 (2024), 196–209

    2024

  70. [78]

    Evaluation of openai o1: Opportunities and challenges of agi

    ZHONG , T., L IU, Z., P AN, Y., Z HANG , Y., Z HOU , Y., LIANG , S., W U, Z., L YU, Y., S HU, P., Y U, X., ET AL . Evaluation of openai o1: Opportunities and challenges of agi. arXiv preprint arXiv:2409.18486 (2024)

    2024

  71. [79]

    Serv- ing large language models on huawei cloudmatrix384

    ZUO, P., LIN, H., DENG , J., ZOU, N., YANG , X., DIAO, Y., GAO, W., X U, K., C HEN , Z., L U, S., ET AL . Serv- ing large language models on huawei cloudmatrix384. arXiv preprint arXiv:2506.12708 (2025)

    2025 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.