Pith. sign in

REVIEW 2 major objections 5 minor 28 references

Migrating large MoE and multimodal inference off CUDA onto Ascend costs twelve plugin patches, disabled high-throughput features, and external watchdogs before the service stays correct.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Serving frontier MoE and multimodal models on Ascend 910 via vLLM-Ascend is feasible but dominated by engineering cost from incomplete operators, fragile parallelism, kernel faults, and weak observability.

T0 review reviewed 2026-07-10 challenge →

load-bearing objection Solid, usable field report: concrete Ascend/vLLM-Ascend failure modes (patches, fault codes, concurrency curves) that teams actually need, scoped honestly. the 2 major comments →

arxiv 2607.08215 v1 pith:UPKI3B3Q submitted 2026-07-09 cs.DC

On the Limitations of Non-GPU AI Accelerators for Large-Model Inference: A Field Study of MoE and Multimodal Serving on Huawei Ascend

Zheng Yu This is my paper
classification cs.DC
keywords AI acceleratorsNPUlarge language modelsinference servingmixture-of-expertsvLLMHuawei Ascendmachine learning systems
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This field study shows that non-GPU AI accelerators can host frontier-scale MoE and multimodal inference, but the migration cost is dominated by engineering, not hardware. On a 16-device Ascend 910 system running CANN and vLLM-Ascend, two demanding workloads—a W8A8 MoE LLM-as-a-judge pipeline and a bf16 multimodal medical vision–language model—were brought to correct service only after twelve source-level patches to the vendor plugin, deliberate disabling of several throughput features, and external operational scaffolding for recurring device faults. The failures fall into eight platform-level classes (operator coverage gaps, fragile multi-axis parallelism, kernel numerical faults, immature graph compilation, unstable advanced features, low concurrency ceilings, weak observability, and ecosystem fragmentation). The paper supplies symptoms, evidence, and likely root causes for each class, quantifies concurrency and end-to-end quality, and distills vendor-agnostic tactics so other teams can budget the true cost of adoption.

Core claim

Deploying two concrete large-model inference workloads on a 16-device Huawei Ascend 910 stack required twelve source patches to the vendor inference plugin, the deliberate disabling of high-throughput features to preserve numerical correctness, and external watchdogs for recurring low-level device faults; the observed failures organize into eight platform-level limitation classes that are properties of the accelerator, its compiler/operator library, and the vendor plugin rather than quirks of the specific models.

What carries the argument

The eight-class catalog of platform limitations, each presented as symptom → evidence → root cause, together with the quantified integration effort (twelve patches), concurrency sweet-spot measurements, and end-to-end MMMU/MMMU-Pro scores that confirm correct service.

Load-bearing premise

The claim that the eight limitation classes and their root causes will generalize beyond one vendor, one engine version line, one 16-device node, and two deliberately hard MoE/multimodal architectures.

What would settle it

A later release of the same stack, or a comparable non-GPU accelerator, successfully serving the same class of W8A8 MoE plus multimodal MoE-decoder workloads at production concurrency without source patches, without disabling throughput features, and without external watchdogs for opaque aicore/vector-core faults.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Teams evaluating non-GPU accelerators for large-model inference must budget engineering effort (patching, correctness-first configuration, fault scaffolding) as the dominant cost, not hardware acquisition.
  • Correctness-first defaults—disable sequence parallelism, fused MoE communication, and speculative decoding until validated—become standard operating procedure on immature stacks.
  • Capacity planning must use measured concurrency sweet spots and multi-minute cold starts rather than theoretical peak FLOPs or memory.
  • Vendor plugins need broader testing of feature cross-products (parallelism × modality × MoE × sparse attention) and graceful fallbacks instead of hard crashes.
  • External health-check watchdogs and log scrapers for known device fault codes become required production scaffolding rather than optional extras.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same coverage-gap pattern will recur on other non-CUDA accelerators until their operator libraries and graph compilers reach CUDA-level maturity on MoE and multimodal paths.
  • Silent numerical corruption that only appears under realistic prefill or multi-axis parallelism implies that differential testing against a trusted CUDA or native-framework oracle should be mandatory in CI for every plugin bump.
  • Ecosystem fragmentation (version-locked env-var matrices and startup patches) may keep total cost of ownership high even after individual kernel bugs are fixed, unless a thin portable abstraction layer becomes common practice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This field study reports the engineering cost of serving two demanding large-model inference workloads on a 16-device Huawei Ascend 910 node with CANN and vLLM-Ascend: a W8A8 MoE LLM-as-a-judge pipeline (DeepSeek-V4-Flash) and a bf16 multimodal VLM (DeepSeek-V4-Flash-Vision) evaluated on MMMU/MMMU-Pro. The authors document twelve source-level patches to the vendor plugin, deliberate disabling of throughput features for correctness, and external watchdogs for recurring aicore/vector-core faults. They organize observations into eight platform-level limitation classes (operator/feature coverage, multi-axis parallelism, kernel numerical faults, graph compilation, advanced features, scalability, observability, ecosystem fragmentation), each with symptom–evidence–root-cause structure, and supply supporting artifacts (patch inventory, fault logs, concurrency curves, memory footprints, end-to-end benchmark matches). They conclude that the hardware is capable but total cost of ownership is dominated by engineering effort, and they distill vendor-agnostic adoption strategies.

Significance. The paper fills a genuine documentation gap: public, artifact-backed accounts of what it actually costs to move frontier-scale MoE and multimodal inference off CUDA are scarce. Strengths include concrete, transferable evidence (Figure 4 patch counts by file; Listings 2–3 sanitized device fault codes; Figures 5–6 non-monotonic concurrency behavior; Figure 2 MMMU/MMMU-Pro scores matching a native-framework reference) and an explicit, usable set of general strategies (differential testing, CI on the target stack, feature-flag matrices, fault-tolerant serving). The work is observational rather than theoretical, but for systems practitioners evaluating non-GPU accelerators the catalog and mitigations are immediately actionable. Threats to validity (§6.1) are stated clearly and do not undermine the reported field evidence itself.

major comments (2)
  1. §5.6 and Figures 5–6: the concurrency “sweet spot” of ~4 in-flight streams and the subsequent throughput collapse are central quantitative claims about scalability ceilings, yet the manuscript gives no request mix, generation-length distribution, or measurement protocol (warmup, duration, number of trials). Without these, the curves cannot be reproduced or compared across engines/versions; a short methods paragraph or appendix table specifying the load generator and statistics would make the result load-bearing rather than anecdotal.
  2. §5.1 / Figure 4: the claim of “twelve source-level patches” is a headline integration-effort metric, but the paper only groups patches by file and lists high-level defect classes. For a reproducible reference, either a supplementary inventory (file, approximate lines, defect type, whether upstreamed) or a public patch set is needed; otherwise the count remains an opaque operational claim that later readers cannot verify or reuse.
minor comments (5)
  1. Table 1 and §4: model names (DeepSeek-V4-Flash, DeepSeek-V4-Flash-Vision) and the “~300 GB / ~540 GB” sizes should be cross-checked against publicly documented checkpoints or explicitly marked as internal/anonymized variants so readers can judge representativeness.
  2. §5.3 Listings 2–3: the sanitized fault excerpts are valuable; adding the approximate CANN/driver version string next to the runtime result codes (507015/507035) would improve long-term archival value.
  3. Figure 2 caption: the golden/prefill gap is correctly attributed to model behavior, but a one-sentence note that the 4-token answer budget is a deliberate evaluation protocol (not an Ascend limit) would prevent misreading.
  4. §7.2 item 9 and the “Asks for the ecosystem” paragraph: these are useful; a short pointer to whether any of the twelve patches have already been submitted upstream would strengthen the collaboration claim.
  5. Minor typography: occasional missing spaces after em-dashes and a few run-on sentences in §5.2 (“…so each additional parallel axis multiplies…”) could be tightened for readability.

Circularity Check

0 steps flagged

No significant circularity: observational field study with no fitted-input-as-prediction or self-definitional loops.

full rationale

This paper is a qualitative field study of engineering effort and platform limitations when serving two large MoE/multimodal workloads on Ascend 910 via CANN and vLLM-Ascend. Its central claims are observational inventories (twelve source patches, disabled features, eight limitation classes, concurrency sweet-spot behavior, and end-to-end MMMU/MMMU-Pro scores used only to confirm numerical correctness of the port). There is no derivation chain that claims a first-principles prediction from fitted parameters, no uniqueness theorem imported from the authors, no ansatz smuggled via self-citation, and no renaming of a known result as a new unification. Benchmark numbers (Figure 2) are explicitly scoped as correctness checks against a native-framework reference, not as independent model-quality claims. Self-citation is minimal and non-load-bearing. The threats-to-validity section already flags the single-vendor, single-engine, two-architecture scope. Consequently the derivation (more accurately: the evidence chain) is self-contained against external benchmarks and does not reduce by construction to its inputs. Score 0 is the honest finding.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

As an observational systems field study the paper introduces almost no free parameters or invented physical entities. Its load-bearing premises are standard domain assumptions about the CANN/vLLM-Ascend stack and the two chosen workloads; the main risk is over-generalization from a single hardware/software snapshot rather than hidden fitted constants.

free parameters (1)
  • concurrency sweet-spot (~4 in-flight streams)
    Empirically observed operating point for the quantized LLM service on this particular node and request mix; used to illustrate non-monotonic throughput but not fitted to support a theoretical claim.
axioms (3)
  • domain assumption The observed failures are properties of the accelerator, its compiler/operator library, and the vendor inference plugin rather than of the two specific DeepSeek-V4-Flash-family models.
    Stated in Scope and non-goals and again in §6; required for the claim that the eight classes generalize beyond the two workloads.
  • domain assumption A successful end-to-end MMMU/MMMU-Pro score matching the native-framework reference implies numerical correctness of the Ascend port.
    Used in §4.2 and Fig. 2 to close the correctness argument for Workload B.
  • standard math Standard tensor/pipeline/expert parallelism decompositions (Megatron-style) apply and their interactions are the relevant stress points.
    Background from cited systems literature; used throughout §5.2.

reviewed 2026-07-10 · how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Limitations of Non-GPU AI Accelerators for Large-Model Inference: A Field Study of MoE and Multimodal Serving on Huawei Ascend." pith.science (2026). https://pith.science/paper/UPKI3B3Q

@misc{pith2026260708215,
  author       = {Pith},
  title        = {Pith review of: On the Limitations of Non-GPU AI Accelerators for Large-Model Inference: A Field Study of MoE and Multimodal Serving on Huawei Ascend},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UPKI3B3Q}},
  note         = {Machine review of arXiv:2607.08215}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Non-GPU AI accelerators are increasingly adopted as alternatives to general-purpose GPUs for large-model inference, but the real engineering cost of migrating demanding workloads beyond CUDA remains poorly documented. We present a field study of deploying two large inference workloads on a 16-device Huawei Ascend 910 system using CANN and vLLM-Ascend: an LLM-as-a-judge safety and alignment evaluation pipeline based on a W8A8 MoE judge model, DeepSeek-V4-Flash, and a multimodal medical vision--language benchmark based on DeepSeek-V4-Flash-Vision for MMMU and MMMU-Pro. Making these workloads reliable required twelve source-level patches to the vendor inference plugin, disabling several high-throughput features to preserve numerical correctness, and adding operational safeguards for recurring device-level failures. We summarize the main platform limitations in eight categories: incomplete operator and feature support, fragile parallelism, numerical faults in low-level kernels, immature graph compilation, unstable advanced features, limited scalability, weak observability, and ecosystem fragmentation. For each category, we report the symptoms, evidence, and likely causes. We also quantify the integration effort, concurrency behavior, and benchmark quality to show that both workloads were served correctly. Our study provides a reproducible reference for teams evaluating or operating non-GPU accelerators for large-model inference.

Figures

Figures reproduced from arXiv: 2607.08215 by Zheng Yu.

Figure 1
Figure 1. Figure 1: Data flow of the multimodal Workload B. A frozen Qwen3.5 vision tower and a frozen DeepSeek [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: End-to-end quality of the multimodal Workload B (fully instruction-tuned checkpoint) on MMMU [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-card memory footprint for the ∼540 GB bf16 multimodal workload split across 16 Ascend 910 devices (TP = 8 × PP = 2). Each card holds roughly a 34 GB decoder shard plus a ∼2 GB replicated vision tower, with the remainder of the ∼59 GB working budget (0.92 of 64 GB) going to KV cache and activations. The large weight footprint leaves little headroom, which interacts with the tensor-parallel cap discussed… view at source ↗
Figure 4
Figure 4. Figure 4: The twelve startup patches applied to the vendor inference plugin, grouped by source file. The [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Aggregate serving throughput (left axis, solid) and steady-state waiting queue depth (right axis, [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Per-stream throughput versus concurrency. Each additional stream past the sweet spot receives a [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 28 canonical work pages · 16 internal anchors

  1. [1]

    Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads.arXiv preprint arXiv:2401.10774, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. InInterna- tional Conference on Learning Representations (ICLR), 2024. arXiv:2307.08691

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    work page 2022

  4. [4]

    DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

    DeepSeek-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    DeepSeek-V3 Technical Report

    DeepSeek-AI. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    work page 2022

  7. [7]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. InInternational Conference on Learning Repre- sentations (ICLR), 2023. arXiv:2210.17323

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    Le, Yonghui Wu, and Zhifeng Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. GPipe: Efficient training of giant neural networks using pipeline parallelism. InAdvances in Neural Information Processing Systems (NeurIPS), 2019

    work page 2019

  9. [9]

    CANN: Compute architecture for neural networks — documentation

    Huawei Technologies Co., Ltd. CANN: Compute architecture for neural networks — documentation. https://www.hiascend.com/en/software/cann, 2024. Accessed 2026

    work page 2024

  10. [10]

    Mixtral of Experts

    AlbertQ.Jiang, AlexandreSablayrolles, AntoineRoux, ArthurMensch, BlancheSavary, ChrisBamford, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Efficient memory management for large language model serving with pagedattention,

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles (SOSP), pages 611–626, 2023. doi: 10.1145/3600006.3613165

    work page doi:10.1145/3600006.3613165 2023

  12. [12]

    GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computa- tion and automatic sharding. InInternational Conference on Learning Representations (ICLR), 2021. arXiv:2006.16668. 13 On the Limitations of Non-GPU AI Accelerators for ...

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  13. [13]

    Fast Inference from Transformers via Speculative Decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. InProceedings of the 40th International Conference on Machine Learning (ICML), 2023. arXiv:2211.17192

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  14. [14]

    EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty.arXiv preprint arXiv:2401.15077, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [15]

    DaVinci: A scalable architecture for neural network computing

    Heng Liao, Jiajin Tu, Jing Xia, and Xiping Zhou. DaVinci: A scalable architecture for neural network computing. InIEEE Hot Chips 31 Symposium (HCS). IEEE, 2019

    work page 2019

  16. [16]

    Ascend: A scalable and unified architecture for ubiquitous deep neural network computing — industry track paper

    Heng Liao, Jiajin Tu, Jing Xia, Hu Liu, Xiping Zhou, Honghui Yuan, and Yuxing Hu. Ascend: A scalable and unified architecture for ubiquitous deep neural network computing — industry track paper. In2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 789–801,

    work page

  17. [17]

    doi: 10.1109/HPCA51647.2021.00071

    work page doi:10.1109/hpca51647.2021.00071 2021

  18. [18]

    AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for LLM compression and acceleration. InProceedings of Machine Learning and Systems (MLSys), 2024. arXiv:2306.00978

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2304.08485

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  20. [20]

    PyTorch: An imperative style, high- performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high- performance deep learning library. InAdvances in Neural Information Processing Systems (NeurIPS),

    work page

  21. [21]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference on Machine Learning (ICML), 2021. arXiv:2103.00020

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  22. [22]

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageouslylargeneuralnetworks: Thesparsely-gatedmixture-of-expertslayer. InInternational Conference on Learning Representations (ICLR), 2017. arXiv:1701.06538

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  23. [23]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-LM: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1909

  24. [24]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems (NeurIPS), 2017

    work page 2017

  25. [25]

    vLLM Ascend plugin (vllm-ascend) documentation.https://vllm-ascend

    vLLM Project. vLLM Ascend plugin (vllm-ascend) documentation.https://vllm-ascend. readthedocs.io/, 2024. Accessed 2026

    work page 2024

  26. [26]

    Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, et al. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. InProceedings of the 40th International Conference on Machine Learning (ICML), 2023. arXiv:2211.10438

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  28. [28]

    Orca: A distributed serving system for Transformer-based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-based generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages521–538, Carlsbad, CA,2022.USENIX Association. 14

    work page 2022

This paper was first reviewed by grok-4.5 on July 10, 2026.