Pith. sign in

REVIEW 4 major objections 6 minor 31 references

CANN Bench gives AI-written Ascend kernels a hardware-anchored score, not just a speedup over a software baseline.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:08 UTC pith:EYE3NG7J

load-bearing objection A well-specified benchmark design for Ascend kernel generation, but the HAP lower bound is unproven and the promised artifact is not yet checkable — worth a serious referee, but only after major revision. the 4 major comments →

arxiv 2607.20518 v1 pith:EYE3NG7J submitted 2026-07-08 cs.AI

CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits

Xue-Jian Gao , Deng Pan , Yueming Su , Jiasheng Li , Bin Du , Fengming Zhu , Chengdi Ma , Junyi Fan
show 7 more authors
Qichen Liao Chengqiu Hu Xinxian Chen Lingchao Zheng Jun Li Jiwei Yang Yuwei Fan
This is my paper
classification cs.AI
keywords CANN BenchAscend NPUkernel generation benchmarkAI agentsHAP limitoperator kernelsreward hackinghardware-anchored performance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces CANN Bench, a benchmark meant to give the Ascend NPU ecosystem a shared, reproducible yardstick for AI agents that write operator kernels. It contains 53 operators and 1060 test cases across four difficulty levels, with a 20-case public plus 80-case hidden split per operator. The paper's main design claim is that performance should be graded between a measured out-of-the-box baseline and an analytical Hardware-Anchored Performance (HAP) limit, so a score retains meaning as software versions drift. Scores combine compilation, functional correctness, and performance into one weighted composite, and the harness includes layered defenses against reward hacking. A sympathetic reader would care because this is the first Ascend-focused benchmark positioned inside the official CANN repository with a hardware-anchored upper bound rather than a relative speedup alone.

Core claim

CANN Bench is an open, versioned benchmark for AI-generated CANN kernels on Ascend 910B2. For each of 1060 test cases it records two reference times: T_baseline, the measured runtime of the vendor's out-of-the-box implementation, and T_HW, the HAP limit computed analytically from the operator's FLOP and byte counts and the published peak rates of the six concurrently executing hardware units (cube, vector, MTE1-3, FixP). A candidate kernel's measured runtime T_cand is converted into a bounded score by Score_i = (T_baseline - T_HW)/((T_cand - T_HW) + (T_baseline - T_HW)), which is 0.5 at baseline and 1 at the hardware limit. Operator scores are linear combinations of compilation success, per-

What carries the argument

The Hardware-Anchored Performance (HAP) limit, T_HW = min_δ max_{k∈K} t_k(δ), where K is the set of six Ascend execution units and t_k(δ) is work divided by peak rate for implementation choice δ. It captures the runtime a kernel would achieve under perfect overlap, peak unit rates, and zero launch overhead, and serves as the per-case upper anchor. The paired anchor (T_baseline, T_HW) plus the bounded score formula is the mechanism that gives CANN Bench its longitudinal stability and its claimed resistance to measurement artifacts.

Load-bearing premise

The entire performance scale rests on T_HW being a true lower bound; if the six units cannot actually overlap perfectly at peak rates, or if the FLOP/byte accounting misses a path, a real kernel could run faster than the 'limit' and every score loses its hardware-grounded meaning.

What would settle it

Find or write a kernel on Ascend 910B2 that, with no delegation or timing tricks, reproducibly runs a CANN Bench case in less time than the published T_HW for that case; if such a kernel exists, Eq. (4) is not a true lower bound. A less adversarial check: independently recompute T_HW from the silicon's published peak rates and measure the fastest known hand-tuned kernel on the same case, and compare.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If T_HW is a genuine lower bound, performance comparisons between kernels remain meaningful across CANN releases, because only the lower anchor drifts with software versions.
  • The 20-public/80-hidden case split means leaderboard scores cannot be gamed by overfitting to disclosed shapes.
  • The three-axis composite makes compilation failure catastrophic (zero score), so agents are rewarded only for kernels that both compile and run correctly.
  • Because MTE dominates the bottleneck labels on 63% of cases, the benchmark highlights data-movement-limited operators as the main optimization target on Ascend.
  • Housed in the official CANN repository and versioned with operator contracts, the benchmark can grow with new operators, precisions, and hardware generations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The HAP scoring scheme could be adapted to other accelerators that expose a small set of concurrently executing hardware units, effectively turning a single silicon's roofline into a benchmark anchor.
  • If adopted widely, hardware-anchored scores would let training pipelines optimize against a stable reward rather than chasing a moving baseline, potentially changing how kernel-generation agents are trained.
  • The paper's precision gate—mean/max relative error with small-value and cancellation fallbacks—is a useful template for other benchmarks that need to distinguish genuine kernel errors from dtype floor noise.
  • The benchmark's own audit signal, any case with T_cand < T_HW triggers review, could in practice surface modeling gaps in the analytical procedure as quickly as it surfaces cheating, making the anchor self-correcting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. CANN Bench is proposed as an open, vendor-maintained benchmark for evaluating AI-generated CANN operator kernels on Huawei Ascend NPUs. The paper reports 53 operators and 1060 test cases in four difficulty tiers, with each operator accompanied by specification, prototype, test cases, golden implementation, and a PyTorch-on-Ascend baseline. A submission is scored on compilation, functional correctness, and performance, combined into a per-operator weighted composite. Performance is graded against an analytical Hardware-Anchored Performance (HAP) limit THW, defined in Eq. (4) as the minimized maximum unit-level execution time, and Eq. (5) interpolates the candidate runtime between baseline and HAP. The paper also describes a layered anti-reward-hacking harness, public/hidden test splits, and plans for a leaderboard, DSL extensions, and broader precision support. The authors position the benchmark inside the official CANN repository and claim it provides a stable, reproducible evaluation scale independent of any training recipe.

Significance. If the manuscript's claims are substantiated, CANN Bench would fill a real gap: kernel-generation benchmarks are dominated by CUDA/Triton targets, and Ascend lacks a widely accepted evaluation scale. The HAP-anchored scoring idea, the detailed correctness gate with region-specific fallbacks, the public/hidden split, and the extensive anti-reward-hacking design are valuable and thoughtfully specified. The paper also gives useful, concrete design criteria (C1-C6) and positions the work carefully against existing benchmarks. However, the manuscript currently demonstrates the design, not the working benchmark: no leaderboard is live, no baseline or THW values are reported, no agent submissions are evaluated, and the core HAP lower-bound claim is asserted rather than validated. Because these are load-bearing for the benchmark's stated purpose, the current version is a design proposal with strong scaffolding rather than a fully evidenced benchmark release.

major comments (4)
  1. [§3.2, Eq. (4)] The HAP limit THW = min_δ max_k t_k(δ) is load-bearing for every performance score, but it is asserted, not demonstrated. The paper gives no closed-form expressions or worked example for t_k(δ), no argument that the structured implementation-choice space δ is exhaustive over feasible schedules, and no empirical validation on real 910B2 kernels. The claim 'No real implementation can violate this bound' depends on the accounting of FLOP/byte counts and peak rates being complete and conservative. The paper even states 'THW ≤ Tbaseline should hold on every case' and treats violations as audit signals, but no audit results or violation counts are reported. Since Eq. (5) and Eq. (6) inherit THW, the entire performance scale is currently unvalidated. I request a detailed derivation for at least a few representative operators, a demonstration that the accounting covers all six units (including M
  2. [Abstract, §1.4, §2.2, Figure 1d] The dataset counts are internally inconsistent. The abstract and §1.4 state 53 operators and 1060 test cases; §2.2 says 'each with about 20 test cases.' But Figure 1d reports per-level case counts n=160, n=320, n=378, n=156, which sum to 1014, not 1060. For L3, 21 operators × 20 public cases = 420, not 378; for L4, 8 × 20 = 160, not 156. Moreover, §2.3 says each operator has a 20-case public split plus an 80-case hidden split, so the total case count should include hidden cases as well (5300 if each operator has 100 total). The paper must clarify whether '1060 test cases' refers only to public cases, correct the figure, and state the exact public and hidden totals. A benchmark's headline dataset statistics cannot be ambiguous.
  3. [§1.4, §1.5, §4] The paper promises a reproducible, benchmarked system but ships no empirical validation. There is no leaderboard URL or live portal (the footnote in §1.4 says it 'will be released in a future update'), no reported baseline_perf_us values, no reported THW values, no bottleneck-distribution table beyond a figure, and no demonstration of the harness scoring an actual agent-generated kernel. The paper says the 'harness and baseline-collection pipeline have completed end-to-end runs on NPU hardware' but provides no measurements. For a benchmark paper, the central deliverable is a working evaluator with published reference values and at least a small set of example scores (e.g., on hand-written kernels or existing agents) that shows the three-axis composite behaves as described. Without such evidence, the claims of reproducibility and a 'shared evaluation scale' cannot be verified by readers.
  4. [§3.2, Figure 2a] The bottleneck distribution (MTE 63%, Vector 26%, Cube 11%) is presented as a dataset finding, but it is derived entirely from the unvalidated HAP analysis. If Eq. (4) underestimates or overestimates any unit's work, these labels are not reliable. The paper should either provide the underlying per-case THW derivations and audit results or present this figure as a preliminary analytical estimate, not as a measured property of the benchmark.
minor comments (6)
  1. [§1.4] The inline formula for EachOperatorScore uses ambiguous parentheses: it is not immediately clear that only the summation term, not the wc·δpass term, is divided by num_of_cases. Eq. (6) clarifies this, but the prose should match exactly.
  2. [Table 4] The FP8 threshold entries are typeset inconsistently (e.g., '2 −3' and '2 −2' appear without superscripts). This makes the table hard to read, especially because the same notation is used for powers of two throughout the paper.
  3. [Figure 1d] The caption says 'provided baseline execution times' while the y-axis is 'total input elements (M, log scale)'. The caption appears mismatched with the data shown; please revise to describe the element-count distribution.
  4. [§2.3] The paper says 'At least 50% non-aligned cases' and 'Within each operator's 20-case public split', but it is not stated whether the same 50% requirement applies to the 80-case hidden split. Please clarify the hidden-split construction guarantees.
  5. [§3.1, Eq. (3)] The ErrorCount ratio uses max(ErrorCountCPU,1), but the statistical behavior of the ratio when ErrorCountCPU is very small or zero is not discussed. A few sentences on the treatment of zero-count CPU runs would improve the precision standard's reproducibility.
  6. [§4] The conclusion repeats the phrase 'ground up' and 'shared evaluation scale' several times; tightening the wording would improve readability.

Circularity Check

0 steps flagged

No significant circularity: HAP anchor is analytically derived and scoring is a definition, not a fitted prediction.

full rationale

The paper's central scoring mechanism is the HAP limit THW in Eq. (4), defined as min_delta max_k t_k(delta) from operator FLOP/byte counts and published hardware peak rates. This is an analytical construction, not a quantity fitted to candidate runtimes; the performance score in Eq. (5) is a definitional mapping of measured Tcand onto a bounded scale anchored by Tbaseline and THW. The paper does not rename an empirical observation as a derivation: its claimed 'prediction' is a benchmark score, and the anchors are set before evaluation. There are no load-bearing self-citations; references to prior hardware-anchored work are acknowledged and not used to justify the current bound. The HAP lower-bound assertion is not empirically validated and is flagged by the paper itself as requiring review when violated, but that is an internal-consistency or correctness risk, not circularity. Benchmark authors defining golden references, precision thresholds, and composite weights is a normative design choice, not an equation reducing a result to its own inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The ledger shows the evaluation design rests on several hand-chosen thresholds and idealized hardware assumptions. No genuinely fitted 'physical constant' is introduced, but the composite weights, precision gates, and the HAP half-space model are all choices made by the authors rather than externally forced facts.

free parameters (4)
  • Composite weights (wc, wf, wp) = 0.2, 0.3, 0.5
    Hand-chosen default weights in Eq. (6); they directly determine operator scores and the balance between compilation, correctness, and performance.
  • Per-dtype mean-RE threshold tau_d = Table 4: FP16 2^-10, BF16 2^-7, FP32 2^-13, etc.
    Hand-picked powers of two anchored to mantissa width; they set the functional-correctness gate and are not derived from data.
  • Region thresholds for small-value and cancellation gate = Appendix A: tau_sv, epsilon_sv, tau_cz, tau_cb
    Hand-chosen absolute-error and cancellation boundaries; they determine which failures are excused in numerically unstable regions.
  • Accuracy-retry perturbation size = 0.01
    Arbitrary perturbation applied in the opt-in two-trial verification mode (§3.1); affects whether cached outputs pass the second trial.
axioms (5)
  • domain assumption CPU FP64 upcast is a reliable ground truth for the functional-correctness gate.
    Invoked throughout §3.1; the Golden path runs on CPU after upcasting to FP64, and all pass/fail decisions depend on that reference.
  • domain assumption The six Ascend execution units can be modeled as independent parallel resources whose work times combine as a maximum under perfect overlap and peak rates.
    Eq. (4) defines THW as min over max of per-unit t_k; if true overlap or rate assumptions fail, THW is not a hardware lower bound.
  • domain assumption Published peak rates and buffer capacities for Ascend 910B2 are correct and sufficient for the analytical HAP procedure.
    The derivation procedure in §3.2 says THW is determined entirely by hardware specification and published peak parameters; errors in those numbers propagate into every performance score.
  • domain assumption Hidden cases are not inferable from public cases and remain free of information leakage through baselines or goldens.
    The anti-reward-hacking design (§2.3, §3.4) relies on the hidden split guarding against overfitting; any leakage weakens the benchmark's core guarantee.
  • domain assumption The baseline runtime T_baseline is a meaningful lower anchor, despite being defined inconsistently as CANN-internal (§3) and PyTorch-on-Ascend (abstract).
    Every HAP score compares candidate runtime to T_baseline; if the baseline is not a fair, stable reference, scores are mis-calibrated.
invented entities (1)
  • Hardware-Anchored Performance (HAP) limit THW independent evidence
    purpose: Analytical per-case upper anchor for performance scoring, derived from the six-unit Ascend architecture and operator FLOP/byte counts.
    THW makes a falsifiable prediction: no real kernel should run faster than it, and Tcand < THW triggers an evaluation review. This is a testable handle outside the paper, though no validation data are reported.

pith-pipeline@v1.3.0-alltime-deepseek · 16190 in / 11945 out tokens · 119435 ms · 2026-08-02T08:08:32.880537+00:00 · methodology

0 comments
read the original abstract

AI agents are now capable of writing, compiling, and iteratively optimizing low-level operator kernels on different hardware platforms. Existing benchmarks, however, focus almost exclusively on CUDA and Triton, leaving hardware ecosystems with less-exposed programming models without a common evaluation baseline. We present CANN Bench, an open benchmark for AI-generated operator code on Huawei's Ascend NPU. The current release covers 53 operators and 1060 test cases organized into four difficulty tiers -- from simple elementwise primitives to MoE dispatch and FlashAttention kernels -- spanning FP16, BF16, FP32, and INT8 precision formats. Evaluation adopts a \textbf{three-dimensional weighted composite score} that treats compilation, functional correctness, and performance as independent axes, providing a principled reward signal for kernel-generation agents. Performance is graded against an out-of-the-box PyTorch-on-Ascend baseline and an analytical per-case Hardware-Anchored Performance (HAP) limit on real NPU hardware, ensuring scores reflect genuine optimization headroom rather than measurement artifacts. The evaluation harness is designed to resist reward hacking from the ground up. CANN Bench is versioned within the official CANN repository and is designed for long-term community co-construction, providing the Ascend ecosystem with a quantitative, reproducible, and sustainably maintained yardstick for AI operator-authoring capability.

Figures

Figures reproduced from arXiv: 2607.20518 by Bin Du, Chengdi Ma, Chengqiu Hu, Deng Pan, Fengming Zhu, Jiasheng Li, Jiwei Yang, Jun Li, Junyi Fan, Lingchao Zheng, Qichen Liao, Xinxian Chen, Xue-Jian Gao, Yueming Su, Yuwei Fan.

Figure 1
Figure 1. Figure 1: Dataset overview of our CANN Bench. (a) Operator count per difficulty level. (b) Operation category [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 10 linked inside Pith

  1. [1]

    CUDA Agent: Large-scale agentic RL for high-performance CUDA kernel generation.arXiv preprint arXiv:2602.24286, 2026

    Weinan Dai, Hanlin Wu, Qiying Yu, Huan-ang Gao, Jiahao Li, Chengquan Jiang, et al. CUDA Agent: Large-scale agentic RL for high-performance CUDA kernel generation.arXiv preprint arXiv:2602.24286, 2026

    arXiv 2026

  2. [2]

    CuTeGen: An LLM-based agentic framework for generation and optimization of high-performance GPU kernels using CuTe.arXiv preprint arXiv:2604.01489, 2026

    Tara Saba, Anne Ouyang, Xujie Si, and Fan Long. CuTeGen: An LLM-based agentic framework for generation and optimization of high-performance GPU kernels using CuTe.arXiv preprint arXiv:2604.01489, 2026

    Pith/arXiv arXiv 2026

  3. [3]

    AutoKernel: Autonomous GPU kernel optimization via iterative agent- driven search.arXiv preprint arXiv:2603.21331, 2026

    Jaber Jaber and Osama Jaber. AutoKernel: Autonomous GPU kernel optimization via iterative agent- driven search.arXiv preprint arXiv:2603.21331, 2026. 17

    arXiv 2026

  4. [4]

    GPU Kernel Scientist: An LLM-driven framework for iterative kernel optimization.arXiv preprint arXiv:2506.20807, 2025

    Martin Andrews and Sam Witteveen. GPU Kernel Scientist: An LLM-driven framework for iterative kernel optimization.arXiv preprint arXiv:2506.20807, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    GEAK: Introducing triton kernel AI agent & evaluation benchmarks.arXiv preprint arXiv:2507.23194, 2025

    Jianghui Wang, Vinay Joshi, Saptarshi Majumder, Xu Chao, Bin Ding, Ziqiong Liu, et al. GEAK: Introducing triton kernel AI agent & evaluation benchmarks.arXiv preprint arXiv:2507.23194, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Kevin: Multi-turn RL for generating CUDA kernels.arXiv preprint arXiv:2507.11948, 2025

    Carlo Baronio, Pietro Marsella, Ben Pan, Simon Guo, and Silas Alberti. Kevin: Multi-turn RL for generating CUDA kernels.arXiv preprint arXiv:2507.11948, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Wei Liu, Jiawei Xu, Yingru Li, Longtao Zheng, Tianjian Li, Qian Liu, et al. Dr. Kernel: Reinforcement learning done right for Triton kernel generations.arXiv preprint arXiv:2602.05885, 2026

    arXiv 2026

  8. [8]

    EvoEngineer: Mastering automated CUDA kernel code evolution with large language models.arXiv preprint arXiv:2509.18570, 2025

    Ping Guo, Chenyu Zhu, Siyuan Chen, Fei Liu, Xi Lin, Zhichao Lu, et al. EvoEngineer: Mastering automated CUDA kernel code evolution with large language models.arXiv preprint arXiv:2509.18570, 2025

    arXiv 2025

  9. [9]

    Towards robust agentic cuda kernel benchmarking, verification, and optimization, 2025

    Robert Tjarko Lange, Qi Sun, Aaditya Prasad, Maxence Faldor, Yujin Tang, and David Ha. Towards robust agentic cuda kernel benchmarking, verification, and optimization, 2025

    2025

  10. [10]

    Z. Wen, Y . Zhang, Z. Li, Z. Liu, L. Xie, and T. Zhang. MultiKernelBench: A multi-platform benchmark for kernel generation.arXiv preprint arXiv:2507.17773, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    AscendKernelGen: A systematic study of LLM-based kernel generation for neural processing units, 2026

    Xinzi Cao, Jianyang Zhai, Pengfei Li, Zhiheng Hu, Cen Yan, Bingxu Mu, et al. AscendKernelGen: A systematic study of LLM-based kernel generation for neural processing units, 2026

    2026

  12. [12]

    Ascend- Craft: Automatic Ascend NPU kernel generation via DSL-guided transcompilation.arXiv preprint arXiv:2601.22760, 2026

    Zhongzhen Wen, Shudi Shao, Zhong Li, Yu Ge, Tongtong Xu, Yuanyi Lin, et al. Ascend- Craft: Automatic Ascend NPU kernel generation via DSL-guided transcompilation.arXiv preprint arXiv:2601.22760, 2026

    arXiv 2026

  13. [13]

    AscendOp- timizer: Episodic agent for Ascend NPU operator optimization.arXiv preprint arXiv:2603.23566, 2026

    Jiehao Wu, Zixiao Huang, Wenhao Li, Chuyun Shen, Junjie Sheng, and Xiangfeng Wang. AscendOp- timizer: Episodic agent for Ascend NPU operator optimization.arXiv preprint arXiv:2603.23566, 2026

    Pith/arXiv arXiv 2026

  14. [14]

    Towards cold-start drafting and continual refining: A value-driven memory approach with application to NPU kernel synthesis.arXiv preprint arXiv:2603.10846, 2026

    Yujie Zheng, Zhuo Li, Shengtao Zhang, Hanjing Wang, Junjie Sheng, Jiaqian Wang, et al. Towards cold-start drafting and continual refining: A value-driven memory approach with application to NPU kernel synthesis.arXiv preprint arXiv:2603.10846, 2026

    arXiv 2026

  15. [15]

    KernelCAT: An expert-level agent for compute acceleration on Ascend NPU

    KernelCAT Team. KernelCAT: An expert-level agent for compute acceleration on Ascend NPU. Zhihu article, 2025. Industrial technical blog post

    2025

  16. [16]

    Microscaling data formats for deep learning.arXiv preprint arXiv:2310.10537, 2023

    Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, et al. Microscaling data formats for deep learning.arXiv preprint arXiv:2310.10537, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Training LLMs with MXFP4.arXiv preprint arXiv:2502.20586, 2025

    Albert Tseng, Tao Yu, and Youngsuk Park. Training LLMs with MXFP4.arXiv preprint arXiv:2502.20586, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Zhang, William Hu, Christopher Ré, et al

    Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, et al. Kernel- Bench: Can LLMs write efficient GPU kernels?arXiv preprint arXiv:2502.10517, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    TritonBench: Benchmarking large language model capabilities for generating Triton operators

    Jianling Li, Shangzhan Wang, Zhenye Zhang, et al. TritonBench: Benchmarking large language model capabilities for generating Triton operators. InFindings of ACL, 2025. arXiv:2502.14752

    Pith/arXiv arXiv 2025

  20. [20]

    SOL-ExecBench: Speed-of-light benchmarking for real-world GPU kernels against hardware limits.arXiv preprint arXiv:2603.19173, 2026

    Edward Lin, Sahil Modi, Siva Kumar Sastry Hari, Qijing Huang, et al. SOL-ExecBench: Speed-of-light benchmarking for real-world GPU kernels against hardware limits.arXiv preprint arXiv:2603.19173, 2026

    arXiv 2026

  21. [21]

    NPUEval: Optimizing NPU kernels with LLMs and open source compilers, 2025

    Sarunas Kalade and Graham Schelle. NPUEval: Optimizing NPU kernels with LLMs and open source compilers, 2025

    2025

  22. [22]

    Kernelcraft: Benchmarking for agentic close-to-metal kernel generation on emerging hardware, 2026

    Jiayi Nie, Haoran Wu, Yao Lai, Zeyu Cao, Cheng Zhang, Binglei Lou, et al. Kernelcraft: Benchmarking for agentic close-to-metal kernel generation on emerging hardware, 2026

    2026

  23. [23]

    KernelBench v0.1

    Scaling Intelligence Lab. KernelBench v0.1. Stanford Scaling Intelligence Lab blog post, blog page, 2025

    2025

  24. [24]

    Hacks and defenses in automatic GPU kernel generation

    DeepReinforce Team. Hacks and defenses in automatic GPU kernel generation. project page, 2025

    2025

  25. [25]

    Fall 2025 KernelBench maintenance and improvement plan

    Scaling Intelligence Lab. Fall 2025 KernelBench maintenance and improvement plan. GitHub Issue #74, issue page, 2025. 18

    2025

  26. [26]

    msprof profiling tool: Ascend CANN auxiliary development toolkit documentation

    Huawei Ascend. msprof profiling tool: Ascend CANN auxiliary development toolkit documentation. Huawei Ascend documentation page, 2023. CANN Commercial 6.0.1

    2023

  27. [27]

    CANN open software license agreement version 2.0

    Huawei. CANN open software license agreement version 2.0. CANN repository LICENSE, 2026

    2026

  28. [28]

    Conditional memory via scalable lookup: A new axis of sparsity for large language models, 2026

    Xin Cheng, Wangding Zeng, Damai Dai, Qinyu Chen, Bingxuan Wang, Zhenda Xie, et al. Conditional memory via scalable lookup: A new axis of sparsity for large language models, 2026

    2026

  29. [29]

    Npu and cuda function alignment

    Huawei. Npu and cuda function alignment. Huawei Technical Support Documentation. Accessed 2026-04-23

    2026

  30. [30]

    Ascend npu performance data collection

    ModelScope SWIFT Contributors. Ascend npu performance data collection. Megatron-SWIFT Documentation. Accessed 2026-04-23

    2026

  31. [31]

    Ascend extension for pytorch

    Ascend. Ascend extension for pytorch. GitHub repository. Accessed 2026-04-23. 19

    2026