REVIEW 10 cited by
MultiKernelBench: A Multi-Platform Benchmark for Kernel Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
MultiKernelBench: A Multi-Platform Benchmark for Kernel Generation
read the original abstract
The automatic generation of deep learning (DL) kernels using large language models (LLMs) has emerged as a promising approach to reduce the manual effort and hardware-specific expertise required for writing high-performance operator implementations. However, existing benchmarks for evaluating LLMs in this domain suffer from limited hardware support, coarse-grained kernel categorization, and imbalanced task coverage. To address these limitations, we introduce MultiKernelBench, the first comprehensive, multi-platform benchmark for LLM-based DL kernel generation. MultiKernelBench spans 285 tasks across 14 well-defined kernel categories and supports three major hardware platforms: Nvidia GPUs, Huawei NPUs, and Google TPUs. To enable future extensibility, we design a modular backend abstraction layer that decouples platform-specific logic from the core benchmarking infrastructure, allowing easy integration of new hardware platforms. We further propose a simple yet effective category-aware one-shot prompting method that improves generation quality by providing in-category exemplars. Through systematic evaluations of seven state-of-the-art LLMs, we reveal significant variation in task difficulty, poor generalization to platforms with less training exposure, and the effectiveness of targeted prompting strategies. MultiKernelBench is publicly available at https://github.com/wzzll123/MultiKernelBench.
Forward citations
Cited by 10 Pith papers
-
FastKernels: Benchmarking GPU Kernel Generation in Production
FastKernels is a production-aligned benchmark covering 96.2% of HuggingFace Transformers that reveals state-of-the-art kernel agents deliver at most 0.94x aggregate speedup.
-
KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?
Under a TF32-enabled baseline and multi-distribution hidden tests, the best LLM kernel generator (GPT-5.5) achieves 0.88x geometric mean speedup versus PyTorch, not the reported 1.43x.
-
AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents
AgentKernelArena is a new open benchmark that measures complete AI agent workflows on 196 GPU kernel tasks with correctness, performance, and generalization checks to unseen configurations.
-
Kernel Contracts: A Specification Language for ML Kernel Correctness Across Heterogeneous Silicon
Kernel Contracts is a specification language that formalizes correctness requirements for ML kernels to ensure consistent results across heterogeneous silicon platforms.
-
KernelGenBench: A Multi-Source and Multi-Chip Benchmark for LLM-based Kernel Generation
A multi-source (ATen/vLLM/cuBLAS, 210 ops) and multi-chip (6 platforms) benchmark shows agentic kernel generation beats sampling but degrades sharply off NVIDIA and costs millions of tokens per success.
-
CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits
CANN Bench is an Ascend-NPU benchmark with 53 operators and 1,060 released test cases that scores agent-generated kernels on compilation, correctness, and hardware-anchored performance.
-
MKEvolve: A Modular Multi-Agent Framework for Kernel Code Generation
MKEvolve iteratively co-evolves a modular decomposition of PyTorch modules and per-subkernel Triton implementations, improving KernelBench L2/L3 correctness and speedup while cutting LLM tokens by up to 35%.
-
Kernel Forge: An Agent Harness for LLM-based Generation and Optimization of CUDA Kernels
A new open-source harness uses an LLM and MCTS to generate CUDA kernels for unmodified PyTorch models, achieving operator-level speedups up to 2.83x over eager mode with guarded fallback.
-
Compiler-Grounded Hierarchical Diagnosis for LLM-Based Triton Kernel Optimization
A hierarchical, compiler-grounded LLM optimization loop reports 4.35x geometric-mean Triton kernel speedups on 37 Ascend NPU benchmark kernels, but with no causal ablation yet.
-
AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization
AscendOptimizer combines kernel rewinding for reusable experience with evolutionary search on hardware feedback to optimize Ascend NPU operators, delivering 1.21x geometric-mean speedup and faster performance on 53.47...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.