Pith. sign in

REVIEW 4 major objections 4 minor 49 references

KernelGenBench unifies evaluation of LLM-generated GPU kernels across 210 operators and six chips, showing agents win with execution feedback, closed-source BLAS caps speedup, and portability collapses.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:48 UTC pith:DTDH5IKN

load-bearing objection A serious multi-source, multi-chip kernel bench with a permissive verifier that may inflate pass rates; fixable, but headline numbers need a grain of salt. the 4 major comments →

arxiv 2607.27231 v1 pith:DTDH5IKN submitted 2026-07-22 cs.AI cs.LG

KernelGenBench: A Multi-Source and Multi-Chip Benchmark for LLM-based Kernel Generation

Peiyu Zang , Jian Tao , Jialing Zhang , Yichen Yuan , Wentao Zhang , Guang Liu , Yonghua Lin This is my paper
classification cs.AI cs.LG
keywords Triton kernelsLLM-generated kernelsagentic code generationGPU benchmarksperformance portabilityheterogeneous hardwaretoken efficiencykernel correctness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

KernelGenBench aims to give the field of LLM-generated accelerator kernels a first rigorous, multi-source, multi-chip measurement standard. It curates 210 production operators from three library types — a framework tensor library, an open LLM-inference library, and a closed linear-algebra library — and requires LLMs and coding agents to write drop-in Triton replacements under a unified execution harness. The paper's central claim is that this harness exposes three repeatable findings: agentic methods with execution feedback outperform static sampling; operator source creates a strict difficulty hierarchy with closed-source BLAS as the hardest ceiling; and generated kernels do not port across hardware, with a specialized agent dropping from 87% accuracy on a familiar GPU to 25% on an alternative platform. These results matter because they shift what 'good kernel generation' means — from single-pass code that compiles to closed-loop, portable, cost-accounted engineering.

Core claim

On its own terms, KernelGenBench establishes a reusable measurement standard for Triton kernel generation. The Multi-Source track covers 210 operators from a framework tensor library, an open-source LLM-inference library, and a closed-source BLAS library; the Multi-Chip track runs a 110-operator subset across six hardware platforms, five of them anonymized alternatives. Using a shared execution sandbox, adversarial verification, and a combinatorially shaped test suite, the paper reports that agentic methods with execution feedback outperform static LLM sampling, that closed-source BLAS operators impose a universal performance ceiling near 0.5x speedup, and that cross-platform transfer fails

What carries the argument

The load-bearing object is the benchmark pipeline itself, specifically its clean-pass gate and three-tier anti-hack design. Generated kernels are mounted into the host tensor framework's dispatch tree, then scored only if they pass numerical validation across all per-operator test cases AND survive three interception layers: an AST static scan for forbidden calls, a ghost-replay check that swaps the kernel for a no-op to prove it was actually invoked, and hardware profiling. Speedup is aggregated by a two-level geometric mean across test cases and operators, and agent cost is measured in total tokens, tokens per success, and wall time. This combination lets the paper attribute differences to

Load-bearing premise

The benchmark's conclusions rest on hand-tuned tolerances and test grids, hand-built prompts, and five anonymized alternative platforms; if those choices — or prior memorization of the requested library source — dominate the scores, the reported difficulty ranks and token costs are properties of the harness, not of the models.

What would settle it

Perturb one-line constants or API spellings in every target operator so any memorized reference implementation would fail, then rerun the full 210-operator suite; if pass rates stay put, generalization is real, and if they collapse, the benchmark is measuring training-data recall.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Benchmarking LLM kernel work must measure execution feedback, not just static code correctness; single-pass sampling systematically undercounts agent value.
  • Operator source is a first-order variable: closed-source BLAS kernels are a universal performance ceiling, so evaluating only generic tensor ops inflates apparent capability.
  • Cross-platform performance portability is a real, measurable bottleneck; scores on one GPU cannot be assumed to transfer to another architecture.
  • Token cost is a first-class metric for agentic systems: at roughly five million tokens per successful operator, autonomous kernel generation is not yet economically scalable.
  • Survivorship bias contaminates speedup comparisons unless conditioned on matched operator subsets; simpler operators are easier to pass and easier to speed up.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the portability collapse stems from immature vendor compilers rather than model skill, then stabilizing low-level Triton backends on non-mainstream hardware may raise scores more than any model improvement.
  • The token-cost finding suggests a cheap fast-fail pre-filter — a lint pass or a conservative compiler check before full verification — could reduce agent budgets by an order of magnitude without sacrificing correctness.
  • A testable extension: a model trained on BLAS operator internals (not just source) should break the ~0.5x closed-library ceiling, because the benchmark identifies the ceiling as algorithmic rather than syntactic.
  • An independent audit that reveals the five anonymized platforms and replicates the 0.18x speedup collapse would confirm whether the portability gap is a hardware reality or a benchmark artifact.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. KernelGenBench is a benchmark for evaluating LLM- and agent-generated Triton kernels. It has two sub-benchmarks: KernelGenBench-MS, covering 210 operators from ATen, vLLM, and cuBLAS on NVIDIA A100, and KernelGenBench-MC, covering 110 ATen operators on six hardware platforms. The paper reports a large-scale evaluation consuming over 15 billion tokens, with three headline findings: agentic methods outperform pure LLM sampling, operator-source difficulty varies sharply (cuBLAS hardest), cross-platform portability degrades severely (e.g., AutoKernel drops from 87% on NVIDIA to 25% on Platform E), and agentic methods are token-expensive (5.11M tokens per successful operator). The evaluation infrastructure includes dispatch via torch.library, a three-tier anti-hack mechanism, combinatorial test grids, adaptive numerical tolerances, and multiple speedup metrics.

Significance. If the results hold, KernelGenBench would be a valuable community resource: it is open-source, uses production-aligned dispatch, explicitly guards against trivial benchmark hacking, and provides a wider operator and hardware surface than prior kernel benchmarks. The paper is also honest about some limitations, including single-trajectory agent measurements and survivorship bias in speedup aggregation. However, the central dependent variable — the clean pass rate in Tables 2 and 3 — depends on a verifier tolerance that appears far too permissive for reduction operators, and several main tables are corrupted and unreadable. These issues must be resolved before the empirical findings can be accepted.

major comments (4)
  1. [Appendix D, "Adaptive Numerical Tolerance"; Appendix F, softmax example] The verifier uses atol = 10^{-4} * D_reduce, where D_reduce is the reduction dimension size. For mean over a dimension of size 40999 (Appendix F shape grid includes (200, 40999, 3)), atol ≈ 4.1 while the output magnitude is O(1); an all-zeros kernel would therefore pass. For softmax over dim of size 2560, atol ≈ 0.256 while typical outputs are ~4e-4; a constant or near-zero kernel also passes. Since the clean pass rate in Tables 2 and 3 is derived from this verifier, the reported difficulty hierarchy and cross-platform accuracy comparisons may be artifacts of the tolerance setting. The paper does not report per-operator error margins, so the number of marginal passes is unknown. Please provide per-operator error distributions, a stricter normalization (e.g., atol independent of D_reduce or scaled sublinearly), or evidence that generated kernels' errors are far below the tolerance.
  2. [Tables 2 and 3] Several cells in the main results tables are corrupted and cannot be read. For example, Table 2, Pass@1 (Qwen3.5-27b) vLLM column reads "22.05" and Table 3, Pass@5 (Qwen3.5-27b) NVIDIA row reads "131.04", which appear to be accuracy and speedup values fused into one cell. Many other rows in Table 3 have the same issue. Because these tables are the empirical basis for Findings 1 and 2, they must be regenerated in a clear, parseable format.
  3. [Section 3.1; Appendix A] The benchmark asks models to implement public ATen and vLLM operators whose source code is openly available in repositories that are likely included in LLM training corpora. The paper provides no contamination analysis (e.g., n-gram overlap with the reference implementations, exact-match checks against GitHub, or ablation on transformed prompts). The ATen prompt injects the full FunctionSchema and official docstrings, which directly identifies the target operator. Without this analysis, the observed pass rates for ATen and vLLM subsets may reflect memorization rather than de novo kernel generation, undermining the difficulty hierarchy. Please add a contamination check or explicitly state the limitation and its expected impact.
  4. [Section 3.2, KernelGenBench-MC] The five non-NVIDIA platforms are anonymized ("Platform A–E"), with identities withheld under confidentiality agreements. This prevents independent verification of the cross-platform claims, including the LLVM IR incompatibilities and 32-bit pointer issues in Section 4.5, and limits the community's ability to reproduce or extend the MC results. While vendor confidentiality is understandable, the manuscript should provide architectural descriptors (ISA class, memory model, compiler/Triton backend version) or a clearly described access mechanism so that the results can be interpreted and reproduced by others.
minor comments (4)
  1. [Abstract / Section 1] The abstract at the top of the paper says "six heterogeneous hardware platforms" while the body says "five alternative hardware platforms" in addition to NVIDIA. Please make the count consistent (NVIDIA + five alternatives = six total).
  2. [Appendix G] The fastp metric is used in Section 4.2 before it is formally defined in Appendix G. Define it at first use or provide a pointer in the main text.
  3. [Figure 2] The left panel is described as a radar chart but appears as a grouped bar/line chart in the rendered text. Please align the caption with the actual figure format.
  4. [Section 4.1] The paper says all agent results are single-trajectory due to cost constraints and interprets cross-method differences below 5 pp cautiously. This is good, but the same caution is not consistently applied when discussing specific gaps (e.g., the 4 pp difference between Claude Code and AKO4all in Section 4.2). Please state the statistical uncertainty more uniformly.

Circularity Check

0 steps flagged

No significant circularity; the benchmark results are externally measured, with only background self-citations.

full rationale

KernelGenBench is an evaluation benchmark, so its derivation chain is a measurement protocol rather than a theorem. The central dependent variables—clean pass rate, speedup, and token cost—are computed by running generated Triton kernels against external baselines (native ATen, vLLM v0.13.0, and libcublas.so loaded directly via ctypes) and an independent anti-hack check. No reported result is obtained by fitting a parameter to the target it is later said to predict; pass/fail labels and speedups are empirical observations on a fixed test grid (Appendix F) and a fixed speedup aggregation formula (Eqs. 1-2). The only same-author items are citation [13], a survey used for related-work framing, and [26], the FlagGems operator library used to motivate drop-in replaceability; neither supplies a load-bearing premise for the empirical findings. The adaptive tolerance formula and hand-built prompts could affect benchmark validity or difficulty, but that is a measurement-quality concern, not circularity: the outputs are not defined to equal the inputs. The stated limitations (single-trajectory runs, joint LLM+scaffold measurement, partial operator coverage) weaken generalization claims without making any result self-confirming. Score 1 reflects harmless background self-citation with an otherwise self-contained empirical evaluation.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The benchmark's scores depend on hand-chosen tolerances, test grids, timeouts, and prompt templates; these are not fitted to produce the conclusions, but they all influence pass rates. The anonymized platforms and lack of contamination analysis are additional measurement assumptions rather than invented physical entities.

free parameters (4)
  • adaptive numerical tolerance constants = atol = 1e-4 * D_reduce; rtol: FP32=1e-5, FP16=1e-3, BF16=0.016
    Hand-set thresholds in Appendix F determine correctness pass/fail and therefore all accuracy numbers; no sensitivity analysis is reported.
  • per-operator test grid cardinality k_i = e.g., softmax k_i=18; shape/dtype/dim grids per operator category
    Number and content of test cases per operator are hand-curated; pass rates depend directly on this choice.
  • wall-clock timeout = 30 min per operator; 300s correctness, 600s performance
    Affects agent completion and token costs; not swept in the paper.
  • max tokens per generation = 16384
    Documented cause of GLM-5.0 truncation; affects pass rates and cost measurements.
axioms (4)
  • domain assumption Triton kernels registered via torch.library can serve as drop-in replacements for ATen/vLLM/cuBLAS operators given matching signatures.
    The entire correctness evaluation and anti-hack design assumes the dispatch mechanism faithfully exercises the generated kernel (Section 3.3).
  • ad hoc to paper Numerical correctness is adequately determined by the hand-set adaptive tolerances and test grids.
    Appendix F sets atol=1e-4*D_reduce and dtype-dependent rtol; pass rates are threshold-dependent and no sensitivity analysis is given.
  • domain assumption L1 AST scan + L2 Ghost Replay suffice as anti-cheat verification on non-NVIDIA platforms where L3 torch.profiler is unavailable.
    Stated in Section 3.3; if a platform-specific cheat bypasses both layers, correctness numbers would be inflated.
  • ad hoc to paper The evaluated LLM/agents do not gain a decisive advantage from pretraining contamination on public ATen/vLLM/cuBLAS code.
    No contamination control or leakage analysis is reported for operator prompts drawn from public libraries.

pith-pipeline@v1.3.0-alltime-deepseek · 53 in / 13834 out tokens · 192594 ms · 2026-08-01T10:48:36.262637+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) have significantly increased the demand for efficient accelerator kernels, but kernel development remains a highly specialized and labor-intensive task. The recent rise of LLMs and agentic frameworks offers a promising pathway toward automatic kernel generation. However, despite rapid progress, there is still no comprehensive benchmark to rigorously evaluate LLM-generated kernels across diverse operator sources or heterogeneous hardware platforms. We present KernelGenBench, a unified benchmark for systematically evaluating LLM- and agent-generated Triton kernels across diverse operator sources and heterogeneous hardware platforms. It comprises two complementary sub-benchmarks: KernelGenBench-MS (Multi-Source), evaluating 210 operators from three sources beyond standard PyTorch-centric tasks, and KernelGenBench-MC (Multi-Chip), measuring performance portability across six heterogeneous hardware platforms using a 110-operator subset. Our large-scale evaluation, consuming over 15 billion tokens, shows: (1) agent-based methods consistently outperform pure LLM sampling methods, while cuBLAS operators are the most challenging across all methods; (2) generation performance varies significantly across hardware platforms, with even recent kernel-specialized agents experiencing severe cross-platform degradation (e.g., AutoKernel drops from 87% on NVIDIA to 25% on Platform E); (3) autonomous kernel generation remains highly cost-intensive, with specialized agent methods averaging 5.11 million tokens per successful operator (AKO4all reaches 5.19 million), orders of magnitude higher than simple LLM sampling approaches.

Figures

Figures reproduced from arXiv: 2607.27231 by Guang Liu, Jialing Zhang, Jian Tao, Peiyu Zang, Wentao Zhang, Yichen Yuan, Yonghua Lin.

Figure 1
Figure 1. Figure 1: Overview of KernelGenBench. The purple and blue dashed boxes represent the two [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Platform A collapses speedup to 0.18× despite high accuracy, while non-NVIDIA platforms incur up to 2× compilation overhead. Left: radar chart showing accuracy and speedup across six platforms (110 ATen, Claude Code Opus-4.6). Right: token and time overhead relative to NVIDIA. 107–123M tokens (1.27–1.46×) and 15–16 hours (1.67–1.78×). This overhead is not algorithmic; it is purely ecosystem friction from i… view at source ↗
Figure 3
Figure 3. Figure 3: Accuracy–speedup divergence: accuracy spans the full range while speedup clusters in a [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study on Opus-4.6 (NVIDIA A100, 210 operators). Execution feedback (+27.2 pp [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Side-by-side comparison of LLM Sampling and Agentic prompts for [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Per-subset ablation breakdown across four subsets (Overall, ATen, vLLM, cuBLAS). Bars [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Cross-turn similarity for 21 jointly failed opera￾tors. Reflection clusters high (local patching); No-Reflect spreads wide (global search). 24 [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 7 linked inside Pith

  1. [1]

    Wei Liu, Jiawei Xu, Yingru Li, Longtao Zheng, Tianjian Li, Qian Liu, and Junxian He. Dr. kernel: Reinforcement learning done right for triton kernel generations.CoRR, abs/2602.05885,

    arXiv

  2. [2]

    Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V . Le, and Charles Sutton. Program synthesis with large language models.CoRR, abs/2108.07732, 2021. URL https://arxiv. org/abs/2108.07732

    Pith/arXiv arXiv 2021

  3. [3]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2...

    2025

  4. [4]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R´e

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher R´e. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, Ne...

    2022

  5. [5]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors,Advances in Neural Information Processing Systems 38: Annual Confere...

    2024

  6. [6]

    Zhang, William Hu, Christopher R´e, and Azalia Mirhoseini

    Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher R´e, and Azalia Mirhoseini. Kernelbench: Can llms write efficient GPU kernels? In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors,Forty-second International Conference on Machine Learning, ICML 20...

    2025

  7. [7]

    Tritonbench: Evaluating triton kernel generation capabilities

    PyTorch Team. Tritonbench: Evaluating triton kernel generation capabilities. https://gith ub.com/meta-pytorch/tritonbench, 2025. GitHub repository

    2025

  8. [8]

    Better performance at lower occupancy

    Vasily V olkov. Better performance at lower occupancy. InProceedings of the GPU technology conference, GTC, volume 10, page 16. San Jose, CA, 2010. URL https://www.nvidia.com /content/gtc-2010/pdfs/2238_gtc2010.pdf

    2010

  9. [9]

    Philippe Tillet, Hsiang-Tsung Kung, and David D. Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Tim Mattson, Abdullah Muzahid, and Armando Solar-Lezama, editors,Proceedings of the 3rd ACM SIGPLAN International 11 Workshop on Machine Learning and Programming Languages, MAPL@PLDI 2019, Phoenix, AZ, USA, June 22,...

    arXiv 2019

  10. [10]

    Jason Ansel, Edward Z. Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael V ozne- sensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Mic...

    arXiv 2024

  11. [11]

    Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K¨opf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-p...

    2019

  12. [12]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace, editors,Proceedings of the 29th Symposium on Operating Systems P...

    arXiv 2023

  13. [13]

    Towards automated kernel generation in the era of llms, 2026

    Yang Yu, Peiyu Zang, Chi Hsu Tsai, Haiming Wu, Yixin Shen, Jialing Zhang, Haoyu Wang, Zhiyou Xiao, Jingze Shi, Yuyu Luo, Wentao Zhang, Chunlei Men, Guang Liu, and Yonghua Lin. Towards automated kernel generation in the era of llms, 2026. URL https://arxiv.or g/abs/2601.15727

    Pith/arXiv arXiv 2026

  14. [14]

    Fisches, Sahan Paliskara, Simon Guo, Alex Zhang, Joe Spisak, Chris Cummins, Hugh Leather, Gabriel Synnaeve, Joe Isaacson, Aram Markosyan, and Mark Saroufim

    Zacharias V . Fisches, Sahan Paliskara, Simon Guo, Alex Zhang, Joe Spisak, Chris Cummins, Hugh Leather, Gabriel Synnaeve, Joe Isaacson, Aram Markosyan, and Mark Saroufim. Kernel- llm: Making kernel development more accessible, 6 2025. URL https://huggingface.co /facebook/KernelLLM

    2025

  15. [15]

    Kevin: Multi-turn RL for generating CUDA kernels.CoRR, abs/2507.11948, 2025

    Carlo Baronio, Pietro Marsella, Ben Pan, Simon Guo, and Silas Alberti. Kevin: Multi-turn RL for generating CUDA kernels.CoRR, abs/2507.11948, 2025. doi: 10.48550/ARXIV.2507.1194

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.1194 2025

  16. [16]

    Incoder-32b: Code foundation model for industrial scenarios.CoRR, abs/2603.16790, 2026

    Jian Yang, Wei Zhang, Jiajun Wu, Junhang Cheng, Shawn Guo, Haowen Wang, Weicheng Gu, Yaxin Du, Joseph Li, Fanglin Xu, Yizhi Li, Lin Jing, Yuanbo Wang, Yuhan Gao, Ruihao Gong, Chuan Hao, Ran Tao, Aishan Liu, Tuney Zheng, Ganqu Cui, Zhoujun Li, Mingjie Tang, Chenghua Lin, Wayne Xin Zhao, Xianglong Liu, Ming Zhou, Bryan Dai, and Weifeng Lv. Incoder-32b: Code...

    doi:10.48550/arxiv.2603.16790 2026

  17. [17]

    CUDA agent: Large-scale agentic RL for high-performance 12 CUDA kernel generation.CoRR, abs/2602.24286, 2026

    Weinan Dai, Hanlin Wu, Qiying Yu, Huan-Ang Gao, Jiahao Li, Chengquan Jiang, Weiqiang Lou, Yufan Song, Hongli Yu, Jiaze Chen, Wei-Ying Ma, Ya-Qin Zhang, Jingjing Liu, Mingxuan Wang, Xin Liu, and Hao Zhou. CUDA agent: Large-scale agentic RL for high-performance 12 CUDA kernel generation.CoRR, abs/2602.24286, 2026. doi: 10.48550/ARXIV.2602.24286. URLhttps://...

    doi:10.48550/arxiv.2602.24286 2026

  18. [18]

    URLhttps://doi.org/10.48550/arXiv.2507.11948

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.11948

  19. [19]

    Hammond, Aram Markosyan, Aman Dontula, Simon Mahns, Zacharias V

    Alec M. Hammond, Aram Markosyan, Aman Dontula, Simon Mahns, Zacharias V . Fis- ches, Dmitrii Pedchenko, Keyur Muzumdar, Natacha Supper, Mark Saroufim, Joe Isaac- son, Laura Wang, Warren Hunt, Kaustubh Gondkar, Roman Levenstein, Gabriel Synnaeve, Richard Li, Jacob Kahn, and Ajit Mathews. Agentic operator generation for ML asics. CoRR, abs/2512.10977, 2025....

    doi:10.48550/arxiv.2512.10977 2025

  20. [20]

    KernelAgent: Hardware-Guided GPU Kernel Optimization via Multi-Agent Orchestration

    PyTorch Team. KernelAgent: Hardware-Guided GPU Kernel Optimization via Multi-Agent Orchestration. https://pytorch.org/blog/kernelagent-hardware-guided-gpu-ker nel-optimization-via-multi-agent-orchestration/ , 2025. PyTorch Blog. Accessed: 2026-05-05

    2025

  21. [21]

    Bringmann, John Tran, Wei Liu, Fung Xie, Michael Lightstone, and Humphrey Shi

    Terry Chen, Zhifan Ye, Bing Xu, Zihao Ye, Timmy Liu, Ali Hassani, Tianqi Chen, Andrew Kerr, Haicheng Wu, Yang Xu, Yu-Jung Chen, Hanfeng Chen, Aditya Kane, Ronny Krashinsky, Ming-Yu Liu, Vinod Grover, Luis Ceze, Roger A. Bringmann, John Tran, Wei Liu, Fung Xie, Michael Lightstone, and Humphrey Shi. A VO: agentic variation operators for autonomous evolution...

    doi:10.48550/arxiv.2603.24517 2026

  22. [22]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=VTF8 yNQM66

    2024

  23. [23]

    Flashinfer-bench: Building the virtuous cycle for ai-driven LLM systems.CoRR, abs/2601.00227, 2026

    Shanli Xing, Yiyan Zhai, Alexander Jiang, Yixin Dong, Yong Wu, Zihao Ye, Charlie Ruan, Yingyi Huang, Yineng Zhang, Liangsheng Yin, Aksara Bayyapu, Luis Ceze, and Tianqi Chen. Flashinfer-bench: Building the virtuous cycle for ai-driven LLM systems.CoRR, abs/2601.00227, 2026. doi: 10.48550/ARXIV.2601.00227. URL https://doi.org/10.485 50/arXiv.2601.00227

    doi:10.48550/arxiv.2601.00227 2026

  24. [24]

    Backendbench: An evaluation suite for testing how well llms and humans can write pytorch backends, 2025

    Mark Saroufim, Jiannan Wang, Bert Maher, Sahan Paliskara, Laura Wang, Shahin Sefati, and Manuel Candales. Backendbench: An evaluation suite for testing how well llms and humans can write pytorch backends, 2025. URL https://github.com/meta-pytorch/BackendBench

    2025

  25. [25]

    Multiker- nelbench: A multi-platform benchmark for kernel generation.CoRR, abs/2507.17773, 2025

    Zhongzhen Wen, Yinghui Zhang, Zhong Li, Zhongxin Liu, Linna Xie, and Tian Zhang. Multiker- nelbench: A multi-platform benchmark for kernel generation.CoRR, abs/2507.17773, 2025. doi: 10.48550/ARXIV.2507.17773. URLhttps://doi.org/10.48550/arXiv.2507.17773

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.17773 2025

  26. [26]

    Flagos/flaggems: An operator library for large language models imple- mented in the triton language., 2024

    FlagOS contributors. Flagos/flaggems: An operator library for large language models imple- mented in the triton language., 2024. URLhttps://github.com/flagos-ai/FlagGems

    2024

  27. [27]

    Sol-execbench: Speed-of-light benchmarking for real-world gpu kernels against hardware limits, 2026

    Edward Lin, Sahil Modi, Siva Kumar Sastry Hari, Qijing Huang, Zhifan Ye, Nestor Qin, Fengzhe Zhou, Yuan Zhang, Jingquan Wang, Sana Damani, Dheeraj Peri, Ouye Xie, Aditya Kane, Moshe Maor, Michael Behar, Triston Cao, Rishabh Mehta, Vartika Singh, Vikram Sharma Mailthody, Terry Chen, Zihao Ye, Hanfeng Chen, Tianqi Chen, Vinod Grover, Wei Chen, Wei Liu, Eric...

    arXiv 2026

  28. [28]

    Autokernel: Autonomous GPU kernel optimization via iterative agent-driven search.CoRR, abs/2603.21331, 2026

    Jaber Jaber and Osama Jaber. Autokernel: Autonomous GPU kernel optimization via iterative agent-driven search.CoRR, abs/2603.21331, 2026. doi: 10.48550/ARXIV.2603.21331. URL https://doi.org/10.48550/arXiv.2603.21331

    doi:10.48550/arxiv.2603.21331 2026

  29. [29]

    AKO4ALL.https://github.com/TongmingLAIC/AKO4ALL, 2026

    TongmingLAIC. AKO4ALL.https://github.com/TongmingLAIC/AKO4ALL, 2026

    2026

  30. [31]

    Tritonrl: Training llms to think and code triton without cheating.CoRR, abs/2510.17891, 2025

    Jiin Woo, Shaowei Zhu, Allen Nie, Zhen Jia, Yida Wang, and Youngsuk Park. Tritonrl: Training llms to think and code triton without cheating.CoRR, abs/2510.17891, 2025. doi: 10.48550/ARXIV.2510.17891. URLhttps://doi.org/10.48550/arXiv.2510.17891

    doi:10.48550/arxiv.2510.17891 2025

  31. [32]

    SPEC CPU 2017 Benchmark Suite

    Standard Performance Evaluation Corporation (SPEC). SPEC CPU 2017 Benchmark Suite. https://www.spec.org/cpu2017/, 2017

    2017

  32. [33]

    Fleming and John J

    Philip J. Fleming and John J. Wallace. How not to lie with statistics: The correct way to summarize benchmark results.Commun. ACM, 29(3):218–221, 1986. doi: 10.1145/5666.5673. URLhttps://doi.org/10.1145/5666.5673

    arXiv 1986

  33. [34]

    cuda-optimized-skill

    KernelFlow-ops. cuda-optimized-skill. https://github.com/KernelFlow-ops/cuda-o ptimized-skill, 2026

    2026

  34. [35]

    Cutlass: Fast linear algebra in cuda c++, 2017

    Andrew Kerr, Duane Merrill, Julien Demouth, and John Tran. Cutlass: Fast linear algebra in cuda c++, 2017. URLhttps://github.com/NVIDIA/cutlass

    2017

  35. [36]

    Gonzalez, Clark W

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zh...

    2024

  36. [37]

    Courville, and Marc G

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors,Advances in Neural Information Processing Systems 34: Annual Conference on Neur...

    2021

  37. [43]

    Operator Spec / Baseline Source Injection

    Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In8th International Conference on Learning Representations, ICLR 2020, Addis ...

    2020

  38. [44]

    Handle broadcasting correctly

  39. [45]

    Use ‘‘‘python‘‘‘ format

    Handle non-contiguous tensors correctly Generate valid Triton code directly without any explanations. Use ‘‘‘python‘‘‘ format. Agentic Framework Prompt # Triton Kernel Implementation Task Operator: add Full name: aten::add GPU ID: 0 ## Runtime Environment CUDA VISIBLE DEVICES=0 ## Operator Specification [same schema and signature as LLM prompt] ## Verific...

  40. [46]

    Write implementation to kernel.py

  41. [47]

    Run verification command

  42. [48]

    If failed, read error and fix

  43. [49]

    Shared content is abbreviated with[...]

    Repeat until all tests pass ## Performance Optimization After correctness is achieved: - Tune BLOCK SIZE, num warps, num stages - Re-run verification to confirm speedup ## Anti-Cheat Rules [same rules as LLM prompt] Figure 5: Side-by-side comparison of LLM Sampling and Agentic prompts for aten::add. Shared content is abbreviated with[...]. Table 6: Full o...

  44. [50]

    Median SortS 1, . . . , SM and take the middle value: Median= S(M+1)/2 Mis odd 1 2 SM/2 +S M/2+1 Mis even The median is immune to extreme outliers and provides a robust central tendency estimate when inter-operator speedup variance is large

  45. [51]

    Interquartile Mean (IQM) Discard the lowest 25% and highest 25% of sorted operators, then take the arithmetic mean of the middle 50%: IQM= 1 |Q3 −Q 1| Q3X i=Q1 S(i) where Q1 =⌊M/4⌋ , Q3 =⌊3M/4⌋ . IQM balances the statistical efficiency of the mean with the robustness of the median, and has been widely adopted in reinforcement learning benchmarks such as A...

  46. [2014]

    URLhttp://arxiv.org/abs/1410.0759

    Pith/arXiv arXiv

  47. [2019]

    URL https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92 f2bfa9f7012727740-Abstract.html

    2019

  48. [2021]

    URLhttps://arxiv.org/abs/2107.03374

    Pith/arXiv arXiv

  49. [2026]

    URL https://doi.org/10.48550/arXiv.260 2.05885

    doi: 10.48550/ARXIV.2602.05885. URL https://doi.org/10.48550/arXiv.260 2.05885

    doi:10.48550/arxiv.2602.05885