Pith. sign in

REVIEW 10 cited by

TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.14752 v1 pith:EFOFE74E submitted 2025-02-20 cs.CL cs.LG

Jianling Li , Shangzhan Li , Zhenye Gao , Qi Shi , Yuxuan Li , Zefan Wang , Jiacheng Huang , Haojie Wang
show 4 more authors
Jianrong Wang Xu Han Zhiyuan Liu Maosong Sun
This is my paper
classification cs.CLcs.LG
keywords tritontritonbenchcodeoperatorsgenerationlanguagealignedconventional
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Triton, a high-level Python-like language designed for building efficient GPU kernels, is widely adopted in deep learning frameworks due to its portability, flexibility, and accessibility. However, programming and parallel optimization still require considerable trial and error from Triton developers. Despite advances in large language models (LLMs) for conventional code generation, these models struggle to generate accurate, performance-optimized Triton code, as they lack awareness of its specifications and the complexities of GPU programming. More critically, there is an urgent need for systematic evaluations tailored to Triton. In this work, we introduce TritonBench, the first comprehensive benchmark for Triton operator generation. TritonBench features two evaluation channels: a curated set of 184 real-world operators from GitHub and a collection of operators aligned with PyTorch interfaces. Unlike conventional code benchmarks prioritizing functional correctness, TritonBench also profiles efficiency performance on widely deployed GPUs aligned with industry applications. Our study reveals that current state-of-the-art code LLMs struggle to generate efficient Triton operators, highlighting a significant gap in high-performance code generation. TritonBench will be available at https://github.com/thunlp/TritonBench.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FastKernels: Benchmarking GPU Kernel Generation in Production

    cs.LG 2026-05 conditional novelty 8.0 of 10

    FastKernels is a production-aligned benchmark covering 96.2% of HuggingFace Transformers that reveals state-of-the-art kernel agents deliver at most 0.94x aggregate speedup.

  2. PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A profiler-guided, verifier-in-the-loop workflow more than doubles the rate at which an off-the-shelf LLM agent matches human-expert speedups on two repository-level code-optimization benchmarks.

  3. CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits

    cs.AI 2026-07 conditional novelty 6.0 of 10

    CANN Bench is an Ascend-NPU benchmark with 53 operators and 1,060 released test cases that scores agent-generated kernels on compilation, correctness, and hardware-anchored performance.

  4. NKI-Agent: Domain-Specific Fine-Tuning and Agentic Tool Use for Neuron Kernel Generation

    cs.LG 2026-07 conditional novelty 6.0 of 10

    An agent with compile/verify tools reaches 77.3% NKI kernel pass rate on real Trn1 hardware with Opus 4.8, versus 6% single-shot; SFT Qwen3-30B hits 25% at ~1/100th cost, and binary-reward GRPO fails to beat SFT.

  5. Learning When to Optimize: Verified Optimization Skills from Expert GPU-Kernel Lineages

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    KLineage derives verified optimization skills from backward lineages of expert GPU kernels to guide LLM agents toward higher-quality and more efficient kernels than memory-based baselines.

  6. AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    AdaExplore improves correctness and speed of Triton kernel generation by converting recurring failures into a memory of rules and organizing search as a tree that mixes local refinements with larger regenerations, yie...

  7. InCoder-32B-Thinking: Industrial Code World Model for Thinking

    cs.AR 2026-04 unverdicted novelty 6.0 of 10

    InCoder-32B-Thinking uses error-feedback synthesized thinking traces and a code world model to reach top open-source scores on general and industrial code benchmarks including 81.3% on LiveCodeBench and 84.0% on CAD-Coder.

  8. SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation

    cs.SE 2025-07 conditional novelty 6.0 of 10

    All 18 evaluated LLMs pass fewer SIMD-intrinsic code-generation tests than scalar-code tests on the new SimdBench benchmark, with the largest drops on SVE and RVV.

  9. MultiKernelBench: A Multi-Platform Benchmark for Kernel Generation

    cs.DC 2025-07 conditional novelty 6.0 of 10

    MultiKernelBench is a 285-task, 14-category benchmark for LLM-generated deep learning kernels on three hardware platforms, with evidence that category-matched one-shot prompts improve accuracy on under-represented platforms.

  10. AutoTriton: Automatic Triton Programming with Reinforcement Learning in LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    An 8B model trained with SFT and GRPO reinforcement learning generates Triton kernels that match or exceed frontier model performance on TritonBench and KernelBench correctness metrics.

Pith tools