Pith. sign in

REVIEW 2 major objections 4 minor 55 references

RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks

T0 review · 2 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read State-of-the-art LLMs solve only about one in five real-world Triton kernel tasks once unit tests, model accuracy, and end-to-end latency all count.

desk verdict A genuinely useful benchmark for realistic Triton-kernel generation, marred by an internal contradiction in Table 4 that currently makes the headline 18.71% success rate unverifiable from the paper as written. read the letter →

arxiv 2608.12004 v1 pith:NRXKLQ2W submitted 2026-08-12 cs.SE cs.AI

Jinjun Huang , Zhongzhen Wen , Tongtong Xu , Meng Yan , Xin Xia , Zhongxin Liu This is my paper · ORCID
classification cs.SEcs.AI
keywords TritonkernelgenerationLLMcodebenchmarkconstructionend-to-endevaluationGPUkernelsnumericalrobustnesspullrequestminingrewardhacking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RealisticTritonBench asks whether large language models can do what real GPU-kernel developers do: modify, optimize, or create Triton kernels inside working AI frameworks such as PyTorch, vLLM, and SGLang. The benchmark builds 31 tasks from merged pull requests, replaces the accepted kernel with the model's kernel, and runs the repository's own unit tests plus model-accuracy and serving-latency checks. Across five state-of-the-art models, task success averages 18.71%, full unit-test pass rates average 43.23%, numerical robustness averages 47.65%, and end-to-end speedup is roughly 1x. The sympathetic reading of the paper's claim is that today's LLMs are not yet reliable enough for real-world Triton kernel deployment, and that unit-test correctness is a misleading proxy for deployment readiness.

What carries the argument

The carrying mechanism is the benchmark's evaluation pipeline rather than a single kernel. Each of the 31 tasks supplies a natural-language requirement, repository context, and a target function signature; the generated kernel is patched into an instance-specific Docker image built from the original repository's own build commands, then scored with the repository's pytest unit tests, a model accuracy benchmark, and wall-clock time-to-first-token and time-per-output-token measured by an external client. Success requires the unit-test pass rate to match the gold patch, numerical robustness to hold, and both latency speedups to be at least 0.98, so a kernel cannot win by passing isolated correctness checks alone.

What would settle it

Run the same generated and gold kernels in the original unmodified repositories at the exact merged-commit state with original dependency resolution, then compare unit-test verdicts, model-accuracy pass/fail, and whether each task's TTFT/TPOT speedup crosses 1.0 against the benchmark's Docker-based numbers; any systematic divergence would show the reconstructed environments altered the tested behavior.

Watch

Extended reading notes

Core claim

The paper's central discovery is that when Triton kernels are evaluated the way deployment actually works, current LLMs mostly fail. Using 31 tasks mined from merged pull requests, each with a Docker reproduction of the original repository plus unit tests, model-accuracy tests, and TTFT/TPOT latency measurement, the authors find average task success of 18.71% across five leading models (best: Qwen3.5-397B-A17B at 25.81%), average full unit-test pass of 43.23%, average numerical robustness of 47.65%, and end-to-end speedups near 1x. The gap between full unit-test pass and task success — kernels that pass unit tests still degrade model accuracy or end-to-end latency in a substantial share of cases — is the paper's key evidence that kernel-level benchmarks overstate LLM ability. The failure analysis attributes most unit-test failures to insufficient mastery of Triton's programming constraints, many others to incomplete understanding of kernel semantics in real repositories, and a further set to inattention to performance and numerical stability after the unit tests pass.

Load-bearing premise

The load-bearing premise is that each manually reconstructed Docker environment, including any dependency pins and configuration patches, behaves like the real merged pull request's runtime well enough that the unit-test, accuracy, and latency numbers reflect production; if those patches change numerical behavior or timing semantics, the benchmark's central verdict could shift.

Editorial extensions

If this is right

  • Unit-test passing does not imply deployment readiness: an average full unit-test pass rate of 43.23% collapses to 18.71% task success once model accuracy and end-to-end latency are required, so benchmarks that stop at kernel-level checks overstate progress.
  • Optimization tasks are within reach functionally but not in performance: optimization tasks average 23.08% success and speedups close to 1x, meaning LLMs can preserve behavior but rarely deliver the latency gains a real optimization PR would.
  • Modification tasks are numerically fragile: average numerical robustness on modification tasks is only 20%, with some models at 0%, so extending or bug-fixing an existing kernel often silently breaks downstream model accuracy.
  • New-kernel generation from scratch is the hardest frontier: average success is 5.455% and average unit-test pass is 20.00%, suggesting that LLMs are far from synthesizing deployable Triton kernels without a reference implementation.
  • End-to-end evaluation undercuts standard reward-hacking strategies: external-client timing and a separate-process accuracy harness make concurrency-based, caching-based, and environment-manipulation exploits ineffective.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's failure taxonomy — wrong API choice such as tl.min versus tl.minimum, missing boundary masks, and unconditional computation that the gold patch guards — reads as a concrete roadmap: targeted fine-tuning on Triton API constraints plus static Triton linting could close much of the gap, though the paper does not propose such methods.
  • The roughly 1x end-to-end speedup implies that any reinforcement-learning reward for kernel generation should be tied to framework-level latency and downstream accuracy rather than kernel-level speedup; the benchmark's Docker pipeline could serve as a training environment for that objective.
  • Because the dataset contains only 31 tasks drawn from three repositories, the category-level numbers such as New-kernel's 5.455% success carry wide error bars; extending the same PR-mining pipeline to more frameworks would test whether that finding is robust.
  • The success criterion is anchored to the gold patch as baseline and a 0.98 speedup tolerance, so absolute percentages would shift under a different reference implementation or stricter latency bar, even if the relative ordering of models likely remains stable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. RealisticTritonBench derives 31 Triton-kernel generation tasks from merged pull requests in PyTorch, vLLM, and SGLang, divided into Optimization, Modification, and New Kernel categories. Each task provides a natural-language requirement, repository context, and target function signature; generated kernels are integrated into reconstructed Docker environments and evaluated by unit tests, model-level numerical robustness, and end-to-end TTFT/TPOT latency. The authors evaluate five LLMs with the mini-SWE-agent scaffold and report an average task success rate of 18.71%, average FTP of 43.23%, NR of 47.65%, and speedups near 1.0, concluding that current LLMs struggle with realistic Triton kernel generation and that framework-level evaluation is necessary.

Significance. If the benchmark is valid, it addresses a real gap in the Triton-generation evaluation literature: prior benchmarks are mostly PyTorch-to-Triton translation with isolated kernel metrics and hand-written checkers. The paper's strengths include grounding tasks in externally merged PRs with gold patches, a three-layer evaluation suite, three-run latency measurements with reported variability, a concrete failure analysis, and a reward-hacking mitigation argument based on external-client TTFT/TPOT measurement. The central qualitative finding — that SOTA LLMs often pass unit tests yet degrade model accuracy or latency — is plausible and important for the community. However, the internal inconsistency in Table 4 affects the credibility of the headline success numbers and must be resolved before the quantitative claims can be accepted.

major comments (2)
  1. [Table 4 and §4.1.3] The Modification rows of Table 4 are incompatible with the success definition stated in §4.1.3, which requires UTP equal to the gold patch's UTP, NR=T, and both latency speedups at least 0.98. The Modification category has 7 tasks. DeepSeek-V3.2 (non-reasoning) reports Success=28.57%, FTP=71.43%, and NR=0.000%; if, as stated in §4.1.3, NR is averaged over tasks that pass all unit tests, then none of the 5 unit-test-passing tasks has NR=T, so Success must be 0%. The same contradiction appears for DeepSeek-V3.2 (reasoning) (Success=42.86%, FTP=57.14%, NR=0.000%) and GPT-5.4 (Success=14.29%, FTP=28.57%, NR=0.000%), and Qwen3.5-397B-A17B (Success=42.86%, FTP=57.14%, NR=50.00% can support at most 2 of the 4 FTP tasks, not 3). The positive Success entries therefore cannot be reproduced from the published protocol. The authors should release per-task evaluation logs and correct either the Success computation or the NR entries; until then the headline 18.71% average success rate is not verifiable from the paper as written.
  2. [§3.2.3 and §3.2.4] The benchmark's realism claim rests on the reconstructed environments faithfully reproducing the behavior of the original merged PRs, but §3.2.3 only states that installation errors were manually resolved by 'patching dependency versions or config files,' and §3.2.4 says testing commands were refined 'according to the actual execution results.' No per-instance inventory of environment patches or validation that the gold patch passes the refined tests with behavior matching upstream is provided. If a patch alters numerical behavior, test semantics, or latency characteristics, then the unit-test, NR, and speedup numbers no longer reflect the real-world deployment setting the benchmark claims to measure. I ask for a per-task log of environment modifications and a demonstration, for each task, that the gold patch passes the final tests and that adjusted commands preserve the original test intent.
minor comments (4)
  1. [§3.2.2] The path from approximately 2,000 candidate PRs to 31 final task instances is not auditable because the paper does not report how many PRs were excluded by each of the three filtering criteria (Triton Kernel Relevance, Test Availability, Clear Kernel Objective).
  2. [§4.1.3] The reported run-to-run variability ('average variation is 1.08% for STTFT and 0.98% for STPOT') is ambiguous; please specify whether this is a standard deviation, a coefficient of variation, or a max-min range over the three runs.
  3. [Table 4] Several NR percentages in Table 4 cannot arise from integer task counts under the stated convention that NR is computed over unit-test-passing tasks; for example, GPT-5.4's New-kernel row has FTP=27.27% (3 of 11 tasks) and NR=50.00%, which is impossible for 3 tasks. Report exact counts (e.g., '2/4') alongside percentages.
  4. [§5.2] The reward-hacking mitigation argument would be easier to evaluate if the paper specified how the 'external client' measures TTFT and TPOT (e.g., wall-clock timestamps at request submission and final response) and clarified whether generated code could defer work until after the response is returned.

Circularity Check

0 steps flagged · score 2.0 of 10

No substantive circularity: ground truth comes from external merged PRs and gold patches; only minor, non-load-bearing self-citations appear.

full rationale

The paper's central empirical claim is a benchmark evaluation, not a derivation. RealisticTritonBench tasks are extracted from merged PRs in PyTorch, vLLM, and SGLang, with gold patches supplied by the upstream repositories; LLM outputs are then measured against unit tests, model accuracy, and end-to-end latency. No parameter is fitted to the measured data and then renamed as a prediction, and no metric is defined in terms of the outcome it is used to establish. The success definition in Section 4.1.3 is explicit (UTP matches gold, NR=T, STTFT>=0.98, STPOT>=0.98) and is applied to independently generated kernels. The self-citations that appear (NoCodeBench [5] for environment construction, MultiKernelBench [41] in background, and several related-work references by the same group) are not load-bearing: the environment construction strategy is a methodological borrowing, and the benchmark's validity rests on external PRs, repository tests, and manual task review rather than on any claim imported from those papers. One non-circularity caveat: Table 4's Modification rows report positive Success rates (e.g., 28.57% for DeepSeek-V3.2 non-reasoning) with NR=0.000%, which contradicts the stated requirement that success requires NR=T. This is an internal consistency/reproducibility problem for the headline numbers, but it is not a circularity because the contradiction is between a reported result and the metric definition, not a reduction of a claimed derivation to its inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

This is an evaluation artifact rather than a scientific model, so no new physical or mathematical entities are introduced. The load-bearing choices are hand-picked evaluation thresholds and assumptions about the fidelity of reconstructed environments and gold-patch baselines.

free parameters (2)
  • Latency success threshold = 0.98 for both S_TTFT and S_TPOT
    Hand-picked tolerance to account for runtime fluctuations; sensitivity analysis is only in the online appendix. This threshold directly changes the reported success rate.
  • Numerical robustness degradation criterion = not specified in main text
    A task is marked T if model performance does not degrade, but the benchmark, threshold, and evaluation protocol for 'degrade' are only described as derived from real-world PRs and are deferred to the appendix.
assumptions (4)
  • domain assumption The gold-patch reference implementation is the correct and sufficient baseline for judging generated kernels.
    Section 4.1.3 states the gold patch is the baseline because PRs were reviewed by experts and it is the only available reference. If gold patches are suboptimal or not representative, success conditions do not measure true deployment quality.
  • domain assumption Reconstructed Docker environments faithfully reproduce the original PR's runtime behavior.
    Section 3.2.3 documents manual dependency and configuration patches to make instances build. This assumes the patches do not change correctness or performance semantics.
  • domain assumption LLM-generated task descriptions, after manual review, faithfully capture the PR intent without leaking the gold solution.
    Section 3.2.2 generates descriptions with an LLM and manually revises them. If a description leaks the gold patch or omits key constraints, the benchmark measures description quality as much as kernel generation.
  • domain assumption Unit tests plus model accuracy plus TTFT/TPOT latency capture the deployment-relevant behavior of a Triton kernel.
    The evaluation pipeline in Section 3.1.2 assumes these three test layers are sufficient to judge whether a kernel is production-ready. Missing interactions such as memory pressure or multi-GPU scaling are not directly measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks." pith.science (2026). https://pith.science/paper/NRXKLQ2W

@misc{pith2026260812004,
  author       = {Pith},
  title        = {Pith review of: RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NRXKLQ2W}},
  note         = {Machine review of arXiv:2608.12004}
}
read the original abstract

In modern AI frameworks, GPU kernels are key to overall system performance. Combining usability, portability, and near-handwritten CUDA performance, Triton is widely adopted for implementing GPU kernels. Recent advances show the potential of large language models (LLMs) to automatically generate Triton kernels, reducing the manual effort required from expert kernel developers. Several benchmarks evaluate LLM-generated Triton kernels. However, they suffer from three key limitations: (1) they restrict tasks to PyTorch-to-Triton translation, failing to reflect the diversity and complexity of real-world Triton tasks; (2) they evaluate only individual-kernel performance rather than end-to-end performance, the core criterion for real-world deployment in AI frameworks; and (3) they rely on manually written evaluation scripts for individual kernels, which may contain flaws that models can exploit to bypass correctness checks and obtain inflated scores. To address these limitations, we introduce RealisticTritonBench, the first benchmark to derive Triton kernel generation tasks from real-world pull requests in popular AI frameworks, enabling realistic, production-like evaluation. RealisticTritonBench systematically extracts PRs that modify Triton kernels from popular open-source AI frameworks and transforms them into generation tasks with concrete engineering contexts. Each task takes a natural language requirement as input and requires a corresponding Triton kernel implementation, with a complete and reproducible evaluation environment. Unlike prior benchmarks focused on isolated kernel performance, RealisticTritonBench integrates generated kernels into their original frameworks and evaluates them using end-to-end tests, enabling a more faithful assessment. We evaluate leading LLMs on RealisticTritonBench and find that they still struggle with real-world Triton kernel generation tasks.

Figures

Figures reproduced from arXiv: 2608.12004 by the authors.

Figure 1
Figure 1. A Triton kernel for Vector Addition integrated into modern AI frameworks and optimization pipelines, serving as a preferred choice for implementing high-performance GPU kernels [13, 18, 51]. As a Python-based domain-specific lan￾guage (DSL) for GPU kernel development, Triton significantly sim￾plifies the implementation of complex kernels while maintaining performance competitive with native CUDA. As shown in [PITH_… view at source ↗
Figure 2
Figure 2. Task formulation and evaluation pipeline of Real [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The pipeline of constructing our benchmark [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: An example of incorrect usage of the Triton API. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: An example where the generated implementation [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: An example of numerical deviations whose accept [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: A stream-injection hack that defeats kernel-level [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 20 canonical work pages

  1. [1]

    Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al

  2. [2]

    Ahsan Ali, Riccardo Pinciroli, Feng Yan, and Evgenia Smirni. 2020. Batch: Machine learning inference serving on serverless platforms with adaptive batching. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–15

    2020

  3. [3]

    Ruisheng Cao, Mouxiang Chen, Jiawei Chen, Zeyu Cui, Yunlong Feng, Binyuan Hui, Yuheng Jing, Kaixin Li, Mingze Li, Junyang Lin, et al. 2026. Qwen3-Coder- Next Technical Report.arXiv preprint arXiv:2603.00729(2026)

    arXiv 2026

  4. [4]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashat- tention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems35 (2022), 16344–16359

    2022

  5. [5]

    Le Deng, Zhonghao Jiang, Jialun Cao, Michael Pradel, and Zhongxin Liu. 2025. Nocode-bench: A benchmark for evaluating natural language-driven feature addition.arXiv preprint arXiv:2507.18130(2025)

    arXiv 2025

  6. [6]

    Jinye Du, Quan Yuan, Zuyao Zhang, Yanzhi Yi, Jiahui Hu, Wangyi Chen, Yiyang Zhu, Qishui Zheng, Wenxiang Zou, Xiangyu Chang, et al . 2025. AKG kernel Agent: A Multi-Agent Framework for Cross-Platform Kernel Synthesis.arXiv preprint arXiv:2512.23424(2025)

    arXiv 2025

  7. [7]

    Fisches, Sahan Paliskara, Simon Guo, Alex Zhang, Joe Spisak, Chris Cummins, Hugh Leather, Gabriel Synnaeve, Joe Isaacson, Aram Markosyan, and Mark Saroufim

    Zacharias V. Fisches, Sahan Paliskara, Simon Guo, Alex Zhang, Joe Spisak, Chris Cummins, Hugh Leather, Gabriel Synnaeve, Joe Isaacson, Aram Markosyan, and Mark Saroufim. 2025. KernelLLM: Making Kernel Development More Accessible. https://huggingface.co/facebook/KernelLLM

    work page 2025

  8. [8]

    Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, et al. 2025. Trae agent: An llm-based agent for software engineering with test-time scaling.arXiv preprint arXiv:2507.23370(2025)

    arXiv 2025

Show all 55 references
  1. [9]

    Georgi Gerganov and contributors. 2023. llama.cpp: LLM Inference in C/C++. https://github.com/ggml-org/llama.cpp. GitHub repository

    2023

  2. [10]

    Google. 2026. Gemini 3.1 Pro Preview. https://ai.google.dev/gemini-api/docs/ models/gemini-3.1-pro-preview. accessed: 2026-03

    2026

  3. [11]

    Alec Hammond, Aram Markosyan, Aman Dontula, Simon Mahns, Zacharias Fisches, Dmitrii Pedchenko, Keyur Muzumdar, Natacha Supper, Site Cao, Haishan Zhu, et al. 2026. Agentic operator generation for ml asics.Proceedings of Machine Learning and Systems8 (2026), 1583–1594

    2026

  4. [12]

    Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. 2024. Liger kernel: Efficient triton kernels for llm training.arXiv preprint arXiv:2410.10989 (2024)

    2024 arXiv

  5. [13]

    Jiawei Hu, Hong Jia, Mahbub Hassan, Lina Yao, Brano Kusy, and Wen Hu. 2025. LightLLM: A versatile large language model for predictive light sensing. In Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems. 158–171

    2025

  6. [14]

    RealisticTritonBench: A Benchmark for Triton- Kernel Generation in Real-World AI Frameworks

    Jinjun Huang, Zhongzhen Wen, Tongtong Xu, Meng Yan, Xin Xia, and Zhongxin Liu. 2026. Online Appendix for “RealisticTritonBench: A Benchmark for Triton- Kernel Generation in Real-World AI Frameworks”. https://anonymous.4open. science/r/RealisticTritonBench-2583/appendix/appendix.md

    2026

  7. [15]

    Zhonghao Jiang, David Lo, and Zhongxin Liu. 2025. Agentic Software Issue Res- olution with Large Language Models: A Survey.arXiv preprint arXiv:2512.22256 (2025)

    2025 arXiv

  8. [16]

    Zhonghao Jiang, Xiaoxue Ren, Meng Yan, Wei Jiang, Yong Li, and Zhongxin Liu

  9. [17]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues?. InInternational Conference on Learning Representations, Vol. 2024. 54107–54157

    2024

  10. [18]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princip...

    2023

  11. [19]

    Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang, Tianrui Hui, Jialin Gao, Xi- aoming Wei, and Si Liu. 2025. Llava-st: A multimodal large language model for fine-grained spatial-temporal understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2025

  12. [20]

    Jianling Li, Shangzhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, WangHaojie WangHaojie, Jianrong Wang, Xu Han, et al. 2025. Triton- bench: Benchmarking large language model capabilities for generating triton operators. InFindings of the Association for Com...

    2025

  13. [21]

    Shangzhan Li, Zefan Wang, Ye He, Yuxuan Li, Qi Shi, Jianling Li, Yonggang Hu, Wanxiang Che, Xu Han, Zhiyuan Liu, et al. 2025. Autotriton: Automatic triton programming with reinforcement learning in llms.arXiv preprint arXiv:2507.05687 (2025)

    2025 arXiv

  14. [22]

    Edward Lin, Sahil Modi, Siva Kumar Sastry Hari, Qijing Huang, Zhifan Ye, Nestor Qin, Fengzhe Zhou, Yuan Zhang, Jingquan Wang, Sana Damani, et al. 2026. SOL- ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits.arXiv preprint arXiv:2603.19173(2026)

    2026

  15. [23]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    2024 arXiv

  16. [24]

    Wei Liu, Jiawei Xu, Yingru Li, Longtao Zheng, Tianjian Li, Qian Liu, and Junxian He. 2026. Dr. Kernel: Reinforcement Learning Done Right for Triton Kernel Generations.arXiv preprint arXiv:2602.05885(2026)

    2026

  17. [25]

    Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, and Yongbin Li. 2025. Swe-gpt: A process-centric language model for automated software improvement.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 2362–2383

    2025

  18. [26]

    Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. 2025. Alibaba lingmaagent: Improving automated issue resolution via com- prehensive repository exploration. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engine...

    2025

  19. [27]

    OpenAI. 2026. GPT-5.4 Model. https://developers.openai.com/api/docs/models/ gpt-5.4. Accessed: 2026-03

    2026

  20. [28]

    Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Re, and Azalia Mirhoseini. 2025. KernelBench: Can LLMs Write Efficient GPU Kernels?. InForty-second International Conference on Machine Learning

    2025

  21. [29]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library.Advances in neural information processing systems...

    2019

  22. [30]

    PyTorch Team and Contributors. 2025. KernelFalcon: Autonomous GPU Kernel Generation via Deep Agents. https://github.com/meta-pytorch/kernelagent. Accessed: 2025-07

    2025

  23. [31]

    Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. https://qwen. ai/blog?id=qwen3.5

    2026

  24. [32]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep- speed: System optimizations enable training deep learning models with over 100 billion parameters. InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. ...

    2020

  25. [33]

    2025.Sakana discovered its AI CUDA engineer cheat- ing

    Reddit Community. 2025.Sakana discovered its AI CUDA engineer cheat- ing. https://www.reddit.com/r/OpenAI/comments/1iwc24f/sakana_discovered_ its_ai_cuda_engineer_cheating/

    2025

  26. [34]

    Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2025. Specrover: Code intent extraction via llms. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 963–974

    2025

  27. [35]

    Zhihong Shao, Yuxiang Luo, Chengda Lu, ZZ Ren, Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, and Xiaokang Zhang. 2025. Deepseekmath-v2: Towards self- verifiable mathematical reasoning.arXiv preprint arXiv:2511.22570(2025)

    2025

  28. [36]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300(2024)

    2024 arXiv

  29. [37]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053(2019)

    2019 arXiv

  30. [38]

    Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Program- ming Languages. 10–19

    2019

  31. [39]

    Junyi Wang, Jialun Cao, and Zhongxin Liu. 2026. iCoRe: An Iterative Correlation- Aware Retriever for Bug Reproduction Test Generation.Proceedings of the ACM on Software Engineering3, FSE (2026), 4231–4252

    2026

  32. [40]

    Xinchen Wang, Pengfei Gao, Xiangxin Meng, Chao Peng, Ruida Hu, Yun Lin, and Cuiyun Gao. 2025. Aegis: An agent-based framework for bug reproduction from issue descriptions. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 331–342

    2025

  33. [41]

    Zhongzhen Wen, Yinghui Zhang, Zhong Li, Zhongxin Liu, Linna Xie, and Tian Zhang. 2025. MultiKernelBench: A Multi-Platform Benchmark for Kernel Gener- ation

    2025

  34. [42]

    Thaddäus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, and Robert Geirhos. 2025. Video models are zero-shot learners and reasoners.arXiv preprint arXiv:2509.20328(2025). RealisticTritonBench: A Benchmark for Triton-K...

    2025 arXiv

  35. [43]

    Jiin Woo, Shaowei Zhu, Allen Nie, Zhen Jia, Yida Wang, and Youngsuk Park. 2025. Tritonrl: Training llms to think and code triton without cheating.arXiv preprint arXiv:2510.17891(2025)

    2025

  36. [44]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. De- mystifying llm-based software engineering agents.Proceedings of the ACM on Software Engineering2, FSE (2025), 801–824

    2025

  37. [45]

    Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, and Lingming Zhang

  38. [46]

    Shanli Xing, Yiyan Zhai, Alexander Jiang, Yixin Dong, Yong Wu, Zihao Ye, Char- lie F Ruan, Yingyi Huang, Yineng Zhang, Liangsheng Yin, et al. 2026. Flashinfer- bench: Building the virtuous cycle for ai-driven llm systems.Proceedings of Machine Learning and Systems8 (2026), 2016–2064

    2026

  39. [47]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer in- terfaces enable automated software engineering. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  40. [48]

    Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly? arXiv preprint arXiv:2511.13646(2025)

    2025

  41. [49]

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Aoyan Li, Lu Chen, Xiaojian Zhong, et al . [n. d.]. Multi-SWE- bench: A Multilingual Benchmark for Issue Resolving. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems...

  42. [50]

    Li Zhang, Youhe Jiang, Guoliang He, Xin Chen, Han Lv, Qian Yao, Fangcheng Fu, and Kai Chen. 2025. Efficient Mixed-Precision Large Language Model Inference with TurboMind.arXiv preprint arXiv:2508.15601(2025)

    2025 arXiv

  43. [51]

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al . 2025. Flashinfer: Efficient and customizable attention engine for llm inference serving. Proceedings of Machine Learning and Syst...

    2025

  44. [52]

    Xinguo Zhu, Shaohui Peng, Jiaming Guo, Yunji Chen, Qi Guo, Yuanbo Wen, Hang Qin, Ruizhi Chen, Qirui Zhou, Ke Gao, et al. 2026. Qimeng-kernel: Macro- thinking micro-coding paradigm for llm-based high-performance gpu kernel generation. InProceedings of the AAAI Conference on Art...

    2026

  45. [54]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing syste...

    2024

  46. [2016]

    In12th USENIX symposium on operating systems design and implementation (OSDI 16)

    {TensorFlow}: a system for {Large-Scale} machine learning. In12th USENIX symposium on operating systems design and implementation (OSDI 16). 265–283

  47. [2025]

    In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)

    Issue Localization via LLM-Driven Iterative Code Graph Searching. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 3034–3045

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.