Pith. sign in

REVIEW 5 major objections 6 minor 22 cited by

Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An agentic pipeline of generate, evaluate, reflect, and optimize, plus parallel sampling, lets frontier LLMs produce correct Triton kernels on AMD GPUs up to 63% of the time, with average speedups up to 2.59x.

desk verdict Useful AMD-focused Triton benchmark and agent pipeline, but the abstract overstates the result: the Reflexion baseline is missing and the pass@k comparison is unequal. read the letter →

arxiv 2507.23194 v1 pith:NGELPXNQ submitted 2025-07-31 cs.CL cs.AIcs.LG

Jianghui Wang , Vinay Joshi , Saptarshi Majumder , Xu Chao , Bin Ding , Ziqiong Liu , Pratik Prabhanjan Brahma , Dong Li
show 2 more authors
Zicheng Liu Emad Barsoum
This is my paper
classification cs.CLcs.AIcs.LG
keywords TritonGPUkernelgenerationagenticcodeinference-timescalingReflexionAMDROCmbenchmarklargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a four-agent pipeline (generator, evaluator, reflector, optimizer) can turn frontier LLMs into practical Triton kernel writers for AMD GPUs by spending inference-time compute instead of training. On a repaired 184-kernel version of TritonBench and a new 30-kernel ROCm benchmark, the GEAK agent reaches up to 54.89% and 63.33% execution accuracy while averaging up to 2.59x speedup on correct kernels, where direct prompting of the same LLMs stays below about 15%. A sympathetic reader should care because the result suggests that hardware-specific kernel expertise can be automated and democratized through agentic loops plus scaling, without fine-tuning. The paper also releases both benchmarks and the agent code.

What carries the argument

The load-bearing mechanism is the inference-time scaling loop: the Evaluator runs generated code against unit tests, the Reflector turns error traces into repair instructions, the Optimizer ranks past code by measured latency and proposes faster variants, and parallel sampling with temperature set to 1 aggregates the best of ten independent runs. A 'debugging trap' guard caps the number of repair attempts per code snippet before forcing a fresh strategy, and one-shot prompting retrieves examples by code similarity. This machinery converts a single LLM call into a search process over kernel space.

What would settle it

Run the same frontier LLM baselines with the same pass@k budget of ten sequential-and-parallel samples and compare execution accuracy; then rerun the evaluation against an expanded test suite with mutation-generated and automatically generated cases to see whether the 'correct' kernels still pass.

Watch

Extended reading notes

Core claim

GEAK is an agent system that combines structured knowledge injection, one-shot retrieval of similar Triton code, a Reflexion-style reflection loop, and a dedicated optimizer LLM to iteratively repair and speed up generated kernels. The central claim is that scaling inference compute along two orthogonal axes, sequential refinement and parallel sampling, yields correct and fast Triton kernels on AMD Instinct MI300X and MI250 from minimal natural-language or reference-code task descriptions, outperforming direct prompting by a large margin. The numbers the paper emphasizes are 54.89% execution accuracy and 2.59x average speedup on TritonBench-revised, and 63.33% execution accuracy on the ROCm benchmark.

Load-bearing premise

The headline comparison assumes the baseline LLMs get a single direct prompt while GEAK gets up to ten sequential attempts and ten parallel samples (pass@10); with the same sampling budget the gap may shrink, and the paper's own disclaimer notes the TritonBench-revised tests are narrow enough that some passes may be false positives.

Editorial extensions

If this is right

  • With a fixed prompt, GEAK's execution accuracy scales roughly log-linearly in the number of parallel runs, so users can trade compute for correctness predictably.
  • Sequential refinement alone more than triples execution accuracy over zero-shot direct prompting on TritonBench-revised, from 13.04% to 44.02% at iteration 19.
  • The optimizer module contributes the main speedup, lifting average speedup to 1.45x when combined with knowledge injection and one-shot prompting; on an individual kernel, GEAK reaches 2.26x over an expert-written flip kernel.
  • Direct prompting of frontier LLMs is insufficient for AMD-targeted Triton code, with GPT-4.1 generating no valid kernels on the ROCm benchmark in the reported setup.
  • The released benchmarks, including the repaired AMD-compatible test harness and the new real-world ROCm kernel set, provide a reusable yardstick for GPU kernel synthesis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fairer side-by-side would give the baseline LLMs the same pass@k budget of ten sequential-and-parallel samples; the paper's own pass@k tables suggest the gap could narrow, though iterative error feedback may still yield an edge beyond pure sampling.
  • The agent architecture is not AMD-specific in design, so the same loop could plausibly be pointed at CUDA, oneAPI, or other accelerator DSLs, though the paper only demonstrates AMD hardware.
  • The appendix's warning that TritonBench-revised tests are narrow implies that adding mutation-based or automatically generated tests could lower the reported pass rates and should be tested before treating the numbers as general correctness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents GEAK, an agentic framework that uses frontier LLMs to generate Triton GPU kernels for AMD GPUs, with a four-module pipeline (generator, evaluator, reflector, optimizer) and inference-time compute scaling via sequential refinement and parallel sampling. It also introduces two evaluation benchmarks: a revised subset of 184 kernels from TritonBench-G and a new ROCm benchmark of 30 kernels from open-source AMD repositories. The headline results are 54.89% execution accuracy on TritonBench-revised and 63.33% on the ROCm benchmark, with an average speedup of up to 2.59X, compared with direct-prompting baselines below 15%.

Significance. If the claims hold, the paper would be a useful contribution to automated GPU-kernel synthesis: it offers an open-sourced agent implementation and evaluation framework, adapts an existing benchmark to AMD hardware, introduces a new ROCm benchmark with real-world kernels, and provides a detailed case study. The appendix also deserves credit for candidly discussing test-coverage limitations. However, the central comparative claims are currently not supported by the experiments as designed: the abstract claims outperformance over Reflexion-based pipelines without running any such baseline, and the direct-prompting comparison is not compute-matched because GEAK is credited with many more attempts.

major comments (5)
  1. [Abstract and Sections 4.2, 5.1] The abstract claims that GEAK "significantly outperformed the baselines of directly prompting frontier LLMs as well as Reflexion-based generation pipelines," but no Reflexion-based baseline is instantiated, run, or reported anywhere in Section 5. Tables 2 and 4 compare only against direct prompting. Since Section 4.2 states that GEAK is adapted from Reflexion-style feedback mechanisms, Reflexion is the natural control for isolating the contribution of the agent design. Without it, the second half of the central claim is unsupported. Please add a Reflexion control or revise the claim.
  2. [Tables 2, 3, 7] The headline 54.89% execution accuracy on TritonBench-revised is a pass@10 result obtained with 10 sequential iterations and 10 parallel runs (Table 7), while the direct-prompting baselines in Table 2 are single-attempt results. Comparing pass@10 with pass@1 is not compute-matched and can materially overstate the advantage. If the baselines were given the same pass@k budget, their accuracy would likely rise. Please report direct-prompting accuracy under the same sampling budget, or restrict the comparative claim to pass@1 settings.
  3. [Section 5.2.1, Table 5] The text states that increasing the number of iterations produces a "monotonic improvement in both call and execution accuracy," but Table 5 does not support monotonicity for call accuracy: for example, call accuracy drops from 57.61% at iter5 to 50.04% at iter6, and from 62.5% at iter8 to 56.52% at iter9. Execution accuracy also plateaus rather than increasing monotonically after early iterations. Please revise the wording to describe the observed trend without claiming monotonicity, and consider analyzing the fluctuations.
  4. [Appendix B and Abstract] Appendix B explicitly concedes that TritonBench-revised retains the original TritonBench-G test harness with limited test coverage, and warns that reported correctness figures around or above 50% may overstate true correctness. The abstract and conclusion report 54.89% and 63.33% without this caveat. The accuracy claims should be qualified as correctness on the specific unit tests used, especially for TritonBench-revised, and the limitations stated in Appendix B should appear in the main results discussion.
  5. [Section 3.2] The ROCm Triton Benchmark was constructed by refactoring kernels and unit tests with assistance from frontier LLMs, and the kernels originate from the authors' employer's ecosystem. The paper does not specify which LLMs were used, what prompts were used, or what human review was performed. If the same model family used by GEAK (e.g., GPT-4.1) was involved in creating the benchmark, the evaluation could be biased in favor of the framework. Please document the refactoring protocol, including model names, prompts, and human verification, or otherwise address this contamination risk.
minor comments (6)
  1. [Table 3] The "Correctly generated kernels" column appears inconsistent with the reported execution accuracy: difficulty 1 shows "2 / 3" but accuracy 66.67 / 100.0, and difficulty 5 shows "1 / 1" but accuracy 20.00 / 20.00. Please clarify how these counts and percentages are computed.
  2. [Section 5.1] The sentence beginning "Table2 shows the direct prompting results on the ROCm benchmark" is imprecise because Table 2 also reports results on the TritonBench-modified benchmark; please rephrase.
  3. [Throughout] The terminology is inconsistent: the paper uses "TritonBench-revised," "TritonBench-modified," and "TritonBench-Modified" to refer to the same benchmark. Please standardize.
  4. [Sections 3.1 and B] Appendix B states that TritonBench-revised "retains the same test harness from TritonBench-G unchanged," but Section 3.1 describes substantive changes including adding missing test-function calls, using tolerance-based tensor comparison, and setting consistent seeds. Please reconcile these two descriptions.
  5. [Abstract and Section 5.1] The phrase "significantly outperformed" is used without any statistical significance testing or confidence intervals; given the small per-difficulty counts and the pass@k methodology, please either add uncertainty estimates or use a less strong formulation.
  6. [Table 5] The table caption does not say whether these numbers are pass@1 or include multiple parallel runs; please clarify the exact evaluation protocol for the sequential-scaling experiment.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports empirical benchmark measurements against external unit tests and reference kernels, and its acknowledged test-coverage limitations do not reduce the central claim to its inputs by construction.

full rationale

This is an empirical systems paper rather than a derivation, so the circularity burden is low. The central claims are benchmark measurements: execution accuracy is the fraction of generated kernels that pass pre-existing unit tests (Section 3.3), and speedup is the ratio of measured median latencies against reference kernels. These are not derived from the method's own assumptions; they are externally observable outcomes. The agent's feedback loop does use the same functionality tests as both refinement signal and evaluation oracle, which means the reported pass rates are best understood as in-distribution fitting quality rather than held-out generalization. However, the paper explicitly acknowledges this in Appendix B: 'this limited coverage may falsely inflate apparent correctness if code is overfitted only to the narrow test cases.' That is an evaluation limitation, not a circular derivation: the final accuracy is an empirical result that could have been lower, and the paper does not claim to derive it from the benchmark definition. There are no load-bearing self-citations, no imported uniqueness theorems, no renamed known results, and no fitted parameter being relabeled as a prediction. The abstract's comparison to a Reflexion baseline is unsupported because no such baseline is run, but that is a missing baseline rather than a circularity. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced. The free parameters are compute-budget hyperparameters that directly affect the reported accuracy and speedup numbers. The axioms are benchmark-validity assumptions that the paper itself partially undermines in Appendix B.

free parameters (4)
  • Sequential refinement iterations = 10 (main result), 19 (sequential scaling)
    Execution accuracy rises with iteration count (Table 5); the main pass@10 result uses 10 sequential iterations.
  • Parallel generation count = 10
    Pass@10 accuracy in Tables 6-8; accuracy scales roughly log-linearly with parallel runs.
  • LLM sampling temperature = 1.0
    Set to maximize diversity across parallel runs; this affects the pass@k results.
  • max_perf_debug_num = not reported
    Limits reflection attempts per code snippet to avoid debugging traps; the actual value is not disclosed.
assumptions (3)
  • domain assumption Passing the revised benchmark's unit tests is a faithful measure of kernel correctness.
    Appendix B acknowledges TritonBench-G has at most six unit tests per kernel and that narrow coverage may inflate pass rates. Section 3.1 claims fixes, but Appendix B says the harness is unchanged.
  • domain assumption ROCm reference kernels and their refactored tests are correct ground truth.
    The authors state the kernels were written by Triton expert engineers, but LLM refactoring could introduce errors into references or tests, and no independent verification is described.
  • domain assumption Latency ratios are stable enough to compare speedups without repeated measurement statistics.
    Speedups are computed from median latency ratios, but no variance, GPU clock control, or measurement methodology is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks." pith.science (2026). https://pith.science/paper/NGELPXNQ

@misc{pith2026250723194,
  author       = {Pith},
  title        = {Pith review of: Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NGELPXNQ}},
  note         = {Machine review of arXiv:2507.23194}
}
abstract

The demand for AI-generated GPU kernels is rapidly growing, influenced by the need for scalable, hardware-optimized solutions in both industry and academia. As deep learning workloads grow in complexity and diversity, it is imperative to automate low-level kernel development to meet performance and productivity demands. Major cloud providers, semiconductor companies, and research institutions are now investing heavily in AI-driven code generation for GPUs, aiming to reduce manual optimization efforts while achieving near-expert performance on hardware like AMD MI300X. The Triton language, a Python-based DSL for GPU programming, has emerged as a popular target for such AI-generated kernels due to its balance of performance and ease-of-coding. In this work, we present an evaluation suite for Triton-based GPU kernels and GEAK (Generating Efficient AI-centric GPU Kernels)-a framework that leverages cutting-edge LLMs to generate performant Triton code specifically for AMD GPUs, including the AMD MI300X and MI250. GEAK leverages inference-time compute scaling to produce Triton-based GPU kernels using a reasoning loop adapted from Reflexion-style feedback mechanisms. On two evaluation benchmarks, GEAK significantly outperformed the baselines of directly prompting frontier LLMs as well as Reflexion-based generation pipelines by achieving correctness up to $63$% and execution speed up of up to $2.59$X. These results highlight the promise of GEAK-like agentic code generation for accelerating the adoption of diverse hardware platforms and democratizing access to expert-level kernel performance.

Figures

Figures reproduced from arXiv: 2507.23194 by the authors.

Figure 1
Figure 1. Illustration of GEAK pipeline. we release two benchmark suites for evaluating AI-generated Triton kernels, enabling measurement of both execution correctness and runtime performance of generated kernels. Key Contributions: 1. We introduce GEAK framework, a modular, agent-based system that uses inference-time compute scaling with frontier LLMs to generate Triton GPU kernels from minimal task descriptions. GEAK incorp… view at source ↗
Figure 2
Figure 2. Inference time compute scaling study on TritonBench-revised benchmark. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Inference time compute scaling study on ROCm benchmark. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FastKernels: Benchmarking GPU Kernel Generation in Production

    cs.LG 2026-05 conditional novelty 8.0 of 10

    FastKernels is a production-aligned benchmark covering 96.2% of HuggingFace Transformers that reveals state-of-the-art kernel agents deliver at most 0.94x aggregate speedup.

  2. The Correctness Illusion in LLM-Generated GPU Kernels

    cs.SE 2026-06 accept novelty 7.0 of 10

    Controlled corpus testing shows that fixed allclose oracles in LLM kernel benchmarks certify transcription-buggy kernels as correct while seeded fuzzing with fp64 references does not.

  3. PassNet: Scaling Large Language Models for Graph Compiler Pass Generation

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    PassNet provides a dataset of 18K graphs and PassBench for LLM-generated compiler passes, with fine-tuned models achieving 2.67x gains on long-tail tasks where TorchInductor underperforms.

  4. From Human Guidance to Autonomy: Agent Skill System for End-to-End LLM Deployment on Spatial NPUs

    cs.LG 2026-05 conditional novelty 7.0 of 10

    A two-stage agent skill system enables autonomous end-to-end deployment of eight decoder-only LLMs on AMD XDNA 2 NPU with numerical correctness in 0.5-4 hours each, generalizing from a human-guided Llama-3.2-1B reference.

  5. AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    AgentKernelArena is a new open benchmark that measures complete AI agent workflows on 196 GPU kernel tasks with correctness, performance, and generalization checks to unseen configurations.

  6. KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    KernelBench-X benchmark shows task category predicts LLM kernel correctness better than method choice, iterative refinement trades performance for higher success rates, and correctness does not ensure efficiency gains...

  7. KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels

    cs.LG 2026-05 conditional novelty 7.0 of 10

    KernelBenchX benchmark shows task category explains nearly three times more variance in LLM kernel correctness than method choice, iterative refinement boosts correctness but reduces performance, and quantization rema...

  8. Kerncap: Automated Kernel Extraction and Isolation for AMD GPUs

    cs.SE 2026-05 conditional novelty 7.0 of 10

    Kerncap automatically extracts isolated, reproducible GPU kernels from large HIP and Triton applications on AMD GPUs by capturing HSA dispatches and producing self-contained reproducer projects that preserve virtual-a...

  9. Kerncap: Automated Kernel Extraction and Isolation for AMD GPUs

    cs.SE 2026-05 conditional novelty 7.0 of 10

    Kerncap automates extraction of faithful, self-contained GPU kernel reproducers from AMD HIP and Triton workloads via HSA interception and address-space closure, delivering 13.6x faster isolated tuning.

  10. AtumAI: A Principled Framework for Agentic Generation of Datacenter Control-Plane Policies

    cs.AI 2026-08 conditional novelty 6.0 of 10

    AtumAI compiles plain-language goals into a formal search problem and uses an LLM-plus-evolution loop to generate datacenter control-plane policies that beat hand-tuned baselines in simulation.

  11. CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits

    cs.AI 2026-07 conditional novelty 6.0 of 10

    CANN Bench is an Ascend-NPU benchmark with 53 operators and 1,060 released test cases that scores agent-generated kernels on compilation, correctness, and hardware-anchored performance.

  12. Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs

    cs.SE 2026-06 conditional novelty 6.0 of 10

    Boundary-only shape sampling is the safe default for tensor-kernel testing (78% recall, 0% false positives); regular 'representative' shapes miss all softmax tail-mask bugs.

  13. Kernel Forge: An Agent Harness for LLM-based Generation and Optimization of CUDA Kernels

    cs.AI 2026-06 conditional novelty 6.0 of 10

    A new open-source harness uses an LLM and MCTS to generate CUDA kernels for unmodified PyTorch models, achieving operator-level speedups up to 2.83x over eager mode with guarded fallback.

  14. Learning When to Optimize: Verified Optimization Skills from Expert GPU-Kernel Lineages

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    KLineage derives verified optimization skills from backward lineages of expert GPU kernels to guide LLM agents toward higher-quality and more efficient kernels than memory-based baselines.

  15. Characterizing Deep Research: A Benchmark and Formal Definition

    cs.CL 2025-08 conditional novelty 6.0 of 10

    Deep research is characterized by high search and reasoning intensity; the new LiveDRBench measures claim-level precision and recall, where the best current model scores 0.55 F1.

  16. Static PTX Metrics Track Structural Kernel Regressions but Miss Semantic Ones

    cs.DC 2026-06 conditional novelty 5.0 of 10

    For this Triton corpus, static PTX deltas separate structural from semantic kernel changes portably across five GPU classes while measured runtime deltas do not.

  17. Static PTX Metrics Track Structural Kernel Regressions but Miss Semantic Ones

    cs.DC 2026-06 conditional novelty 5.0 of 10

    A nine-pair corpus shows static PTX instruction/register deltas are identical across five GPU classes and flag structural bugs, but constant-only semantic bugs produce byte-identical PTX and zero static delta.

  18. Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs

    cs.SE 2026-06 unverdicted novelty 5.0 of 10

    Boundary shape sampling for tensor kernel testing achieves 78% recall on seeded bugs with 0% false positives on correct kernels, while adversarial value sampling reaches 99% recall at the cost of 94% false positives.

  19. The Correctness Illusion in LLM-Generated GPU Kernels

    cs.SE 2026-06 conditional novelty 5.0 of 10

    Fixed-shape allclose checks in LLM kernel benchmarks can certify seeded buggy Triton kernels as correct; a schema-aware fp64 fuzzer catches all seeded bugs on five GPUs and passes all correct controls.

  20. AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization

    cs.LG 2026-03 unverdicted novelty 5.0 of 10

    AscendOptimizer combines kernel rewinding for reusable experience with evolutionary search on hardware feedback to optimize Ascend NPU operators, delivering 1.21x geometric-mean speedup and faster performance on 53.47...

  21. Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels

    cs.LG 2026-06 conditional novelty 4.0 of 10

    Per-operator tolerances calibrated from the correct kernel's own error distribution catch 229 more seeded bugs on the gpuemu corpus at the cost of 20 false positives.

  22. Towards Automated Kernel Generation in the Era of LLMs

    cs.LG 2026-01 conditional novelty 4.0 of 10

    A structured survey of LLM-based and agentic approaches for GPU kernel generation, plus a catalog of datasets and benchmarks for the field.

Reference graph

Works this paper leans on

21 extracted references · 15 canonical work pages · cited by 17 Pith papers

  1. [1]

    Tritonbench: Benchmarking large language model capabilities for generating triton operators, 2025

    Jianling Li, Shangzhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, Haojie Wang, Jianrong Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. Tritonbench: Benchmarking large language model capabilities for generating triton operators, 2025. 10

    work page 2025

  2. [2]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021

    2021

  3. [3]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023

    2023

  4. [4]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024

    2024

  5. [5]

    Learning performance-improving code edits, 2024

    Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. Learning performance-improving code edits, 2024

    work page 2024

  6. [6]

    Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M. Zhang. Effibench: Benchmarking the efficiency of automatically generated code, 2025

    work page 2025

  7. [7]

    How efficient is llm-generated code? a rigorous & high-standard benchmark, 2025

    Ruizhong Qiu, Weiliang Will Zeng, James Ezick, Christopher Lott, and Hanghang Tong. How efficient is llm-generated code? a rigorous & high-standard benchmark, 2025

    work page 2025

  8. [8]

    Davis, Zhaojun Xie, Arjun Rajaram, and Abhinav Bhatele

    Daniel Nichols, Joshua H. Davis, Zhaojun Xie, Arjun Rajaram, and Abhinav Bhatele. Can large language models write parallel code? In Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing, HPDC ’24, page 281–294. ACM, June 2024

    work page 2024

Show all 21 references
  1. [9]

    Hpc-coder-v2: Studying code llms across low-resource parallel languages, 2024

    Aman Chaturvedi, Daniel Nichols, Siddharth Singh, and Abhinav Bhatele. Hpc-coder-v2: Studying code llms across low-resource parallel languages, 2024

    2024

  2. [10]

    Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini

    Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. Kernelbench: Can llms write efficient gpu kernels?, 2025

    2025

  3. [11]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y . Wu, Y . K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024

    2024

  4. [12]

    Qwen2.5- coder technical report, 2024

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou...

    2024

  5. [13]

    Reflexion: Language agents with verbal reinforcement learning, 2023

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023

    2023

  6. [14]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V . Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers, 2024. A Case study: Flip kernel We studied kernel test_triton_f lip.py, from ROCm Triton Benchmark, which achieved 2.26x speedup over Triton exper...

    2024

  7. [15]

    load ( X + off2d ) # Load entire block 2 x = tl

    Double Memory Access Pattern: 1 x = tl . load ( X + off2d ) # Load entire block 2 x = tl . flip ( x ) # Flip in r eg is ter s 3 tl . store ( Z + off2d , x ) # Store entire block • Loads the entire (N×M) block into registers • Performs flip operation on register data • Stores t...

  8. [16]

    Register Pressure: • Must hold the entire block in registers simultaneously • For large blocks, this can exceed register capacity • May cause register spilling to local memory

  9. [17]

    Limited Flexibility: • ‘tl.flip()‘ behavior may not be optimally tuned for all tensor shapes • Less control over the exact memory access pattern GEAK generated code advantages:

  10. [18]

    load ( x_ptrs , mask = mask , other =0) 5 tl

    Optimized Memory Access Pattern: 1 f l i p p e d _ c o l s = M - 1 - cols 2 x_ptrs = X + rows [: , None ] * M + f l i p p e d _ c o l s [ None , :] # Read from flipped p osi ti on s 3 z_ptrs = Z + rows [: , None ] * M + cols [ None , :] # Write to normal po si ti on s 4 vals =...

  11. [19]

    Better Memory Efficiency: • Lower register usage since it doesn’t need to hold entire blocks • Reduced memory bandwidth requirements • Better cache utilization due to direct addressing 12

  12. [20]

    Explicit Masking: 1 mask = mask_row [: , None ] & mask_col [ None , :] • Handles boundary conditions explicitly • Prevents out-of-bounds memory accesses • More robust for arbitrary tensor sizes

  13. [21]

    Evaluating AI-generated code must be paired with broad and well-crafted unit tests

    Coalesced Memory Access: • The addressing pattern ‘rows[:, None] * M + cols[None, :]‘ maintains good memory coalescing • Sequential threads access nearby memory locations Performance Implications: • Memory Bandwidth: GEAK generated kernel uses ∼ 50% less memory bandwidth (sing...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.