Pith. sign in

REVIEW 2 major objections 5 minor 43 references

An LLM-driven agent harness that captures operators from unmodified PyTorch models can generate CUDA kernels that beat PyTorch eager mode on 14 operators within 50 optimization iterations each, reaching a 2.83× speedup on a softmax variant

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 12:25 UTC pith:EGUTUORF

load-bearing objection A well-scoped, honest systems paper whose headline win count is undermined by missing error bars and best-of-N selection; deserves peer review with revision. the 2 major comments →

arxiv 2607.24762 v1 pith:EGUTUORF submitted 2026-06-02 cs.AI cs.PF

Kernel Forge: An Agent Harness for LLM-based Generation and Optimization of CUDA Kernels

Joshua Brodsky , Dhravid Kumar , Savini Kashmira , Jayanaka Danatanarayana , Jason Mars , Krisztian Flautner , Lingjia Tang This is my paper
classification cs.AI cs.PF
keywords CUDA kernel generationLLM agentsMonte Carlo Tree SearchPyTorchoperator captureguarded dispatchGPU kernel optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that LLM-driven CUDA kernel generation can work inside real PyTorch workloads, not just on standalone benchmarks. Kernel Forge captures the operators that actually run when an unmodified model executes, groups them into concrete variants, and uses an LLM agent guided by Monte Carlo Tree Search to write and refine CUDA kernels for those variants. Within 50 optimization iterations per kernel, the system beats PyTorch eager mode on 14 captured operators across a vision, a diffusion, and two language models, with speedups from 1.05× to 2.83×. Crucially, the paper also argues that the wins are mostly on operators with small runtime share, while dominant operators usually remain on the PyTorch eager path, so the value of the system lies as much in knowing when not to replace a kernel as in generating fast ones.

Core claim

Kernel Forge claims to be an end-to-end replacement, not just a kernel-writing aid: it runs an unmodified PyTorch model on sample inputs, records the operators that execute, groups calls into concrete variants with captured shapes, dtypes, arguments, and reference outputs, then uses an LLM agent guided by Monte Carlo Tree Search to write and refine CUDA kernels. In 50 optimization iterations per variant, it reports 14 variants that run faster than the PyTorch eager path, with per-operator speedups of 1.05–2.83× across ResNet-50, Stable Diffusion 3.5 Medium, Gemma 4 E2B, and Qwen 3.5 35B-A3B. It also makes a second, more cautionary claim: the large speedups land on operators with small runtim

What carries the argument

The central mechanism is the operator card: a captured, workload-specific record of an operator family and its concrete variants (shapes, dtypes, layouts, arguments, call counts, eager latency, reference outputs). Around that card, the paper builds a loop: an LLM generator writes the first CUDA candidate; a validator compiles, launches, and checks it against PyTorch eager outputs on captured inputs; a benchmarker compares latencies; and a Monte Carlo Tree Search controller keeps a revision tree, propagating best-subtree latency and selecting the next parent to revise. The MCTS tree lets the search revisit temporarily slower candidates. The final piece is guarded dispatch: an exported package

Load-bearing premise

The whole evidence base rests on the assumption that the user-provided sample inputs are representative enough that kernels validated on them will stay correct and fast under real deployment shapes and memory conditions.

What would settle it

Take an exported package, run it against inputs outside the captured variant set (different sequence length, batch size, tensor shape, or memory layout), and compare outputs to PyTorch eager with the same tolerance; if correctness fails or the speedup disappears on those inputs, the guarded dispatch claim is falsified for that deployment.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • With a fixed budget of 50 optimization iterations per kernel, the harness finds faster-than-eager CUDA for 14 captured variants across four model families, so per-operator optimization cost is bounded.
  • For every operator where generated CUDA loses to eager, the guarded package falls back, so the captured operator region is protected from regressions.
  • Wins concentrate on open-source/native PyTorch operator paths; vendor-backed linear, convolution, and attention paths are rarely beaten, so replacing those kernels is not the target.
  • Because top speedups are on small-runtime-share operators, operator-region speedups should not be read as end-to-end model speedups; end-to-end numbers are not claimed by the paper.
  • Larger fixed search budgets are poorly aligned with runtime impact: opt50 adds cost on operators that may already have strong baselines, so budget-allocation policy matters.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the harness were instead given a variable budget that stops early on low-runtime-share operators and spends more on dominant native operators, the measured operator-region benefit per dollar would rise; the paper points toward this but does not implement it.
  • Editorial inference: because validation is confined to recorded example inputs, deployment safety depends on the gap between sample inputs and real traffic; a natural extension is re-capturing and re-validating at deployment time or adding shape-aware guards.
  • Editorial inference: the combination of MCTS with an LLM generator could be tested as a general search strategy for other code-evolution tasks, but this paper's evidence is specific to CUDA kernels on one GPU.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents Kernel Forge, an open-source agentic harness that captures PyTorch operator execution from unmodified models, groups calls into workload-specific variants, generates CUDA candidates via an LLM, validates them against eager outputs, and searches over candidate revisions with MCTS. Optimized kernels are packaged with guarded dispatch that falls back to eager when a candidate is not faster on the captured variant. The evaluation covers ResNet-50, Stable Diffusion 3.5 Medium, and two LLMs on a DGX Spark, reporting that 14 operator-level kernels exceed PyTorch eager at the opt50 budget, with the largest speedup being 2.83x on Gemma softmax. The paper also reports API costs across search arms.

Significance. If the quantitative claims hold, Kernel Forge is a useful contribution to the growing area of LLM-based kernel optimization. Its strengths are the end-to-end integration with real PyTorch models, the explicit handling of workload-specific operator variants, the MCTS search over multiple refinement paths, the audit labels that separate generated CUDA from wrappers/fallbacks, and the guarded-dispatch packaging that avoids regressions on dominant operators. The paper is also careful in scoping its claims to the captured operator region rather than claiming end-to-end speedups. The main weakness is that the central empirical claim—14 kernels beating eager—rests on operator-level latency measurements that are never characterized statistically, allowing best-of-N selection over noisy timings to create apparent wins.

major comments (2)
  1. [§3.4, Eq. (2), §3.5, Figures 3–6] The paper's headline claim of 14 opt50 wins is not protected against measurement noise. Section 3.4 selects candidates using the lowest measured latency (Eq. 2), and §3.5 admits any candidate whose measured latency is below the measured eager latency. The evaluation never reports repeats, standard deviations, or confidence intervals for these operator-level timings; Table 1's warmup/timed-batch policy is described for full-model runs, not for per-operator variant benchmarking. Several reported wins are within typical benchmark noise: relu 1.004x, embedding 1.009x, layer_norm 1.049x, silu 1.052x. With many noisy candidate measurements, selecting the minimum can produce apparent speedups at parity. The authors should report repeated measurements with confidence intervals or significance tests, and either substantiate or remove wins whose effect size is below the noise floor. This directly
  2. [§4.1, Table 1, §4.3] The per-operator benchmarking protocol is under-specified. Table 1 gives a timing policy for model-level contexts (20 warmup, 100 timed), but the text does not say whether the same policy applies to the operator-card benchmarks used in Figures 3–6, how many repetitions were taken for each candidate and eager baseline, whether measurements were interleaved to control for thermal/clock drift, or how outliers were handled. Without this detail, the reader cannot judge whether the reported 1.0x-level differences are real. Please specify the exact measurement procedure for operator-level latency, including repetition counts, warmup, stream synchronization, and any outlier filtering.
minor comments (5)
  1. [Abstract] The phrase 'optimizes 14 kernels to outperform PyTorch eager mode' is easily read as an end-to-end claim. Since the paper's evidence is operator-region only and several of the wins are on small-runtime operators, the abstract should say 'operator-level' or 'on captured operator variants' to avoid overstatement.
  2. [§3.5, §4.3] The distinction between 'open-source/native' and 'proprietary/vendor-backed' paths is important, but the definitions are not given. Please state explicitly which operators are classified as wrapper/backend (e.g., those dispatching to cuDNN/cuBLAS/ATen) and confirm that all 'generated-CUDA wins' are custom CUDA rather than wrapper paths.
  3. [Figures 3–6] The speedup labels are sometimes hard to read (e.g., 0.021x in Figure 4). Consider using a log scale or a table with exact values, especially for the near-1.0x results that the statistical analysis must address.
  4. [Table 1] The 'Captured calls' column should clarify whether it counts operator invocations or grouped variants. For ResNet-50, 2,800 calls is plausible, but the mapping to the displayed operator families (and to the 14-win count) is not explicit.
  5. [Abstract] The code availability line 'The code is available at: Kernel Forge' does not include a URL or repository identifier. Please add a link or footnote.

Circularity Check

0 steps flagged

No circular derivation: Kernel Forge's speedups are direct empirical measurements, with explicit workload scoping and no load-bearing self-citations.

full rationale

Kernel Forge's central claims are empirical system measurements rather than a derivation from assumptions. The opt50 speedups in §4.3 are obtained by compiling, validating, and benchmarking generated CUDA candidates against the PyTorch eager path for captured operator variants; no fitted parameter is subsequently renamed as a prediction. The MCTS configuration (C=1.0, α anneal from 0.5 to 0.3, Eq. 1–2) is a search policy, not a constant tuned to reproduce the reported speedups. The guarded export policy (§3.5) is an integration safety rule, and the paper explicitly distinguishes generated-CUDA wins from wrapper/backend/fallback paths. Citations to related systems are external (KernelBench, Astra, CudaForge, AutoComp, GEAK, etc.); no load-bearing argument depends on a self-citation or on a 'uniqueness theorem' imported from the authors' prior work. The paper itself repeatedly scopes results to the captured operator region and recorded example inputs, and acknowledges that dominant operators often fall back to eager. The remaining concern raised by a skeptical reader — that best-of-N selection over noisy per-operator timings can overstate wins — is a statistical-validity / benchmarking-practice issue, not a circularity of definition or a fitted-input-as-prediction reduction. Under the stated circularity criteria, no specific step reduces a claimed result to its own inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on the representativeness of captured operator variants (from four specific input sets), on PyTorch eager as a stable baseline, and on workload-specific validation being sufficient for guarded dispatch. The only tuned numeric constants are MCTS configuration values (C=1.0, alpha annealing schedule), which are chosen by the authors rather than fitted to the reported speedups; the 50-iteration budget is a reporting choice. No new physical or conceptual entities are introduced.

free parameters (4)
  • MCTS exploration constant C = 1.0
    Used in the UCT-style child score (Eq. 2); chosen by configuration, not fitted to reported results.
  • Progressive widening annealing alpha = 0.5 to 0.3 over first 1000 root visits
    Eq. 1; anneals branching factor; configuration choice.
  • Numerical validation tolerance = Not specified in paper
    Configurable per operator/dtype; without the value, correctness thresholds are unstated.
  • Optimization budget = 50 iterations per kernel
    Chosen budget for opt50 arms; affects results and cost.
axioms (4)
  • domain assumption Captured operator variants are representative of the workload's runtime behavior and deployment inputs
    The entire operator-region evaluation and guarded dispatch rely on the traced calls from example inputs (ImageNetV2, T2I-CompBench, MT-Bench/etc.) being representative. §4.1, Table 1.
  • domain assumption PyTorch eager latency is a stable and fair baseline
    All speedups are ratios against eager timings; no variance is reported. §4.2.
  • domain assumption Numerical validation on recorded examples suffices to certify replacement kernels
    The paper explicitly calls this a workload-specific filter, not a correctness proof (§3.3); guarded dispatch depends on it.
  • domain assumption LLM-generated kernels can be specialized to CUDA without delegating to libraries
    The generator prompt asks for custom CUDA; the system audits wrapper/backend paths, so this is an assumption about the LLM's output distribution. §3.2.

pith-pipeline@v1.3.0-alltime-deepseek · 14682 in / 13686 out tokens · 123794 ms · 2026-08-02T12:25:48.062726+00:00 · methodology

0 comments
read the original abstract

Machine learning models are increasingly embedded in everyday software, and most of their runtime is spent in a small set of compute kernels such as matrix multiplication, convolution, and normalization. Optimizing these kernels is one of the most direct ways to reduce latency and cost, but it has traditionally required expert engineers to hand-write low-level GPU code. Agentic systems built on large language models (LLMs) can now generate and optimize kernels with far less human effort, yet existing tools are largely evaluated on randomly generated tensors and isolated kernels, emit standalone CUDA code that developers must manually reintegrate, mostly target only LLM PyTorch models, and offer limited support for inspecting and debugging results. We present Kernel Forge, an open-source, end-to-end agentic harness that accepts any unmodified PyTorch model in place. Kernel Forge supports vision, diffusion, and LLM workloads, uses Monte Carlo Tree Search (MCTS) to explore multiple optimization paths rather than a single linear refinement chain, and ships with a graphical user interface for monitoring progress, inspecting candidate kernels, and debugging failures. We evaluate Kernel Forge on four PyTorch models spanning vision, diffusion, and LLM workloads on an NVIDIA DGX Spark with GB10 GPU. With only 50 optimization iterations per kernel, it optimizes 14 kernels to outperform PyTorch eager mode, reaching $1.52\times$ on adaptive\_avgpool2d in ResNet-50, $1.70\times$ on group\_norm in Stable Diffusion 3.5 Medium, $2.83\times$ on softmax in Gemma 4 E2B, and $1.54\times$ on softmax in Qwen 3.5 35B-A3B.

Figures

Figures reproduced from arXiv: 2607.24762 by Dhravid Kumar, Jason Mars, Jayanaka Danatanarayana, Joshua Brodsky, Krisztian Flautner, Lingjia Tang, Savini Kashmira.

Figure 1
Figure 1. Figure 1: Kernel Forge graphical interface. The numbered screenshots show the main workflow: (1) the project launcher lists saved optimization projects; (2) project creation lets users load models and weights directly from Hugging Face, configure the target backend, and provide validation inputs without modifying model code; (3) the operator workbench shows captured kernels, their runtime prominence, and their optim… view at source ↗
Figure 3
Figure 3. Figure 3: ResNet-50 opt50 operator-level results. Bars report [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Stable Diffusion 3.5 Medium opt50 operator-level [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Gemma 4 E2B opt50 operator-level results. Bars [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qwen 3.5 35B-A3B opt50 operator-level results. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 9
Figure 9. Figure 9: Incremental LLM API cost by optimization arm [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Cumulative LLM API cost by optimization arm [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 3 linked inside Pith

  1. [1]

    Efficient processing of deep neural networks: A tutorial and survey,

    V. Sze, Y. H. Chen, T. J. Yang, and J. S. Emer, “Efficient processing of deep neural networks: A tutorial and survey, ”Proceedings of the IEEE, vol. 105, no. 12, pp. 2295–2329, 2017

    2017

  2. [2]

    In-datacenter performance analysis of a tensor processing unit,

    N. P. Jouppiet al., “In-datacenter performance analysis of a tensor processing unit, ” inProceedings of the International Symposium on Computer Architecture, 2017, pp. 1–12

    2017

  3. [3]

    A systematic characterization of LLM inference on GPUs,

    H. Wang, X. Xiao, M. Yan, Z. Zhu, D. Han, D. Wang, W. Li, X. Ye, C. Hu, H. Chen, and G. Sun, “A systematic characterization of LLM inference on GPUs, ”arXiv preprint arXiv:2512.01644, 2025

    arXiv 2025

  4. [4]

    Triton: An intermediate language and compiler for tiled neural network computations,

    P. Tillet, H. T. Kung, and D. Cox, “Triton: An intermediate language and compiler for tiled neural network computations, ” inACM SIG- PLAN International Workshop on Machine Learning and Programming Languages, 2019

    2019

  5. [5]

    FlashAttention: Fast and memory-efficient exact attention with IO-awareness,

    T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Re, “FlashAttention: Fast and memory-efficient exact attention with IO-awareness, ” in Advances in Neural Information Processing Systems, 2022

    2022

  6. [6]

    Autocomp: A powerful and portable code optimizer for tensor accelerators,

    C. Hong, S. Bhatia, A. Cheung, and Y. S. Shao, “Autocomp: A powerful and portable code optimizer for tensor accelerators, ” arXiv preprint arXiv:2505.18574, 2025. [Online]. Available: https: //arxiv.org/abs/2505.18574

    arXiv 2025

  7. [7]

    GEAK: Introducing Triton kernel AI agent and evaluation benchmarks,

    J. Wanget al., “GEAK: Introducing Triton kernel AI agent and evaluation benchmarks, ”arXiv preprint arXiv:2507.23194, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    KernelBench: Can LLMs write efficient GPU kernels?

    A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. Re, and A. Mirhoseini, “KernelBench: Can LLMs write efficient GPU kernels?” inInternational Conference on Machine Learning, 2025

    2025

  9. [9]

    Kernel- BenchX: A comprehensive benchmark for evaluating LLM-generated GPU kernels,

    H. Wang, J. Zhang, K. Jiang, H. Wang, J. Chen, and J. Zhu, “Kernel- BenchX: A comprehensive benchmark for evaluating LLM-generated GPU kernels, ”arXiv preprint arXiv:2605.04956, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    TritonBench: Benchmarking large language model capabilities for generating Triton operators,

    J. Liet al., “TritonBench: Benchmarking large language model capabilities for generating Triton operators, ” inFindings of the Association for Computational Linguistics, 2025. 10

    2025

  11. [11]

    MultiKernelBench: A multi-platform benchmark for kernel generation,

    Z. Wenet al., “MultiKernelBench: A multi-platform benchmark for kernel generation, ”arXiv preprint arXiv:2507.17773, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    CUDABench: Benchmarking LLMs for text-to-CUDA generation,

    J. Zhuet al., “CUDABench: Benchmarking LLMs for text-to-CUDA generation, ”arXiv preprint arXiv:2603.02236, 2026

    arXiv 2026

  13. [13]

    BackendBench: An evaluation suite for testing how well LLMs and humans can write PyTorch backends,

    M. Saroufim, J. Wang, B. Maher, S. Paliskara, L. Wang, S. Sefati, and M. Candales, “BackendBench: An evaluation suite for testing how well LLMs and humans can write PyTorch backends, ” https: //github.com/meta-pytorch/BackendBench, 2025, accessed: 2026-05- 22

    2025

  14. [14]

    FlashInfer- Bench: Building the virtuous cycle for AI-driven LLM systems,

    S. Xing, Y. Zhai, A. Jiang, Y. Dong, Y. Wu, Z. Ye, C. Ruan, Y. Huang, Y. Zhang, L. Yin, A. Bayyapu, L. Ceze, and T. Chen, “FlashInfer- Bench: Building the virtuous cycle for AI-driven LLM systems, ”arXiv preprint arXiv:2601.00227, 2026

    arXiv 2026

  15. [15]

    CudaForge: An agent framework with hardware feedback for CUDA kernel optimization,

    Z. Zhang, R. Wang, S. Li, Y. Luo, M. Hong, and C. Ding, “CudaForge: An agent framework with hardware feedback for CUDA kernel optimization, ”arXiv preprint arXiv:2511.01884, 2025

    arXiv 2025

  16. [16]

    Astra: A multi-agent system for GPU kernel performance optimization,

    A. Wei, T. Sun, Y. Seenichamy, H. Song, A. Ouyang, A. Mirhoseini, K. Wang, and A. Aiken, “Astra: A multi-agent system for GPU kernel performance optimization, ”arXiv preprint arXiv:2509.07506, 2025

    arXiv 2025

  17. [17]

    Towards robust agentic CUDA kernel benchmarking, verification, and optimization,

    R. T. Lange, Q. Sun, A. Prasad, M. Faldor, Y. Tang, and D. Ha, “Towards robust agentic CUDA kernel benchmarking, verification, and optimization, ”arXiv preprint arXiv:2509.14279, 2025

    arXiv 2025

  18. [18]

    KernelAgent: Multi-agent GPU kernel synthesis and optimization,

    Meta PyTorch, “KernelAgent: Multi-agent GPU kernel synthesis and optimization, ” https://github.com/meta-pytorch/KernelAgent, 2025, accessed: 2026-05-22

    2025

  19. [19]

    Complete anytime beam search,

    W. Zhang, “Complete anytime beam search, ” inProceedings of the Fifteenth National Conference on Artificial Intelligence. AAAI Press, 1998, pp. 425–430

    1998

  20. [20]

    Bandit based Monte-Carlo planning,

    L. Kocsis and C. Szepesvari, “Bandit based Monte-Carlo planning, ” inEuropean Conference on Machine Learning, 2006

    2006

  21. [21]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition, ” inIEEE Conference on Computer Vision and Pattern Recognition, 2016

    2016

  22. [22]

    Stable diffusion 3.5 medium model card,

    Stability AI, “Stable diffusion 3.5 medium model card, ” Hugging Face, 2024, [Online]. Available: https://huggingface.co/stabilityai/stable-dif fusion-3.5-medium

    2024

  23. [23]

    google/gemma-4-E2B-it model card,

    Google, “google/gemma-4-E2B-it model card, ” Hugging Face, 2026, [Online]. Available: https://huggingface.co/google/gemma-4-E2B-it; accessed 2026-05-22

    2026

  24. [24]

    Qwen/Qwen3.5-35B-A3B model card,

    Qwen, “Qwen/Qwen3.5-35B-A3B model card, ” Hugging Face, 2026, [Online]. Available: https://huggingface.co/Qwen/Qwen3.5-35B-A3B; accessed 2026-05-22

    2026

  25. [25]

    DGX Spark user guide: Hardware overview,

    NVIDIA, “DGX Spark user guide: Hardware overview, ” https://docs.n vidia.com/dgx/dgx-spark/hardware.html, 2026, accessed: 2026-05-22

    2026

  26. [26]

    PyTorch: An imperative style, high-performance deep learning library,

    A. Paszkeet al., “PyTorch: An imperative style, high-performance deep learning library, ” inAdvances in Neural Information Processing Systems, 2019

    2019

  27. [27]

    CUDA programming guide,

    NVIDIA Corporation, “CUDA programming guide, ” https://docs.nvi dia.com/cuda/cuda-programming-guide/, 2026, accessed: 2026-05-21

    2026

  28. [28]

    CUDA C++ best practices guide,

    ——, “CUDA C++ best practices guide, ” https://docs.nvidia.com/cuda /cuda-c-best-practices-guide/, 2026, accessed: 2026-05-21

    2026

  29. [29]

    Tvm: an automated end-to-end optimizing compiler for deep learning,

    T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, M. Cowan, H. Shen, L. Wang, Y. Hu, L. Ceze, C. Guestrin, and A. Krishnamurthy, “Tvm: an automated end-to-end optimizing compiler for deep learning, ” inProceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’18. USA: USENIX Association, 2018, p. 579–594

    2018

  30. [30]

    Learning to optimize tensor programs,

    T. Chenet al., “Learning to optimize tensor programs, ” inAdvances in Neural Information Processing Systems, 2018

    2018

  31. [31]

    Ansor: Generating high-performance tensor programs for deep learning,

    L. Zhenget al., “Ansor: Generating high-performance tensor programs for deep learning, ” inUSENIX Symposium on Operating Systems Design and Implementation, 2020

    2020

  32. [32]

    torchvision.models.resnet50,

    PyTorch, “torchvision.models.resnet50, ” TorchVision documen- tation, 2026, [Online]. Available: https://docs.pytorch.org/vision/ma in/models/generated/torchvision.models.resnet50.html; accessed 2026-05-22

    2026

  33. [33]

    Do ImageNet classifiers generalize to ImageNet?

    B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, “Do ImageNet classifiers generalize to ImageNet?” inProceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 97, 2019, pp. 5389–5400

    2019

  34. [34]

    Diffusers: State-of-the-art diffusion models,

    P. von Platen, S. Patil, A. Lozhkov, P. Cuenca, N. Lambert, K. Rasul, M. Davaadorj, D. Nair, S. Paul, W. Berman, Y. Xu, S. Liu, and T. Wolf, “Diffusers: State-of-the-art diffusion models, ” https://github.com/hug gingface/diffusers, 2022

    2022

  35. [35]

    T2I-CompBench: A comprehensive benchmark for open-world compositional text-to-image generation,

    K. Huanget al., “T2I-CompBench: A comprehensive benchmark for open-world compositional text-to-image generation, ” inAdvances in Neural Information Processing Systems Datasets and Benchmarks Track, 2023

    2023

  36. [36]

    Judging LLM-as-a-judge with MT-Bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-Bench and chatbot arena, ”Advances in Neural Information Processing Systems, vol. 36, 2023

    2023

  37. [37]

    ShareGPT Prompts Annotated,

    lewtun, “ShareGPT Prompts Annotated, ” https://huggingface.co/datas ets/lewtun/sharegpt_prompts_annotated, 2023, accessed: 2026-05-22

    2023

  38. [38]

    LongBench: A bilingual, multitask benchmark for long context understanding,

    Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li, “LongBench: A bilingual, multitask benchmark for long context understanding, ” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024, pp. 3119–3137

    2024

  39. [39]

    Introducing Claude Opus 4.7,

    Anthropic, “Introducing Claude Opus 4.7, ” https://www.anthropic.co m/news/claude-opus-4-7, 2026, accessed: 2026-05-22

    2026

  40. [40]

    ComputeEval: Evaluating large language models on CUDA,

    NVIDIA, “ComputeEval: Evaluating large language models on CUDA, ” https://github.com/NVIDIA/compute-eval, 2025, accessed: 2026-05-22

    2025

  41. [41]

    TritonGym: A benchmark for agentic LLM work- flows in Triton GPU code generation,

    Y. Guanet al., “TritonGym: A benchmark for agentic LLM work- flows in Triton GPU code generation, ” OpenReview, 2025, [Online]. Available: https://openreview.net/forum?id=oaKd1fVgWc

    2025

  42. [42]

    KernelFalcon: Deep agent architecture for autonomous GPU kernel generation,

    L. Wang and the PyTorch Team at Meta, “KernelFalcon: Deep agent architecture for autonomous GPU kernel generation, ” PyTorch Blog, 2025, [Online]. Available: https://pytorch.org/blog/kernelfalcon-a utonomous-gpu-kernel-generation-via-deep-agents/; accessed 2026-05-22

    2025

  43. [43]

    torch.nn.functional.scaled_dot_product_attention,

    PyTorch, “torch.nn.functional.scaled_dot_product_attention, ” PyTorch documentation, 2026, [Online]. Available: https://docs.pytorch.org/d ocs/stable/generated/torch.nn.functional.scaled_dot_product_attent ion.html; accessed 2026-05-22. 11

    2026