{"id":"03aa32f0-9165-493e-8636-037942304839","arxiv_id":"2507.23194","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GEAK, an agentic LLM framework, generates Triton kernels for AMD GPUs with up to 63% unit-test pass rate and up to 2.59x speedup on a revised benchmark.","lead":"GEAK is an agent that uses LLMs to write Triton GPU kernels for AMD hardware, and its authors release two benchmarks for judging such kernels. On their tests, GEAK passes more unit tests and produces faster kernels than direct prompting, though the comparison favors GEAK by giving it many more attempts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing Reflexion baseline and pass@10-vs-pass@1 comparison leave the abstract's outperformance claim unsupported.","rationale":"The reader's weakest_assumption focuses on the unequal pass@k budget, which is a valid and concrete fairness concern for the direct-prompting comparison. My read agrees with that concern but identifies an even more direct gap: the Abstract explicitly promises a comparison against Reflexion-based generation pipelines, and no such baseline exists in the experiments. This is a missing control for a stated part of the central claim, not merely an asymmetry in compute. Because GEAK is described as adapting Reflexion, a Reflexion baseline is essential to establish that the agent's design, rather than the Reflexion loop or additional inference-time compute, drives the reported gains. The paper otherwise presents plausible execution results, and the ROCm benchmark with more thorough tests supports a narrower claim. Therefore the correct disposition is conditional acceptance pending the missing baseline and a fair pass@k comparison, which aligns with the reader's CONDITIONAL verdict; no change to the verdict is needed.","tokens_in":9797,"tokens_out":4521,"duration_ms":48447,"concrete_test":"Run a single controlled experiment on both benchmarks using GPT-4.1 with three arms: (1) GEAK as reported (10 sequential iterations × 10 parallel runs, pass@10), (2) an independently implemented Reflexion-based agent with the same 10×10 budget and same LLM, and (3) direct prompting with 10 independent samples (pass@10) and with one sample (pass@1). Compare execution accuracy and speedup across arms. If Reflexion matches or exceeds GEAK, the abstract's outperformance claim fails; if direct pass@10 approaches GEAK's accuracy, the reported advantage is largely inference-time compute rather than agentic refinement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The Abstract claims GEAK 'significantly outperformed the baselines of directly prompting frontier LLMs as well as Reflexion-based generation pipelines,' but no Reflexion-based pipeline is instantiated, run, or reported anywhere in Section 5. Table 2 and Table 4 only compare against direct prompting; the promised Reflexion control is absent. This matters because GEAK is itself 'adapted from Reflexion-style feedback mechanisms' (§4.2), making Reflexion the natural baseline for isolating the contribution of the agent design. Without it, the second half of the central claim is unsupported. Separately, the direct-prompting comparison is not compute-matched: GEAK's headline 54.89% on TritonBench-revised is pass@10 with 10 sequential iterations and 10 parallel runs (Table 7), while the baselines in Table 2 are pass@1 single prompts. If direct prompting were given the same pass@k budget, its execution accuracy would likely rise, potentially narrowing or eliminating the reported gap. Appendix B further concedes that the TritonBench-revised harness retains limited test coverage, so the 54.89% figure may overstate true correctness; the ROCm benchmark's 63.33% is more credible but the Abstract does not restrict the claim to that benchmark.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents GEAK, an agentic framework that uses frontier LLMs to generate Triton GPU kernels for AMD GPUs, with a four-module pipeline (generator, evaluator, reflector, optimizer) and inference-time compute scaling via sequential refinement and parallel sampling. It also introduces two evaluation benchmarks: a revised subset of 184 kernels from TritonBench-G and a new ROCm benchmark of 30 kernels from open-source AMD repositories. The headline results are 54.89% execution accuracy on TritonBench-revised and 63.33% on the ROCm benchmark, with an average speedup of up to 2.59X, compared with direct-prompting baselines below 15%.","tokens_in":10157,"tokens_out":4649,"duration_ms":49054,"significance":"If the claims hold, the paper would be a useful contribution to automated GPU-kernel synthesis: it offers an open-sourced agent implementation and evaluation framework, adapts an existing benchmark to AMD hardware, introduces a new ROCm benchmark with real-world kernels, and provides a detailed case study. The appendix also deserves credit for candidly discussing test-coverage limitations. However, the central comparative claims are currently not supported by the experiments as designed: the abstract claims outperformance over Reflexion-based pipelines without running any such baseline, and the direct-prompting comparison is not compute-matched because GEAK is credited with many more attempts.","major_comments":[{"comment":"The abstract claims that GEAK \"significantly outperformed the baselines of directly prompting frontier LLMs as well as Reflexion-based generation pipelines,\" but no Reflexion-based baseline is instantiated, run, or reported anywhere in Section 5. Tables 2 and 4 compare only against direct prompting. Since Section 4.2 states that GEAK is adapted from Reflexion-style feedback mechanisms, Reflexion is the natural control for isolating the contribution of the agent design. Without it, the second half of the central claim is unsupported. Please add a Reflexion control or revise the claim.","section":"Abstract and Sections 4.2, 5.1"},{"comment":"The headline 54.89% execution accuracy on TritonBench-revised is a pass@10 result obtained with 10 sequential iterations and 10 parallel runs (Table 7), while the direct-prompting baselines in Table 2 are single-attempt results. Comparing pass@10 with pass@1 is not compute-matched and can materially overstate the advantage. If the baselines were given the same pass@k budget, their accuracy would likely rise. Please report direct-prompting accuracy under the same sampling budget, or restrict the comparative claim to pass@1 settings.","section":"Tables 2, 3, 7"},{"comment":"The text states that increasing the number of iterations produces a \"monotonic improvement in both call and execution accuracy,\" but Table 5 does not support monotonicity for call accuracy: for example, call accuracy drops from 57.61% at iter5 to 50.04% at iter6, and from 62.5% at iter8 to 56.52% at iter9. Execution accuracy also plateaus rather than increasing monotonically after early iterations. Please revise the wording to describe the observed trend without claiming monotonicity, and consider analyzing the fluctuations.","section":"Section 5.2.1, Table 5"},{"comment":"Appendix B explicitly concedes that TritonBench-revised retains the original TritonBench-G test harness with limited test coverage, and warns that reported correctness figures around or above 50% may overstate true correctness. The abstract and conclusion report 54.89% and 63.33% without this caveat. The accuracy claims should be qualified as correctness on the specific unit tests used, especially for TritonBench-revised, and the limitations stated in Appendix B should appear in the main results discussion.","section":"Appendix B and Abstract"},{"comment":"The ROCm Triton Benchmark was constructed by refactoring kernels and unit tests with assistance from frontier LLMs, and the kernels originate from the authors' employer's ecosystem. The paper does not specify which LLMs were used, what prompts were used, or what human review was performed. If the same model family used by GEAK (e.g., GPT-4.1) was involved in creating the benchmark, the evaluation could be biased in favor of the framework. Please document the refactoring protocol, including model names, prompts, and human verification, or otherwise address this contamination risk.","section":"Section 3.2"}],"minor_comments":[{"comment":"The \"Correctly generated kernels\" column appears inconsistent with the reported execution accuracy: difficulty 1 shows \"2 / 3\" but accuracy 66.67 / 100.0, and difficulty 5 shows \"1 / 1\" but accuracy 20.00 / 20.00. Please clarify how these counts and percentages are computed.","section":"Table 3"},{"comment":"The sentence beginning \"Table2 shows the direct prompting results on the ROCm benchmark\" is imprecise because Table 2 also reports results on the TritonBench-modified benchmark; please rephrase.","section":"Section 5.1"},{"comment":"The terminology is inconsistent: the paper uses \"TritonBench-revised,\" \"TritonBench-modified,\" and \"TritonBench-Modified\" to refer to the same benchmark. Please standardize.","section":"Throughout"},{"comment":"Appendix B states that TritonBench-revised \"retains the same test harness from TritonBench-G unchanged,\" but Section 3.1 describes substantive changes including adding missing test-function calls, using tolerance-based tensor comparison, and setting consistent seeds. Please reconcile these two descriptions.","section":"Sections 3.1 and B"},{"comment":"The phrase \"significantly outperformed\" is used without any statistical significance testing or confidence intervals; given the small per-difficulty counts and the pass@k methodology, please either add uncertainty estimates or use a less strong formulation.","section":"Abstract and Section 5.1"},{"comment":"The table caption does not say whether these numbers are pass@1 or include multiple parallel runs; please clarify the exact evaluation protocol for the sequential-scaling experiment.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is authored almost entirely by AMD employees, and the new ROCm benchmark is drawn from AMD repositories. This is not a reason to reject, but it increases the need for external scrutiny of the benchmark construction and for transparency in the refactoring protocol. The most serious gap is the missing Reflexion baseline combined with the compute mismatch between GEAK and the direct-prompting baselines; both are needed to support the abstract's central claim. If the authors cannot run a Reflexion baseline or a compute-matched direct-prompting baseline, the abstract must be rewritten to state what was actually compared. I would not recommend reject at this stage because the framework and benchmarks are potentially useful and the issues are addressable within a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate systems paper with a real new benchmark and a sensible agent design, but the headline claim is not supported as written. The missing Reflexion baseline and pass@10-vs-pass@1 comparison are not nitpicks; they are right there in the paper.\n\nWhat is actually new: the ROCm Triton Benchmark, 30 kernels from AMD repositories, is a useful addition to a thin area. The revised 184-kernel TritonBench subset, fixed for AMD, is also plausible. The GEAK pipeline combines known pieces—1-shot retrieval, knowledge injection, Reflexion-style reflection, LLM-as-optimizer, parallel sampling—but the combination and the AMD focus are a real contribution. The ablation in Table 9 is informative: knowledge injection gives the biggest correctness lift, and the optimizer is what pushes speedup above 1.0. The flip-kernel case study is concrete and honest about why the generated kernel is faster.\n\nNow the soft spots, in proportion. The abstract claims GEAK \"significantly outperformed\" Reflexion-based pipelines, but no Reflexion baseline is instantiated or run anywhere. Since GEAK is itself adapted from Reflexion, that baseline is the one you most need to isolate the contribution. Just as important, the direct-prompting baselines in Table 2 appear to be pass@1, while GEAK's headline 54.89% is pass@10 after 10 sequential iterations and 10 parallel runs (Table 7). That is an unequal compute budget, and equalizing it would very likely shrink the gap. How much, we don't know, because the data isn't there. Appendix B then concedes that TritonBench-revised retains the limited unit tests of the original, so the 54.89% figure may also overstate true correctness. I would treat the 63.33% on the ROCm benchmark as the more credible number, but the missing baseline still applies. The GitHub links are placeholders, so nothing can be verified or reused yet. Also, the 2.59x speedup is on the narrower benchmark; on the ROCm benchmark the average speedup is 0.92x, a fact the abstract does not foreground.\n\nWho is this for? People working on LLM-based kernel generation for AMD hardware and anyone building evaluation suites for non-NVIDIA targets. They will get value from the benchmark and the ablation data. The paper does deserve a serious referee—the work is real and the flaws are fixable—but it needs major revision before the central claim can stand. I would send it to review with a clear request for fair pass@k baselines, a real Reflexion control, working code links, and a headline that matches what the data actually shows.","headline":"Useful AMD-focused Triton benchmark and agent pipeline, but the abstract overstates the result: the Reflexion baseline is missing and the pass@k comparison is unequal.","tokens_in":10597,"tokens_out":1770,"would_cite":false,"duration_ms":21796,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An agentic pipeline of generate, evaluate, reflect, and optimize, plus parallel sampling, lets frontier LLMs produce correct Triton kernels on AMD GPUs up to 63% of the time, with average speedups up to 2.59x.","keywords":["Triton","GPU kernel generation","agentic code generation","inference-time scaling","Reflexion","AMD ROCm","benchmark","large language models"],"falsifier":"Run the same frontier LLM baselines with the same pass@k budget of ten sequential-and-parallel samples and compare execution accuracy; then rerun the evaluation against an expanded test suite with mutation-generated and automatically generated cases to see whether the 'correct' kernels still pass.","tokens_in":1376,"feed_emoji":"⚡","tokens_out":1971,"duration_ms":75090,"temperature":0.7,"pith_summary":"The paper claims that a four-agent pipeline (generator, evaluator, reflector, optimizer) can turn frontier LLMs into practical Triton kernel writers for AMD GPUs by spending inference-time compute instead of training. On a repaired 184-kernel version of TritonBench and a new 30-kernel ROCm benchmark, the GEAK agent reaches up to 54.89% and 63.33% execution accuracy while averaging up to 2.59x speedup on correct kernels, where direct prompting of the same LLMs stays below about 15%. A sympathetic reader should care because the result suggests that hardware-specific kernel expertise can be automated and democratized through agentic loops plus scaling, without fine-tuning. The paper also releases both benchmarks and the agent code.","feed_headline":"Agent-written Triton kernels hit 63% accuracy on AMD GPUs","feed_subtitle":"GEAK's reflection loop and parallel sampling beat direct LLM prompting, with up to 2.59x speedups.","key_machinery":"The load-bearing mechanism is the inference-time scaling loop: the Evaluator runs generated code against unit tests, the Reflector turns error traces into repair instructions, the Optimizer ranks past code by measured latency and proposes faster variants, and parallel sampling with temperature set to 1 aggregates the best of ten independent runs. A 'debugging trap' guard caps the number of repair attempts per code snippet before forcing a fresh strategy, and one-shot prompting retrieves examples by code similarity. This machinery converts a single LLM call into a search process over kernel space.","core_discovery":"GEAK is an agent system that combines structured knowledge injection, one-shot retrieval of similar Triton code, a Reflexion-style reflection loop, and a dedicated optimizer LLM to iteratively repair and speed up generated kernels. The central claim is that scaling inference compute along two orthogonal axes, sequential refinement and parallel sampling, yields correct and fast Triton kernels on AMD Instinct MI300X and MI250 from minimal natural-language or reference-code task descriptions, outperforming direct prompting by a large margin. The numbers the paper emphasizes are 54.89% execution accuracy and 2.59x average speedup on TritonBench-revised, and 63.33% execution accuracy on the ROCm benchmark.","pith_inferences":["A fairer side-by-side would give the baseline LLMs the same pass@k budget of ten sequential-and-parallel samples; the paper's own pass@k tables suggest the gap could narrow, though iterative error feedback may still yield an edge beyond pure sampling.","The agent architecture is not AMD-specific in design, so the same loop could plausibly be pointed at CUDA, oneAPI, or other accelerator DSLs, though the paper only demonstrates AMD hardware.","The appendix's warning that TritonBench-revised tests are narrow implies that adding mutation-based or automatically generated tests could lower the reported pass rates and should be tested before treating the numbers as general correctness."],"forward_implications":["With a fixed prompt, GEAK's execution accuracy scales roughly log-linearly in the number of parallel runs, so users can trade compute for correctness predictably.","Sequential refinement alone more than triples execution accuracy over zero-shot direct prompting on TritonBench-revised, from 13.04% to 44.02% at iteration 19.","The optimizer module contributes the main speedup, lifting average speedup to 1.45x when combined with knowledge injection and one-shot prompting; on an individual kernel, GEAK reaches 2.26x over an expert-written flip kernel.","Direct prompting of frontier LLMs is insufficient for AMD-targeted Triton code, with GPT-4.1 generating no valid kernels on the ROCm benchmark in the reported setup.","The released benchmarks, including the repaired AMD-compatible test harness and the new real-world ROCm kernel set, provide a reusable yardstick for GPU kernel synthesis."],"supporting_citations":[{"why":"Supplies the TritonBench-G kernels and evaluation metrics that the revised benchmark adapts and repairs.","marker":"[1]"},{"why":"Defines the KernelBench GPU kernel correctness and runtime benchmark that motivates the evaluation approach.","marker":"[10]"},{"why":"Provides the Reflexion feedback loop that the agent uses for iterative debugging.","marker":"[13]"},{"why":"Underlies the LLM-as-Optimizer module that proposes performance-improving variants.","marker":"[14]"},{"why":"Motivates rigorous test-based evaluation and the caution about narrow test suites in the appendix.","marker":"[3]"}],"fun_headline_variants":["Triton AI agent hits 63% accuracy, 2.59x speedup on AMD","GEAK agent writes Triton kernels: 63% correct, 2.59x faster","Reflection-loop agent beats direct LLM for Triton kernels","Agentic Triton gen: 63% accuracy, 2.59x speedup on AMD","GEAK's reflection loop nails 63% Triton kernel accuracy"],"cache_read_input_tokens":12800,"weakest_assumption_plain":"The headline comparison assumes the baseline LLMs get a single direct prompt while GEAK gets up to ten sequential attempts and ten parallel samples (pass@10); with the same sampling budget the gap may shrink, and the paper's own disclaimer notes the TritonBench-revised tests are narrow enough that some passes may be false positives.","fun_headline_variants_meta":{"raw":{"variants":["Triton AI agent hits 63% accuracy, 2.59x speedup on AMD","GEAK agent writes Triton kernels: 63% correct, 2.59x faster","Reflection-loop agent beats direct LLM for Triton kernels","Agentic Triton gen: 63% accuracy, 2.59x speedup on AMD","GEAK's reflection loop nails 63% Triton kernel accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000727,"raw_usage":{"total_tokens":3266,"prompt_tokens":963,"completion_tokens":2303,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":2191}},"tokens_in":579,"tokens_out":2303,"duration_ms":15819,"temperature":1.0,"reasoning_tokens":2191,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:57:37.275590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same frontier LLM baselines with the same pass@k budget of ten sequential-and-parallel samples and compare execution accuracy; then rerun the evaluation against an expanded test suite with mutation-generated and automatically generated cases to see whether the 'correct' kernels still pass.","supporting_citations":[{"cited_title":"Tritonbench: Benchmarking large language model capabilities for generating triton operators, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the TritonBench-G kernels and evaluation metrics that the revised benchmark adapts and repairs."},{"cited_title":"Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini","cited_arxiv_id":null,"evidence_quote":"Defines the KernelBench GPU kernel correctness and runtime benchmark that motivates the evaluation approach."},{"cited_title":"Le, Denny Zhou, and Xinyun Chen","cited_arxiv_id":null,"evidence_quote":"Underlies the LLM-as-Optimizer module that proposes performance-improving variants."}],"review_version":1}