{"id":"7fc54ae9-a3d2-4a98-9dfd-4331db02740d","arxiv_id":"2608.12004","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new benchmark derives 31 Triton kernel generation tasks from real pull requests and finds current LLMs pass only 18.71% of them end to end.","lead":"The authors built a benchmark that turns real pull requests from AI frameworks like PyTorch and vLLM into Triton kernel coding tasks, then tested leading AI models on them. The models solved few tasks end to end, which shows that generating production-ready GPU kernels from natural language remains hard.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's Modification rows report positive Success rates where NR=0, contradicting the stated success definition; the headline 18.71% success rate is not reproducible from the published protocol.","rationale":"I read the paper as a benchmark-plus-evaluation contribution whose central empirical claim is the 18.71% average task success rate. The environment-faithfulness concern in the reader's verdict is legitimate, but the most load-bearing issue I found is internal: Table 4 reports logically impossible combinations of Success and NR for the Modification category. Under the published success rule (NR must be T), Success>0 and NR=0 cannot co-occur; for the three rows cited, this means either the NR column is not what the text says, the success rule was not applied, or the rows are misaligned. This directly undermines the reproducibility of the headline numbers, not just their interpretation. A concrete re-computation from the released per-task logs is the right test. I am not alleging fraud; a column mislabel would be sufficient. If the contradiction is a typo, a conditional accept could follow after correction, but as written the central quantitative claim is not verifiable, so I hold the verdict at UNVERDICTED until the per-task data confirm the reported numbers. I also note the baseline-for-speedup issue (gold patch vs pre-PR implementation) as a secondary concern that affects interpretation, but it is less decisive than the internal inconsistency.","tokens_in":19474,"tokens_out":12342,"duration_ms":115737,"concrete_test":"Download the released per-task logs from https://doi.org/10.5281/zenodo.19221469 or the GitHub repository. For each Modification task and each of the five models, recompute UTP (vs gold), NR (T/F under the paper's accuracy-degradation criterion), STTFT, STPOT, and Success according to the Section 4.1.3 definition. Cross-tabulate Success against NR: if any row shows Success=1 with NR=F, or any category row has Success>0 while NR=0, the published success metric was not implemented as described. Report the corrected values and confirm whether the Table 4 contradiction is a typo or a systematic flaw in the evaluation protocol.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"In Section 4.1.3, a task counts as successful only if UTP equals the gold patch's UTP, NR=T, and both latency speedups are at least 0.98. Table 4's Modification rows contradict this definition: DeepSeek-V3.2 (non-reasoning) shows Success=28.57% with NR=0.000%, DeepSeek-V3.2 (reasoning) shows Success=42.86% with NR=0.000%, and GPT-5.4 shows Success=14.29% with NR=0.000%. The Modification category contains 7 tasks, and the FTP of 71.43% for these rows means 5 tasks pass all unit tests. If NR=0.000% among unit-test-passing tasks, then zero tasks satisfy the NR=T requirement, so no task can be successful. Conversely, if two or three tasks truly succeeded, at least 28.6% to 42.9% of tasks must have NR=T, making the NR=0.000% entries impossible. This is not a minor rounding issue: either the NR column is mislabeled, success was computed without the NR condition, or the table rows are misaligned. Until the per-task logs are released and this contradiction is resolved, the central quantitative claim (average task success 18.71%) cannot be verified from the paper as written. The environment-reconstruction concern raised by the reader is real, but this internal inconsistency is more immediately load-bearing because it affects the headline numbers themselves.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"RealisticTritonBench derives 31 Triton-kernel generation tasks from merged pull requests in PyTorch, vLLM, and SGLang, divided into Optimization, Modification, and New Kernel categories. Each task provides a natural-language requirement, repository context, and target function signature; generated kernels are integrated into reconstructed Docker environments and evaluated by unit tests, model-level numerical robustness, and end-to-end TTFT/TPOT latency. The authors evaluate five LLMs with the mini-SWE-agent scaffold and report an average task success rate of 18.71%, average FTP of 43.23%, NR of 47.65%, and speedups near 1.0, concluding that current LLMs struggle with realistic Triton kernel generation and that framework-level evaluation is necessary.","tokens_in":19754,"tokens_out":7706,"duration_ms":73607,"significance":"If the benchmark is valid, it addresses a real gap in the Triton-generation evaluation literature: prior benchmarks are mostly PyTorch-to-Triton translation with isolated kernel metrics and hand-written checkers. The paper's strengths include grounding tasks in externally merged PRs with gold patches, a three-layer evaluation suite, three-run latency measurements with reported variability, a concrete failure analysis, and a reward-hacking mitigation argument based on external-client TTFT/TPOT measurement. The central qualitative finding — that SOTA LLMs often pass unit tests yet degrade model accuracy or latency — is plausible and important for the community. However, the internal inconsistency in Table 4 affects the credibility of the headline success numbers and must be resolved before the quantitative claims can be accepted.","major_comments":[{"comment":"The Modification rows of Table 4 are incompatible with the success definition stated in §4.1.3, which requires UTP equal to the gold patch's UTP, NR=T, and both latency speedups at least 0.98. The Modification category has 7 tasks. DeepSeek-V3.2 (non-reasoning) reports Success=28.57%, FTP=71.43%, and NR=0.000%; if, as stated in §4.1.3, NR is averaged over tasks that pass all unit tests, then none of the 5 unit-test-passing tasks has NR=T, so Success must be 0%. The same contradiction appears for DeepSeek-V3.2 (reasoning) (Success=42.86%, FTP=57.14%, NR=0.000%) and GPT-5.4 (Success=14.29%, FTP=28.57%, NR=0.000%), and Qwen3.5-397B-A17B (Success=42.86%, FTP=57.14%, NR=50.00% can support at most 2 of the 4 FTP tasks, not 3). The positive Success entries therefore cannot be reproduced from the published protocol. The authors should release per-task evaluation logs and correct either the Success computation or the NR entries; until then the headline 18.71% average success rate is not verifiable from the paper as written.","section":"Table 4 and §4.1.3"},{"comment":"The benchmark's realism claim rests on the reconstructed environments faithfully reproducing the behavior of the original merged PRs, but §3.2.3 only states that installation errors were manually resolved by 'patching dependency versions or config files,' and §3.2.4 says testing commands were refined 'according to the actual execution results.' No per-instance inventory of environment patches or validation that the gold patch passes the refined tests with behavior matching upstream is provided. If a patch alters numerical behavior, test semantics, or latency characteristics, then the unit-test, NR, and speedup numbers no longer reflect the real-world deployment setting the benchmark claims to measure. I ask for a per-task log of environment modifications and a demonstration, for each task, that the gold patch passes the final tests and that adjusted commands preserve the original test intent.","section":"§3.2.3 and §3.2.4"}],"minor_comments":[{"comment":"The path from approximately 2,000 candidate PRs to 31 final task instances is not auditable because the paper does not report how many PRs were excluded by each of the three filtering criteria (Triton Kernel Relevance, Test Availability, Clear Kernel Objective).","section":"§3.2.2"},{"comment":"The reported run-to-run variability ('average variation is 1.08% for STTFT and 0.98% for STPOT') is ambiguous; please specify whether this is a standard deviation, a coefficient of variation, or a max-min range over the three runs.","section":"§4.1.3"},{"comment":"Several NR percentages in Table 4 cannot arise from integer task counts under the stated convention that NR is computed over unit-test-passing tasks; for example, GPT-5.4's New-kernel row has FTP=27.27% (3 of 11 tasks) and NR=50.00%, which is impossible for 3 tasks. Report exact counts (e.g., '2/4') alongside percentages.","section":"Table 4"},{"comment":"The reward-hacking mitigation argument would be easier to evaluate if the paper specified how the 'external client' measures TTFT and TPOT (e.g., wall-clock timestamps at request submission and final response) and clarified whether generated code could defer work until after the response is returned.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The Table 4 contradiction is the key blocker: the positive Modification success rates with NR=0.000% are internally inconsistent under the published success definition. I agree with the stress-test note that this is more immediately load-bearing than the environment-reconstruction concern. Requiring per-task logs and a corrected table should be a condition of revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — worth reading, but with a calculator. The benchmark idea is genuinely useful: tasks mined from merged Triton PRs in PyTorch, vLLM, and SGLang, covering optimization, modification, and new-kernel work, with an end-to-end pipeline that measures unit tests, model accuracy, and TTFT/TPOT. Ground truth comes from external gold patches, so there is no circularity. The reward-hacking discussion is thoughtful, and the failure analysis with concrete API misuse and kernel-semantics examples is the strongest part of the paper.\n\nThe soft spots: the Modification rows in Table 4 contradict the paper's own success definition. Section 4.1.3 says a task is successful only if UTP equals the gold patch, NR=T, and both latency speedups are at least 0.98. Yet DeepSeek-V3.2 (non-reasoning) shows 28.57% success with NR=0.000%, DeepSeek-V3.2 (reasoning) shows 42.86% success with NR=0.000%, and GPT-5.4 shows 14.29% success with NR=0.000%. If NR is truly zero among tasks that pass all unit tests, none of those can be successful. The headline 18.71% average includes these rows, so it is not reproducible from the published protocol. This is not a rounding artifact; either the NR column is mislabeled, success was computed without the NR condition, or the rows are misaligned. Per-task logs are needed before I would trust any of the aggregate numbers.\n\nThe other caveats are mostly normal for a benchmark: only 31 tasks remain after manual filtering with no per-criterion exclusion counts; the NR criterion \"does not degrade\" lacks an explicit threshold; the 0.98 latency cutoff is hand-picked; and the reconstructed Docker environments may not faithfully reproduce the original PR behavior. That environment concern is real, but the Table 4 contradiction is more immediately load-bearing.\n\nThis paper deserves a serious referee, not a desk reject. The benchmark is a genuine contribution for the LLM kernel-generation community. But the authors should be required to release per-task logs and resolve the Table 4 contradiction before acceptance. If that is fixed, I would cite it and probably bring it to a reading group; as it stands, the central numbers are not yet credible.","headline":"A genuinely useful benchmark for realistic Triton-kernel generation, marred by an internal contradiction in Table 4 that currently makes the headline 18.71% success rate unverifiable from the paper as written.","tokens_in":20293,"tokens_out":4260,"would_cite":false,"duration_ms":40932,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"State-of-the-art LLMs solve only about one in five real-world Triton kernel tasks once unit tests, model accuracy, and end-to-end latency all count.","keywords":["Triton kernel generation","LLM code generation","benchmark construction","end-to-end evaluation","GPU kernels","numerical robustness","pull request mining","reward hacking"],"falsifier":"Run the same generated and gold kernels in the original unmodified repositories at the exact merged-commit state with original dependency resolution, then compare unit-test verdicts, model-accuracy pass/fail, and whether each task's TTFT/TPOT speedup crosses 1.0 against the benchmark's Docker-based numbers; any systematic divergence would show the reconstructed environments altered the tested behavior.","tokens_in":19277,"feed_emoji":"⚙️","tokens_out":5664,"duration_ms":52333,"temperature":0.7,"pith_summary":"RealisticTritonBench asks whether large language models can do what real GPU-kernel developers do: modify, optimize, or create Triton kernels inside working AI frameworks such as PyTorch, vLLM, and SGLang. The benchmark builds 31 tasks from merged pull requests, replaces the accepted kernel with the model's kernel, and runs the repository's own unit tests plus model-accuracy and serving-latency checks. Across five state-of-the-art models, task success averages 18.71%, full unit-test pass rates average 43.23%, numerical robustness averages 47.65%, and end-to-end speedup is roughly 1x. The sympathetic reading of the paper's claim is that today's LLMs are not yet reliable enough for real-world Triton kernel deployment, and that unit-test correctness is a misleading proxy for deployment readiness.","feed_headline":"LLMs crack only 18.7% of real-world Triton kernel tasks","feed_subtitle":"A 31-task benchmark from real merged PRs shows unit-test passes do not guarantee accuracy or speed in production.","key_machinery":"The carrying mechanism is the benchmark's evaluation pipeline rather than a single kernel. Each of the 31 tasks supplies a natural-language requirement, repository context, and a target function signature; the generated kernel is patched into an instance-specific Docker image built from the original repository's own build commands, then scored with the repository's pytest unit tests, a model accuracy benchmark, and wall-clock time-to-first-token and time-per-output-token measured by an external client. Success requires the unit-test pass rate to match the gold patch, numerical robustness to hold, and both latency speedups to be at least 0.98, so a kernel cannot win by passing isolated correctness checks alone.","core_discovery":"The paper's central discovery is that when Triton kernels are evaluated the way deployment actually works, current LLMs mostly fail. Using 31 tasks mined from merged pull requests, each with a Docker reproduction of the original repository plus unit tests, model-accuracy tests, and TTFT/TPOT latency measurement, the authors find average task success of 18.71% across five leading models (best: Qwen3.5-397B-A17B at 25.81%), average full unit-test pass of 43.23%, average numerical robustness of 47.65%, and end-to-end speedups near 1x. The gap between full unit-test pass and task success — kernels that pass unit tests still degrade model accuracy or end-to-end latency in a substantial share of cases — is the paper's key evidence that kernel-level benchmarks overstate LLM ability. The failure analysis attributes most unit-test failures to insufficient mastery of Triton's programming constraints, many others to incomplete understanding of kernel semantics in real repositories, and a further set to inattention to performance and numerical stability after the unit tests pass.","pith_inferences":["The paper's failure taxonomy — wrong API choice such as tl.min versus tl.minimum, missing boundary masks, and unconditional computation that the gold patch guards — reads as a concrete roadmap: targeted fine-tuning on Triton API constraints plus static Triton linting could close much of the gap, though the paper does not propose such methods.","The roughly 1x end-to-end speedup implies that any reinforcement-learning reward for kernel generation should be tied to framework-level latency and downstream accuracy rather than kernel-level speedup; the benchmark's Docker pipeline could serve as a training environment for that objective.","Because the dataset contains only 31 tasks drawn from three repositories, the category-level numbers such as New-kernel's 5.455% success carry wide error bars; extending the same PR-mining pipeline to more frameworks would test whether that finding is robust.","The success criterion is anchored to the gold patch as baseline and a 0.98 speedup tolerance, so absolute percentages would shift under a different reference implementation or stricter latency bar, even if the relative ordering of models likely remains stable."],"forward_implications":["Unit-test passing does not imply deployment readiness: an average full unit-test pass rate of 43.23% collapses to 18.71% task success once model accuracy and end-to-end latency are required, so benchmarks that stop at kernel-level checks overstate progress.","Optimization tasks are within reach functionally but not in performance: optimization tasks average 23.08% success and speedups close to 1x, meaning LLMs can preserve behavior but rarely deliver the latency gains a real optimization PR would.","Modification tasks are numerically fragile: average numerical robustness on modification tasks is only 20%, with some models at 0%, so extending or bug-fixing an existing kernel often silently breaks downstream model accuracy.","New-kernel generation from scratch is the hardest frontier: average success is 5.455% and average unit-test pass is 20.00%, suggesting that LLMs are far from synthesizing deployable Triton kernels without a reference implementation.","End-to-end evaluation undercuts standard reward-hacking strategies: external-client timing and a separate-process accuracy harness make concurrency-based, caching-based, and environment-manipulation exploits ineffective."],"supporting_citations":[{"why":"Defines Triton, the DSL whose kernels the benchmark asks models to generate.","marker":"[38]"},{"why":"Supplies the SWE-bench methodology of providing a runnable Docker environment per instance, which RealisticTritonBench adapts.","marker":"[17]"},{"why":"Provides the two-level environment construction strategy (shared base image plus instance-specific image) used to build reproducible test environments.","marker":"[5]"},{"why":"Represents the prior PyTorch-to-Triton benchmark whose isolated kernel-level evaluation RealisticTritonBench extends and contrasts with.","marker":"[20]"},{"why":"Represents KernelBench, the isolated kernel-generation benchmark whose unit-test-only evaluation the paper argues overstates model ability.","marker":"[28]"},{"why":"Represents FlashInferBench, a framework-integrated benchmark whose task formulation and metrics the paper distinguishes from its own PR-derived tasks and TTFT/TPOT evaluation.","marker":"[46]"},{"why":"Supplies the reward-hacking taxonomy (concurrency, state/caching, environment manipulation) used to argue that end-to-end evaluation mitigates evaluation exploits.","marker":"[22]"},{"why":"Provides the SWE-agent scaffold from which the paper's evaluation agent is derived and whose bash-only interface motivates the scaffold choice.","marker":"[47]"}],"fun_headline_variants":["LLMs fail 81% of real-world Triton tasks","Unit-test pass doesn't mean production-ready Triton","RealisticTritonBench: only 18.7% LLM success on real kernels","Real-world Triton tasks stump LLMs despite unit tests","LLMs score 18.7% on production-style Triton benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each manually reconstructed Docker environment, including any dependency pins and configuration patches, behaves like the real merged pull request's runtime well enough that the unit-test, accuracy, and latency numbers reflect production; if those patches change numerical behavior or timing semantics, the benchmark's central verdict could shift.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fail 81% of real-world Triton tasks","Unit-test pass doesn't mean production-ready Triton","RealisticTritonBench: only 18.7% LLM success on real kernels","Real-world Triton tasks stump LLMs despite unit tests","LLMs score 18.7% on production-style Triton benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1476,"prompt_tokens":1089,"completion_tokens":387,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":705,"completion_tokens_details":{"reasoning_tokens":295}},"tokens_in":705,"tokens_out":387,"duration_ms":3732,"temperature":1.0,"reasoning_tokens":295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:19:01.655359+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same generated and gold kernels in the original unmodified repositories at the exact merged-commit state with original dependency resolution, then compare unit-test verdicts, model-accuracy pass/fail, and whether each task's TTFT/TPOT speedup crosses 1.0 against the benchmark's Docker-based numbers; any systematic divergence would show the reconstructed environments altered the tested behavior.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the prior PyTorch-to-Triton benchmark whose isolated kernel-level evaluation RealisticTritonBench extends and contrasts with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents KernelBench, the isolated kernel-generation benchmark whose unit-test-only evaluation the paper argues overstates model ability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents FlashInferBench, a framework-integrated benchmark whose task formulation and metrics the paper distinguishes from its own PR-derived tasks and TTFT/TPOT evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SWE-agent scaffold from which the paper's evaluation agent is derived and whose bash-only interface motivates the scaffold choice."}],"review_version":1}