{"id":"fc3df721-93a7-420d-b0fc-6a2deee5a394","arxiv_id":"2607.17979","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A human-designed evaluation harness with a profile-backed controller lets LLM coding agents produce GPU kernels that beat FlashInfer baselines by up to 29.7x in a contest setting.","lead":"This paper describes a harness-centered workflow for using AI coding agents to generate GPU kernels, tested in the MLSys 2026 FlashInfer contest on NVIDIA Blackwell GPUs. The authors report 1.1x–29.7x speedups over provided baselines across five operators, and find that adding human-designed constraints and references beats a fully autonomous agent search.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DSA top-k headline speedup is known-incorrect under the paper's own gate: retained tag v50 fails high-trial replay on workload 70d53807 (3/200), so the abstract's 18.05x claim overstates what contest validation would yield.","rationale":"The reader's weakest assumption is exactly the n=3 correctness gate; Appendix E's v50 data makes that concern concrete and internal. The DSA top-k 18.05x figure is the second-largest headline speedup and is associated with a known failing workload under high-trial replay. The paper's own formula makes any workload failure fatal at definition level. The fallback exists, but its measured speedup is not reported in Table 3, so the headline cannot be recovered from current data. I also considered the Agent-Assisted vs Full-Agent budget asymmetry; that undermines the comparative conclusion but is secondary because the paper's primary quantitative claim is the speedup list. The paper deserves credit for surfacing the replay evidence, but the abstract and Table 3 still overstate correctness. The reader's conditional verdict already captures the required fix: replace or separate the DSA top-k artifact, and state the budget asymmetry explicitly. No verdict change is needed beyond that.","tokens_in":33265,"tokens_out":4967,"duration_ms":49885,"concrete_test":"Run the exact FlashInfer-Bench contest evaluator on DSA top-k tag v50 over all 128 workloads with a high-trial budget (e.g., 100 trials/workload, matching Appendix E's replay), and do the same for fallback tag v37. If any v50 workload fails correctness, the definition-level score for v50 is zero under §4's cd=0 rule; then recompute the headline speedup using v37 or report the DSA top-k column as 'no valid retained artifact'. If all 128 workloads pass for v50 at the high-trial budget, the 18.05x claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central speedup list includes 18.05x for DSA top-k, but that number comes from the 'low-trial retained artifact' (tag v50), which Appendix E shows is not reliably correct. On workload 70d53807, tag v50 passes an official-style 3-trial run (0/3) but fails a 100-trial replay (3/200 mismatches, max error 6.033). The paper's own §4 scoring rule is cd=0 if any workload fails correctness; under that rule, an artifact with a known failing workload has definition-level score zero, not 18.05x. The paper is transparent about the replay evidence and offers a conservative fallback tag v37 (0/1206 mismatches), but the abstract and Table 3 still headline the low-trial artifact. Because the correctness gate is per-workload and trial-sensitive, the headline claim requires replacing the v50 speedup with the fallback's measured speedup or removing the DSA top-k entry from the retained summary. The n=3 default is not merely a validation-cost trade-off; it changes whether the central claimed artifact is correct under the contest's all-workload correctness gate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a harness-engineering system for LLM-driven GPU kernel generation in the MLSys 2026 FlashInfer contest on NVIDIA B200 GPUs. The system separates an evaluation harness from a profile-backed controller, uses human-authored skills to constrain Codex/Claude Code agents, and enforces compilation, correctness, official-aligned timing, workload-distribution sweeps, and artifact archival before promotion. Across five operator definitions (MoE FP8, DSA top-k indexer, DSA sparse attention, GDN decode, GDN prefill), the retained Agent-Assisted artifacts are reported to achieve mean-latency speedups over the supplied FlashInfer baselines of 1.62x, 18.05x, 29.68x, 1.12x, and 13.70x. Full-Agent LoongFlow artifacts are evaluated under a matched final-evaluation protocol and are slower. Appendix E discloses a trial-sensitivity issue in DSA top-k and identifies a conservative fallback tag v37.","tokens_in":33566,"tokens_out":4405,"duration_ms":42583,"significance":"If the speedup claims hold, this is one of the more detailed and honest case studies of LLM-driven GPU kernel optimization with a human-designed harness. The paper releases artifacts, gives per-workload latency tables for all five definitions, includes profiler evidence, and explicitly discusses a rare correctness failure in Appendix E. The Agent-Assisted vs Full-Agent comparison is a useful negative result for fully autonomous search in this contest setting. The contribution is an engineering workflow/template rather than a new kernel algorithm, and the paper appropriately frames itself that way.","major_comments":[{"comment":"The headline 18.05x DSA top-k speedup is attributed to the low-trial retained artifact tag v50. Appendix E shows that v50 fails 100-trial replay on workload 70d53807 (3/200 mismatches, max error 6.033), while the conservative fallback v37 passes 0/1206 repeated checks. Section 4 defines cd=0 if any workload fails correctness, so under the paper's own scoring rule a definition containing a failing workload would score zero, not 18.05x. The authors are transparent about this in the text, but the abstract's unqualified list overstates the reliability of the retained DSA top-k artifact. The revision should either report the v37 fallback's measured speedup in Table 3/Figure 2 or add a prominent abstract-level caveat, and should quantify the fallback's latency impact.","section":"Abstract, Table 3, Figure 2; Appendix E, Table 8"},{"comment":"The 'Trial-sensitive correctness' paragraph explains the n=3 vs high-trial issue and identifies v37 as a conservative fallback, but it never reports the fallback's speedup. Without that number, readers cannot determine whether the 18.05x figure is materially affected by moving to the correctness-conservative artifact. Please add the measured mean-latency speedup for v37 (or the v50-to-v37 latency change) and state explicitly whether DSA top-k is included in the cross-definition summary under v37 or only under the low-trial v50.","section":"§5.1, Appendix E"},{"comment":"Table 3 reports mean-latency speedups without error bars, standard deviations, or repetition counts, despite the paper's emphasis on measured, noise-resistant promotion. Given that DSA top-k is shown to be trial-sensitive, the main comparison table should at least report the number of repetitions and a variance/range measure, or a footnote explaining why repeated-run variability is negligible for these B200 measurements.","section":"Table 3, §4"}],"minor_comments":[{"comment":"Consider adding one clause such as 'with DSA top-k subject to the high-trial validation caveat in Appendix E' near the speedup list, so the abstract and the appendix are consistent at first reading.","section":"Abstract"},{"comment":"The phrase 'Contest definitions covered by this Agent-Assisted paper' is awkward; 'Definitions covered' would suffice.","section":"Table 1"},{"comment":"'A VO' should be 'A VO' with a space (the reference lists 'A VO: Agentic variation operators'). Minor typographical issue.","section":"Introduction"},{"comment":"The DSA top-k table header includes 'Fixed: page count = 11923.' If this is a fixed property of all listed workloads, state it in the main text near Table 1 as well; otherwise clarify the condition.","section":"Appendix I.2"},{"comment":"The sentence 'Following the FlashInfer-Bench evaluation protocol ... speedup is correctness-gated' is followed by a reporting normalization that is not correctness-gated. Consider making this distinction more prominent, e.g., by explicitly labeling Table 3 as 'uncorrected reporting normalization' in the caption.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually transparent: the appendix-level disclosure of the DSA top-k trial-sensitivity problem and the conservative fallback is a strength, not an attempt to hide a flaw. The problem is purely in the framing of the headline results, which do not carry the caveat into the abstract or the main speedup table. I would not reject; a revision that moves the fallback (or an explicit caveat) into the main reporting would resolve the load-bearing issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you're building a harness for LLM-driven kernel optimization. It's a genuine engineering contribution: the authors separate the evaluation harness from the optimization controller, ground shape dispatch in workload JSON and profiler data, and archive rejected probes as memory. The per-workload latency tables and appendix material are exhaustive, and the code is released. The trial-sensitivity analysis in Appendix E—a simple probabilistic model for why n=3 misses rare top-k mismatches—is the kind of thing I'd want to see in more agent-workflow papers.\n\nThe five headline speedups are not all equal. The DSA top-k 18.05x number comes from tag v50, which passes the official-style 3-trial gate but fails a 100-trial replay on workload 70d53807 (3/200 mismatches). Under the paper's own scoring rule (cd=0 if any workload fails), v50's definition-level score is zero, not 18.05x. The authors are fully transparent about this and provide a conservative fallback tag v37 with 0/1206 mismatches, but the abstract and Table 3 still headline the low-trial artifact. That's the main thing to fix: either report v37's speedup for DSA top-k or pull that entry until the artifact is validated. This is a load-bearing caveat for one of the five numbers, not for the whole paper.\n\nThe second soft spot is the Agent-Assisted vs Full-Agent comparison. It's a reasonable empirical baseline, but the budgets are not matched: Full-Agent traces ran on far fewer workloads (e.g., 12 vs 128 for DSA top-k) and far fewer iterations. So the claim that expert-provided directions remain critical is plausible, but this particular comparison doesn't prove it. The paper should either match search budgets or soften the conclusion.\n\nMinor: the main speedup table has no error bars, though the per-workload tables let you reconstruct variance. The n=3 correctness gate is acknowledged as a speed/accuracy trade-off; the appendix owns that.\n\nWho's this for: anyone working on LLM agents for systems/kernel code, and contest participants. It deserves a serious referee—the detailed measurements, released artifacts, and honest appendix outweigh the DSA headline issue. I'd recommend sending it out, with the DSA top-k fix as a required revision.","headline":"Solid, unusually transparent contest engineering paper; the 18.05x DSA top-k headline is known to be from an artifact that fails its own high-trial correctness gate, so that one number is overstated.","tokens_in":34023,"tokens_out":3427,"would_cite":true,"duration_ms":31855,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A human-authored harness with strict gates and profiler feedback, not the LLM alone, is what makes LLM-generated GPU kernels correct and fast: across five operators, retained kernels beat supplied baselines by 1.12x to 29.68x, while fully a","keywords":["harness engineering","LLM code agents","GPU kernel optimization","profile-backed search","correctness gates","trial budget","workload-shape dispatch","Blackwell kernels"],"falsifier":"Run a 500-trial correctness replay of the retained DSA top-k artifact over every workload in the contest distribution. If any workload produces one exact-match-ratio mismatch, the 18.05x speedup claim for that artifact is false; the paper's own appendix already observed 3/200 mismatches on one workload at 100 trials, so this is a concrete check.","tokens_in":33149,"feed_emoji":"⚡","tokens_out":6175,"duration_ms":57416,"temperature":0.7,"pith_summary":"This paper tries to establish that the reliability of LLM-driven GPU kernel generation comes from harness engineering, not from the model alone. The system separates a measurement harness that enforces compilation, correctness, timing, and archival from a profile-backed optimization controller that turns profiler evidence into bounded, one-direction search decisions. Human-authored skills, reference-first reconnaissance, workload-shape dispatch, and conservative promotion turn agent output into auditable kernels. On five operators spanning mixture-of-experts, sparse attention, and gated-delta recurrence, the retained artifacts achieved mean-latency speedups of 1.62x, 18.05x, 29.68x, 1.12x, and 13.70x over supplied baselines, and the human-assisted artifacts were faster than fully autonomous search artifacts in matched evaluation. The paper also reports its own trial-budget caveat: the default three-trial correctness gate can miss rare top-k boundary mismatches, which only appear under high-trial replay.","feed_headline":"Human-built harness yields 1.12–29.68× GPU kernel speedups","feed_subtitle":"Strict gates and profiler feedback let LLM coding agents beat reference baselines on five kernels.","key_machinery":"The central mechanism is the harness/controller separation. The evaluation harness packages candidates, compiles them in the target environment, runs correctness checks, measures official-aligned latency, and archives artifacts; the optimization controller converts profiler output into bottleneck state, selects one bounded optimization direction per round, supervises plateaus, and records accepted and rejected evidence. Reusable skills encode the loop as a playbook: reference-first scanning, workload-derived shape regimes, paired same-round gates, profile matrices, supervisor memory, and promotion only after a full distribution sweep without correctness regressions. Shape-aware dispatch is i","core_discovery":"On its own terms, the central claim is that when you separate the evaluation harness from the optimization controller and bind LLM coding agents with human-authored skills, reference-first reconnaissance, paired baseline gates, profile-derived bottleneck state, workload-shape dispatch, and full-sweep promotion, the agents can produce correct, fast GPU kernels. The retained kernels improve mean latency over the supplied baselines by 1.12x to 29.68x across five operator definitions, with the largest gains coming from structural route changes such as sparse-attention route specialization and a Blackwell chunked recurrent prefill path. The paper further claims that expert-provided optimization d","pith_inferences":["The paper does not say so, but the trial-budget fragility likely extends beyond this operator: any randomized exact-match validator with small n can certify kernels that fail on rare boundary inputs. A testable fix is adaptive trial counts driven by the observed top-k boundary margin.","The paper leaves the official scoring dependence implicit: if final contest validation uses a different trial budget than the n=3 used here, the DSA top-k result may not hold, and the ranking across the five definitions could shift.","A transferable pattern is the strict separation of measurement from decision-making, with rejection archives; this recipe can apply to other LLM-generated systems code where correctness and performance are both gated.","The paper does not quantify the human effort spent authoring skills and curating references, so the practical trade-off between harness investment and agent autonomy remains an open cost question."],"forward_implications":["Retained kernels improve mean latency over supplied baselines by 1.62x, 18.05x, 29.68x, 1.12x, and 13.70x across the five operator definitions, so LLM-driven kernel generation can be practical when bounded by a harness.","Human-assisted search outperforms fully autonomous search in matched final evaluation; two fully autonomous artifacts fall below the supplied baseline, indicating that human-curated references and directions are decisive in this setting.","Large gains come from structural, hardware-aware route changes rather than micro-tuning alone, such as sparse-attention route specialization and replacing a broad dispatch table with a Blackwell chunked recurrent kernel.","Correctness gates based on few trials are not sufficient for boundary-sensitive operators; high-trial replay is needed as a final validation gate before an artifact is treated as conservative and release-ready.","Archiving rejected probes, profiler reports, and promotion decisions prevents repeated exploration of failed routes and makes the overall promotion process auditable."],"fun_headline_variants":["Harness-tamed LLMs write kernels 1.12–29.68× faster","Human harness, not AI alone, lifts kernel speed 1.12–29.68×","LLM kernel gains need harness: 1.12–29.68× speedups","Separate harness, human skills: GPU kernels 1.12–29.68×","Kernel speedups 1.12–29.68× with harness-first LLM guide"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The results rest on the assumption that passing the harness's low-trial correctness gate with n=3 trials per workload is enough to certify the retained kernels as correct; the paper's own appendix shows this gate can pass artifacts that fail on rare top-k boundary cases.","fun_headline_variants_meta":{"raw":{"variants":["Harness-tamed LLMs write kernels 1.12–29.68× faster","Human harness, not AI alone, lifts kernel speed 1.12–29.68×","LLM kernel gains need harness: 1.12–29.68× speedups","Separate harness, human skills: GPU kernels 1.12–29.68×","Kernel speedups 1.12–29.68× with harness-first LLM guide"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000404,"raw_usage":{"total_tokens":1933,"prompt_tokens":731,"completion_tokens":1202,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":1084}},"tokens_in":475,"tokens_out":1202,"duration_ms":10707,"temperature":1.0,"reasoning_tokens":1084,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:26:39.248098+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a 500-trial correctness replay of the retained DSA top-k artifact over every workload in the contest distribution. If any workload produces one exact-match-ratio mismatch, the 18.05x speedup claim for that artifact is false; the paper's own appendix already observed 3/200 mismatches on one workload at 100 trials, so this is a concrete check.","supporting_citations":[],"review_version":1}