{"id":"cffcf2f8-4af2-440d-9dd8-06148f06166e","arxiv_id":"2607.20908","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CudaPerf adds an offline-learned structural code score to execution-based RL rewards and reports higher correctness and speedup for CUDA kernel generation than several strong baselines.","lead":"CudaPerf trains a large language model to write CUDA code by rewarding it both for running fast on test inputs and for matching structural patterns that expert GPU code tends to have. The method beats several strong code-generation baselines on C-to-CUDA and PyTorch-to-CUDA benchmarks, but the paper releases no code or data and reports no error bars.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The structural reward's predictive validity is never measured; without a ranker-generalization check, the Table 3 ablation gains could be from any dense extra reward rather than from CUDA structure.","rationale":"The reader's weakest_assumption is exactly the one I would flag: the structural ranker's out-of-distribution validity is unmeasured. The paper's own text (Section 2.1, Appendix A.4-A.11) shows the features are crude, thresholded proxies, and Section 3 says ranker training data comes from Qwen-3-32B and from BabelTower, which is also used as an evaluation set. The strongest evidence for the central claim, Table 3, cannot distinguish between 'structural knowledge helps' and 'any dense auxiliary reward helps'. I also note secondary issues that reinforce the conditional verdict but do not by themselves invalidate the method: the abstract's 5X speedup claim is inconsistent with Table 1 (11.02X vs. 6.2X is a 1.78x ratio, and 6.41X vs. 3.09X is a 2.07x ratio), and no code, data, or error bars are released. These considerations keep the verdict CONDITIONAL rather than moving it to REJECT, because the reported approach is plausible and the verifiable reward alone may explain much of the gain. The proposed ranker-correlation and random-reward tests would settle the load-bearing question about whether the structural reward is genuinely doing causal work.","tokens_in":15470,"tokens_out":6279,"duration_ms":57086,"concrete_test":"Compute the Spearman rank correlation between Rstr(y) and measured speedup on held-out kernels generated by a model other than Qwen-3-32B (e.g., CUDA Agent outputs or KernelBench reference kernels) that were not used to train the ranker; report rho with confidence intervals. Then rerun the Table 3 combined-reward condition with Rstr replaced by a random score matched in mean and variance to the real Rstr. If rho is near zero or the random-score run reproduces the 89%, 94%, and 93% correctness rows with similar speedups, the structural feature content is not the cause of the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Rstr contributes to correctness and speedup rests on Eq. 1's ranker being a reliable proxy for runtime performance on kernels beyond its training distribution. Section 2.1 trains s_psi on pairwise preferences from Qwen-3-32B-generated CUDA variants, using hand-crafted features with fixed thresholds (Appendix A.4-A.11: coalescing stride scores, occupancy proxy 1.0 for TPB in [128,256] and -0.3 otherwise, etc.). The paper reports no validation of this ranker: no rank correlation with measured speedups, no held-out accuracy, no error bars. Table 3 shows that removing Rstr lowers geomean speedup by 1.72X, 2.92X, and 2.11X across the three benchmarks, but if the score does not track true performance on the eval distributions, those gains may reflect a generic dense reward effect (broader exploration, changed entropy) or overfitting to feature statistics shared with training and evaluation data rather than structural understanding. Because BabelTower is used both for ranker training and for evaluation in Section 3, the BabelTower gain is especially exposed to feature-distribution leakage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CudaPerf, a two-stage RL framework for CUDA kernel generation. In an offline stage, an MLP ranker is trained with a pairwise logistic loss on measured speedups to produce a structural reward Rstr(y) from hand-crafted source-level features (coalescing, arithmetic intensity, occupancy, divergence, synchronization, etc.). In an online stage, GRPO optimizes a composite reward combining this structural reward with an execution-based verifiable reward Rver(y,x) that includes compilation, correctness, speedup, and stability terms. The method also uses multi-turn refinement with execution feedback and structural feature values. The paper reports large gains in correctness and speedup over strong baselines on a new C-to-CUDA dataset, BabelTower, and KernelBench, plus ablations showing both reward components contribute.","tokens_in":15716,"tokens_out":6223,"duration_ms":52511,"significance":"The core idea is timely and potentially valuable: injecting performance-relevant structural priors into RLVR for kernel generation, rather than relying solely on sparse execution outcomes, could improve both sample efficiency and final performance. The curated dataset of 2.9k C-to-CUDA and 1k PyTorch-to-CUDA programs with multiple CUDA implementations is a useful resource if released, and the systematic model-size study in Table 2 is informative. However, the empirical claims are not yet rigorously supported: all tables report single runs without variance or significance testing, the offline ranker is never validated, and the BabelTower evaluation appears contaminated by ranker training data. If these issues are addressed, the paper would be a solid contribution to the RL-for-code-generation literature.","major_comments":[{"comment":"The offline ranker is trained using BabelTower data, as stated in Section 3: 'We use existing dataset like BabelTower ... to some extent'. CudaPerf is then evaluated on the BabelTower evaluation set in Table 1 (94% correctness, 9.02X speedup). This is a training/evaluation overlap that directly contaminates the BabelTower results and can inflate the reported improvements over Qwen-3-32B. The paper must either train the ranker without BabelTower or evaluate on a genuinely held-out set with a clear data split; the current presentation does not support the BabelTower claims.","section":"Section 3 (Training offline ranker) and Table 1 (BabelTower row)"},{"comment":"The structural reward Rstr is a fitted MLP trained on pairwise preferences derived from measured speedups. No validation of this ranker is reported: no held-out accuracy, no rank correlation with measured speedups, no comparison with a linear baseline. Because Table 3 attributes large gains (1.72X-2.92X speedup and 7-9% correctness) to the presence of Rstr, the results are equally consistent with a generic dense-reward effect that improves RL credit assignment. Please report ranker validation on held-out tasks, and add a control condition where the structural reward is replaced by a reward-matched dense signal that does not encode CUDA structure, to support the interpretation that structure, not merely density, drives the gains.","section":"Section 2.1, Eq. (1), Appendix A.13, Table 3"},{"comment":"The pass@5 protocol is not defined. The text states only that correctness means passing all test cases and performance is the geomean speedup, but it does not specify how the 5 candidates are sampled, how correctness is aggregated across them (e.g., any-of-5 vs majority), or which candidate's speedup enters the geomean (e.g., fastest correct candidate vs average over correct candidates). Without this definition, the headline speedups (11.02X, 9.02X, 6.41X) and all comparisons to baselines are not reproducible. Please specify the protocol and, ideally, report per-task distributions or confidence intervals.","section":"Section 3 (Metrics) and Tables 1-4"},{"comment":"The structural reward is defined inconsistently: Section 2.1 says Rstr = sψ(φ(y)) with an MLP containing hidden layers, while Appendix A.11 gives Rstr(y) = w^T (φ(y)-μ)/σ, a linear form. This discrepancy must be resolved because it affects the reproducibility of the reward and the claim that the ranker is nonlinear. Also, the fixed thresholds in Appendix A.4, A.7, and A.10 and the α weights in Section 2.2.1 are set empirically without sensitivity analysis; given that Table 3's central ablation depends on the calibration of these values, at least a brief robustness check is needed.","section":"Section 2.1 vs Appendix A.11"}],"minor_comments":[{"comment":"The abstract reports 'up to 5X & 3.32X improvements', while the introduction's contribution list says 'up to 5× and 6× speedups over strong baselines such as Qwen-3-32B and CUDA-Agent'; these numbers are inconsistent and should be reconciled.","section":"Abstract and Section 1"},{"comment":"In the performance-reward definition, 'TCPU(y)' is listed twice; the second term should presumably be 'TGPU(y)'.","section":"Section 2.2.1, Verifiable Reward"},{"comment":"There are several typos: 'Arithmatic' should be 'Arithmetic', and 'perforamnce' should be 'performance'.","section":"Abstract and Section 3 (Prompt Construction)"},{"comment":"The text refers to 'Table A.16', but the appendix table numbering is not consistent with the text reference; please correct the cross-reference.","section":"Appendix A.16"},{"comment":"The phrase 'white-box reasoning over program performance' is an overstatement, since Rstr is a learned MLP on hand-crafted features rather than an interpretable performance model; consider softening this claim.","section":"Section 2.2.1"}],"recommendation":"major_revision","confidential_remarks":"The BabelTower training/evaluation contamination is the most serious issue and should be verified by the editor. The missing pass@5 definition and the lack of ranker validation also need thorough attention. The paper's central idea is sound and the dataset is potentially useful, so a major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core idea is new enough: pairing an offline pairwise ranker over static CUDA features with execution-based GRPO does not appear in the cited prior work, which sticks to correctness and speedup rewards. Second, the paper is currently a good idea with incomplete verification. The central claim—that adding a structural reward to the verifiable reward helps both correctness and speed—is plausible and the ablations show it directionally, but the evidence as reported does not pin down whether the gain comes from the CUDA-specific structure or just from a denser reward.\n\nWhat is good: the recipe is concrete and reproducible in principle. The feature extractor is described in detail in the appendix, the MLP ranker is small and cheap, the RL setup follows standard GRPO, and the evaluation covers C-to-CUDA and PyTorch-to-CUDA with several strong baselines. The ablation on reward components (Table 3) and the pass@1/3/5 breakdown in the appendix are the right things to report, and the results are consistent in direction across all three benchmarks. The related-work coverage is adequate and the limitations paragraph is honest about training cost and the model's ceiling.\n\nSoft spots, in rough order of severity. No code or data are released, so none of the numbers can be checked. The structural ranker is never validated: no rank correlation with measured speedups, no held-out accuracy, no seed variance. Since Rstr is a fitted surrogate of Rver, the \"white-box reasoning\" claim in Section 2.2.1 is overstated. BabelTower is used both to train the ranker and as an evaluation benchmark, and the paper does not say how the eval set was excluded from training. Also, the headline results are pass@5 with no error bars; Table 4 shows that at pass@1 the differences shrink, and on BabelTower the baseline QiMeng-MuPa is actually ahead in pass@1 correctness (82 vs 76). One minor thing: the abstract's \"5X\" is the additive speedup difference (11.02 - 6.2), so it is consistent with Table 1, just awkwardly phrased.\n\nI do not think any of this sinks the paper. The stress-test worry about reward hacking is real but unproven; the ablation gains could come from generic dense reward, but they could also come from structure, and the paper as written cannot distinguish the two.\n\nWho this is for: researchers working on RLVR for code generation and GPU kernel synthesis. It deserves a serious referee, provided the first requests are for code, data, and a ranker-generalization analysis. I would send it out rather than desk reject.","headline":"CudaPerf's composite reward idea is genuinely new and plausibly useful, but the paper's evidential base needs a ranker-generalization check and released code before the big numbers can be trusted.","tokens_in":16253,"tokens_out":3677,"would_cite":true,"duration_ms":33368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CudaPerf argues that combining an offline-learned structural code score with execution rewards makes RL-generated CUDA kernels both more correct and faster, with 89% correctness and 11.02x geomean speedup reported on its C-to-CUDA…","keywords":["CUDA kernel generation","reinforcement learning with verifiable rewards","structural code rewards","memory coalescing","pairwise ranking","GRPO","C to CUDA translation","PyTorch to CUDA translation"],"falsifier":"Take a held-out set of CUDA kernels produced by a different generator or written by hand, measure their actual speedup, and compute the rank correlation between $R_{\\mathrm{str}}$ and measured speedup; if the correlation is not clearly positive, the structural proxy is not doing the claimed work. A complementary check is to retrain CudaPerf with the ranker's pairwise labels randomized or inverted and see whether the reported gains in Table 3 survive.","tokens_in":15240,"feed_emoji":"⚡","tokens_out":12367,"duration_ms":99825,"temperature":0.7,"pith_summary":"The paper tries to show that reinforcement learning for CUDA kernel generation should reward program structure, not just measured outcomes. Its framework, CudaPerf, adds an offline-learned structural score over source features such as memory coalescing, occupancy, arithmetic intensity, and synchronization counts to a verifiable execution reward based on compilation, test pass rate, and speedup. The policy is trained with group-relative policy optimization and up to five refinement turns that feed compiler errors, failed test cases, measured speedup, and structural feature values back into the prompt. The central empirical claim is that both reward components matter: ablations report that removing the structural reward drops correctness by 7–9 percentage points and geomean speedup by 1.72–2.92X across three benchmarks, and removing the verifiable reward drops them further. If this is right, RL-based code generation can be guided by interpretable performance features in addition to execution feedback, which matters because execution feedback alone is sparse and expensive.","feed_headline":"CUDA kernels hit 11x speedup when RL rewards code structure","feed_subtitle":"A ranker scores memory and occupancy, and the combined reward beats execution-only training on all benchmarks.","key_machinery":"The load-bearing mechanism is the structural reward $R_{\\mathrm{str}}(y) = s_{\\psi}(\\hat{\\phi}(y))$, produced by an offline pairwise ranker. The ranker is an MLP trained with a logistic pairwise ranking loss to assign higher scores to CUDA variants drawn from the top measured-speedup quartile than from the bottom quartile, using an 11-dimensional normalized feature vector $\\phi(y) = [\\mathrm{coal}, \\mathrm{ai}, \\mathrm{occ}, \\mathrm{div}, \\mathrm{xfer}, \\mathrm{atomics}, \\mathrm{sync}, \\mathrm{kernels}, \\mathrm{tpb}, \\mathrm{gmem}, \\mathrm{ops}]$. These features approximate memory coalescing, arithmetic intensity, occupancy, control divergence, data transfer overhead, atomic and synchronization counts, kernel count, threads per block, global memory accesses, and operation counts. During RL this score is added to the verifiable reward $R_{\\mathrm{ver}}(y,x)$, which applies a compile penalty, a correctness penalty proportional to failed tests, a log-scaled and clipped speedup reward, and a runtime variance penalty. The combined reward is normalized across each group of 16 generated candidates and fed into the group-relative policy objective; the structural term is what provides a dense, execution-free signal that can steer the policy before a correct, fast kernel is found.","core_discovery":"The central discovery, stated on the paper's own terms, is that a composite reward $R_{\\mathrm{ver}}(y,x) + R_{\\mathrm{str}}(y)$ drives a Qwen-3-based policy to generate CUDA kernels that are both correct more often and faster than baselines that use only execution rewards or only structural rewards. On the paper's C-to-CUDA evaluation split it reports 89% correctness and 11.02X geomean speedup; on BabelTower, 94% and 9.02X; and on KernelBench, 93% and 6.41X, with the corresponding comparisons to strong baselines given in Table 1. The ablation in Table 3 shows each reward component contributes: with structural reward only, correctness is 75–81% and speedup 3.7–6.9X; with verifiable reward only, 81–87% and 4.3–9.3X; with both, 89–94% and 6.41–11.02X. The paper interprets this as evidence that the model learns both what works and why it works.","pith_inferences":["A direct test of the paper's proxy assumption would be to measure the rank correlation between $R_{\\mathrm{str}}$ and actual runtime speedup on kernels written by a different generator or by hand; positive correlation would support transfer, while near-zero correlation would suggest the structural reward is overfit to the training distribution.","The 11-feature vector omits several known GPU bottlenecks, including shared-memory bank conflicts, cache reuse, and tensor-core utilization; adding such features could widen the reported speedup gains on memory- and tensor-heavy workloads.","The paper's Section 5 limitations note that execution-based training is expensive and that iterative refinement is bounded by the model and reward design; a natural extension is to anneal the structural reward's weight as the verifiable reward becomes informative, reducing both cost and reward-hacking risk.","The prompt construction sorts few-shot examples by measured speedup, so an evaluation on a benchmark whose distribution differs strongly from the training pool would test whether the gains reflect learned optimization principles rather than familiar example patterns."],"forward_implications":["Removing either reward component degrades every benchmark: without structural rewards correctness falls 7–9 points and geomean speedups fall 1.72–2.92X, while without verifiable rewards the drops are larger, so both signals are load-bearing in the reported system.","The same two-stage recipe—offline ranker, then online RL with composite reward—is applied successfully to both C-to-CUDA and PyTorch-to-CUDA translation, suggesting the approach is not tied to a single translation direction.","Larger base policy models yield stronger results: scaling the Qwen-3 policy from 8B to 32B raises correctness and geomean speedup on all three evaluation sets, consistent with the claim that a stronger prior helps RL for kernel generation.","Because the ranker is trained once offline and the structural reward is execution-free, the method turns a sparse outcome signal into a dense per-candidate score during RL, which is what the ablation attributes the extra gains to."],"supporting_citations":[{"why":"Supplies the GPU optimization principles (memory coalescing, occupancy, synchronization) that justify the structural feature set.","marker":"[NVIDIA, 2023]"},{"why":"Provides the roofline-style reasoning behind the arithmetic-intensity feature in the structural vector.","marker":"[Williams et al., 2009]"},{"why":"Motivates the occupancy proxy by relating thread occupancy to achievable performance.","marker":"[Volkov, 2010]"},{"why":"Kevin is a strong PyTorch-to-CUDA baseline and the source of the multi-turn refinement and beta-DKL=0 training recipe.","marker":"[Baronio et al., 2025]"},{"why":"Supplies the group-relative policy optimization objective used for the online RL update.","marker":"[Guo et al., 2025]"},{"why":"Provides the BabelTower evaluation set used to measure C-to-CUDA correctness and speedup.","marker":"[Wen et al., 2022]"},{"why":"Provides the KernelBench evaluation set used for PyTorch-to-CUDA translation.","marker":"[Ouyang et al., 2025]"},{"why":"CUDA Agent is the main PyTorch-to-CUDA agentic-RL baseline that CudaPerf claims to outperform.","marker":"[Dai et al., 2026]"},{"why":"Qwen-3-32B is the base policy model for RL and the generator of training CUDA variants for the offline ranker.","marker":"[Qwen3-Team, 2025]"}],"fun_headline_variants":["RL with structural rewards writes CUDA kernels 11x faster","CudaPerf: code-structure rewards lift CUDA speedups to 11x","Structural-aware RL beats execution-only for CUDA code gen","Better CUDA kernels from RL that scores memory and occupancy","Reward code structure to get 11x CUDA speedups from RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 11 hand-crafted source-level features and the ranker trained on them are a reliable enough proxy for runtime performance on kernels outside the Qwen-3-generated training distribution; if that proxy fails, the structural reward can push the policy toward code that looks optimized rather than code that runs fast, and the paper's own Section 5 acknowledges the approach is limited by execution cost and by the model and reward design.","fun_headline_variants_meta":{"raw":{"variants":["RL with structural rewards writes CUDA kernels 11x faster","CudaPerf: code-structure rewards lift CUDA speedups to 11x","Structural-aware RL beats execution-only for CUDA code gen","Better CUDA kernels from RL that scores memory and occupancy","Reward code structure to get 11x CUDA speedups from RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1832,"prompt_tokens":1092,"completion_tokens":740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":708,"completion_tokens_details":{"reasoning_tokens":647}},"tokens_in":708,"tokens_out":740,"duration_ms":6939,"temperature":1.0,"reasoning_tokens":647,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:30:47.658597+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of CUDA kernels produced by a different generator or written by hand, measure their actual speedup, and compute the rank correlation between $R_{\\mathrm{str}}$ and measured speedup; if the correlation is not clearly positive, the structural proxy is not doing the claimed work. A complementary check is to retrain CudaPerf with the ranker's pairwise labels randomized or inverted and see whether the reported gains in Table 3 survive.","supporting_citations":[],"review_version":2}