{"id":"ce52c204-f9cb-46c4-a5ae-bc2b2cf45c4b","arxiv_id":"2506.10401","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A TVM-based pipeline generates 20,000 CUDA-CPU code pairs, and fine-tuning code LLMs on this data improves transpilation success and CPU performance, with an average speedup improvement of 43.8%.","lead":"This paper builds a large dataset of paired CUDA and CPU code using the TVM compiler, then fine-tunes code LLMs on it to improve automatic CUDA-to-CPU translation. The authors report an average 43.8% speedup improvement for the translated CPU operators and release a benchmark called HPCTransEval.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on unverified dataset correctness, and the paper's own Listing 3 shows a concrete indexing bug in the dependency-free CPU output; without independent ground-truth validation, gains may reward copying flawed TVM output.","rationale":"The reader's weakest_assumption targeted exactly the right region: correctness is checked against TVM's own baseline rather than against an independent ground truth. I agree with that general concern, but I want to sharpen it with concrete evidence from the manuscript itself. Listing 3, which is presented as the dependency-free CPU output, contains an indexing error that makes the kernel compute the wrong result. If this reflects the actual generated data, training labels are unreliable, and the Execute Pass metric would reward models that mimic the buggy TVM output. That is the most load-bearing weakness of the central claim: the dataset's quality is the precondition for every downstream result. I also notice that the abstract's 43.8% average improvement appears to use a nonstandard normalization (1 minus the before/after ratio) rather than the conventional relative improvement, making the headline number ambiguous. Neither issue would be disqualifying if artifacts and an independent reference validation were provided, so I do not move to REJECT; the paper's direction is promising and the KernelBench results partly sidestep the TVM-as-ground-truth problem. But the manuscript as submitted cannot support the central claim with confidence, and a conditional acceptance requiring the artifact release plus independent numerical validation is the honest verdict.","tokens_in":17254,"tokens_out":5950,"duration_ms":67134,"concrete_test":"Obtain the released repository (or, if added, the exact commit and dataset download) and run an independent reference check. For every HPCTransEval case and a stratified sample of the 20k training pairs, execute the TVM-free CPU code and the CUDA code on random inputs with the same shapes, and compare outputs against PyTorch/NumPy CPU references with a tight tolerance. Specifically, check whether Listing 3's indexing bug appears in the generated dependency-free code. Then re-tune Qwen2.5-Coder-14B on only the labels that pass this independent check and re-run the Table 2 and Table 3 evaluations. If the Execute Pass or speedup deltas change materially, the central claim that fine-tuning on this dataset improves transpilation is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The pipeline has a single source of truth: TVM generates both the CUDA and CPU code, and Section 3.6 validates LLM output only against TVM's own expected results. If the TVM-generated CPU label is wrong, the 'correct' target is wrong and the evaluation propagates that error. The paper's own Listing 3 gives a concrete example: inside the inner loop over i2 (0..8), it writes compute[i1] = cosf(data[i1*9+i2]);, which stores only the last i2 per i1 and loses 323 of 324 outputs, whereas the CUDA code and Listing 2 compute every element. If that listing is representative of the dependency-free code used as training data, many labels are semantically incorrect, and a model that reproduces the same buggy reduction would pass the Execute Pass check. The headline 43.8% average improvement also deserves scrutiny: Table 4's per-operator improvements appear to be computed as 1 - (before/after) rather than the standard (after-before)/before. With the standard definition, M1's average across operator types is roughly 124%, so the reported metric is ambiguous and the baseline for 'original code supported by PyTorch or TVM' is never pinned down. No commit hash, dataset download, or code for the modified TVM is provided, so none of these issues can be checked from the paper alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HPCTransCompile, a framework that uses the TVM compiler and auto-scheduler to generate paired high-performance CUDA and CPU C code, augmented by a graph-based construction method, and introduces HPCTransEval, a benchmark for CUDA-to-CPU transpilation. The authors fine-tune three open-source LLMs (Qwen2.5-Coder-14B, DeepSeek-Coder-V2-Lite, OpenCoder-8B) on 20k generated code pairs using LoRA, and report that fine-tuning improves compile rate, execute rate, and speedup ratio on both HPCTransEval and an extended KernelBench-C benchmark, with the abstract claiming an average speedup-ratio improvement of 43.8%.","tokens_in":17510,"tokens_out":8893,"duration_ms":92256,"significance":"If the generated dataset is semantically correct and the evaluation is sound, the framework is a useful contribution: it provides a scalable method to synthesize transpilation training data from an AI compiler, a new benchmark, and evidence that domain-specific fine-tuning can improve CUDA-to-CPU code quality. The inclusion of KernelBench-C as an external benchmark is a strength, and the paper's overall experimental design (clear before/after comparison, multiple models) is commendable. However, the paper's own Listing 3 shows a concrete indexing error in the dependency-free CPU code, the correctness validation in Section 3.6 is anchored to TVM's own outputs, and the headline average improvement is not defined and appears to use a nonstandard formula. These issues put the central claims of dataset quality and performance improvement into question.","major_comments":[{"comment":"Listing 3, presented as the 'CPU C code without TVM dependency', is semantically incorrect relative to Listing 2. The inner loop over i2 (0..8) writes compute[i1] = cosf(data[i1*9+i2]) each iteration, so only the last i2 value survives; the output is a reduction over i2 rather than the elementwise cosine shown in Listing 2 and Listing 1. If this listing is representative of the dependency-removal pass output, the 20k training labels in HPCTransData are systematically wrong. Because Section 3.6 validates LLM-generated code only against TVM's own expected results, a model that reproduces the same buggy reduction would pass the Execute Pass check. The authors must correct this example, provide an automated correctness check against an independent reference (e.g., PyTorch CPU or hand-written kernels) for at least a random subset of the dataset, and report how many pairs survive such a check.","section":"§3.3, Listing 3"},{"comment":"The headline 'average improvement of 43.8%' in the abstract is not defined anywhere in the paper. The operator-type percentages in Section 5.2 (e.g., 15.5%, 69.9%, 225.9%, 131.5%, 340%) use the standard formula (after-before)/before; averaging these 15 values from Table 4 gives approximately 121%, not 43.8%. I can reproduce 43.8% only by averaging 1 - (before/after) over those same 15 entries. The paper must state the exact formula used for the headline number, report the baseline values, and present standard relative improvements in tables and text.","section":"Abstract and §5.2 / Table 4"},{"comment":"No error bars, number of repeated runs, or statistical significance tests are reported for any compile, execute, or speedup metric. Execution time is inherently noisy, and the speedup claims (e.g., M1 Level 1 speedup ratio 2.90 to 6.75 in Table 2) are single-point estimates. The authors should report the mean/median across multiple runs with variance, and state how the execution time was measured (e.g., number of trials, warm-up, Turbo/boost settings). Without this, the speedup improvements cannot be reliably assessed.","section":"§4.2, Tables 2–4, Figures 3–5"},{"comment":"The evaluation is partially circular: the expected results used to judge correctness come from TVM's own generated code, and HPCTransEval is composed of TVM-generated code, while the training data is also TVM-generated. This means the fine-tuned models may be rewarded for reproducing TVM's compilation artifacts and any TVM errors. The inclusion of KernelBench-C mitigates transfer concerns, but the correctness-validation circularity remains. The authors should validate correctness on a subset against an independent ground-truth implementation (e.g., PyTorch CPU or a manually verified reference) and report the agreement rate.","section":"§3.6, §4.1"},{"comment":"The modified TVM passes and the generated dataset are not made available in sufficient detail: only a repository URL is given, with no commit hash, dataset download link, or description of the modified compilation passes. Because the central contribution is the dataset and benchmark derived from these modifications, and because Listing 3 raises a correctness question, the artifact must be released with versioning and enough implementation detail for the community to reproduce the dependency-removal step and the 20k pairs. The current reproducibility level is insufficient to verify the paper's claims.","section":"§3.3, §4.1, §6"}],"minor_comments":[{"comment":"The word 'improvemnet' is a typo for 'improvement'; the same issue appears in the abstract and in Section 4.2 ('libirary' for 'library').","section":"Abstract"},{"comment":"The text says 'The framework is illustrated in Figure 2', but the framework is shown in Figure 1 and the Source Code Evaluation Runtime is shown in Figure 2; the cross-references are swapped.","section":"§3 and Figure captions"},{"comment":"The sentence 'Even fused operations (Level 2) see a 50% increase in Compile Pass ... for M1' is contradicted by Table 2, which shows M1 Level 2 Compile Pass increasing from 0.42 to 0.51 (21.4%); the 50% figure corresponds to M3. Please correct the claim.","section":"§5.1"},{"comment":"The phrases 'a 0.9x speedup' and similar are ambiguous: they could mean a speedup factor of 0.9 (a slowdown) or an improvement of 0.9×. Specify whether the reported values are speedup factors relative to PyTorch, and define the baseline per figure.","section":"§5.2 and Figures 3–4"},{"comment":"HPCTransEval is described as containing 10 complex-model building blocks, but no quantitative results are reported for these blocks; Section 5.1 only states that none ran correctly. Please provide explicit numbers (e.g., compile/execute pass rates) or state why they were excluded from the tables.","section":"§4.1"},{"comment":"The search is configured with 'the number of measurement trials to 200' but there is no discussion of how sensitive the final schedule is to this choice or how TVM's measurement noise is handled; please clarify whether the 200 trials are per candidate or total.","section":"§3.5"},{"comment":"The evaluation metric 'Speedup Ratio' is defined as comparing to 'the original code supported by PyTorch or TVM', but the baseline is not pinned down per benchmark; for KernelBench-C, state explicitly that the baseline is e.g., PyTorch CPU inference, and for HPCTransEval, state the TVM-generated CPU reference.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The Listing 3 example is a red flag: it demonstrates a semantic error in the very artifact the paper contributes, and the evaluation procedure would not catch it because expected results come from TVM. The 43.8% headline metric also appears to be computed with a nonstandard formula, which makes me cautious about the reliability of the quantitative claims. These issues are fixable in principle, but they require substantial additional validation and artifact release, not just local edits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a careful read, but the headline result is not yet supported.\n\nWhat's new: a pipeline that uses TVM to auto-generate CUDA-CPU code pairs, strips TVM-specific dependencies from the CPU code, applies graph-based augmentation to expand the dataset, and a new benchmark (HPCTransEval). Fine-tuning three LLMs on this data does produce meaningful gains on both HPCTransEval and the external KernelBench, so the direction is plausible.\n\nThe main soft spot is dataset correctness. The evaluation validates generated CPU code against TVM's own expected output, so any TVM bug propagates silently. Listing 3 is a concrete red flag: the 'dependency-free' CPU code writes compute[i1] inside the inner loop over i2, so only the last i2 value survives. The correct output, as in Listing 2, is compute[i1*9+i2]. If that listing is representative of the training data, the labels are systematically wrong, and a model that copies the bug passes the Execute Pass check.\n\nThe headline '43.8% average improvement' is also murkier than it looks. Table 4's per-operator numbers don't match the standard improvement definition. Layout Transform goes from 0.74 to 1.53, which is a 107% relative gain, not the claimed 225.9%. The paper doesn't pin down the baseline for 'original code supported by PyTorch or TVM', and there are no error bars or multiple runs.\n\nOn the plus side, the experimental design is mostly sound: three models, two benchmarks, a scaling study. The dependency-removal engineering is real, and the graph augmentation idea is a reasonable way to increase diversity. The paper is not circular in the worst sense, since KernelBench is external, but the TVM-only benchmark is self-referential.\n\nReproducibility is a problem. The GitHub link is given, but no commit hash, dataset download, or code for the modified TVM is in the paper, so none of these issues can be checked.\n\nBottom line: the idea is worth referee time, but the paper needs a major revision and artifact release. The Listing 3 bug alone should be enough to require a re-check of the entire dataset generation. Send it to review, but with a strong request for fixes.","headline":"A promising pipeline for generating CUDA-CPU training data via TVM, but the headline speedup is undercut by questionable dataset correctness and inconsistent metric definitions.","tokens_in":18116,"tokens_out":3458,"would_cite":false,"duration_ms":37464,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning code models on compiler-generated CUDA-CPU pairs improves translated CPU kernels' speedup by an average of 43.8%.","keywords":["CUDA transpilation","high-performance computing","LLM fine-tuning","TVM auto-scheduler","performance portability","code generation dataset","HPCTransEval","KernelBench"],"falsifier":"Take a set of hand-verified CUDA kernels with independently computed outputs, run the fine-tuned models on them, and compare the translated CPU code against those independent outputs and against hand-optimized CPU references; if execute-pass rates or speedups fall to baseline once the compiler's output is no longer the oracle, the claimed gains would be an artifact of the evaluation.","tokens_in":17017,"feed_emoji":"⚡","tokens_out":6205,"duration_ms":69653,"temperature":0.7,"pith_summary":"This paper claims that the main obstacle to good CUDA-to-CPU translation by large language models is not model capability but the absence of high-quality training pairs, and that a compiler can supply those pairs automatically. Using the TVM compiler's auto-scheduler, the authors generate 20,000 matched high-performance CUDA and CPU C code pairs, strip the CPU code of compiler-library dependencies, and add graph-based augmentation to widen coverage. After LoRA fine-tuning on this dataset, three open code models produce CPU translations that are more often compilable and correct, with a 43.8% average improvement in CPU speedup ratio. The paper also releases HPCTransEval, a benchmark of 100 primitive operators, 100 computation graphs, and 10 model building blocks, as an architecture-independent test for CUDA transpilation.","feed_headline":"CUDA-to-CPU speedups rise 43.8% after LLM fine-tuning","feed_subtitle":"A compiler-generated dataset of matched kernel pairs turns base LLMs into faster CPU code translators.","key_machinery":"The generating mechanism is a modified TVM pipeline: TVM's auto-scheduler searches optimal scheduling configurations for each operator or computation graph, then emits a CUDA kernel and a CPU C kernel from the same computation description. The paper strips the CPU code of TVM library dependencies, labels each pair with hardware context and optimization strategy, and builds a source-code evaluation runtime using LLVM ORC JIT to compile and execute translated code in place of the reference kernel. Dataset diversity comes from a graph-based augmentation algorithm that expands operators into directed acyclic computation graphs through random connections, branch expansion, and path merging, while fine-tuning uses LoRA to update only a small fraction of model parameters.","core_discovery":"The central discovery is that compiler-generated high-performance code pairs provide enough training signal for LLMs to learn hardware-aware CUDA-to-CPU transpilation. On HPCTransEval Level 1, the strongest model's speedup ratio rises from 0.83 before fine-tuning to 2.35 after, with execute pass rising from 0.30 to 0.62; on KernelBench Level 1, the same model's execute pass rises from 0.28 to 0.51 and its speedup ratio from 2.90 to 6.75. The paper attributes the gains to clean, dependency-free code pairs and graph-based augmentation, which let fine-tuned models adopt compiler-style optimizations such as replacing per-thread Mersenne Twister construction with lightweight hash-based randomness and replacing GPU-style shared-memory and tiling patterns with cache-friendly CPU loops.","pith_inferences":["The same pipeline should transfer to other backends the compiler supports, such as AMD GPUs or other accelerators, which would turn CUDA-to-platform portability largely into a data-generation problem.","Because correctness is judged against the compiler's own reference output, benchmark scores conflate 'translates the computation' with 'reproduces the compiler's numerical choices'; an independent numerical oracle would sharpen the claim.","The case-study shift from Mersenne Twister to a hash-based PRNG suggests fine-tuning teaches LLMs compiler-style micro-optimizations; a testable extension is to check whether such substitutions preserve statistical quality in dropout under adversarial seed patterns.","The small Level 3 gains suggest model-level translation bottlenecks are about API selection rather than kernel optimization, so adding graph-level pairs with explicit Torch-call choices could extend the method to whole networks."],"forward_implications":["If the central claim is right, LLM-based transpilation can be improved without manual kernel rewrites, by changing the training data rather than the architecture or prompt.","Compiler-generated pairs cover operator categories that are hard to curate by hand; the largest measured HPCTransEval gains are on layout-transform operators, reaching 225.9%, and compute-intensive operators, exceeding 340%.","Dataset size mostly affects execution speed rather than correctness: correctness plateaus around 5,000 training pairs while speedups continue to improve up to 20,000 pairs.","Gains concentrate at the operator and fused-kernel level; the paper reports that HPCTransEval's complex model building blocks do not execute correctly after fine-tuning, marking a clear boundary for current LLM transpilation ability."],"supporting_citations":[{"why":"Supplies the compiler that generates the high-performance CUDA and CPU code pairs.","marker":"[13]"},{"why":"Provides the auto-scheduler that searches the optimized schedules used to produce the paired kernels.","marker":"[42]"},{"why":"Supplies the external KernelBench task suite used to test generalization to unseen CUDA workloads.","marker":"[31]"},{"why":"Provides the low-rank adaptation method used for parameter-efficient fine-tuning of the models.","marker":"[18]"},{"why":"One of the base models fine-tuned in the experiments and the model showing the largest gains.","marker":"[20]"},{"why":"One of the base models used to evaluate whether the dataset improves a mixture-of-experts code model.","marker":"[45]"},{"why":"One of the base models used to evaluate whether the dataset improves an open-code-focused LLM.","marker":"[19]"},{"why":"Supplies the ORC JIT runtime used to compile and execute translated CPU code in the evaluation harness.","marker":"[8]"}],"fun_headline_variants":["LLM fine-tuning lifts CUDA-to-CPU speedups by 43.8%","Compiler-generated dataset boosts LLM CUDA transpilation","HPCTransEval: LLMs gain 2.35x speedup on CUDA-CPU port","AI-compiler dataset turns LLMs into CUDA-to-CPU translators","Graph-augmented code pairs sharpen LLM CUDA porting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The translations are judged correct when they match the output of the compiler-generated reference code, so the evaluation inherits whatever numerical behavior, including any bugs, the compiler itself has.","fun_headline_variants_meta":{"raw":{"variants":["LLM fine-tuning lifts CUDA-to-CPU speedups by 43.8%","Compiler-generated dataset boosts LLM CUDA transpilation","HPCTransEval: LLMs gain 2.35x speedup on CUDA-CPU port","AI-compiler dataset turns LLMs into CUDA-to-CPU translators","Graph-augmented code pairs sharpen LLM CUDA porting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000363,"raw_usage":{"total_tokens":2003,"prompt_tokens":1040,"completion_tokens":963,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":860}},"tokens_in":656,"tokens_out":963,"duration_ms":9980,"temperature":1.0,"reasoning_tokens":860,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:26:27.520344+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of hand-verified CUDA kernels with independently computed outputs, run the fine-tuned models on them, and compare the translated CPU code against those independent outputs and against hand-optimized CPU references; if execute-pass rates or speedups fall to baseline once the compiler's output is no longer the oracle, the claimed gains would be an artifact of the evaluation.","supporting_citations":[{"cited_title":"TVM: An automated end-to-end optimizing compiler for deep learning","cited_arxiv_id":null,"evidence_quote":"Supplies the compiler that generates the high-performance CUDA and CPU code pairs."},{"cited_title":"H., Haj-Ali, A., Wang, Y., Yang, J., Zhuo, D., Sen, K., et al","cited_arxiv_id":null,"evidence_quote":"Provides the auto-scheduler that searches the optimized schedules used to produce the paired kernels."},{"cited_title":"https://llvm.org/docs/ORCv2.html, LLVM.2018","cited_arxiv_id":null,"evidence_quote":"Supplies the ORC JIT runtime used to compile and execute translated CPU code in the evaluation harness."}],"review_version":1}