{"id":"a0d6e8d5-48af-4d22-82bd-46c4dd13cf47","arxiv_id":"2507.14403","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"NPUEval evaluates LLM-generated NPU kernels on real AMD hardware, finding that frontier models rarely produce vectorized code, with average vectorization scores near 10%.","lead":"NPUEval is a new benchmark that asks large language models to write efficient AIE kernels for AMD NPUs and tests them on real hardware. Early results show most models can produce correct but scalar code, while even frontier models average only about 10% vectorization, suggesting LLM-driven NPU kernel optimization is still an open problem.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Vectorization score is uncalibrated: reference vectorized kernels score only 13–30%, so the ~10% LLM average may not demonstrate failure without per-kernel reference ceilings.","rationale":"The reader's weakest assumption correctly identified the vectorization score as a potential weak point; my concern sharpens that into a specific calibration problem: the paper provides only two reference scores (13% and 30%) and does not report per-kernel achievable vectorization for the benchmark's 102 operators. Because the metric is raw VPU-cycle share rather than speedup relative to a fixed reference, low absolute scores do not by themselves imply poor LLM vectorization. The paper's own Section 7 reference range of 10-30% makes this tension explicit. I do not challenge the correctness tolerances (Section 3.2.2) as the primary issue, since the benchmark's main empirical contribution is about efficiency. The benchmark is genuinely novel and the open-source evaluation stack is a real strength, but the central interpretation needs a calibration table. Since the reader already conditioned acceptance on validation of this metric, my analysis does not move the verdict; it adds a concrete test that should be required before the headline claim is accepted as strong evidence.","tokens_in":13102,"tokens_out":3508,"duration_ms":44852,"concrete_test":"For all 102 NPUEval prompts, compile and run the dataset's canonical C++ solution (or a hand-written vectorized reference matched to the prompt's data movement) through the released harness, and record per-kernel vectorization scores; also record scores for scalar implementations for calibration. Then compare the reported LLM averages in Figure 5 against this per-kernel ceiling distribution. If the median reference score is below ~20%, the ~10% LLM average is close to the achievable range and the 'challenging dataset' conclusion needs reframing; if the median reference score is above ~40%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLMs average only ~10% vectorization rests on Section 6.3's metric: vectorization score = VPU cycles / total cycles. This is an unnormalized utilization fraction, not a normalized speedup, and the paper's own reference vectorized kernels in Appendix A score only 13% (eltwise_add_bf16_vector) and 30% (conv2dk1_i8_vector). If many of the 102 operators are memory-bound or dominated by scalar address arithmetic, even a perfectly vectorized kernel may score near 10-20%. Section 7 even states that SoTA open source kernels 'typically see a vectorization factor of 10-30%.' The LLM average of ~10% therefore sits near the bottom of the reference range, not obviously far below it, so the headline 'LLMs are bad at vectorization' is under-supported. Additionally, Section 6.3 scores kernels that fail functional tests as 0%, conflating correctness failure with vectorization failure. Without per-kernel reference vectorization scores, the main empirical claim is not interpretable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"NPUEval introduces a benchmark for evaluating LLM-generated kernel code for AMD NPUs (AIE tiles), consisting of 102 common ML operators. The evaluation harness compiles generated C++ kernels with LLVM-AIE/MLIR-AIE, runs them on actual NPU hardware, checks functional correctness against NumPy behavioral models, and computes a vectorization score defined as the fraction of cycles spent on the VPU. The paper evaluates a range of proprietary and open-weight LLMs under zero-shot, compiler-feedback, and RAG conditions, reporting that average vectorization scores remain around 10% even with feedback and vectorized examples, while select kernels reach 50%+ scores. The authors position NPUEval as the first benchmark for LLM-based NPU kernel generation and plan to release the dataset and code.","tokens_in":13319,"tokens_out":3811,"duration_ms":45253,"significance":"If the empirical claims are robust, NPUEval fills a real gap: existing code-generation benchmarks emphasize functional correctness and GPU kernels, while NPU programming is fragmented and underrepresented in LLM training data. The paper's strengths include evaluation on real hardware with an open-source compiler stack, a relatively large operator set (102 kernels), and a detailed failure analysis with concrete examples. The result that frontier LLMs achieve only modest vectorization on specialized NPU kernels is a useful data point for the community. However, the central quantitative claim is currently weakened by an uncalibrated performance metric, potential confounds between correctness and vectorization scoring, and the absence of public code/data. The paper would be much stronger with per-kernel reference ceilings, a normalized or conditional metric, and a released artifact.","major_comments":[{"comment":"The vectorization score (VPU cycles / total cycles) is not calibrated against per-kernel reference implementations. The paper's own reference vectorized kernels achieve only 13% (eltwise_add_bf16_vector, Listing 1) and 30% (conv2dk1_i8_vector, Listing 2), and Section 7 states that state-of-the-art open-source kernels 'typically see a vectorization factor of 10-30%'. An LLM average of roughly 10% therefore sits near the bottom of the reference range rather than clearly below it. Without per-kernel reference vectorization scores for all 102 operators, the headline claim that current LLMs 'struggle' at NPU vectorization is not quantitatively established. Please report reference scalar and vectorized scores per kernel, or normalize the metric by the observed reference ceiling, and discuss the distribution of attainable scores.","section":"Section 6.3, Appendix A, Section 7"},{"comment":"Scoring kernels that fail functional tests as 0% conflates correctness failure with vectorization performance. Because functional pass rates in Table 1 are well below 100% for most models (e.g., 20-70% even after recompilation), the average vectorization scores in Figure 5 are depressed by correctness failures rather than reflecting the VPU utilization of the generated code. Please report vectorization scores separately for functionally passing kernels, or present correctness and vectorization as two independent axes, to avoid attributing a correctness failure to a vectorization failure.","section":"Section 6.3"},{"comment":"The correctness tolerances of 1e-2 to 3e-2 absolute error are quite loose, and the paper does not justify these thresholds per operator or provide a sensitivity analysis. Since functional correctness is one of the two primary benchmark axes and failing kernels are assigned a 0% vectorization score, the interaction between loose tolerances and the reported scores should be quantified. Please show how many kernels pass or fail when the tolerance is tightened (e.g., 1e-3), or provide per-operator reasoning for the chosen thresholds.","section":"Section 3.2.2"}],"minor_comments":[{"comment":"The caption lists the examples in an order that does not match the figure. Figure 6a is GPT-4o (scalar loop), 6b is Llama-3.1-70b (hallucinated API), and 6c is Claude 3.7 Sonnet (scalar loop with conditional), but the caption states 'scalar loop with conditional (claude-3-7-sonnet), incomplete vectorization (gpt-4o), and hallucinated API usage (llama-3.1-70b)', which is inconsistent.","section":"Figure 6 caption"},{"comment":"The word 'grevious' should be 'grievous' in the sentence discussing the effect of omitting dataflow information.","section":"Appendix B"},{"comment":"The paper states that the dataset and evaluation code 'will be released', but no link or public artifact is provided. For a benchmark paper, an anonymous or persistent repository link would substantially aid reproducibility and community adoption.","section":"Abstract / Section 7"},{"comment":"All performance results appear to be based on single hardware runs with greedy decoding. The paper notes this in Section 7, but adding variance information (e.g., repeated runs or pass@k for at least a subset of models) would strengthen the quantitative claims.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim about low LLM vectorization scores is plausible but currently rests on an uncalibrated metric. The authors' own reference kernels score 13-30%, so the ~10% LLM average is less damning than presented. The fix is within scope: provide per-kernel reference scores or normalize. I also note that the benchmark and RAG database are AMD-authored; this is a reasonable tooling choice, but independent validation or a clear statement of potential conflict would help. The lack of a public artifact is a concern for a benchmark paper, though the authors state it will be released."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read if you care about LLM code generation for accelerators. The paper introduces NPUEval, a benchmark of 102 AIE kernel prompts for AMD NPUs, with on-hardware correctness and cycle-level performance metrics. That is a genuine first: KernelBench and TritonBench target mature GPU ecosystems, while NPU programming is fragmented and underrepresented in training data. The harness is built on open source tools and the dataset is promised under a permissive license. That is real value. The authors also report honestly, including the surprising DeepSeek R1 degradation with RAG, and include a prompt-ablation study in Appendix B.\n\nThe core claim, that frontier LLMs average about 10% vectorization and therefore struggle at NPU vectorization, needs calibration. The vectorization score is VPU cycles over total cycles — a utilization fraction, not a normalized speedup. The paper's own reference vectorized kernels score 13% and 30% (Appendix A), and the paper notes that SoTA open source kernels typically see a vectorization factor of 10–30%. An LLM average of 10% sits at the bottom of that range, not clearly below it. Without per-kernel reference scores for all 102 operators, the headline overclaims; many kernels may be memory-bound or dominated by scalar address arithmetic, so even perfect vectorization could land near 20%. The issue is compounded by scoring failed functional tests as 0% for performance, which conflates correctness with efficiency. That is the main soft spot.\n\nThe other limitations are more standard: no released artifacts yet, no error bars, a single hardware platform, one compiler backend, and loose correctness tolerances. These are fixable, and the paper is honest about them.\n\nThis deserves peer review, not desk rejection. A referee should ask for per-kernel reference ceilings, a separation of correctness from vectorization, and ideally the artifact release. If those land, this becomes a useful benchmark for a growing area. I would bring it to a reading group; the metric discussion alone is worth the hour.","headline":"Useful first benchmark for NPU kernel generation, but the headline vectorization claim needs per-kernel reference ceilings before it shows LLMs are bad at vectorization.","tokens_in":13800,"tokens_out":3104,"would_cite":true,"duration_ms":32071,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NPUEval is a benchmark that measures LLM-generated NPU kernels on real hardware and reports only about 10% average vectorization for frontier models.","keywords":["NPU kernel generation","LLM code generation benchmark","vectorization score","AI Engine","compiler feedback","retrieval-augmented generation","kernel correctness evaluation","AMD NPU"],"falsifier":"Re-run the 102 kernel prompts with tightened correctness tolerances (for example 1e-3) and compare vectorization scores against measured wall-clock throughput; if many previously passing kernels fail under stricter tolerances, or if high vectorization scores do not track faster execution, the reported LLM capability numbers would be misleading.","tokens_in":12924,"feed_emoji":"🧠","tokens_out":4008,"duration_ms":45004,"temperature":0.7,"pith_summary":"NPUEval is a benchmark of 102 common machine-learning operators written as AMD AI Engine kernel prompts, evaluated on real NPU hardware through an open-source compiler stack. The paper's central claim is that frontier LLMs, even when given compiler error feedback and retrieved vectorized examples, average only about 10% on its vectorization score, making NPU kernel optimization a hard and meaningful open problem. A few kernels do reach above 50% vectorization with out-of-the-box reasoning models, showing partial latent knowledge of NPU programming. If the benchmark is right, it gives the field a reproducible target and exposes a clear gap between functionally correct code generation and hardware-efficient code generation.","feed_headline":"New NPU benchmark: frontier LLMs vectorize only ~10 percent","feed_subtitle":"NPUEval runs 102 kernels on real AMD hardware; best models hit 50 percent on a few but average near 10 percent.","key_machinery":"The load-bearing object is the vectorization score, defined as the fraction of total NPU cycles spent executing vector instructions on the AIE vector processing unit, measured cycle-accurately on real hardware; kernels that fail functional tests count as zero. The supporting machinery is the evaluation harness built from NumPy behavioral models, including ml_dtypes for bfloat16, the LLVM-AIE compiler with up to ten feedback-retry iterations, MLIR-AIE for graph and data-movement configuration, and a retrieval-augmented generation pipeline that supplies open-source vectorized kernel examples.","core_discovery":"The paper introduces NPUEval as, to its knowledge, the first benchmark for evaluating LLMs on vectorized NPU kernel generation, and reports that state-of-the-art LLMs achieve only roughly 10% average vectorization across all 102 kernels, with any kernel that fails functional correctness scored as zero. Functional pass rates improve substantially with recompilation, reaching 60-70% for several models after five attempts, while vectorization stays low overall. The strongest out-of-the-box result comes from the DeepSeek R1 reasoning model, which exceeds 50% vectorization on selected kernels without retrieved examples; adding retriever-augmented examples improved most models but hurt R1 because the retrieved kernels used compiler-specific pragmas that LLVM-AIE ignores. The paper presents this gap as evidence that LLM-assisted NPU kernel optimization is still in its early stages.","pith_inferences":["The same prompt-and-behavioral-model methodology could be extended to other NPU or accelerator ISAs, with compiler-backend-specific retrieval as a first testable improvement.","The 10% average partly reflects a zero-score policy for any functionally failing kernel; pass@k or partial-credit scoring might produce different model rankings.","A direct wall-clock speedup study on end-to-end workloads would test whether the vectorization score truly predicts real efficiency rather than merely measuring VPU activity.","NPUEval could serve as a fine-tuning dataset for making LLMs fluent in NPU-specific APIs, not just as an evaluation set."],"forward_implications":["If the vectorization score is accepted as a proxy for kernel efficiency, frontier LLM output is far from production-ready for NPUs: the reported average is roughly 10%.","Compiler feedback materially improves functional pass rates for several models, so iterative recompilation should be a standard component in accelerator code benchmarks.","Smaller models more often write scalar code and pass functional tests, while stronger models attempt vectorization and hallucinate APIs; NPUEval can distinguish these failure modes.","Retrieval-augmented generation with vectorized examples raises average VPU utilization for most models, but can be counterproductive when retrieved examples use incompatible compiler pragmas.","The fully open-source stack runs on commodity laptops, so the benchmark can be reproduced and extended to other accelerator families."],"supporting_citations":[{"why":"Supplies the default absolute error tolerance of 1e-2 used in correctness evaluation and frames the related line of GPU kernel-generation benchmarks.","marker":"[17]"},{"why":"Provides the open-source MLIR-AIE framework used to configure the AIE array, build kernels, and define data movement.","marker":"[22]"},{"why":"Provides the open-source LLVM-AIE compiler used both to compile generated kernels and to produce compiler feedback for iterative retries.","marker":"[25]"},{"why":"Supplies the IRON bindings used by the harness to configure the NPU and move data in and out of the AIE tiles.","marker":"[20]"},{"why":"Provides the HumanEval-inspired prompt and behavioral-test structure that NPUEval adapts to kernel code.","marker":"[1]"},{"why":"Supplies the ml_dtypes library used to generate NumPy-compatible bfloat16 behavioral models for kernels.","marker":"[21]"},{"why":"Provides open-source AIE kernel examples used to build the retrieval-augmented generation database.","marker":"[26]"}],"fun_headline_variants":["LLMs average 10% vectorization on new NPU kernel benchmark","NPUEval: first NPU kernel benchmark for LLMs yields ~10% average","Frontier LLMs score ~10% on NPU kernel optimization benchmark","New benchmark NPUEval shows LLMs struggle with NPU kernels","AI coding gap: LLMs achieve only 10% on NPU kernel benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's conclusions rest on the assumption that the vectorization score, the fraction of cycles spent on vector instructions, is a good proxy for kernel efficiency, and that correctness tolerances of 1e-2 to 3e-2 are strict enough to validate outputs.","fun_headline_variants_meta":{"raw":{"variants":["LLMs average 10% vectorization on new NPU kernel benchmark","NPUEval: first NPU kernel benchmark for LLMs yields ~10% average","Frontier LLMs score ~10% on NPU kernel optimization benchmark","New benchmark NPUEval shows LLMs struggle with NPU kernels","AI coding gap: LLMs achieve only 10% on NPU kernel benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000279,"raw_usage":{"total_tokens":1689,"prompt_tokens":1006,"completion_tokens":683,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":583}},"tokens_in":622,"tokens_out":683,"duration_ms":7769,"temperature":1.0,"reasoning_tokens":583,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:56:07.026879+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 102 kernel prompts with tightened correctness tolerances (for example 1e-3) and compare vectorization scores against measured wall-clock throughput; if many previously passing kernels fail under stricter tolerances, or if high vectorization scores do not track faster execution, the reported LLM capability numbers would be misleading.","supporting_citations":[{"cited_title":"Fork of LLVM to support AMD AIEngine processors, https://github.com/Xilinx/mlir-aie, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the open-source MLIR-AIE framework used to configure the AIE array, build kernels, and define data movement."},{"cited_title":"An MLIR-based toolchain for AMD AI engine-enabled devices, https://github.com/Xilinx/llvm-aie, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the open-source LLVM-AIE compiler used both to compile generated kernels and to produce compiler feedback for iterative retries."},{"cited_title":"Efficiency, expressivity, and extensibility in a close-to-metal npu programming interface, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the IRON bindings used by the harness to configure the NPU and move data in and out of the AIE tiles."},{"cited_title":"A stand-alone implementation of several NumPy dtype extensions used in machine learning, https://github.com/jax-ml/ml_dtypes, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the ml_dtypes library used to generate NumPy-compatible bfloat16 behavioral models for kernels."},{"cited_title":"An open-source exploration framework for first time users of the AMD Ryzen AI Neural Processing Unit, https://riallto.ai/, 2024","cited_arxiv_id":null,"evidence_quote":"Provides open-source AIE kernel examples used to build the retrieval-augmented generation database."}],"review_version":1}