{"id":"a07f5569-7d0c-4b08-8a4f-d58f06aa9403","arxiv_id":"2607.20518","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CANN Bench is an Ascend-NPU benchmark with 53 operators and 1,060 released test cases that scores agent-generated kernels on compilation, correctness, and hardware-anchored performance.","lead":"A Huawei team introduces CANN Bench, a benchmark that scores AI-generated operator kernels on Ascend NPUs by compilation, correctness, and speed relative to both a default PyTorch baseline and an analytical hardware-speed limit. It aims to be the standard yardstick for kernel-writing agents on Ascend hardware.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HAP lower bound Eq. (4) is asserted, not derived; no validation against real 910B2 kernels is reported, so the entire performance-score scale rests on an unverified inequality.","rationale":"The reader's weakest_assumption correctly identifies Eq. (4)'s HAP limit as the most fragile premise. I agree with that identification and with the CONDITIONAL verdict: the benchmark design is plausible, but the central claim of hardware-anchored, artifact-free performance scores depends on THW being a genuine lower bound. The paper provides no derivation details, no worked examples, no validation, and no audit results for violations, so the claim is currently unverified. My recommended verdict remains CONDITIONAL rather than REJECT because the gap is addressable: a focused validation study on a handful of operators would settle whether THW is sound, and the rest of the benchmark (dataset, correctness gate, anti-reward-hacking) can stand independently. I did not find any other concern more load-bearing: the artifact availability gap is real but explicitly disclosed and does not invalidate the design; the dataset-count inconsistency (53 operators × 20 public cases = 1060, but 20+80 hidden per operator would imply 5300 total cases) is a documentation issue about the hidden split, not about the central claim; and the correctness gate's three-region fallback is conservative. The HAP unvalidated lower bound is the single point where the entire numerical output of the benchmark could lose meaning.","tokens_in":16443,"tokens_out":1717,"duration_ms":14846,"concrete_test":"Take 5 representative operators (one per bottleneck class: MTE-dominated, Vector-dominated, Cube-dominated, and two mixed L4 fusions) and 10 cases per operator from the public split. Independently implement the HAP procedure from §3.2 using the stated 910B2 peak parameters, then (a) hand-write an optimized Ascend C kernel per case using the production engineering practices the benchmark targets, and (b) measure Tcand under the paper's own timing protocol. Check whether any Tcand < THW, and how close the best kernel gets to THW (e.g., within 10%, 50%, or 2×). If any Tcand < THW, the lower-bound claim in Eq. (4) is false and the score scale must be re-anchored. If no kernel approaches THW, the bound may be too tight to serve as a meaningful upper anchor.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central scoring innovation is the HAP limit THW = min_δ max_k t_k(δ) in Eq. (4), claimed to be a true lower bound that no real implementation can beat. The load-bearing concern is that this bound is not actually demonstrated. The derivation procedure is described only qualitatively ('estimates tk from FLOP and byte counts and that unit's peak rate'), with no worked example, no closed-form expressions for tk(δ), no proof that the structured implementation-choice space δ covers all feasible schedules, and no empirical validation that THW is attainable or not violable on 910B2. The paper even states 'THW ≤ Tbaseline should hold on every case' and treats violations as audit signals, but no audit results or violation counts are reported. If the analytical accounting omits any unit, overlap constraint, or memory-system effect—e.g., MTE2/MTE3 contention, L2 bandwidth limits, or non-overlappable synchronization bubbles—THW could be either too optimistic (scores below 1 become unachievable and the scale is distorted) or not a lower bound at all (scores above 1 become meaningless). Since Eq. (5) maps scores to a 0–1 scale anchored on THW, every numerical claim about performance headroom inherits this unvalidated assumption. This is an internal-consistency risk, not a disagreement with consensus: the paper itself flags the bound as load-bearing but supplies no evidence for it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CANN Bench is proposed as an open, vendor-maintained benchmark for evaluating AI-generated CANN operator kernels on Huawei Ascend NPUs. The paper reports 53 operators and 1060 test cases in four difficulty tiers, with each operator accompanied by specification, prototype, test cases, golden implementation, and a PyTorch-on-Ascend baseline. A submission is scored on compilation, functional correctness, and performance, combined into a per-operator weighted composite. Performance is graded against an analytical Hardware-Anchored Performance (HAP) limit THW, defined in Eq. (4) as the minimized maximum unit-level execution time, and Eq. (5) interpolates the candidate runtime between baseline and HAP. The paper also describes a layered anti-reward-hacking harness, public/hidden test splits, and plans for a leaderboard, DSL extensions, and broader precision support. The authors position the benchmark inside the official CANN repository and claim it provides a stable, reproducible evaluation scale independent of any training recipe.","tokens_in":16827,"tokens_out":6023,"duration_ms":61958,"significance":"If the manuscript's claims are substantiated, CANN Bench would fill a real gap: kernel-generation benchmarks are dominated by CUDA/Triton targets, and Ascend lacks a widely accepted evaluation scale. The HAP-anchored scoring idea, the detailed correctness gate with region-specific fallbacks, the public/hidden split, and the extensive anti-reward-hacking design are valuable and thoughtfully specified. The paper also gives useful, concrete design criteria (C1-C6) and positions the work carefully against existing benchmarks. However, the manuscript currently demonstrates the design, not the working benchmark: no leaderboard is live, no baseline or THW values are reported, no agent submissions are evaluated, and the core HAP lower-bound claim is asserted rather than validated. Because these are load-bearing for the benchmark's stated purpose, the current version is a design proposal with strong scaffolding rather than a fully evidenced benchmark release.","major_comments":[{"comment":"The HAP limit THW = min_δ max_k t_k(δ) is load-bearing for every performance score, but it is asserted, not demonstrated. The paper gives no closed-form expressions or worked example for t_k(δ), no argument that the structured implementation-choice space δ is exhaustive over feasible schedules, and no empirical validation on real 910B2 kernels. The claim 'No real implementation can violate this bound' depends on the accounting of FLOP/byte counts and peak rates being complete and conservative. The paper even states 'THW ≤ Tbaseline should hold on every case' and treats violations as audit signals, but no audit results or violation counts are reported. Since Eq. (5) and Eq. (6) inherit THW, the entire performance scale is currently unvalidated. I request a detailed derivation for at least a few representative operators, a demonstration that the accounting covers all six units (including M","section":"§3.2, Eq. (4)"},{"comment":"The dataset counts are internally inconsistent. The abstract and §1.4 state 53 operators and 1060 test cases; §2.2 says 'each with about 20 test cases.' But Figure 1d reports per-level case counts n=160, n=320, n=378, n=156, which sum to 1014, not 1060. For L3, 21 operators × 20 public cases = 420, not 378; for L4, 8 × 20 = 160, not 156. Moreover, §2.3 says each operator has a 20-case public split plus an 80-case hidden split, so the total case count should include hidden cases as well (5300 if each operator has 100 total). The paper must clarify whether '1060 test cases' refers only to public cases, correct the figure, and state the exact public and hidden totals. A benchmark's headline dataset statistics cannot be ambiguous.","section":"Abstract, §1.4, §2.2, Figure 1d"},{"comment":"The paper promises a reproducible, benchmarked system but ships no empirical validation. There is no leaderboard URL or live portal (the footnote in §1.4 says it 'will be released in a future update'), no reported baseline_perf_us values, no reported THW values, no bottleneck-distribution table beyond a figure, and no demonstration of the harness scoring an actual agent-generated kernel. The paper says the 'harness and baseline-collection pipeline have completed end-to-end runs on NPU hardware' but provides no measurements. For a benchmark paper, the central deliverable is a working evaluator with published reference values and at least a small set of example scores (e.g., on hand-written kernels or existing agents) that shows the three-axis composite behaves as described. Without such evidence, the claims of reproducibility and a 'shared evaluation scale' cannot be verified by readers.","section":"§1.4, §1.5, §4"},{"comment":"The bottleneck distribution (MTE 63%, Vector 26%, Cube 11%) is presented as a dataset finding, but it is derived entirely from the unvalidated HAP analysis. If Eq. (4) underestimates or overestimates any unit's work, these labels are not reliable. The paper should either provide the underlying per-case THW derivations and audit results or present this figure as a preliminary analytical estimate, not as a measured property of the benchmark.","section":"§3.2, Figure 2a"}],"minor_comments":[{"comment":"The inline formula for EachOperatorScore uses ambiguous parentheses: it is not immediately clear that only the summation term, not the wc·δpass term, is divided by num_of_cases. Eq. (6) clarifies this, but the prose should match exactly.","section":"§1.4"},{"comment":"The FP8 threshold entries are typeset inconsistently (e.g., '2 −3' and '2 −2' appear without superscripts). This makes the table hard to read, especially because the same notation is used for powers of two throughout the paper.","section":"Table 4"},{"comment":"The caption says 'provided baseline execution times' while the y-axis is 'total input elements (M, log scale)'. The caption appears mismatched with the data shown; please revise to describe the element-count distribution.","section":"Figure 1d"},{"comment":"The paper says 'At least 50% non-aligned cases' and 'Within each operator's 20-case public split', but it is not stated whether the same 50% requirement applies to the 80-case hidden split. Please clarify the hidden-split construction guarantees.","section":"§2.3"},{"comment":"The ErrorCount ratio uses max(ErrorCountCPU,1), but the statistical behavior of the ratio when ErrorCountCPU is very small or zero is not discussed. A few sentences on the treatment of zero-count CPU runs would improve the precision standard's reproducibility.","section":"§3.1, Eq. (3)"},{"comment":"The conclusion repeats the phrase 'ground up' and 'shared evaluation scale' several times; tightening the wording would improve readability.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The authors have a well-structured benchmark design and a credible engineering plan, but the manuscript currently functions more as a design/announcement than as an evaluated benchmark. The missing HAP validation and missing empirical results are not cosmetic; they are necessary to support the title's claim of 'Benchmarking Agent Generated Kernels.' I would encourage the editor to require a revision with actual measurements, a worked HAP derivation, and corrected dataset statistics before considering acceptance. If the authors cannot provide such validation within a revision cycle, the paper may be more appropriate as a systems/design contribution or a shorter position paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a thoughtful benchmark design, but the paper is ahead of the artifact. The abstract promises an open, reproducible, vendor-maintained benchmark; the body describes a design, with the leaderboard “forthcoming,” the hidden cases “being prepared,” and the HAP limit asserted rather than demonstrated. The useful novelty is real: co-locating the benchmark in the official CANN repo, the 20+80 public/hidden split, and a hardware-anchored score that could survive software baseline drift. The functional-correctness section is the strongest part — the three-region fallback and the CPU same-precision comparison show genuine care.\n\nThe soft spots are exactly where the reader flagged them. Equation (4) defines T_HW as a min over implementation choices of a max over six execution units, but the paper gives no closed-form t_k, no proof that the implementation space covers all feasible schedules, and no validation on real 910B2 kernels. The claim that no real implementation can violate the bound is a hypothesis, not a result. The paper sets up an audit signal for violations but reports zero audit outcomes. If that bound is wrong, every performance score inherits the error. That is load-bearing.\n\nThere are also concrete internal inconsistencies: 53 operators × 100 cases (20 public + 80 hidden) is 5300, not the 1060 in the abstract. And the abstract says the baseline is PyTorch-on-Ascend, while §3 says T_baseline is the CANN-internal implementation. Both are easy fixes but need to be made before the numbers are trustworthy.\n\nThe artifact is not yet what the paper claims. The repo URL is given, but there is no commit hash, no leaderboard, no published baseline values, no visible public case files. The authors are honest about the leaderboard being in preparation, so this is a status report rather than a finished release, but it means the benchmark does not exist in a checkable form yet.\n\nNone of this is fatal. The core design is sound, and the HAP bound can be either derived properly or downgraded to a reference point. This paper deserves a serious referee, but the referee should send it back with a request for major revision: derive or empirically validate T_HW, fix the count and baseline inconsistencies, and ship at least the public split and the harness so third parties can run it. As it stands, I would not cite it as a working benchmark, but I would cite it as a design proposal for the Ascend kernel-generation community.","headline":"A well-specified benchmark design for Ascend kernel generation, but the HAP lower bound is unproven and the promised artifact is not yet checkable — worth a serious referee, but only after major revision.","tokens_in":17339,"tokens_out":3198,"would_cite":false,"duration_ms":30903,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CANN Bench gives AI-written Ascend kernels a hardware-anchored score, not just a speedup over a software baseline.","keywords":["CANN Bench","Ascend NPU","kernel generation benchmark","AI agents","HAP limit","operator kernels","reward hacking","hardware-anchored performance"],"falsifier":"Find or write a kernel on Ascend 910B2 that, with no delegation or timing tricks, reproducibly runs a CANN Bench case in less time than the published T_HW for that case; if such a kernel exists, Eq. (4) is not a true lower bound. A less adversarial check: independently recompute T_HW from the silicon's published peak rates and measure the fastest known hand-tuned kernel on the same case, and compare.","tokens_in":16359,"feed_emoji":"⚡","tokens_out":4948,"duration_ms":45017,"temperature":0.7,"pith_summary":"The paper introduces CANN Bench, a benchmark meant to give the Ascend NPU ecosystem a shared, reproducible yardstick for AI agents that write operator kernels. It contains 53 operators and 1060 test cases across four difficulty levels, with a 20-case public plus 80-case hidden split per operator. The paper's main design claim is that performance should be graded between a measured out-of-the-box baseline and an analytical Hardware-Anchored Performance (HAP) limit, so a score retains meaning as software versions drift. Scores combine compilation, functional correctness, and performance into one weighted composite, and the harness includes layered defenses against reward hacking. A sympathetic reader would care because this is the first Ascend-focused benchmark positioned inside the official CANN repository with a hardware-anchored upper bound rather than a relative speedup alone.","feed_headline":"AI-written Ascend kernels now scored against hardware limits","feed_subtitle":"CANN Bench anchors performance to a six-unit analytical bound, keeping scores meaningful as software evolves.","key_machinery":"The Hardware-Anchored Performance (HAP) limit, T_HW = min_δ max_{k∈K} t_k(δ), where K is the set of six Ascend execution units and t_k(δ) is work divided by peak rate for implementation choice δ. It captures the runtime a kernel would achieve under perfect overlap, peak unit rates, and zero launch overhead, and serves as the per-case upper anchor. The paired anchor (T_baseline, T_HW) plus the bounded score formula is the mechanism that gives CANN Bench its longitudinal stability and its claimed resistance to measurement artifacts.","core_discovery":"CANN Bench is an open, versioned benchmark for AI-generated CANN kernels on Ascend 910B2. For each of 1060 test cases it records two reference times: T_baseline, the measured runtime of the vendor's out-of-the-box implementation, and T_HW, the HAP limit computed analytically from the operator's FLOP and byte counts and the published peak rates of the six concurrently executing hardware units (cube, vector, MTE1-3, FixP). A candidate kernel's measured runtime T_cand is converted into a bounded score by Score_i = (T_baseline - T_HW)/((T_cand - T_HW) + (T_baseline - T_HW)), which is 0.5 at baseline and 1 at the hardware limit. Operator scores are linear combinations of compilation success, per-","pith_inferences":["The HAP scoring scheme could be adapted to other accelerators that expose a small set of concurrently executing hardware units, effectively turning a single silicon's roofline into a benchmark anchor.","If adopted widely, hardware-anchored scores would let training pipelines optimize against a stable reward rather than chasing a moving baseline, potentially changing how kernel-generation agents are trained.","The paper's precision gate—mean/max relative error with small-value and cancellation fallbacks—is a useful template for other benchmarks that need to distinguish genuine kernel errors from dtype floor noise.","The benchmark's own audit signal, any case with T_cand < T_HW triggers review, could in practice surface modeling gaps in the analytical procedure as quickly as it surfaces cheating, making the anchor self-correcting."],"forward_implications":["If T_HW is a genuine lower bound, performance comparisons between kernels remain meaningful across CANN releases, because only the lower anchor drifts with software versions.","The 20-public/80-hidden case split means leaderboard scores cannot be gamed by overfitting to disclosed shapes.","The three-axis composite makes compilation failure catastrophic (zero score), so agents are rewarded only for kernels that both compile and run correctly.","Because MTE dominates the bottleneck labels on 63% of cases, the benchmark highlights data-movement-limited operators as the main optimization target on Ascend.","Housed in the official CANN repository and versioned with operator contracts, the benchmark can grow with new operators, precisions, and hardware generations."],"fun_headline_variants":["CANN Bench: AI kernels vs real NPU limits","AI kernel scores hit real Ascend hardware limits","CANN Bench grades AI agents on real NPU limits","Benchmark anchors AI kernel scores to hardware limits"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire performance scale rests on T_HW being a true lower bound; if the six units cannot actually overlap perfectly at peak rates, or if the FLOP/byte accounting misses a path, a real kernel could run faster than the 'limit' and every score loses its hardware-grounded meaning.","fun_headline_variants_meta":{"raw":{"variants":["CANN Bench: AI kernels vs real NPU limits","AI kernel scores hit real Ascend hardware limits","CANN Bench grades AI agents on real NPU limits","Benchmark anchors AI kernel scores to hardware limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000827,"raw_usage":{"total_tokens":3482,"prompt_tokens":806,"completion_tokens":2676,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":2613}},"tokens_in":550,"tokens_out":2676,"duration_ms":15794,"temperature":1.0,"reasoning_tokens":2613,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:08:32.880537+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find or write a kernel on Ascend 910B2 that, with no delegation or timing tricks, reproducibly runs a CANN Bench case in less time than the published T_HW for that case; if such a kernel exists, Eq. (4) is not a true lower bound. A less adversarial check: independently recompute T_HW from the silicon's published peak rates and measure the fastest known hand-tuned kernel on the same case, and compare.","supporting_citations":[],"review_version":1}