{"id":"f785efc1-f6e3-4334-a450-cd694bfc73b8","arxiv_id":"2608.05033","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SparseDitto uses LLM agents guided by structural matrix features and target-GPU measurements to generate custom CUDA kernels for SpMV, SpMM, and SpGEMM, beating cuSPARSE by 2.68x to 2.79x on average.","lead":"SparseDitto is an automated system that writes custom GPU code for sparse matrix operations, tailoring the code to each matrix, operator, and GPU. It reports mean speedups of 2.68x and 2.79x over NVIDIA's cuSPARSE library on two GPUs, with maximum gains over 100x on some inputs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing success-rate and run-to-run variance reporting leaves the headline speedup potentially conditional on lucky LLM draws.","rationale":"After reading the paper in good faith, the architecture is coherent and the evaluation is careful: correctness is validated against cuSPARSE with a predeclared tolerance, the baseline is the fastest cuSPARSE algorithm per task, and the ablation shows a consistent progression. I considered whether the 36% 'outside vocabulary' results or the exclusion of failed SpGEMM baselines indicate cherry-picking; they do not, since the out-of-vocabulary wins are identified post hoc but still represent genuine search outcomes, and excluding tasks where cuSPARSE has no valid reference is standard practice. The reader's weakest assumption about LLM success within budget is on target, but I sharpen it: the missing quantity is not just the success rate but the run-to-run variance. Because GPT-5.6-terra is stochastic and proprietary, a point estimate from one run per task is not reproducible. The paper's bootstrap CI is over tasks and does not capture generation randomness. This is the load-bearing concern because the central claim is an empirical speedup number; if that number is a lucky draw, the claim is not established. A multi-seed rerun on a subset would settle it. I do not see an internal inconsistency that would warrant rejection, so the verdict remains CONDITIONAL (UNCHANGED), pending the requested evidence.","tokens_in":18263,"tokens_out":10306,"duration_ms":112507,"concrete_test":"Re-run the full SparseDitto pipeline on a stratified 20-task subset (5 SpMV, 5 SpMM at K=32, 5 SpMM at K=128, 5 SpGEMM) with 10 independent seeds, varying the LLM sampling temperature around the API default (e.g., 0.0, 0.2, 0.8, 1.0) while keeping all other settings fixed. For each task and seed, record whether a valid kernel was produced within the fixed budget and its speedup over cuSPARSE. Report (a) the per-task success rate, and (b) the mean and coefficient of variation of the per-task speedup across seeds. If the success rate falls below 100% or the CV of per-task speedups exceeds roughly 20%, the headline geometric mean should be qualified as conditional on successful generation and accompanied by a confidence interval over runs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a geometric-mean speedup of 2.68x (RTX PRO 6000) and 2.79x (H200) over cuSPARSE, computed across 350 tasks. Each task is the output of a stochastic LLM-based agent (GPT-5.6-terra, API defaults) running a fixed search budget. The paper reports exactly one run per task and nowhere states how many tasks, if any, failed to yield a numerically valid kernel within budget, nor what happens when all branches fail. If the per-task success rate is below 100%, the reported mean is conditional on success and overstates the expected speedup for a fresh input. The problem is sharpest for the 36% of tasks where the winning kernel is 'outside the selector vocabulary' (Section 4.5): these are the cases where the LLM's random exploration is most influential, and a single lucky branch can produce the 146.61x maximum. Because the LLM is proprietary and its sampling is not seeded, the reported numbers are one draw from a distribution, and the paper provides no evidence about that distribution's spread. The 95% bootstrap interval in Section 4.1 resamples tasks, not generation runs, so it does not address this. Without a success-rate report and a multi-run variance estimate, the headline speedup cannot be distinguished from a favorable sample.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents SparseDitto, an LLM-based agentic system that produces a custom CUDA kernel for a given sparse matrix, operator, and target GPU. The system computes 36 structural features, uses an additive energy model trained on an offline corpus to rank known strategies, and then runs a hierarchical planner, coding agent, and verification agent that compile, numerically validate, and profile candidate kernels on the target GPU. The evaluation covers 60 SuiteSparse matrices across SpMV, SpMM at four widths, and SpGEMM on an RTX PRO 6000 and an H200, reporting geometric-mean speedups of 2.68x and 2.79x over the fastest applicable cuSPARSE configuration, with a maximum of 146.61x, and an end-to-end GCN training speedup of up to 3.39x.","tokens_in":18584,"tokens_out":6963,"duration_ms":76963,"significance":"If the reported results are representative, SparseDitto would be a significant advance: it is the first system in the paper's scope to unify SpMV, SpMM, and SpGEMM under one adaptive framework, and it appears to beat both cuSPARSE and several recent specialized systems on the evaluated workloads. The evaluation is in many ways careful: the baseline is the fastest valid cuSPARSE configuration per task, the tolerance-based validation rule is predeclared and uniformly applied, bootstrap confidence intervals are reported, the training and test matrix sets are disjoint, and an ablation separates the contributions of pattern analysis and architecture-aware planning. The main weakness is stochasticity: because each task is generated by a nondeterministic LLM pipeline with no reported success rate or multi-run variance, the headline numbers may be conditional on successful draws.","major_comments":[{"comment":"The headline geometric-mean speedups rest entirely on the stochastic agentic pipeline, but the paper reports exactly one generation run per task and never states how many tasks produced no numerically valid kernel within the fixed budget (three branches, up to five generate-compile-benchmark iterations, then five more for the two fastest branches). Section 4.1 only reports speedups 'over the validated tasks.' If any task fails all branches, the mean is conditional on success and overstates the expected speedup for a fresh input; the problem is sharpest for the 127 outside-vocabulary tasks in Table 3, where a single lucky branch can dominate (e.g., the 146.61x maximum). The 95% bootstrap intervals resample tasks, not generation runs, so they do not address run-to-run variance. The paper should report the per-operator success rate, the distribution of attempts and failures, what happens when all branches fail, and a multi-seed repeat of a representative subset with a confidence interval over generation runs.","section":"§3.4 and §4.1"},{"comment":"The reported task count is internally inconsistent. Section 4.1 describes 60 matrices with SpMV, four SpMM widths, and SpGEMM, i.e., 360 tasks, and says only that seven large SpGEMM tasks were excluded because cuSPARSE ran out of resources; that would leave 353 tasks, yet Table 3 and the text report exactly 350 tasks (60 SpMV + 238 SpMM + 52 SpGEMM). The manuscript does not explain the missing three SpMM/SpGEMM tasks. In addition, excluding tasks where the baseline cannot run removes exactly the largest and hardest cases and therefore biases the aggregate and the SpGEMM speedup upward relative to a user who still needs those products; those tasks should be reported separately, ideally with an alternative reference or with SparseDitto's absolute performance.","section":"§4.1 and Table 3"},{"comment":"Table 3 shows that tasks whose winning kernel is outside the selector vocabulary have higher mean speedup than inside-vocabulary tasks, and the paper presents this as evidence that the search is not confined to its prior. Because the final strategy is selected after measuring candidates, this comparison is outcome-dependent: the outside-vocabulary set consists of branches that won, so a higher mean is expected even if exploration contributed nothing on average. A fairer test would compare, per task, the best inside-vocabulary candidate against the best outside-vocabulary candidate under identical budgets, or report the success rate of outside-vocabulary branches. As written, the claim in Section 1 that 'those cases carry a higher mean speedup' overstates what Table 3 establishes.","section":"§4.5"}],"minor_comments":[{"comment":"The paper repeatedly references 'supplementary material' (e.g., Section 3.3 and Section 4.4), but the arXiv submission contains no supplementary file and no code or data availability statement; without the generated kernels, prompts, and measurement scripts, the results cannot be reproduced or audited.","section":"Throughout"},{"comment":"The energy objective depends on hyperparameters such as the operator weights lambda_o, the softmax temperature T, and the basis dimensions, but the manuscript reports none of their values and no sensitivity study; please add them.","section":"§3.2, Eq. (9)"},{"comment":"The 'Task only' ablation condition is described only as receiving the task description; specify exactly what the planner and coder receive in that condition so that the ablation is interpretable.","section":"§4.3"},{"comment":"The title of the CB-SpMV reference contains a duplicated 'for for'; please fix the typo.","section":"Reference [4]"},{"comment":"The table would be easier to read if the ratio columns were explicitly labeled 'ours/theirs' and if the Delta-L1 and Delta-L2 columns were identified as percentage points; also define 'L1 req.' in the caption.","section":"Table 1"},{"comment":"The paper notes that the aggregate confidence intervals on the two GPUs overlap, but the per-operator intervals are only shown graphically; please report the numeric intervals for each operator and width.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The main gate for acceptance is the success-rate and variance analysis; the task-count discrepancy should also be fixed. I would also ask the authors to release logs, prompts, and generated kernels before final acceptance, since the LLM is proprietary and the current submission is not independently verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the short version: this is a real systems paper with an unusually careful evaluation, but the headline 2.7x speedup is measured over tasks that produced a valid kernel, and the paper never says how many tasks failed. That gap needs closing before I'd trust the number.\n\nWhat's actually new: SparseDitto is the first LLM-based kernel generator I've seen that covers SpMV, SpMM, and SpGEMM under one framework, with structural features guiding an additive energy model that ranks known strategies, a hierarchical planner that seeds candidate designs, and coding/verification agents that iterate on the target GPU. The 36-feature profiler and the energy model's interpretability are nice pieces. The analysis of the 36% of winning kernels that fall outside the selector vocabulary is worth attention—it shows the search goes beyond its prior, and those out-of-vocabulary wins carry higher speedups.\n\nThe evaluation deserves credit: fastest cuSPARSE config per task, predeclared tolerance-based validation, bootstrap confidence intervals, an ablation, and NCU cache/bandwidth analysis. The comparisons with CB-SpMV, DTC-SpMM, SparseTIR, and HSMU-SpGEMM are the right ones.\n\nSoft spots, in proportion. The biggest is the stress-test concern: one run per task, no success rate, no seeds, proprietary LLM. The bootstrap interval resamples tasks, not generation runs, so it says nothing about LLM variance. If even 5% of tasks fail to produce a valid kernel, the geometric mean over successes overstates what a user should expect. This is fixable—report per-task success, maybe a second run with a different seed, and what the fallback is when no branch yields a valid kernel. Also, the energy model is trained on one GPU and applied to another without retraining; the H200 results being nearly identical could be luck or could mean the ranking generalizes—worth a sentence. Minor: SpGEMM only covers A*A and A*A^T, and the K=128 GCN result is a 1% effect.\n\nI'd send this to peer review. The system is substantial, the claims are important if true, and the missing reproducibility data is addressable in revision. Anyone working on LLM-based kernel generation or sparse auto-tuning should read it, but they should treat the speedup as conditional pending the success-rate data.","headline":"SparseDitto's headline speedups are plausible and the evaluation is careful, but missing success-rate and run-to-run variance data make the numbers conditional on the LLM drawing a favorable sample.","tokens_in":19106,"tokens_out":4490,"would_cite":true,"duration_ms":46472,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM agent that builds a fresh GPU kernel for each sparse matrix, operator, and target GPU outperforms the vendor sparse library by a geometric mean of 2.68x to 2.79x, with wins up to 146x.","keywords":["sparse matrix computation","GPU kernels","kernel generation","large language models","SpMV","SpMM","SpGEMM","sparsity patterns"],"falsifier":"Run the identical pipeline on the same 350-task suite with a different capable code-generation model under the same search budget; if the geometric-mean speedup over the vendor library falls close to 1x, the reported gains are attributable to the specific model's coding ability rather than to the structural analysis and measurement loop.","tokens_in":18090,"feed_emoji":"⚡","tokens_out":10850,"duration_ms":110662,"temperature":0.7,"pith_summary":"SparseDitto claims that no fixed GPU sparse kernel can stay fast across sparsity patterns, operators, and GPU generations, and that a system which builds a fresh kernel for each matrix, operator, and target GPU can do better. It combines structural analysis of the input matrix with a small interpretable ranking model and an LLM-based agentic loop that writes CUDA kernels, checks them numerically, and profiles them on the target hardware. Across 350 tasks spanning three sparse operations, the system reaches a geometric-mean speedup of 2.68x over the vendor sparse library on one GPU and 2.79x on another, with a maximum of 146.61x. This matters because a mismatched sparse representation can cost orders of magnitude: for a single matrix the vendor library itself spans a 350x gap between two of its formats, so adapting the kernel to the pattern is a practical route to speedups that fixed libraries and specialized systems leave on the table.","feed_headline":"Per-matrix GPU kernels beat a tuned sparse library 2.7x on average","feed_subtitle":"An LLM agent tailors a CUDA kernel to each sparsity pattern and chip, cutting time across SpMV, SpMM, and SpGEMM.","key_machinery":"The load-bearing mechanism is an iterative measurement loop wrapped around a structured design space. Each candidate is a configuration tuple \\(\\Pi=\\langle R,S,\\theta_H\\rangle\\): representation \\(R=\\langle L,I\\rangle\\) (value layout plus sparse metadata), schedule \\(S=\\langle P,D\\rangle\\) (parallel decomposition plus dataflow), and hardware mapping \\(\\theta_H\\) (tile sizes, launch geometry, shared-memory usage). Before code generation, a 36-feature structural profile feeds an additive energy model that ranks established strategies, so the search starts near designs that already work for matrices like the current one; after code generation, numerical validation and GPU profiling feed back into the planner, so the target hardware's measurements, not the model's confidence, decide what is kept. This combination is what lets the system both reuse prior knowledge and invent unlabeled strategies.","core_discovery":"The paper's central claim is that sparse-kernel construction should be treated as a joint design problem over representation, execution schedule, and hardware mapping, and that this space is best searched with measurement-driven LLM agents rather than fixed libraries or hand-written rules. Each candidate is a configuration tuple \\(\\Pi=\\langle R,S,\\theta_H\\rangle\\), where \\(R\\) bundles the data layout and sparse metadata, \\(S\\) bundles the parallel decomposition and dataflow, and \\(\\theta_H\\) carries hardware-specific parameters such as tile size and launch geometry. A structural profiler computes 36 features, an additive energy model ranks known strategies from an offline measurement corpus, and a planner seeds several branches that a coding agent implements and a verification agent measures on the target GPU. The reported results support the claim: the generated kernels beat the vendor library in 94% of evaluated tasks, and the largest wins come from kernels that no predefined strategy label describes, which suggests the search escapes its own prior.","pith_inferences":["The same design could be pointed at other sparse operations such as triangular solves or sparse tensor contractions, since the planner already consumes device property tables and the verification loop is measurement-based rather than format-specific.","The 36% of wins outside the predefined vocabulary suggests the ranking model is a warm start rather than an upper bound; a planner that learns to mutate strategy families from structural features might find still more unlabeled winners.","The headline speedups should be read as a property of the current code-generation model under the fixed search budget: a weaker model would reduce the number of valid candidates and would likely shift which matrices lose, so reproducibility depends on model capability, not just on the planning and profiling machinery."],"forward_implications":["If a user runs many sparse workloads on one GPU, the per-matrix kernels beat the vendor library in 94% of evaluated cases, so choosing the kernel itself, not just the library, is where the speedup is.","On a different GPU generation the pipeline regenerates kernels from scratch and retains essentially the same aggregate speedup, so the approach transfers without hand-tuning to new hardware.","Because 36% of the winning kernels implement strategies outside the ranking model's vocabulary and those wins carry higher speedups, the search does not merely pick among known formats.","Replacing sparse aggregation kernels with generated ones improves end-to-end GCN training time by up to 3.39x, so the gains survive the surrounding training loop and are not confined to isolated microbenchmarks."],"supporting_citations":[{"why":"Supplies the cache-friendly blocked SpMV baseline whose matched inputs the generated kernels beat, and provides an adaptive blocking strategy that the system reuses.","marker":"[4]"},{"why":"Supplies the public sparse matrix collection used for both training the selector and constructing the 60-matrix evaluation set.","marker":"[5]"},{"why":"Supplies the Tensor-Core SpMM baseline and the end-to-end GCN protocol; its poor transfer to the test GPU motivates hardware-aware regeneration.","marker":"[7]"},{"why":"Provides the per-task vendor reference implementation and baselines, including the observed 350x performance spread across its formats that motivates the work.","marker":"[22]"},{"why":"Provides the row-wise product dataflow analysis used for the SpGEMM intermediate-work estimator and row binning structure.","marker":"[29]"},{"why":"Supplies the shared-memory hash SpGEMM baseline and the square/rectangular product protocol used in the evaluation.","marker":"[33]"},{"why":"Supplies the sparse-compiler baseline whose enumerated rule set limits coverage, motivating open-ended LLM-based synthesis beyond pre-encoded formats.","marker":"[34]"}],"fun_headline_variants":["LLM agent builds custom kernels, beating cuSPARSE by 2.7x","Per-matrix GPU kernels from LLM outpace sparse library 2.7x","SparseDitto: LLM tailors CUDA kernels per matrix, 2.7x faster","Custom GPU kernels per sparsity pattern: 2.7x speedup"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline depends on the language model succeeding at writing and fixing correct CUDA kernels on most tasks within the fixed search budget; if the model fails too often, no amount of planning and profiling can produce the reported speedups.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent builds custom kernels, beating cuSPARSE by 2.7x","Per-matrix GPU kernels from LLM outpace sparse library 2.7x","SparseDitto: LLM tailors CUDA kernels per matrix, 2.7x faster","Custom GPU kernels per sparsity pattern: 2.7x speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2895,"prompt_tokens":1034,"completion_tokens":1861,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":1767}},"tokens_in":650,"tokens_out":1861,"duration_ms":13708,"temperature":1.0,"reasoning_tokens":1767,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:51:50.687893+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical pipeline on the same 350-task suite with a different capable code-generation model under the same search budget; if the geometric-mean speedup over the vendor library falls close to 1x, the reported gains are attributable to the specific model's coding ability rather than to the structural analysis and measurement loop.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the cache-friendly blocked SpMV baseline whose matched inputs the generated kernels beat, and provides an adaptive blocking strategy that the system reuses."},{"cited_title":"Davis and Yifan Hu","cited_arxiv_id":null,"evidence_quote":"Supplies the public sparse matrix collection used for both training the selector and constructing the 60-matrix evaluation set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the per-task vendor reference implementation and baselines, including the observed 350x performance spread across its formats that motivates the work."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the row-wise product dataflow analysis used for the SpGEMM intermediate-work estimator and row binning structure."}],"review_version":1}