{"id":"cd02e110-6bd6-40ef-bc84-94f4fb15a300","arxiv_id":"2608.08085","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Providing LLMs with specific optimization hints and letting them generate plain C code outperforms using Halide, Exo, and Noarr abstractions for PolyBench optimization in the tested non-iterative setting.","lead":"This paper tests whether giving AI models special programming frameworks helps or hurts when the AI optimizes scientific computing code. It finds that letting the model write plain C code with detailed performance hints beats all three frameworks tested.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The validation tolerance in Section 3.4 (absolute 1e-2, relative 2e-7) is loose enough that semantically incorrect optimized code could pass as valid, which would inflate both the reported validity rates and the best-of-k speedups built only from 'valid' candidates.","rationale":"I read the paper as a carefully scoped empirical comparison of LLM-guided optimization workflows, and I agree with the reader that the central claim is conditional on the validity gate. Among the possible soft spots, the validation tolerance is the most upstream: it affects not only the reported validity rates but also the performance aggregates, because invalid candidates are excluded and the best valid candidate within a budget is selected. A workflow confound (framework results require an extra LLM translation step while direct C starts from a known-good baseline) is real and worth noting, but it is transparently part of the evaluated workflow and would mainly narrow the scope of the claim. The validation concern, by contrast, could change the numbers themselves. I did not find an internal inconsistency or a fatal flaw; the paper provides a replication package and clearly states its assumptions. The reader's conditional verdict is therefore the right default, and the concrete test above would determine whether the concern actually lands. If the stricter check reveals many false positives concentrated in direct C, I would move toward REJECT or UNVERDICTED; if it does not, the conditional acceptance should stand.","tokens_in":14319,"tokens_out":7494,"duration_ms":85332,"concrete_test":"Execute the replication package's validation harness on all implementations that were reported as passing, but compare against an independent reference computed in long double (or with the original C run under a far stricter threshold, e.g., relative 1e-12 and absolute 0). For each benchmark, also run a few out-of-distribution inputs (random values, including zeros and negative numbers) because the PolyBench-provided inputs may be too forgiving. Recompute the validity rates and best valid speedup curves after excluding every candidate that fails the stricter check. If direct C loses more false-positive candidates than Exo/Halide/Noarr, or if the all-hints C advantage in Figure 4 shrinks by more than the reported differences, the headline claim needs to be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 declares a generated implementation valid if its output matches the original C output within absolute tolerance 1e-2 and relative tolerance 2e-7 for all tested input sizes. This tolerance is much looser than double-precision noise; for a reduction kernel whose output terms are O(1), dropping a single small term—e.g., an off-by-one loop bound or a skipped update in a corner tile—produces an error that can sit below the absolute component of the tolerance. The paper argues (Section 3.4) that non-symmetric inputs and dimension sizes non-divisible by common tiling factors make passing 'unlikely,' but that argument is unquantified and does not address small systematic errors such as a missing boundary element or a slightly wrong scaling. The concern is load-bearing because validity is the gateway to everything else: invalid implementations are excluded from performance aggregates, and performance is reported as the best valid output within a budget. If a non-trivial fraction of the 1,258 passing candidates are false positives, the validity-rate advantage of direct C over the DSL workflows and the speedups selected from those candidates are both overstated. The 91 numerical-mismatch failures confirm that the tolerance does catch some errors, but they do not measure false negatives; there is no independent oracle for the remaining passes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a large empirical comparison of LLM-guided code optimization workflows on the PolyBench suite. The workflows differ along two axes: the code representation to which the LLM applies optimizations (direct C, or a translation into Halide, Exo, or Noarr) and the prompting strategy (naive, all-hints, choose-hints, or from-plan). Using GPT-5.1 in a non-iterative setting with five independent attempts per condition, the authors report that direct C generation with specific optimization hints achieves the best geometric-mean speedups and the highest validity rates, that the from-plan approach does not help, and that the DSL abstractions do not provide clear benefits in this setting. The paper includes a replication package and excludes the four benchmarks used for prompt development from the evaluation.","tokens_in":14591,"tokens_out":3287,"duration_ms":34119,"significance":"If the result holds, it is practically significant: it challenges the assumption that established optimization DSLs are the right interface for LLM-guided optimization and suggests that prompt specificity matters more than the representation. The study has notable strengths: a relatively large corpus of 2,080 optimization attempts, four different prompting approaches, three frameworks plus direct C, automated validation, and a public replication package with prompts, code, and results. The main risk is that the validation tolerance (Section 3.4) is loose enough to admit semantically incorrect code as valid; because invalid outputs are excluded from all performance aggregates and the C baseline is used as a fallback, a non-negligible false-positive rate would overstate both the validity-rate comparisons and the best-of-k speedups. The paper itself flags this as a limitation but does not quantify the false-negative risk.","major_comments":[{"comment":"The validation tolerance is load-bearing and is not sufficiently justified. The paper declares a generated implementation valid if its output matches the baseline within an absolute tolerance of 1e-2 and a relative tolerance of 2e-7 for all tested input sizes. For double-precision kernels whose outputs are O(1), a missing boundary term, an off-by-one loop bound, or a slightly wrong scaling can produce errors below 1e-2, and the argument that non-symmetric inputs and non-divisible sizes make such cases \"unlikely\" is unquantified. The 91 numerical-mismatch failures demonstrate that the tolerance catches some errors, but they do not measure false negatives. Since Section 4.2 builds all performance aggregates from valid candidates only and uses the C baseline as a fallback, an inflated validity rate could directly inflate the reported speedups. Please provide evidence that the tolerance is discriminating, for example by running the same validation on intentionally mutated or perturbed versions of the benchmarks, or by re-running the evaluation with a stricter tolerance and reporting whether the conclusions are stable.","section":"Section 3.4"},{"comment":"The central quantitative claims are reported without any measure of uncertainty. Execution times are averaged over three runs after two warm-ups, but the paper reports no confidence intervals, standard deviations, or significance tests for the geometric-mean speedups or for the validity-probability curves. The aggregation in Figure 4 uses best-of-k speedups with the C baseline as a fallback, and the validity denominator includes all attempts; these are reasonable choices, but the headline differences between all-hints direct C and the framework-based approaches could be driven by a few outliers or by measurement noise. Please report per-benchmark results, inter-run variance, and either paired tests or bootstrap confidence intervals to establish that the observed ordering is not a chance artifact of the specific 26-benchmark sample.","section":"Section 4.2, Figure 4"}],"minor_comments":[{"comment":"The translation selection step chooses the candidate whose measured performance most closely matches the original C code; with only three timing runs, this selection can be noisy. It would be helpful to state whether the selection was stable across repeated runs or to use a more robust criterion.","section":"Section 3.1"},{"comment":"The comparison to Merouani et al. is not apples-to-apples: the baselines differ (original PolyBench C versus a Tiramisu re-implementation), the number of LLM invocations differs (5 versus up to 150), and the hardware and model versions are not identical. The sentence claiming results of \"similar order of magnitude\" should be softened or accompanied by a direct re-benchmarking under the same baseline.","section":"Section 4.3"},{"comment":"The assertion that PolyBench inputs are \"intentionally designed with non-symmetric input data and dimension sizes non-divisible by common tiling factors\" is a factual claim about the benchmark suite; please provide a citation or a concrete example in the replication package to substantiate it.","section":"Section 3.4"},{"comment":"The failure-category tags are described as diagnostic and non-disjoint, but the counts (454 compilation errors, 276 runtime errors, 91 numerical mismatches, 10 timeouts) are presented without the overlap information. Reporting the number of attempts that fail in multiple categories would clarify the failure analysis.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution for a systems/CSE venue, and the replication package is a clear strength. The main concern is the validation tolerance in Section 3.4, which is central to the validity-rate and speedup claims; the authors should be asked to provide a quantitative false-negative analysis or re-run under a stricter tolerance. The lack of uncertainty quantification in Section 4.2 is a second key issue. There is no indication of circularity or problematic citation practices; the self-citations to the Noarr paper are topical and do not constrain the outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful empirical comparison. The design—crossing four prompting strategies with four code representations (direct C, Exo, Halide, Noarr) on the same 26 PolyBench benchmarks—is not in the prior literature, and the authors are careful to scope their claims to a non-iterative, single-model setting. The finding that direct C with specific optimization hints beats the DSL pipelines is plausible and worth taking seriously. The replication package is a real asset, and excluding the prompt-tuning benchmarks from evaluation shows good hygiene.\n\nThe soft spots are real but not fatal. The validation tolerance (absolute 1e-2, relative 2e-7) is loose enough that small systematic errors—a dropped boundary term, an off-by-one in a tile—could slip through. The paper's argument that non-symmetric inputs make this unlikely is unquantified, and the 91 numerical mismatches show the tolerance catches some errors but say nothing about false negatives. Since validity is the gateway to both the validity-rate and speedup aggregates, this could overstate the C advantage and the reported best-of-k speedups. The same tolerance applies to all conditions, so the central ranking is probably robust; the precise magnitudes and validity-rate gaps are the uncertain parts.\n\nI also miss confidence intervals or significance tests. With 26 benchmarks and five attempts, a difference like the Exo vs C validity at budget five could be noise. The paper reports aggregate geometric means but no per-benchmark variability, so the reader can't judge that. The translation selection rule (pick the candidate closest to baseline C performance) is a reasonable choice but could bias the downstream comparison—a translation that is already fast is not a neutral starting point. And the comparison to Merouani et al. uses a different baseline, model, and budget, so it's context, not an apples-to-apples benchmark.\n\nWho this is for: people working on LLM-guided code optimization and anyone designing verifiable optimization pipelines. It's evidence that prompt specificity matters more than the DSL abstraction for this generation of models in a non-iterative setting. The paper deserves a serious referee. I'd send it out; in revision I'd ask for tighter validation or a small manual audit, proper error bars, and a more careful framing of the prior-work comparison. The central result will likely hold up.","headline":"A useful, honest head-to-head of direct C vs three DSLs for LLM-guided optimization; the main ranking likely holds, but the loose validation tolerance and missing statistics undercut the precise numbers.","tokens_in":15107,"tokens_out":6354,"would_cite":true,"duration_ms":50018,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Direct C with specific hints outperformed Halide, Exo, and Noarr for LLM-guided optimization.","keywords":["large language models","code optimization","domain-specific languages","prompt engineering","high-performance computing","PolyBench","loop transformations","LLM-guided optimization"],"falsifier":"Run the same all-hints direct-C protocol on the 26 PolyBench kernels, but validate every candidate against many randomly generated inputs whose dimensions are non-divisible and whose values are non-symmetric, in addition to the suite's fixed inputs. If a substantial share of candidates that pass the paper's tolerance fail on the random inputs, or if the speedup ranking between direct C and the three DSL representations flips, the central claim is contradicted.","tokens_in":14138,"feed_emoji":"⚡","tokens_out":9826,"duration_ms":93330,"temperature":0.7,"pith_summary":"This paper asks whether the abstractions built for human-guided code optimization—Halide's separation of algorithm and schedule, Exo's verified rewrite rules, Noarr's composable layouts—help or hinder when a large language model does the optimizing. In a non-iterative, best-of-five protocol on 26 PolyBench kernels with GPT-5.1, it compares four code representations (plain C and LLM-written translations into the three DSLs) under four prompting strategies. Its central finding is that direct C generation with a prompt listing four concrete optimization goals (cache locality, temporary-buffer structure, arithmetic reduction, parallelism) yields the fastest validated code and the highest validity rate, while the DSL representations provide no clear advantage. The paper also finds that asking the model to externalize an abstract optimization plan before writing code does not improve performance and can reduce validity, so prompt specificity matters more than the choice of representation. If correct, these results argue that verifiable LLM-guided optimization should be developed around the LLM's own code output and prompting, not retrofitted onto existing DSL toolchains.","feed_headline":"Plain C beats DSL pipelines for LLM-guided optimization","feed_subtitle":"Across 26 PolyBench kernels, detailed prompts plus raw C gave the fastest valid code and the highest validity rate.","key_machinery":"The engine of the comparison is a fixed-cost best-of-five protocol: for each of 26 PolyBench kernels, the LLM gets five independent tries per configuration; translations into a DSL are selected by closeness to the original C's runtime on SMALL, MEDIUM, and LARGE inputs, and optimized candidates are validated by output matching against the baseline C within an absolute tolerance of $10^{-2}$ and a relative tolerance of $2\\times10^{-7}$ on all tested input sizes. The four prompting strategies are the naïve request, the all-hints request listing the four optimization categories, the choose-hints variant that asks the model to pick suitable hints, and the from-plan pipeline that first produces an externalized abstract schedule in the spirit of Halide/Exo separation and then implements it. The all-hints direct-C cell is the configuration that carries the paper's main conclusion.","core_discovery":"The paper claims that, in its evaluated setting, the best-performing and most often valid configuration is plain C plus an all-hints prompt: the model is told to improve cache locality, use limited temporary buffers, reduce arithmetic work, and parallelize, while preserving semantics. None of the three DSL workflows beats this on measured speedup at EXTRALARGE sizes, and direct C has the highest validity rate regardless of prompting approach. The from-plan workflow, which forces the model to first write an abstract schedule and then follow it, produces no performance benefit over direct optimization and lowers the validity rate for Exo. The paper takes this as evidence that existing optimization abstractions restrict the LLM more than they protect it, and that the model's own adaptive reasoning—guided by explicit hints but not coerced into a fixed plan—is what drives performance.","pith_inferences":["Inference: the paper deliberately excludes compiler/runtime feedback loops; with feedback, DSL-based approaches might close the gap because iterative repair can correct translation errors, so the direct-C advantage should not be assumed to carry over to agentic settings.","Inference: the from-plan result suggests a testable design principle—let the model choose its own reasoning depth—which could be probed by varying how much of the plan is forced and where in the pipeline it is inserted.","Inference: the validation logic is an assumption, not a proof; stress-testing it with adversarial or randomized inputs could change which candidates count as valid and, with it, the speedup aggregates.","Inference: if the ranking generalizes to other LLMs and benchmark suites, the productive direction for verifiable LLM optimization is validating the LLM's free-form C output, rather than constraining the LLM to existing DSL templates."],"forward_implications":["Prompt specificity is the dominant lever: giving the LLM concrete optimization directions consistently outperforms a generic optimize-this request, so prompt design should be treated as a first-class component of LLM optimization tools.","Current DSL abstractions do not pay for themselves in a non-iterative workflow: they neither raise validity nor unlock better speedups, and the translation step can lose performance before optimization begins.","Forcing an explicit plan step is not a reliable way to improve reasoning; the paper's data indicate it can hurt, so future prompting schemes should test plan externalization before adopting it.","A cheap, non-iterative best-of-five strategy with direct C reaches the same order of magnitude as an iterative agentic scheduler on several benchmarks (for example 103× on 3MM), implying that iteration cost is not the only route to high speedups.","Validity is mostly a function of representation: direct C has the highest validity rate, Exo approaches it after five attempts, and Noarr stays slightly lower, giving a concrete target for verifiability research."],"supporting_citations":[{"why":"Supplies Halide, one of the three DSL representations whose pipeline and schedule abstractions are compared against direct C.","marker":"[30]"},{"why":"Supplies Exo, the Python-based rewrite-rule framework used as the second DSL comparison point and the one closest to C in validity after five attempts.","marker":"[15]"},{"why":"Supplies Noarr, the layout-abstraction library used as the third comparison representation.","marker":"[18]"},{"why":"Supplies the PolyBench kernels, input sizes, and output-matching validation workloads on which all speedups and validity rates are measured.","marker":"[29]"},{"why":"Provides the iterative agentic Tiramisu baseline whose speedups and validity rates the best direct-C results are compared against.","marker":"[21]"},{"why":"Identifies the GPT-5.1 model that produced all translations, plans, and optimized code in the experiments.","marker":"[26]"},{"why":"Motivates the from-plan prompting design by claiming that instructing the model to reason stepwise improves task performance.","marker":"[19]"}],"fun_headline_variants":["Plain C with hints beats LLM schedule pipelines in speed test","LLM optimization: direct C outperforms DSL workflows","Abstractions limit LLM optimizers: direct C yields fastest code","Hints and plain C beat schedule pipelines for LLM code tuning","LLM code tuning: raw C and goals win over pipeline abstractions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that matching the original outputs within an absolute tolerance of $10^{-2}$ and a relative tolerance of $2\\times10^{-7}$ on PolyBench's non-symmetric inputs makes it unlikely that semantically wrong code passes validation; if that tolerance is too loose, the reported validity rates and the speedups built only from valid candidates overstate correctness.","fun_headline_variants_meta":{"raw":{"variants":["Plain C with hints beats LLM schedule pipelines in speed test","LLM optimization: direct C outperforms DSL workflows","Abstractions limit LLM optimizers: direct C yields fastest code","Hints and plain C beat schedule pipelines for LLM code tuning","LLM code tuning: raw C and goals win over pipeline abstractions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000719,"raw_usage":{"total_tokens":3223,"prompt_tokens":935,"completion_tokens":2288,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":2213}},"tokens_in":551,"tokens_out":2288,"duration_ms":29366,"temperature":1.0,"reasoning_tokens":2213,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:25:36.108106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same all-hints direct-C protocol on the 26 PolyBench kernels, but validate every candidate against many randomly generated inputs whose dimensions are non-divisible and whose values are non-symmetric, in addition to the suite's fixed inputs. If a substantial share of candidates that pass the paper's tolerance fail on the random inputs, or if the speedup ranking between direct C and the three DSL representations flips, the central claim is contradicted.","supporting_citations":[{"cited_title":"ACM SIGPLAN Notices48(6), 519–530 (2013)","cited_arxiv_id":null,"evidence_quote":"Supplies Halide, one of the three DSL representations whose pipeline and schedule abstractions are compared against direct C."},{"cited_title":"In: Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation","cited_arxiv_id":null,"evidence_quote":"Supplies Exo, the Python-based rewrite-rule framework used as the second DSL comparison point and the one closest to C in validity after five attempts."},{"cited_title":"Parallel Computing p","cited_arxiv_id":null,"evidence_quote":"Supplies Noarr, the layout-abstraction library used as the third comparison representation."},{"cited_title":"net/projects/polybench/files/polybench-c-4.2.1-beta.tar.gz/download","cited_arxiv_id":null,"evidence_quote":"Supplies the PolyBench kernels, input sizes, and output-matching validation workloads on which all speedups and validity rates are measured."},{"cited_title":"In: 34th International Conference on Parallel Architectures and Compilation Techniques","cited_arxiv_id":null,"evidence_quote":"Provides the iterative agentic Tiramisu baseline whose speedups and validity rates the best direct-C results are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies the GPT-5.1 model that produced all translations, plans, and optimized code in the experiments."},{"cited_title":"Advances in Neural Information Processing Systems35, 22199–22213 (2022) PREPRINT 20 J","cited_arxiv_id":null,"evidence_quote":"Motivates the from-plan prompting design by claiming that instructing the model to reason stepwise improves task performance."}],"review_version":1}