Pith. sign in

REVIEW 4 major objections 9 minor 33 references

Coding agents write correct parallel code with little prompting, but real speedups depend on the algorithm, the language, and the model—and one strong model gets none.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 11:29 UTC pith:OP37S3RV

load-bearing objection Useful bake-off showing correctness is easy and speedup is sparse, but the Sonnet-7/15 vs GPT-0 scoreboard sits on noisy single-shot timings. the 4 major comments →

arxiv 2607.26083 v1 pith:OP37S3RV submitted 2026-07-26 cs.SE

Cross-Model Cross-Language AI Coding Agent Performance: Accuracy and Speed of Parallel CLRS Algorithms

Shiqi Cheng , Evelyne Ringoot , Rabab Alomairy , Alan Edelman This is my paper
classification cs.SE
keywords LLMAgentic AIHPCParallel AlgorithmsJuliacode generationruntime efficiencyCLRS
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether today’s AI coding agents can turn ordinary serial algorithms into working parallel code, and whether that code actually runs faster. It tests three agents on twelve classical sorting, graph, and search algorithms in C++, Python, and Julia, counting how many user prompts it takes to get correct code and then measuring speedup against serial baselines. The agents usually reach correctness with modest effort, but meaningful speedup is uneven: one model produces the strongest gains, another produces none despite being correct, and no language wins every category. The point is practical—if agents are going to help with parallel software, accuracy alone is not enough; runtime efficiency has to be treated as a first-class measure of success.

Core claim

Modern coding agents can generate functionally correct multi-threaded versions of CLRS sorting, graph, and search algorithms across C++, Python, and Julia with only modest user-level prompting, yet turning that correctness into real parallel speedup is strongly algorithm- and language-dependent. Sonnet 4.6 delivers the strongest overall performance gains, GPT 5.4 yields no measurable speedups despite consistent correctness, C++ is most consistently helpful on graph algorithms, and Python and Julia show their largest gains on search—while sometimes regressing on graphs. Correctness is therefore an incomplete score for parallel code generation; runtime efficiency must sit beside it.

What carries the argument

An iterative, agent-in-the-IDE protocol: start from a fixed serial baseline and a shared parallelization prompt, feed back benchmark or error output until correctness, then issue a bounded number of efficiency prompts, counting user-level prompts while measuring speedup versus custom serial code and third-party libraries on the same inputs.

Load-bearing premise

That this particular prompting loop—fixed start, error feedback, occasional human hints when stuck, and at most two non-improving efficiency prompts—fairly reveals each agent’s true ability to produce fast parallel code.

What would settle it

Rerun the same twelve algorithms and three languages with a stricter no-human-hint protocol, or with free choice of parallel cutoffs and backends, and check whether GPT still never speeds up and whether Sonnet still leads on the same language–algorithm pairs at matched problem sizes and thread counts.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Benchmarks for coding agents on parallel work should score runtime speedup, not only pass/fail correctness.
  • Model choice matters for performance engineering even when all models can write correct parallel code.
  • Language choice should be matched to algorithm class: C++ for graph-style work, Python/Julia more often for search-style gains.
  • Agent-generated parallel code can still regress versus serial on some graph and search cases, so human performance review remains necessary.
  • Performance-aware prompting and automated profiling feedback are natural next levers if the correctness–speed gap is to close.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Training and eval sets that reward only compilable parallel patterns may entrench “correct but slow” agents like the GPT result here.
  • Forbidding serializing cutoffs in the prompt may hide the practical parallelizations engineers actually ship, so a follow-up with allowed thresholds would test ecological validity.
  • The recurring Julia type and threading-package failures suggest language tooling and docs quality shape agent success as much as model skill.
  • Extending the same protocol to GPUs and larger problem sizes would show whether the correctness–speed gap widens or narrows under heavier parallelism.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 9 minor

Summary. The manuscript reports an empirical evaluation of three coding agents (Cursor Composer 2.0, GPT 5.4, Claude Sonnet 4.6) generating parallel multi-threaded implementations of twelve CLRS algorithms (sorting, graph traversal, search) in C++, Python, and Julia. Using an iterative prompting protocol in the Cursor IDE — an initial fixed prompt, error-log feedback, optional human causal hints, and at most two non-improving efficiency prompts — the authors measure iterations to first correctness and parallel speedup against custom serial baselines and (for Julia graphs) Graphs.jl. The headline findings are: (i) GPT 5.4 and Sonnet 4.6 reach correctness on the first user prompt in all languages, while Composer 2.0 needs more iterations, especially in Julia; (ii) correctness does not translate into speedup — GPT 5.4 achieves no parallel speedup in any language, while Sonnet 4.6 achieves speedup in 7 of 15 language–algorithm combinations; (iii) no language dominates: C++ is most consistent on graphs, Python and Julia do best on search, and both regress on some graph algorithms.

Significance. If the results hold, this is a useful addition to the growing parallel-code-generation benchmarking literature (ParEval, PCEBench), and its cross-language × cross-model design on non-embarrassingly-parallel algorithms (sorts, graph traversals with priority queues, selection) addresses a genuine gap: prior multi-language work (e.g., Valero-Lara et al.) covered massively parallel kernels without speedup measurement. The paper deserves credit for several honest practices: it reports negative and regressive results plainly (Table IV/VI regressions), it flags its own anomalous C++/Sonnet N=10^4 spike and attributes it to hardware oversubscription, it justifies the Polyester.jl choice with a direct comparison against Base.Threads (Table V), it includes a third-party library baseline (Graphs.jl, Fig. 4), and it is transparent about the ambiguity of counting user-level prompts versus internal tool calls. The central qualitative message — that correctness is cheap but parallel performance is not, and that runtime efficiency should be a first-class benchmark metric — is well motivated and likely to be of interest to the SE/HPC community. However, the quantitative scoreboard underlying the cr

major comments (4)
  1. [§III.B–III.C, Tables IV and VI] Tables IV and VI report exactly one serial and one parallel runtime per (language, algorithm), yet the study evaluates three agents. It is never stated which agent's generated code produced these numbers, or whether the table shows the best agent, a fixed agent, or an aggregate. This is load-bearing: the abstract's claims that 'C++ is most consistently parallelizable for graph algorithms' and that 'Python and Julia achieve the largest speedups on search algorithms' rest on these tables, and the cross-model scoreboard (Sonnet 7/15, GPT 0/15) cannot be reconciled with tables that contain no per-agent breakdown. Please either report per-agent runtimes/speedups for graph and search as was done for sorting (Table III), or state explicitly what each row represents.
  2. [§II.C, Tables IV and VI] The timing protocol — one warm-up run plus the mean of three timed runs, each on a *different* dataset — conflates dataset variance with machine noise, and no spread (min/max or standard deviation) is reported anywhere. Several cells that determine the win counts sit at the noise floor: Julia BFS 1.01×, Julia DFS 1.03×, Python DFS 1.07×, Python Minimum 1.08×, C++ Dijkstra 1.09×, Python BFS 1.00×. The paper itself documents instability of single-configuration timings (the C++/Sonnet N=10^4 spike present at 32 threads but not 16). Worse, the Julia search narrative ('uneven', 'dramatically slower') is built on microsecond-scale cells: Binary Search 0.020 vs 0.015 ms (a 5 µs difference reported as 1.29×) and Minimum 0.014 vs 0.031 ms (0.43×). At that scale, timer resolution, JIT warm-up state, and GC dominate a 3-run mean. The speedup>1 classification of these cells, and hence part of the 7/
  3. [§II.B–II.D, §III] Each (agent, language, algorithm) cell appears to be a single generation attempt from a stochastic model, with a human in the loop providing error-cause hints 'if the model continues to fail' (§II.D). The headline cross-model comparison — Sonnet 7/15, GPT 0/15 — therefore rests on n=1 samples per cell, and the frequency and content of human hints per agent is not reported, so prompting effort is not held constant across agents. The adaptive efficiency-prompt stopping rule (continue while improving; stop after two non-improving prompts) further favors agents whose first efficiency prompt happens to help. The correctness claims (first-prompt success) are plausibly robust to this, but the speedup scoreboard is not. At minimum, report the number and nature of human interventions per cell and replicate each cell with a small number of independent sessions (e.g., 3–5 fresh conversations) to sh
  4. [§II, Tables III–VI] The provenance of the serial baselines is unclear. Speedup is defined against 'custom serial baselines,' and §II.D says the agents were given 'serial implementations of the algorithms' — but it is not stated who wrote these, whether the identical serial code is used for all three agents and all timing runs, or whether the agents modified the serial code during the session. If the serial baseline differs across agents or was itself agent-touched, cross-agent speedup comparisons are confounded (an agent paired with a slow serial baseline wins by construction). Please state the baseline provenance explicitly and confirm a single fixed serial implementation per (language, algorithm) is used in all speedup ratios.
minor comments (9)
  1. [§III.C vs §III.B/Tables] The '7 out of 15' Sonnet claim is stated 'at moderate array sizes of N=10^4', but the graph results (Table IV) are at n=10^5 and search results (Table VI) at n=10^6. Please clarify which sizes enter the 15-cell count, and note that 5 sorting + 4 graph + 3 search = 12 algorithms, so the composition of the 15 cells needs explaining.
  2. [§II.D] Typo in the initial prompt block: 'C+++' should presumably be 'C++'. Also 'Ensure the continued usePolyester.jl' is missing 'of'.
  3. [§III.C] Typo: 'mulitthreading' → 'multithreading'. Also 'Base.Threads.jl' is not the conventional name; 'Base.Threads' suffices.
  4. [§II.A] 'fromIntroduction to Algorithms' is missing a space. More substantively: CLRS [27] is a benchmark of algorithmic *reasoning traces* for neural algorithmic reasoning, not a source of reference serial implementations. Please clarify exactly what was taken from CLRS (the algorithm list only, apparently) and where the serial code came from.
  5. [Table VI] Caption says 'at n=10^6 nodes' — these are arrays, so 'elements' is meant. Same issue in §III.C text ('array size n=10^6, Python achieves...').
  6. [Fig. 1] The y-axis label 'Iterations to first correctness' with values like '0 2 4 3 5 11 1 11 1 1' in the figure description is hard to parse; the figure would benefit from grouped bars per language with clear tick labels, and from stating whether '1 prompt' includes internal tool-call self-corrections (the text notes this ambiguity — consider reflecting it in the caption).
  7. [§II.E] Python uses ProcessPoolExecutor (processes, not threads) while Julia and C++ use threads; since the abstract and conclusions frame the study as 'parallel multi-threaded implementation', please discuss whether the process-based Python results are comparable to the thread-based ones, particularly given pickling overhead at these sizes.
  8. [References] Reference [29] (LASSI) is incomplete ('IEEE.' with no venue/year). Reference [7] has a stray '&' artifact. Several 2026-dated arXiv references should be checked for stability.
  9. [General] No artifact (generated code, prompts logs, timing scripts, raw data) is mentioned. Given that the central claims are empirical and the community standard in cs.SE is moving toward artifact availability, releasing the benchmark harness and generated implementations would substantially strengthen the paper.

Circularity Check

0 steps flagged

Empirical bake-off with no derivation circularity: speedups are measured serial/parallel ratios, not fitted inputs renamed as predictions.

full rationale

This paper is a cross-model, cross-language empirical evaluation of coding agents on parallel CLRS algorithms, not a first-principles derivation. Correctness is checked against algorithm-specific tests; speedup is defined as serial/parallel wall-clock time on stated input sizes, averaged over timed runs, and in places compared to third-party baselines (e.g., Graphs.jl). Nothing in the claim chain reduces by construction to a fitted parameter, a self-definition, or a load-bearing uniqueness theorem from the authors. Mandating Polyester.jl for Julia after an internal Base.Threads microbench (Table V) and using custom serial baselines are methodological choices that can affect absolute numbers, but they do not make the reported inequalities (Sonnet wins some cells; GPT wins none; language/algorithm dependence) true by definition. No circular steps meet the quote-and-reduce standard.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The central claims rest on standard empirical-benchmark assumptions plus several protocol choices that shape who appears to “achieve speedup.” There are no invented physical entities. Free parameters are experimental knobs (sizes, threads, stopping rules, backend libraries) rather than fitted theory constants. Domain assumptions about what counts as a valid parallelization (no serializing thresholds; mandated Polyester/ProcessPoolExecutor/std::thread-OpenMP) strongly affect measured speedups.

free parameters (3)
  • Efficiency-prompt stopping rule (continue while faster; else stop after two) = stop after 2 non-improving efficiency prompts
    Directly caps how hard agents may optimize; GPT’s “no speedups” partly depends on this shallow search.
  • Benchmark sizes (e.g. N=10^4 sorts, n=10^5 graphs, n=10^6 search) and 32 threads = N in {1e3..1e5+}; 32 CPU threads
    Chosen evaluation points; small kernels and possible oversubscription change whether thread overhead dominates.
  • Timing protocol (1 warmup + mean of 3 runs) = mean of 3 timed runs
    Defines reported milliseconds and speedup ratios without variance estimates.
axioms (6)
  • ad hoc to paper User-level iterative prompting in Cursor, including optional human error hypotheses, is a valid operational measure of coding-agent parallelization effort and ability.
    Methods §§II.B–II.D define the interaction loop and explicitly note ambiguity of internal tool calls; claims about modest effort and model ranking depend on this.
  • ad hoc to paper Forbidding size thresholds or other structures that would “make the parallel version essentially serial” is required for a fair parallel test.
    Initial prompt in §II.D; this can force unprofitable parallelization on small or irregular workloads and depress speedups.
  • domain assumption Language-specific backends (Python ProcessPoolExecutor despite GIL; Julia Polyester.@batch non-recursively; C++ std::thread/OpenMP) are appropriate vehicles for comparing parallel code generation quality.
    §II.E; backend limits are confounded with agent skill, especially Python scalability and Polyester misuse.
  • domain assumption Speedup versus the paper’s custom serial baselines (and selective third-party libs) indicates whether agents achieved “meaningful” parallel performance.
    Used throughout §III; weak serial baselines can inflate speedup, strong library baselines can reframe “success.”
  • domain assumption CLRS sorting, graph traversal, and search tasks are informative probes of non-massively-parallel reasoning for agents.
    §II.A motivating choice relative to AXPY-style kernels in prior multi-language work.
  • standard math Average wall-clock time from language-native timers on one AMD EPYC node is sufficient to rank parallel implementations.
    Standard HPC microbenchmark practice as described in §II.C and Table II; still omits statistical error bars.

pith-pipeline@v1.2.0-grok45-kimik3 · 15154 in / 3794 out tokens · 82404 ms · 2026-07-30T11:29:57.233082+00:00 · methodology

0 comments
read the original abstract

AI coding agents have quickly become omnipresent in software engineering. Their serial performance, both in terms of accuracy and speed, has been extensively covered. However, recent initial results suggest their parallel programming capabilities lag behind serial programming capabilities. This paper presents a cross-language evaluation of three coding agents -- Cursor's Composer 2.0, GPT 5.4, and Claude Sonnet 4.6 -- on parallel code generation across three algorithm categories -- sorting, graph traversal, and search -- in C++, Python, and Julia. For each algorithm and language pair, we prompt a coding agent to produce a parallel implementation from a serial baseline, track the prompting effort required to achieve both functional correctness and performance improvements, and measure speedup against both custom serial baselines and third-party library implementations. We find that coding agents can produce correct parallel implementations with modest prompting effort, but that achieving meaningful speedup is heavily algorithm- and language-dependent. Sonnet 4.6 delivers the strongest overall performance gains, whereas GPT 5.4 produces no measurable speedups despite consistent correctness. C++ is most consistently parallelizable for graph algorithms, while Python and Julia achieve the largest speedups on search algorithms: no single language dominates across all categories. Python and Julia each achieve speedup on some graph algorithms but regress on others. These findings underscore the impact of including runtime performance efficiency as a main LLM performance metric, in addition to accuracy, particularly for parallel implementations.

Figures

Figures reproduced from arXiv: 2607.26083 by Alan Edelman, Evelyne Ringoot, Rabab Alomairy, Shiqi Cheng.

Figure 1
Figure 1. Figure 1: First iteration index at which correctness is achieved [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Best parallel runtime for each language/coding agent [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Graph algorithm runtimes on a logarithmic scale. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 10 linked inside Pith

  1. [1]

    Ai agentic programming: A survey of techniques, challenges, and opportunities,

    H. Wang, J. Gong, H. Zhang, J. Xu, and Z. Wang, “Ai agentic programming: A survey of techniques, challenges, and opportunities,”

  2. [2]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...

    Pith/arXiv arXiv 2021

  3. [3]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

    Pith/arXiv arXiv 2021

  4. [4]

    DS-1000: A natural and reliable benchmark for data science code generation,

    Y . Lai, C. Li, Y . Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.-T. Yih, D. Fried, S. Wang, and T. Yu, “DS-1000: A natural and reliable benchmark for data science code generation,” inProceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato...

    2023

  5. [5]

    Multipl-e: A scalable and polyglot approach to benchmarking neural code generation,

    F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y . Zi, C. J. Anderson, M. Q. Feldman, A. Guha, M. Greenberg, and A. Jangda, “Multipl-e: A scalable and polyglot approach to benchmarking neural code generation,”IEEE Transactions on Software Engineering, vol. 49, no. 7, pp. 3675–3691, 2023

    2023

  6. [6]

    A survey on code generation with LLM-based agents,

    Y . Dong, X. Jiang, J. Qian, T. Wang, K. Zhang, Z. Jin, and G. Li, “A survey on code generation with LLM-based agents,” 2025. [Online]. Available: https://arxiv.org/abs/2508.00083

    Pith/arXiv arXiv 2025

  7. [7]

    How efficient is llm-generated code? a rigorous & high-standard benchmark,

    R. Qiu, W. Zeng, J. Ezick, C. Lott, and H. Tong, “How efficient is llm-generated code? a rigorous & high-standard benchmark,” in International Conference on Learning Representations, Y . Yue, A. Garg, N. Peng, F. Sha, and R. Yu, Eds., vol. 2025, 2025, pp. 2233–2261. [Online]. Available: https://proceedings.iclr.cc/paper files/paper/2025/ file/06694da0...

    2025

  8. [8]

    Lm4hpc: Towards effective language model appli- cation in high-performance computing,

    L. Chen, P.-H. Lin, T. Vanderbruggen, C. Liao, M. Emani, and B. De Supinski, “Lm4hpc: Towards effective language model appli- cation in high-performance computing,” inInternational Workshop on OpenMP. Springer, 2023, pp. 18–33

    2023

  9. [9]

    Scope is all you need: Transforming llms for hpc code,

    T. Kadosh, N. Hasabnis, V . A. V o, N. Schneider, N. Krien, A. Wasay, N. Ahmed, T. Willke, G. Tamir, Y . Pinter, T. Mattson, and G. Oren, “Scope is all you need: Transforming llms for hpc code,” 2023. [Online]. Available: https://arxiv.org/abs/2308.09440

    Pith/arXiv arXiv 2023

  10. [10]

    Mod- eling parallel programs using large language models,

    D. Nichols, A. Marathe, H. Menon, T. Gamblin, and A. Bhatele, “Mod- eling parallel programs using large language models,”arXiv preprint arXiv:2306.17281, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Ompgpt: A generative pre-trained transformer model for openmp,

    L. Chen, A. Bhattacharjee, N. Ahmed, N. Hasabnis, G. Oren, V . V o, and A. Jannesari, “Ompgpt: A generative pre-trained transformer model for openmp,” inEuro-Par 2024: Parallel Processing, J. Carretero, S. Shende, J. Garcia-Blas, I. Brandic, K. Olcoz, and M. Schreiber, Eds. Cham: Springer Nature Switzerland, 2024, pp. 121–134

    2024

  12. [12]

    Can large language models write parallel code?

    D. Nicholset al., “Can large language models write parallel code?”

  13. [13]

    Pcebench: A multi-dimensional benchmark for evaluating large language models in parallel code generation,

    L. Chen, N. Ahmed, M. Capot ˘a, T. Willke, N. Hasabnis, and A. Jan- nesari, “Pcebench: A multi-dimensional benchmark for evaluating large language models in parallel code generation,” in2025 IEEE Interna- tional Parallel and Distributed Processing Symposium (IPDPS), 2025, pp. 546–557

    2025

  14. [14]

    Llm & hpc: Benchmarking deepseek’s performance in high-performance computing tasks,

    N. Naderet al., “Llm & hpc: Benchmarking deepseek’s performance in high-performance computing tasks,” 2025. [Online]. Available: https://arxiv.org/abs/2504.03665

    arXiv 2025

  15. [15]

    Using chatgpt for converting sequential python programs into parallel code,

    D. B. Czejdo, W. B. Daszczuk, and W. Grabski, “Using chatgpt for converting sequential python programs into parallel code,” inAdvances in Dependable Systems and Networks, W. Zamojski, J. Mazurkiewicz, J. Sugier, T. Walkowiak, and J. Kacprzyk, Eds. Cham: Springer Nature Switzerland, 2025, pp. 41–50

    2025

  16. [16]

    Kernelbench: Can llms write efficient gpu kernels?

    A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. R ´e, and A. Mirhoseini, “Kernelbench: Can llms write efficient gpu kernels?”

  17. [17]

    Fine-tuning gpt-5 for gpu kernel generation,

    A. Tehrani, Y . Emara, E. Wissam, W. Paluch, W. Atallah, Łukasz Dudziak, and M. S. Abdelfattah, “Fine-tuning gpt-5 for gpu kernel generation,” 2026. [Online]. Available: https://arxiv.org/abs/2602.11000

    arXiv 2026

  18. [18]

    Peak: A performance engineering ai-assistant for gpu kernels powered by natural language transformations,

    M. U. Tariq, A. Jangda, A. Moreira, M. Musuvathi, and T. Sorensen, “Peak: A performance engineering ai-assistant for gpu kernels powered by natural language transformations,” 2025. [Online]. Available: https://arxiv.org/abs/2512.19018

    arXiv 2025

  19. [19]

    Available: https://arxiv.org/abs/2502.10517

    [Online]. Available: https://arxiv.org/abs/2502.10517

    Pith/arXiv arXiv

  20. [20]

    Kernelfoundry: Hardware-aware evolutionary gpu kernel optimization,

    N. Wiedemann, Q. Leboutet, M. Paulitsch, D. Wofk, and B. Ummenhofer, “Kernelfoundry: Hardware-aware evolutionary gpu kernel optimization,” 2026. [Online]. Available: https: //arxiv.org/abs/2603.12440

    arXiv 2026

  21. [21]

    Comparing llama-2 and gpt-3 llms for hpc kernels generation,

    P. Valero-Lara, A. Huante, M. Al Lail, W. F. Godoy, K. Teranishi, P. Balaprakash, and J. S. Vetter, “Comparing llama-2 and gpt-3 llms for hpc kernels generation,” inInternational Workshop on Languages and Compilers for Parallel Computing. Springer, 2023, pp. 20–32

    2023

  22. [22]

    Autokernel: Autonomous gpu kernel optimization via iterative agent-driven search,

    J. Jaber and O. Jaber, “Autokernel: Autonomous gpu kernel optimization via iterative agent-driven search,” 2026. [Online]. Available: https://arxiv.org/abs/2603.21331

    arXiv 2026

  23. [23]

    Hintpilot: Llm-based compiler hint synthesis for code optimization,

    H. Jiang, P. Yao, K. Li, T. Lin, C. Wang, and K. Ren, “Hintpilot: Llm-based compiler hint synthesis for code optimization,” 2026. [Online]. Available: https://arxiv.org/abs/2604.15041

    Pith/arXiv arXiv 2026

  24. [24]

    Should ai optimize your code? a comparative study of current large language models versus classical optimizing compilers

    M. R. Rosas, M. T. Sanchez, and R. Eigenmann, “Should ai optimize your code? a comparative study of current large language models versus classical optimizing compilers.” The International Symposium on Code Generation and Optimization, 2024

    2024

  25. [25]

    Performance-aligned llms for generating fast hpc code,

    D. Nichols, P. Polasam, H. Menon, A. Marathe, T. Gamblin, and A. Bhatele, “Performance-aligned llms for generating fast hpc code,” IEEE Transactions on Parallel and Distributed Systems, vol. 37, no. 6, pp. 1330–1341, 2026

    2026

  26. [26]

    Evaluating ai-generated code for c++, fortran, go, java, julia, matlab, python, r, and rust,

    P. Diehl, N. Nader, S. Brandt, and H. Kaiser, “Evaluating ai-generated code for c++, fortran, go, java, julia, matlab, python, r, and rust,” inEuro-Par 2024: Parallel Processing Workshops, S. Caino-Lores, D. Zeinalipour, T. D. Doudali, D. E. Singh, G. E. M. Garz ´on, L. Sousa, D. Andrade, T. Cucinotta, D. D’Ambrosio, P. Diehl, M. F. Dolz, A. Jukan, R. Mon...

    2024

  27. [27]

    The CLRS-text algorithmic reasoning language benchmark,

    L. Markeevaet al., “The CLRS-text algorithmic reasoning language benchmark,” 2024. [Online]. Available: https://arxiv.org/abs/2406.04229

    Pith/arXiv arXiv 2024

  28. [28]

    Do large language models understand performance optimization?

    B. Cui, T. Ramesh, O. Hernandez, and K. Zhou, “Do large language models understand performance optimization?” 2025. [Online]. Available: https://arxiv.org/abs/2503.13772

    Pith/arXiv arXiv 2025

  29. [29]

    LASSI: An LLM-based automated self-correcting pipeline for translating parallel scientific codes

    M. T. Dearinget al., “LASSI: An LLM-based automated self-correcting pipeline for translating parallel scientific codes.” IEEE

  30. [30]

    Parallel code generation with large language model,

    X. Huanget al., “Parallel code generation with large language model,” inProceedings of the 1st FastCode Programming Challenge. ACM, 2025

    2025

  31. [31]

    Cursor: The AI code editor,

    Anysphere, Inc., “Cursor: The AI code editor,” 2024, accessed: April 24, 2026. [Online]. Available: https://cursor.com

    2024

  32. [2024]

    Available: https://arxiv.org/abs/2401.12554

    [Online]. Available: https://arxiv.org/abs/2401.12554

    Pith/arXiv arXiv

  33. [2025]

    Available: https://arxiv.org/abs/2508.11126

    [Online]. Available: https://arxiv.org/abs/2508.11126

    arXiv