Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation

Alexis Huante; Jeffrey S. Vetter; Keita Teranishi; Mustafa Al Lail; Pedro Valero-Lara; Prasanna Balaprakash; William F. Godoy

Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.07103 v1 pith:4ZNIRAXM submitted 2023-09-12 cs.SE cs.AIcs.DCcs.PL

Pedro Valero-Lara , Alexis Huante , Mustafa Al Lail , William F. Godoy , Keita Teranishi , Prasanna Balaprakash , Jeffrey S. Vetter This is my paper

classification cs.SEcs.AIcs.DCcs.PL

keywords llama-2openmpgpt-3kernelsaccuracycodescopilotcuda

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

We evaluate the use of the open-source Llama-2 model for generating well-known, high-performance computing kernels (e.g., AXPY, GEMV, GEMM) on different parallel programming models and languages (e.g., C++: OpenMP, OpenMP Offload, OpenACC, CUDA, HIP; Fortran: OpenMP, OpenMP Offload, OpenACC; Python: numpy, Numba, pyCUDA, cuPy; and Julia: Threads, CUDA.jl, AMDGPU.jl). We built upon our previous work that is based on the OpenAI Codex, which is a descendant of GPT-3, to generate similar kernels with simple prompts via GitHub Copilot. Our goal is to compare the accuracy of Llama-2 and our original GPT-3 baseline by using a similar metric. Llama-2 has a simplified model that shows competitive or even superior accuracy. We also report on the differences between these foundational large language models as generative AI continues to redefine human-computer interactions. Overall, Copilot generates codes that are more reliable but less optimized, whereas codes generated by Llama-2 are less reliable but more optimized when correct.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

KernelBench: Can LLMs Write Efficient GPU Kernels?
cs.LG 2025-02 accept novelty 7.0 of 10

KernelBench shows that even the best current LLMs generate correct and faster-than-baseline GPU kernels in fewer than 20 percent of realistic ML workloads.
LLM4VV: Evaluating Cutting-Edge LLMs for Generation and Evaluation of Directive-Based Parallel Programming Model Compiler Tests
cs.SE 2025-07 conditional novelty 5.0 of 10

In a six-model comparison, DeepSeek-Coder-33B generated the most passable directive-based compiler tests (Pass@1=0.434) and Qwen2.5-Coder-32B judged test validity best (F1=0.735).

Pith tools