LLM4VV: Evaluating Cutting-Edge LLMs for Generation and Evaluation of Directive-Based Parallel Programming Model Compiler Tests

Rahul Patel; Saieda Ali Zada; Sunita Chandrasekaran; Zachariah Sollenberger

REVIEW 3 major objections 6 minor 41 references

LLM4VV: Evaluating Cutting-Edge LLMs for Generation and Evaluation of Directive-Based Parallel Programming Model Compiler Tests

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that six open-source LLMs can take on the compiler-testing roles for directive-based parallel models, with Deepseek-Coder-33B-Instruct generating the most compilable and runnable tests and Qwen2.5-Coder-32B-Instruct…

desk verdict Useful empirical sweep of open-source LLMs for compiler test generation, but the time-budgeted evaluation makes the cross-model ranking uninterpretable without per-instance coverage data. read the letter →

arxiv 2507.21447 v2 pith:5PPJA7YU submitted 2025-07-29 cs.SE cs.ET

Zachariah Sollenberger , Rahul Patel , Saieda Ali Zada , Sunita Chandrasekaran This is my paper

classification cs.SEcs.ET

keywords LLM4VVcompilertestingOpenMPACCdirective-basedparallelprogrammingLLM-as-a-Judgeretrieval-augmentedgenerationtest

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether existing open-source large language models can take over the labor-intensive job of writing and checking compiler tests for directive-based parallel programming models such as OpenMP and OpenACC. It proposes a dual-LLM system in which one model generates tests from directives and clauses while a second model judges whether each test is valid, and it evaluates six instruction-tuned LLMs on both roles. The results single out Deepseek-Coder-33B-Instruct as the strongest generator, with a Pass@1 of 0.434, and Qwen2.5-Coder-32B-Instruct as the strongest judge, with an F1-score of 0.735 and MCC of 0.447. A sympathetic reader would take this as evidence that fully automated compiler test creation and verification is within reach for these programming models.

What carries the argument

The load-bearing mechanism is the dual-LLM pipeline. The generative side uses four prompting strategies (few-shot, one-shot, detailed prompting, and retrieval-augmented generation with specification text) to produce compiler tests; the discriminative side uses two LLM-as-a-Judge frameworks, the Validation Pipeline (compile, execute, then judge) and the LLMJ-Agent with RAG, and two prompting styles (parameter-based and description-based). The argument's support structure is a set of ten metrics, with Pass@1 for generation and F1-score and MCC for discrimination as the headline numbers.

What would settle it

A controlled rerun that gives every LLM the same fixed set of directive-clause prompts and the same fixed set of valid and invalid test files, with each model required to finish every instance, would settle whether the reported rankings reflect capability rather than time budget; if Deepseek's Pass@1 and Qwen's F1-score do not stay on top under identical instances, the paper's headline conclusions fail.

Watch

Extended reading notes

Core claim

The central claim is that a division of labor between two LLMs can automate compiler test development for directive-based parallel programming models: a generative agent writes tests for supported directive-clause combinations, and a discriminative agent filters them through a validation pipeline that compiles, executes, and then judges the code. Over 36 unit experiments on six open-source instruction-tuned models, the authors report that Deepseek-Coder-33B-Instruct produced the highest fraction of generated tests that compile and run successfully, while Qwen2.5-Coder-32B-Instruct most accurately separated valid tests from tests with injected syntax, semantic, and parallel-programming errors. The paper further reports that code-trained medium-sized models outperformed both smaller and larger non-code-trained models, and that discriminative accuracy rises sharply when an LLM judge is given compilation and execution results rather than reviewing code alone.

Load-bearing premise

The rankings depend on the assumption that every LLM was evaluated on the same or statistically equivalent set of test instances, but each node only processed as many cases as it could within the time budget, with no randomization or completion guarantee mentioned.

Editorial extensions

If this is right

If the reported rankings hold, the next practical step is to fine-tune Deepseek-Coder-33B-Instruct and Qwen2.5-Coder-32B-Instruct into the fixed generative and discriminative agents of the dual-LLM system.
The finding that compilation and execution data sharply improve LLM judgment suggests that any autonomous test-suite pipeline should keep automated checks in front of the LLM judge rather than asking the model to reason about raw code alone.
The observed drop in performance on Fortran compared with C and C++ implies that directive-based compiler test generation in Fortran needs extra training data or specialized prompting before it can be automated at the same quality.
Because the best generative and discriminative models are different, a practical system should not expect one LLM to serve both roles at peak quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct testable extension would be to chain the two winners into a closed loop, letting Qwen reject Deepseek's outputs and feed them back for revision, then measuring how much the final pass rate improves over generation without discrimination.
The comparison's fairness is not fully established: each unit experiment ran for a fixed seven-hour wall-clock budget and processed as many instances as it could, so slower models may have been scored on a smaller or easier subset of instances than faster ones.
The RAG retrieval over OpenMP and OpenACC specifications could be adapted to other directive-based models or to the next spec revisions, since the same cosine-similarity clustering machinery does not depend on the directive vocabulary.
One implicit consequence is that leaderboard coding scores do not predict discriminative accuracy well: the model that generated best was not the best judge, so evaluations of LLMs for code should separate the two capabilities.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a useful first step for anyone picking open-source LLMs for compiler test generation and triage, but the headline ranking of models shouldn't be trusted as-is. The core weakness is the time-budgeted design: each unit experiment ran for 7 hours on a node and covered as many instances as it could, with no randomization or fixed ordering. If Deepseek-Coder's Pass@1 of 0.434 came from a different, perhaps easier, subset of prompts than Qwen2.5-Coder's F1 0.735, the comparison collapses. The paper doesn't report per-model instance counts or coverage statistics, so nothing rules that out. This is a load-bearing flaw for the central claim.

What's genuinely new here is the scope: six instruction-tuned open-source models across three size classes, four prompting techniques for generation, two discriminative frameworks, and ten metrics. That's a broader evaluation than their prior LLM4VV papers, and the dataset of valid/invalid compiler tests is larger. The task itself matters — directive-based parallel programming models badly need automated validation. The paper is also transparent about its own limitations: it explicitly says the passing criterion ignores semantic correctness, and it explains why they swapped out models after preliminary tests.

The softer spots: no released code or data, which makes replication hard; no confidence intervals or significance tests, so similar scores like the F1 range from .705 to .735 are treated as meaningfully different without support; and the model selection after preliminary testing introduces a selection bias they mitigate somewhat with explanation but can't fully remove.

The stress-test concern holds up on reading. Section III.D is exactly as you described. The authors' claim that combining results makes evaluation methods apply equally to all LLMs doesn't address instance-set comparability. That said, this is an honest, well-written empirical study with a real gap. A serious referee should engage, but the authors need to report per-model instance counts or randomize the order, and ideally release the artifacts.

For a reader in the compiler-validation or HPC-LLM space, this is worth skimming. I wouldn't cite it for the ranking, but I'd cite it as an example of the dual-LLM approach. I'd send it to peer review with a request for major revision.

Referee Report

3 major / 6 minor

Summary. The paper evaluates six open-source instruction-tuned LLMs for two roles in an automated compiler-testing pipeline for directive-based parallel programming models (OpenMP and OpenACC): generation of tests from directive-clause combinations, and discrimination of valid versus invalid tests. The generative evaluation measures compilation rate, returned-0 rate, and Pass@1 over tests produced with four prompting methods (few-shot, one-shot, detailed, RAG). The discriminative evaluation uses a validation pipeline and an LLM-as-a-judge RAG agent with two prompting styles, computing ten metrics. Based on aggregated results, the authors conclude that Deepseek-Coder-33B-Instruct is the best generator (Pass@1 0.434) and Qwen2.5-Coder-32B-Instruct is the best judge (F1 0.735, MCC 0.447).

Significance. If the comparison is valid, this is a useful empirical contribution to automated compiler test generation: it provides a head-to-head comparison of current open-source LLMs and identifies concrete base models for fine-tuning, which is directly actionable. Strengths include the orthogonal evaluation of generation and discrimination, the use of multiple prompting strategies including RAG over the actual specifications, and the authors' explicit acknowledgment that their 'passing' criterion does not detect semantic errors. The main weakness is that the central ranking is built on asynchronous, time-budgeted runs in which each LLM may have seen a different subset of instances, with no evidence of statistical equivalence; the ranking should therefore not be interpreted as established until this is addressed.

major comments (3)

[Section III.D and Table IV] The asynchronous design described in Section III.D is load-bearing for the paper's central claim. Each unit experiment ran for a fixed 7-hour wall-clock budget, and the text states that each node 'cover[ed] as many testing instances as it could in its allotted time' with no mention of randomization, fixed instance ordering, or completion guarantees. Since Pass@1, F1-Score, and MCC are proportions computed over whatever subset each LLM happened to finish, a slower model may have been scored on a different, possibly easier or harder, prefix of the test list than a faster model. This makes the cross-model comparisons in Table IV and the Section V ranking uninterpretable as measures of model capability. The authors should either evaluate all models on the identical fixed instance set, use randomized ordering with a fixed completion threshold, or analyze only the common completed prefix, and in all cases report per-LLM instance counts and instance overlap.
[Section III.B.2 and Table IV] The generative 'passing' criterion is compilation plus a return code of 0; the authors explicitly state that semantic errors are 'significantly more difficult to detect automatically.' Consequently Pass@1 does not measure whether a generated test actually exercises the requested directive-clause combination or would detect a faulty compiler implementation. The conclusion that Deepseek-Coder-33B-Instruct shows the 'best generative capabilities' should be restated as best at producing compilable, runnable code. To support the stronger claim, the authors should supplement the automated metric with a manual semantic review of a sample of generated tests or apply their discriminative agent to the generated files and report agreement.
[Section IV and Figures 1-3] All reported metrics are point estimates with no confidence intervals, significance tests, or per-LLM sample sizes. Given that the number of instances each model saw could differ substantially under the time-budget design, differences such as the F1 gap between Qwen2.5-Coder-32B-Instruct (0.735) and Mistral-7B-Instruct-v0.3 (0.719) may be within noise. At minimum, the authors should report the number of files judged or generated for each LLM and each condition, and provide bootstrap confidence intervals or a paired-comparison analysis on the overlapping subset of instances.

minor comments (6)

[Section III.D] The text contains a typo: 'BFLOA T-16s' should be 'BFLOAT16'.
[Section III.D] The paper does not state which inference framework, software versions, quantization, or batch settings were used for the LLMs. These details matter for both runtime and reproducibility, and they are directly relevant to interpreting the time-budget results.
[Section III.B.2] The RAG implementation is described only at a high level; the number of retrieved sentences per query, the query template, and whether the retrieved text was inserted before or after the prompt are not specified. These details are needed for replication.
[Section III.A and Table II] The selection process from 15 initial candidates to 6 final models is described only qualitatively; a table listing the screened models and the exclusion criterion for each would improve transparency.
[Section IV] The narrative refers to 'Qwen-Coder-33B' while Table III and the conclusion use 'Qwen2.5-Coder-32B-Instruct'; please make the naming consistent.
[General] No artifact or data availability statement is included. Releasing the prompts, per-file judgments, and aggregate per-LLM counts would greatly strengthen the reproducibility of this empirical study.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the LLM ranking is an empirical evaluation with standard metrics, and the paper's self-citations supply tooling and prior evidence rather than the result itself.

full rationale

The paper's central claim (Deepseek-Coder-33B-Instruct best for generation, Qwen2.5-Coder-32B-Instruct best for discrimination) is an empirical ranking produced by compiling and executing generated files and by applying standard formulas (Eqs. 1-10) to per-file judgments. No parameter is fitted to force a particular winner, and no 'prediction' is defined in terms of the measured outcome. The self-cited prior work ([11], [12]) supplies the Validation Pipeline, the LLMJ-Agent, and the syntactic/semantic invalidation heuristics used as measurement tools, but those tools are not the target result and their provenance does not by construction determine the Pass@1, F1, or MCC values reported in Table IV. The fine-grained results are reconstructed from the same measured outcomes, so there is no derivation chain in which an output variable is equal to an input variable by definition. Potential threats such as the time-budgeted unit experiments in Section III.D (where each node covered 'as many testing instances as it could in its allotted time', allowing different instance subsets across models) and the acknowledged residual semantic-error risk in Section III.B.2 ('Despite this potential for semantic errors in "passing" files') are validity concerns about comparability and measurement, not circular equations. Therefore, the paper contains no circular step; at most there are minor self-citations that are not load-bearing for the ranking claim.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The evaluation rests on several unstated or partially stated modeling choices. The most important are the use of the V&V suites as ground truth, the compilation-and-return-0 proxy for test quality, the assumption that corrupted copies are genuinely invalid, and the assumption that time-bounded subsets of instances yield comparable metrics. None of these are free parameters; they are domain assumptions about the validity and generalizability of the measurements.

assumptions (4)

domain assumption The OpenMP and OpenACC V&V suites correctly implement the specifications; any directive-clause combination found in them is valid.
Used in Section III.B.1 to build the generative task list and the discriminative ground truth. If a combination in the suite is invalid or incomplete, the evaluation targets are wrong.
domain assumption A compiler test that compiles and returns exit code 0 is considered passing; semantic errors are ignored.
Stated in Section III.B.2: 'we considered tests that compiled and ran successfully without runtime errors to be passing since semantic errors would be significantly more difficult to detect automatically.' This proxy underlies the generative metrics.
domain assumption The corrupted copies of the V&V tests are genuinely invalid, and the originals are genuinely valid, as ground truth.
Section III.B.1: invalid copies are created by introducing syntax, semantic, or parallel programming errors. If some corruptions do not change validity, or some originals are already invalid, the discriminative labels are noisy.
domain assumption For each LLM experiment, processing as many instances as fit in the 7-hour budget still yields comparable metrics across LLMs.
Section III.D: 'each unit experiment... cover as many testing instances as it could in its allotted time'. This assumes the subset seen does not bias the comparison.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLM4VV: Evaluating Cutting-Edge LLMs for Generation and Evaluation of Directive-Based Parallel Programming Model Compiler Tests." pith.science (2026). https://pith.science/paper/5PPJA7YU

@misc{pith2026250721447,
  author       = {Pith},
  title        = {Pith review of: LLM4VV: Evaluating Cutting-Edge LLMs for Generation and Evaluation of Directive-Based Parallel Programming Model Compiler Tests},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5PPJA7YU}},
  note         = {Machine review of arXiv:2507.21447}
}

read the original abstract

The usage of Large Language Models (LLMs) for software and test development has continued to increase since LLMs were first introduced, but only recently have the expectations of LLMs become more realistic. Verifying the correctness of code generated by LLMs is key to improving their usefulness, but there have been no comprehensive and fully autonomous solutions developed yet. Hallucinations are a major concern when LLMs are applied blindly to problems without taking the time and effort to verify their outputs, and an inability to explain the logical reasoning of LLMs leads to issues with trusting their results. To address these challenges while also aiming to effectively apply LLMs, this paper proposes a dual-LLM system (i.e. a generative LLM and a discriminative LLM) and experiments with the usage of LLMs for the generation of a large volume of compiler tests. We experimented with a number of LLMs possessing varying parameter counts and presented results using ten carefully-chosen metrics that we describe in detail in our narrative. Through our findings, it is evident that LLMs possess the promising potential to generate quality compiler tests and verify them automatically.

Figures

Figures reproduced from arXiv: 2507.21447 by the authors.

**Figure 2.** LLM Programming Model Performance - Correlation [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** This figure has four 3D fine-grained plots (a-d) that de [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 35 canonical work pages

[1]

Using a larg e language model as a building block to generate usablevalidation and v eriﬁcation suite for openmp,

S. Pophale, W. Elwasif, and D. E. Bernholdt, “Using a larg e language model as a building block to generate usablevalidation and v eriﬁcation suite for openmp,” in Proceedings of the International Conference on High Performance Computing in Asia-Paciﬁc Region , 2025, pp. 131– 141

work page 2025
[2]

Llm & hpc: Be nchmarking deepseek’s performance in high-performance computing tas ks,

N. Nader, P . Diehl, S. Brandt, and H. Kaiser, “Llm & hpc: Be nchmarking deepseek’s performance in high-performance computing tas ks,” arXiv preprint arXiv:2504.03665, 2025

arXiv 2025
[3]

Can large language models write parallel code?

D. Nichols, J. H. Davis, Z. Xie, A. Rajaram, and A. Bhatele , “Can large language models write parallel code?” in Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing, 2024, pp. 281–294

work page 2024
[4]

The landscape and challenges of hpc research and llms,

L. Chen, N. K. Ahmed, A. Dutta, A. Bhattacharjee, S. Y u, Q. I. Mahmud, W. Abebe, H. Phan, A. Sarkar, B. Butler et al. , “The landscape and challenges of hpc research and llms,” arXiv preprint arXiv:2402.02018 , 2024

arXiv 2024
[5]

An assessment of large langu age models for openmp-based code parallelization: a user perspective ,

M. Miˇ si´ c and M. Dodovi´ c, “An assessment of large langu age models for openmp-based code parallelization: a user perspective ,” Journal of Big Data , vol. 11, no. 1, p. 161, 2024

work page 2024
[6]

Comparing llama-2 and gpt -3 llms for hpc kernels generation,

P . V alero-Lara, A. Huante, M. A. Lail, W. F. Godoy, K. Tera nishi, P . Balaprakash, and J. S. V etter, “Comparing llama-2 and gpt -3 llms for hpc kernels generation,” arXiv preprint arXiv:2309.07103 , 2023

arXiv 2023
[7]

Large language model evaluation for high-perform ance com- puting software development,

W. F. Godoy, P . V alero-Lara, K. Teranishi, P . Balaprakas h, and J. S. V etter, “Large language model evaluation for high-perform ance com- puting software development,” Concurrency and Computation: Practice and Experience , vol. 36, no. 26, p. e8269, 2024

work page 2024
[8]

Boosting llm-based software generation by aligning code with requirements,

T. Y aacov, A. Elyasaf, and G. Weiss, “Boosting llm-based software generation by aligning code with requirements,” in 2024 IEEE 32nd International Requirements Engineering Conference W orks hops (REW) . IEEE, 2024, pp. 301–305

work page 2024

Show all 41 references

[9]

Generative software engineering,

Y . Huang, Y . Chen, X. Chen, J. Chen, R. Peng, Z. Tang, J. Hua ng, F. Xu, and Z. Zheng, “Generative software engineering,” arXiv preprint arXiv:2403.02583, 2024

2024 arXiv
[10]

V altest: Automated vali dation of language model generated test cases,

H. Taherkhani and H. Hemmati, “V altest: Automated vali dation of language model generated test cases,” arXiv preprint arXiv:2411.08254 , 2024

2024
[11]

Llm4vv : Developing llm-driven testsuite for compiler validation,

C. Munley, A. Jarmusch, and S. Chandrasekaran, “Llm4vv : Developing llm-driven testsuite for compiler validation,” Future Generation Com- puter Systems , vol. 160, pp. 1–13, 2024

2024
[12]

Llm4vv: Exploring llm-as-a-judge for valida tion and veriﬁ- cation testsuites,

Z. Sollenberger, J. Patel, C. Munley, A. Jarmusch, and S . Chan- drasekaran, “Llm4vv: Exploring llm-as-a-judge for valida tion and veriﬁ- cation testsuites,” in SC24-W: W orkshops of the International Conference for High Performance Computing, Networking, Storage and An alysis...

2024
[13]

Hugging Face – The AI community building the future. — h ugging- face.co,

“Hugging Face – The AI community building the future. — h ugging- face.co,” https://huggingface.co/, [Accessed 29-04-202 5]
[14]

library — ollama.com,

“library — ollama.com,” https://ollama.com/library ?sort=popular, [Ac- cessed 29-04-2025]

2025
[15]

LLM Leaderboard - Compare GPT-4o, Llama 3, Mistral, Ge mini & other models — Artiﬁcial Analysis — artiﬁcialanalysis.ai ,

“LLM Leaderboard - Compare GPT-4o, Llama 3, Mistral, Ge mini & other models — Artiﬁcial Analysis — artiﬁcialanalysis.ai ,” https: //artiﬁcialanalysis.ai/leaderboards/models, [Accesse d 29-04-2025]

2025
[16]

EvalPlus Leaderboard — evalplus.github.io,

“EvalPlus Leaderboard — evalplus.github.io,” https: //evalplus.github.io/ leaderboard.html, [Accessed 29-04-2025]

2025
[17]

The Evolving Landscape of LLM Training Data — alibabacloud.com,

“The Evolving Landscape of LLM Training Data — alibabacloud.com,” https://www.alibabacloud.com/blog / the-evolving-landscape-of-llm-training-data 602104, [Accessed 13-06-2025]

2025
[18]

EleutherAI/gpt-neo-2.7B · Hugging Face — huggingfac e.co,

“EleutherAI/gpt-neo-2.7B · Hugging Face — huggingfac e.co,” https:// huggingface.co/EleutherAI/gpt-neo-2.7B, [Accessed 29- 04-2025]

2025
[19]

Mistral 7B — Mistral AI — mistral.ai,

“Mistral 7B — Mistral AI — mistral.ai,” https://mistra l.ai/news/ announcing-mistral-7b, [Accessed 29-04-2025]

2025
[20]

nvidia/NVLM-D-72B · Hugging Face — huggingface.co,

“nvidia/NVLM-D-72B · Hugging Face — huggingface.co,” https:// huggingface.co/nvidia/NVLM-D-72B, [Accessed 29-04-202 5]
[21]

meta-llama/Llama-4-Maverick-17B-128E-Instruct · Hugging Face — huggingface.co,

“meta-llama/Llama-4-Maverick-17B-128E-Instruct · Hugging Face — huggingface.co,” https://huggingface.co/meta-ll ama/ Llama-4-Maverick-17B-128E-Instruct, [Accessed 29-04-2 025]
[22]

mistralai/Mistral-7B-Instruct-v0.3 · Hugging Face — huggingface.co,

“mistralai/Mistral-7B-Instruct-v0.3 · Hugging Face — huggingface.co,” https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3, [Accessed 29- 04-2025]

2025
[23]

The best small LLM-instruct models - a ehristoforu Col lec- tion — huggingface.co,

“The best small LLM-instruct models - a ehristoforu Col lec- tion — huggingface.co,” https://huggingface.co/collect ions/ehristoforu/ the-best-small-llm-instruct-models-669e89c6263d0188 8798cb7a, [Ac- cessed 29-04-2025]

2025
[24]

EvalPerf: Evaluating Language Models for Efﬁcient Co de Generation — evalplus.github.io,

“EvalPerf: Evaluating Language Models for Efﬁcient Co de Generation — evalplus.github.io,” https://evalplus.github.io/eva lperf.html, [Accessed 30-04-2025]

2025
[25]

Qwen/Qwen2.5-7B-Instruct · Hugging Face — huggingfa ce.co,

“Qwen/Qwen2.5-7B-Instruct · Hugging Face — huggingfa ce.co,” https: //huggingface.co/Qwen/Qwen2.5-7B-Instruct, [Accessed 30-04-2025]

2025
[26]

Qwen/Qwen2.5-Coder-32B-Instruct · Hugging Face — hu gging- face.co,

“Qwen/Qwen2.5-Coder-32B-Instruct · Hugging Face — hu gging- face.co,” https://huggingface.co/Qwen/Qwen2.5-Coder- 32B-Instruct, [Accessed 29-04-2025]

2025
[27]

deepseek-ai/deepseek-coder-33b-instruct · Huggin g Face — huggingface.co,

“deepseek-ai/deepseek-coder-33b-instruct · Huggin g Face — huggingface.co,” http://huggingface.co/deepseek-ai/ deepseek-coder-33b-instruct, [Accessed 29-04-2025]

2025
[28]

Phind/Phind-CodeLlama-34B-v2 · Hugging Face — huggi ngface.co,

“Phind/Phind-CodeLlama-34B-v2 · Hugging Face — huggi ngface.co,” https://huggingface.co/Phind/Phind-CodeLlama-34B-v2, [Accessed 30- 04-2025]

2025
[29]

OpenAI Platform — platform.openai.com,

“OpenAI Platform — platform.openai.com,” https://pl atform.openai. com/docs/models/gpt-4-turbo, [Accessed 30-04-2025]

2025
[30]

meta-llama/Llama-3.3-70B-Instruct · Hugging Face — hugging- face.co,

“meta-llama/Llama-3.3-70B-Instruct · Hugging Face — hugging- face.co,” https://huggingface.co/meta-llama/Llama-3. 3-70B-Instruct, [Accessed 29-04-2025]

2025
[31]

nvidia/Llama-3.1-Nemotron-70B-Instruct-HF · Hugg ing Face — huggingface.co,

“nvidia/Llama-3.1-Nemotron-70B-Instruct-HF · Hugg ing Face — huggingface.co,” https://huggingface.co/nvidia/Llam a-3. 1-Nemotron-70B-Instruct-HF, [Accessed 29-04-2025]

2025
[32]

meta-llama/Llama-3.1-405B-Instruct · Hugging Face — hugging- face.co,

“meta-llama/Llama-3.1-405B-Instruct · Hugging Face — hugging- face.co,” https://huggingface.co/meta-llama/Llama-3. 1-405B-Instruct, [Accessed 30-04-2025]

2025
[33]

Introducing Claude 3.5 Sonnet — anthropic.com,

“Introducing Claude 3.5 Sonnet — anthropic.com,” http s://www. anthropic.com/news/claude-3-5-sonnet, [Accessed 30-04 -2025]

2025
[34]

GitHub - lmarena/arena-hard-auto: Arena-Hard-Auto : An auto- matic LLM benchmark. — github.com,

“GitHub - lmarena/arena-hard-auto: Arena-Hard-Auto : An auto- matic LLM benchmark. — github.com,” https://github.com/l marena/ arena-hard-auto, [Accessed 30-04-2025]

2025
[35]

AlpacaEval Leaderboard — tatsu-lab.github.io,

“AlpacaEval Leaderboard — tatsu-lab.github.io,” htt ps://tatsu-lab.github. io/alpaca eval/, [Accessed 30-04-2025]

2025
[36]

MT Bench - a Hugging Face Space by lmsys — huggingface.c o,

“MT Bench - a Hugging Face Space by lmsys — huggingface.c o,” https: //huggingface.co/spaces/lmsys/mt-bench, [Accessed 30- 04-2025]

2025
[37]

“GitHub - OpenMP-Validation-and-Veriﬁcation/OpenM P VV: OpenMP Ofﬂoading V alidation & V eriﬁcation Suite; Ofﬁcial r epository. We have migrated from bitbucket!! For documentation, resul ts, publica- tion and presentations, please check out our website -¿ — git hub.com,” https...

2025
[38]

GitHub - OpenACCUserGroup/OpenACCV-V: OpenACC Vali dation and Veriﬁcation Testsuite repository — github.com,

“GitHub - OpenACCUserGroup/OpenACCV-V: OpenACC Vali dation and Veriﬁcation Testsuite repository — github.com,” https ://github.com/ OpenACCUserGroup/OpenACCV -V, [Accessed 13-06-2025]

2025
[39]

C-pack: P ackaged resources to advance general chinese embedding,

S. Xiao, Z. Liu, P . Zhang, and N. Muennighoff, “C-pack: P ackaged resources to advance general chinese embedding,” 2023

2023
[40]

BERT: pre -training of deep bidirectional transformers for language understan ding,

J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre -training of deep bidirectional transformers for language understan ding,” CoRR, vol. abs/1810.04805, 2018. [Online]. Available: http://a rxiv.org/abs/ 1810.04805

2018 arXiv
[41]

Architecture - NERSC Documentation — docs.ner sc.gov,

NERSC, “Architecture - NERSC Documentation — docs.ner sc.gov,” https://docs.nersc.gov/systems/perlmutter/architecture/, [Accessed 14- 06-2025]

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Using a larg e language model as a building block to generate usablevalidation and v eriﬁcation suite for openmp,

S. Pophale, W. Elwasif, and D. E. Bernholdt, “Using a larg e language model as a building block to generate usablevalidation and v eriﬁcation suite for openmp,” in Proceedings of the International Conference on High Performance Computing in Asia-Paciﬁc Region , 2025, pp. 131– 141

work page 2025

[2] [2]

Llm & hpc: Be nchmarking deepseek’s performance in high-performance computing tas ks,

N. Nader, P . Diehl, S. Brandt, and H. Kaiser, “Llm & hpc: Be nchmarking deepseek’s performance in high-performance computing tas ks,” arXiv preprint arXiv:2504.03665, 2025

arXiv 2025

[3] [3]

Can large language models write parallel code?

D. Nichols, J. H. Davis, Z. Xie, A. Rajaram, and A. Bhatele , “Can large language models write parallel code?” in Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing, 2024, pp. 281–294

work page 2024

[4] [4]

The landscape and challenges of hpc research and llms,

L. Chen, N. K. Ahmed, A. Dutta, A. Bhattacharjee, S. Y u, Q. I. Mahmud, W. Abebe, H. Phan, A. Sarkar, B. Butler et al. , “The landscape and challenges of hpc research and llms,” arXiv preprint arXiv:2402.02018 , 2024

arXiv 2024

[5] [5]

An assessment of large langu age models for openmp-based code parallelization: a user perspective ,

M. Miˇ si´ c and M. Dodovi´ c, “An assessment of large langu age models for openmp-based code parallelization: a user perspective ,” Journal of Big Data , vol. 11, no. 1, p. 161, 2024

work page 2024

[6] [6]

Comparing llama-2 and gpt -3 llms for hpc kernels generation,

P . V alero-Lara, A. Huante, M. A. Lail, W. F. Godoy, K. Tera nishi, P . Balaprakash, and J. S. V etter, “Comparing llama-2 and gpt -3 llms for hpc kernels generation,” arXiv preprint arXiv:2309.07103 , 2023

arXiv 2023

[7] [7]

Large language model evaluation for high-perform ance com- puting software development,

W. F. Godoy, P . V alero-Lara, K. Teranishi, P . Balaprakas h, and J. S. V etter, “Large language model evaluation for high-perform ance com- puting software development,” Concurrency and Computation: Practice and Experience , vol. 36, no. 26, p. e8269, 2024

work page 2024

[8] [8]

Boosting llm-based software generation by aligning code with requirements,

T. Y aacov, A. Elyasaf, and G. Weiss, “Boosting llm-based software generation by aligning code with requirements,” in 2024 IEEE 32nd International Requirements Engineering Conference W orks hops (REW) . IEEE, 2024, pp. 301–305

work page 2024