CompilerGPT: Leveraging Large Language Models for Analyzing and Acting on Compiler Optimization Reports

Chunhua Liao; Peter Pirkelbauer

REVIEW 3 major objections 6 minor 17 references

CompilerGPT: Leveraging Large Language Models for Analyzing and Acting on Compiler Optimization Reports

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read When an LLM reads the compiler's optimization report and rewrites the code in a test-guided loop, it can achieve up to 6.5x speedups over -O3.

desk verdict Worth a serious referee, but the paper's central claim — that optimization reports drive the speedups — is untested until we see a no-report control. read the letter →

arxiv 2506.06227 v2 pith:L7GWGXGA submitted 2025-06-06 cs.PL

Peter Pirkelbauer , Chunhua Liao This is my paper

classification cs.PL

keywords compileroptimizationreportslargelanguagemodelsiterativecodepromptengineeringperformancetuningOpenMPClangGCC

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CompilerGPT tests a specific hypothesis: that a large language model, fed the compiler's own optimization report, can decide which missed optimization matters most and rewrite the code to fix it, with each rewrite checked by a user-supplied test harness before the next round begins. The paper reports speedups up to 6.5x on a prefix-sum benchmark, 3.1x on matrix multiplication, and modest gains on Smith-Waterman and two NAS OpenMP kernels, but also runs that produced no gain. The framework matters because compiler optimization reports are dense, compiler-specific, and hard for programmers to act on; if the approach holds, it turns those reports into a directly actionable optimization loop.

What carries the argument

The load-bearing mechanism is the iterative prompt loop. In each round CompilerGPT compiles the current code with optimization remarks enabled, sends the LLM a chain-of-thought prompt made of the code snippet, a runtime score, and the optimization report, and asks it to prioritize the top three issues and rewrite only the code tied to the highest-impact one. The rewritten code is compiled and run through a user-defined harness; failures feed back as error prompts, and successes feed into a fresh report for the next iteration. Conversation history is preserved across rounds so later changes build on earlier context, while negative prompting blocks unwanted constructs such as adding OpenMP directives.

What would settle it

Run the identical CompilerGPT loop on the same five benchmarks with the optimization report removed from every prompt while keeping code, runtime scores, and test feedback; if the speedups remain the same or larger, the central role of the report collapses.

Watch

Extended reading notes

Core claim

The central claim is that coupling an LLM to compiler-generated optimization reports in an iterative loop improves code optimization beyond what the compiler alone achieves. The paper demonstrates the claim in the small: across five benchmarks, two compilers (Clang and GCC), and two LLMs (GPT-4o and Claude Sonnet), the best runs produced 1.1x to 6.5x speedups over an -O3 baseline, with the largest gains on codes where the LLM spotted algorithmic and memory-layout issues that the compiler's own report only hinted at. The paper's own results are uneven: GPT-4o produced no speedup on prefix sum, and few runs beat the baseline on the NAS kernels. What the paper is trying to establish is not that every run wins, but that the report-plus-LLM loop is a viable route to automatic performance tuning.

Load-bearing premise

The reported speedups are attributed to the LLM reading the compiler's optimization report, but the paper never ran a version of CompilerGPT without that report, so the gains could come from the LLM's general rewriting ability instead.

Editorial extensions

If this is right

If the central claim holds, compiler optimization reports become a practical steering signal for LLM-driven performance tuning rather than a documentation aid for experts.
The same iterative report-fix-test loop could be applied to other compiler families and languages, since the framework only requires a report and a test harness.
Even where speedups do not occur, the LLM's prioritized issue lists give a programmer a readable map of what the compiler thinks it missed.
At 0.01 to 1.31 USD per run, the approach is cheap enough to be used as a routine optimization pass before human review.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An untested control condition means the strongest version of the claim, that the optimization report specifically causes the gains, is not yet established; the paper admits it ran no report-free configuration, so report-independent LLM rewrites remain a live alternative explanation.
A natural next experiment is to ablate the report: run the same loop with only code and runtime as prompts. If speedups persist, the framework's value simplifies to iterative LLM rewriting with correctness gates.
The prefix-sum result suggests the biggest wins come from restructuring memory management and parallelism, not from enabling vectorization; this points to combining CompilerGPT with a profiler to target hotspots in larger codebases.
Because the harness is user-supplied, a breakdown in correctness checking would let silent semantic errors through; tool support for generating property-based tests would be the load-bearing upgrade.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick read: this is the first system I know of that points an LLM at real Clang/GCC optimization reports and runs an iterative rewrite loop around them. The authors deserve credit for shipping the code, documenting the prompts, and being explicit about what they did not test. But the main causal claim is not yet supported. Section 4.5 concedes there is no configuration without the optimization report, so the reported speedups — up to 6.5x on prefix sum — could come from the LLM's generic ability to spot inefficiencies plus runtime feedback. The stress-test note lands.

What's genuinely new: using the report as a prioritization filter for the LLM, the chain-of-thought and negative prompting design, and the integration with user-provided correctness tests. The empirical setup is small but real: two LLMs, two compilers, five OpenMP-oriented benchmarks, five runs per configuration. The cost and wall-clock data are useful for anyone building on this. The best-case speedups are plausible, and the authors do not oversell them.

The soft spots are proportionate to a first feasibility study, but they matter. The missing no-report ablation is the big one. The paper's justification — that such prompts wouldn't constrain the model the same way — is really an admission that the constraint's effect is untested. Some of the best transformations were inferred from issues not in the report (Smith-Waterman contention, prefix-sum OpenMP overhead), which further weakens the attribution. I'd also like to see more than max/avg over five runs; a per-run distribution would help given LLM stochasticity. None of this is fatal. A serious revision could add a code-only control, a report-only control, and more runs, and the claims would be much easier to evaluate.

Bottom line: send it to peer review. The topic is timely, the artifact is open source, and the missing control is exactly what refs should demand. I wouldn't cite it as evidence for report-driven speedups yet, but I'd cite it as related work if I were working on LLM-guided optimization. For a reading group, it would spark a good discussion about ablation design in LLM systems. Serious thinker: yes — the paper is coherent, honest, and doesn't hide its weaknesses.

Referee Report

3 major / 6 minor

Summary. CompilerGPT iteratively couples a compiler's optimization reports with an LLM: the LLM receives the code, the report, a runtime score, and a chain-of-thought-style prompt, rewrites the code, and the loop repeats with compile/test feedback. The paper evaluates the framework with GPT-4o and Claude Sonnet 3.7, Clang and GCC, and five benchmark programs, reporting speedups up to 6.5x but with considerable inconsistency across runs and configurations. The claimed contribution is that LLMs can interpret compiler optimization reports and act on them to improve code performance. The manuscript also describes prompt-engineering strategies (chain-of-thought, negative prompting) and discusses limitations such as hallucination, context-window constraints, and the need for user-provided test harnesses.

Significance. If supported, the framework would be a useful instantiation of LLM-guided performance engineering and would make compiler diagnostics more actionable. The paper's strengths are its open-source artifact, transparent workflow description, honest reporting of inconsistent results, and explicit treatment of correctness via user harnesses. The central empirical claim, however, is currently underdetermined: the missing no-report control leaves the attribution to optimization reports unsupported, and the statistical basis is thin. The results are suggestive rather than conclusive.

major comments (3)

[Section 4.5] Section 4.5 states: "We did not compare the results with CompilerGPT configurations that omit the optimization reports." This is the load-bearing missing control for the paper's central claim. The research hypothesis in Section 1 concerns "accurately interpreting compiler optimization reports," and Section 5 positions the novelty as "directly interpreting and acting upon compiler optimization reports." Without a condition that removes only the report (keeping the code, runtime score, and evaluation harness), the speedups cannot be attributed to report interpretation rather than to the LLM's generic code-optimization ability or to lucky rewrites. The paper's own examples support this worry: Section 4.2's best prefix-sum speedup follows from the LLM's inference from a low-level "__builtin_GOMP_parallel" message, and Section 4.3 reports that the LLM "infers issues that are not mentioned in the optimization report." The stated justification that report-free prompts "would not constrain AI models in the same way" does not resolve the attribution problem; it concedes that the constraint's contribution is untested. I consider this missing ablation a load-bearing gap.
[Table 3] Table 3 reports only Max and Avg speedup over five runs, with no standard deviations, per-run values, confidence intervals, or measurement-noise estimates. Given that LLM outputs are stochastic and that several rows have Num = 0 or Num = 1 (e.g., Prefix/GPT-4o, SW/GPT-4o/Clang, NAS/BT/GPT-4o/Clang), the reported "up to 6.5x" result and even the averages may be driven by one or two exceptional runs. The paper should report the full distribution of the five runs for each configuration, the median speedup, the number of runs improving beyond a defined threshold, and repeated timing of the final versions to separate LLM sampling variability from runtime measurement noise.
[Sections 4.1 and 4.5] The speedup measurements are only meaningful if the rewritten code is behaviorally correct, but the only correctness oracle is the user-supplied evaluation harness, whose coverage is not characterized. The matrix-multiply harness is described in Section 4.1, but for Smith-Waterman and the NAS benchmarks the paper does not state which checks are performed beyond using the benchmark suite's tests. Section 4.5 explicitly warns that "If the unit tests are not comprehensive, software engineers need to validate the optimized code." Without a coverage analysis or validation of the final optimized versions against an independent reference, some reported speedups could be inflated by transformations that alter behavior outside the tested inputs. The paper should document each harness's checks and, where feasible, validate final versions on additional inputs or against a reference implementation.

minor comments (6)

[Section 4] The text says "4.3 min with GPT-o" and "0.01 USD (GPT4-o)"; the model name should be spelled consistently as GPT-4o.
[Table 2] In the prompt table, "T ask 1", "T ask 2", and "T ask 3" appear as separate words, and the Success Prompt contains stray ellipsis formatting in "milliseconds.. . .".
[Section 4.5] "larger codes seem to be pose more difficulties" is ungrammatical; it should read "larger codes seem to pose more difficulties."
[Table 3] Please define what "any speedup" means for the Num column (e.g., speedup strictly greater than 1.0 versus some other threshold), and state how runs that fail to produce complete code are counted.
[Section 3.1] "The LLMs is tasked" should be "The LLM is tasked" or "The LLMs are tasked".
[Section 4] Only conversation histories of the best runs appear to be linked; making all five runs' logs available for each configuration would strengthen reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the speedup claims are measured against external baselines; the missing report-free control is an experimental attribution threat, not a definitional or fitted-parameter circularity.

full rationale

CompilerGPT makes an empirical capability claim: LLMs guided by compiler optimization reports and an evaluation harness can rewrite code to obtain speedups of up to 6.5x. The evaluation is external wall-clock performance against compiler-only baselines on five benchmark codes, with correctness enforced by user-provided tests. There is no fitted parameter renamed as a prediction, no target quantity defined in terms of the output, and no formal derivation whose conclusion is an input by construction. The success prompt feeds the measured runtime back to the LLM, but that is iterative feedback, not statistical fitting, and the speedup itself is not defined as the model's own output. The paper contains no load-bearing self-citation chain and invokes no uniqueness theorem from the authors' prior work. The only relevant self-referential limitation is explicit in Section 4.5: 'We did not compare the results with CompilerGPT configurations that omit the optimization reports.' This missing ablation is a genuine threat to attributing the speedups to report interpretation rather than to generic LLM optimization ability, and the paper's own examples show the LLM sometimes acting on issues not stated in the report. However, a missing control is a validity concern, not circularity: the measured speedups are not constructed from the report, and the central claim is not equivalent to its inputs by definition. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fit to data. The claims rest on domain assumptions about benchmark representativeness, the reliability of the performance metric, and the usefulness of compiler optimization reports; these are empirical premises, not derived results. No new physical or conceptual entities are introduced.

assumptions (3)

domain assumption The five selected benchmark codes and their unit tests are sufficient to detect functional regressions in LLM-generated code.
Section 4.1 describes tests for matrix multiply (prime dimensions, datatype check, exception behavior), but comprehensive correctness tests for the other four benchmarks are not fully enumerated; correctness is assumed if the benchmark suite passes.
domain assumption The sum of ten execution times on the Intel Xeon node is a stable and representative performance score.
Section 4 states the scoring method but provides no variance, no repetition beyond the five framework runs, and no statistical analysis; the score is treated as a deterministic measure of performance.
domain assumption Clang/GCC -Rpass-missed optimization reports contain actionable, correct information about missed optimizations.
The framework's premise, stated in Section 2, assumes these reports are meaningful and that acting on them improves performance. The paper even notes the LLM sometimes infers issues not present in the report, which weakens this assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CompilerGPT: Leveraging Large Language Models for Analyzing and Acting on Compiler Optimization Reports." pith.science (2026). https://pith.science/paper/L7GWGXGA

@misc{pith2026250606227,
  author       = {Pith},
  title        = {Pith review of: CompilerGPT: Leveraging Large Language Models for Analyzing and Acting on Compiler Optimization Reports},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L7GWGXGA}},
  note         = {Machine review of arXiv:2506.06227}
}

read the original abstract

Current compiler optimization reports often present complex, technical information that is difficult for programmers to interpret and act upon effectively. This paper assesses the capability of large language models (LLM) to understand compiler optimization reports and automatically rewrite the code accordingly. To this end, the paper introduces CompilerGPT, a novel framework that automates the interaction between compilers, LLMs, and user defined test and evaluation harness. CompilerGPT's workflow runs several iterations and reports on the obtained results. Experiments with two leading LLM models (GPT-4o and Claude Sonnet), optimization reports from two compilers (Clang and GCC), and five benchmark codes demonstrate the potential of this approach. Speedups of up to 6.5x were obtained, though not consistently in every test. This method holds promise for improving compiler usability and streamlining the software optimization process.

Figures

Figures reproduced from arXiv: 2506.06227 by the authors.

**Figure 3.** An excerpt of a Clang/LLVM optimization report for code in Fig. 1. The clang version is 18.1.8 [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** Comparison of maximum and average speedups achieved by GPT-4o and Claude Sonnet 3.7 across [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Sonnet optimized code for Clang: Iteration 6 yielded the best result. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 11 canonical work pages

[1]

[Accessed 15-05-2025]

A parallel implementation of a left-associative prefix sum using OpenMP.https://github.com/ robfarr/openmp-prefix-sum. [Accessed 15-05-2025]

work page 2025
[2]

[Accessed 15-05-2025]

Faster with Compiler Optimization Reports — intel.com.https://www.intel.com/content/www/us/ en/developer/articles/technical/compiler-optimization-report-news-2025.html. [Accessed 15-05-2025]

work page 2025
[3]

[Accessed 15-05-2025]

TheFighters/Smith-Waterman: Parallel implementation of Smith–Waterman using OpenMP.https: //github.com/TheFighters/Smith-Waterman. [Accessed 15-05-2025]

work page 2025
[4]

Cognition | Introducing Devin, the first AI software engineer — cognition.ai.https: //www.cognition.ai/blog/introducing-devin

cognition.ai. Cognition | Introducing Devin, the first AI software engineer — cognition.ai.https: //www.cognition.ai/blog/introducing-devin. [Accessed 15-05-2025]

work page 2025
[5]

Large language models for compiler optimization.arXiv preprint arXiv:2309.07062, 2023

Chris Cummins, Volker Seeker, Dejan Grubisic, Mostafa Elhoushi, Youwei Liang, Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Kim Hazelwood, Gabriel Synnaeve, et al. Large language models for compiler optimization.arXiv preprint arXiv:2309.07062, 2023

arXiv 2023
[6]

Meta large language model compiler: Foundation models of compiler optimization

Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Synnaeve, and Hugh Leather. Meta large language model compiler: Foundation models of compiler optimization. arXiv preprint arXiv:2407.02524, 2024

arXiv 2024
[7]

Faros: A frame- work to analyze openmp compilation through benchmarking and compiler optimization analysis

Giorgis Georgakoudis, Johannes Doerfert, Ignacio Laguna, and Thomas RW Scogland. Faros: A frame- work to analyze openmp compilation through benchmarking and compiler optimization analysis. In OpenMP: Portable Multi-Level Parallelism on Modern Systems: IWOMP 2020, USA, 2020, pages 3–17. Springer, 2020

work page 2020
[8]

Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology, 2023

Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology, 2023

work page 2023

Show all 17 references

[9]

The openmp implementation of nas parallel bench- marks and its performance

Hao-Qiang Jin, Michael Frumkin, and Jerry Yan. The openmp implementation of nas parallel bench- marks and its performance. 1999

1999
[10]

A comparison of the effectiveness of chatgpt and co-pilot for generating quality python code solutions

Nikolaos Nikolaidis, Karolos Flamos, Khanak Gulati, Daniel Feitosa, Apostolos Ampatzoglou, and Alexander Chatzigeorgiou. A comparison of the effectiveness of chatgpt and co-pilot for generating quality python code solutions. In2024 SANER-C, pages 93–101. IEEE, 2024

2024
[11]

Navigating compiler errors with ai assistance-a study of gpt hints in an introductory programming course

Maciej Pankiewicz and Ryan S Baker. Navigating compiler errors with ai assistance-a study of gpt hints in an introductory programming course. InProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1, pages 94–100. 2024

2024
[12]

Identification of common molecular subsequences.Journal of molecular biology, 147(1):195–197, 1981

Temple F Smith, Michael S Waterman, et al. Identification of common molecular subsequences.Journal of molecular biology, 147(1):195–197, 1981

1981
[13]

Dcc–help: Transforming the role of the compiler by generating context-aware error explanations with large language models

Andrew Taylor, Alexandra Vassar, Jake Renzella, and Hammond Pearce. Dcc–help: Transforming the role of the compiler by generating context-aware error explanations with large language models. InProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, ...

2024
[14]

Openhands: An open platform for ai software developers as generalist agents.arXiv preprint arXiv:2407.16741, 2024

Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents.arXiv preprint arXiv:2407.16741, 2024

2024 arXiv
[15]

Addressing compiler errors: Stack overflow or large language models?arXiv preprint arXiv:2307.10793, 2023

Patricia Widjojo and Christoph Treude. Addressing compiler errors: Stack overflow or large language models?arXiv preprint arXiv:2307.10793, 2023. 11

2023 arXiv
[16]

Swe-agent: Agent-computer interfaces enable automated software engineering.arXiv preprint arXiv:2405.15793, 2024

John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering.arXiv preprint arXiv:2405.15793, 2024

2024 arXiv
[17]

Demystifyingpractices, challenges and expected features of using github copilot.arXiv preprint arXiv:2309.05687, 2023

BeiqiZhang, PengLiang, XiyuZhou, AakashAhmad, andMuhammadWaseem. Demystifyingpractices, challenges and expected features of using github copilot.arXiv preprint arXiv:2309.05687, 2023. 12

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

[Accessed 15-05-2025]

A parallel implementation of a left-associative prefix sum using OpenMP.https://github.com/ robfarr/openmp-prefix-sum. [Accessed 15-05-2025]

work page 2025

[2] [2]

[Accessed 15-05-2025]

Faster with Compiler Optimization Reports — intel.com.https://www.intel.com/content/www/us/ en/developer/articles/technical/compiler-optimization-report-news-2025.html. [Accessed 15-05-2025]

work page 2025

[3] [3]

[Accessed 15-05-2025]

TheFighters/Smith-Waterman: Parallel implementation of Smith–Waterman using OpenMP.https: //github.com/TheFighters/Smith-Waterman. [Accessed 15-05-2025]

work page 2025

[4] [4]

Cognition | Introducing Devin, the first AI software engineer — cognition.ai.https: //www.cognition.ai/blog/introducing-devin

cognition.ai. Cognition | Introducing Devin, the first AI software engineer — cognition.ai.https: //www.cognition.ai/blog/introducing-devin. [Accessed 15-05-2025]

work page 2025

[5] [5]

Large language models for compiler optimization.arXiv preprint arXiv:2309.07062, 2023

Chris Cummins, Volker Seeker, Dejan Grubisic, Mostafa Elhoushi, Youwei Liang, Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Kim Hazelwood, Gabriel Synnaeve, et al. Large language models for compiler optimization.arXiv preprint arXiv:2309.07062, 2023

arXiv 2023

[6] [6]

Meta large language model compiler: Foundation models of compiler optimization

Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Synnaeve, and Hugh Leather. Meta large language model compiler: Foundation models of compiler optimization. arXiv preprint arXiv:2407.02524, 2024

arXiv 2024

[7] [7]

Faros: A frame- work to analyze openmp compilation through benchmarking and compiler optimization analysis

Giorgis Georgakoudis, Johannes Doerfert, Ignacio Laguna, and Thomas RW Scogland. Faros: A frame- work to analyze openmp compilation through benchmarking and compiler optimization analysis. In OpenMP: Portable Multi-Level Parallelism on Modern Systems: IWOMP 2020, USA, 2020, pages 3–17. Springer, 2020

work page 2020

[8] [8]

Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology, 2023

Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology, 2023

work page 2023