Pith. sign in

REVIEW 2 major objections 6 minor 47 references

No coding agent yet matches human developers on both correctness and profitability when implementing real LLVM peephole optimizations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 07:43 UTC pith:S7DALXEM

load-bearing objection Solid empirical systems paper: a leakage-resistant InstCombine benchmark plus a clear validity/profitability trade-off that no current agent closes. the 2 major comments →

arxiv 2607.02684 v1 pith:S7DALXEM submitted 2026-07-02 cs.SE

Can Coding Agents Implement Missed Compiler Optimizations? Evaluating LLM Agents on LLVM Peephole Optimizations

Hongxu Xu , Chunhao Liao , Xintong Zhou , Chengnian Sun This is my paper
classification cs.SE
keywords coding agentsLLVMInstCombinepeephole optimizationcompiler optimizationLLM evaluationbehavioral validityAlive2
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether today's coding agents can implement genuine compiler optimizations, not merely produce patches that pass ordinary software tests. It builds PeepholeBench from 19 real missed peephole optimizations reported against LLVM's InstCombine pass, giving each agent only the issue context that existed before the human fix. Measured against those human-written patches, every evaluated agent falls short on at least one of two requirements: semantic validity or non-regressive profitability. The typical failures are patches that overfit the reported example and mishandle LLVM-specific rules—one-use guards, poison flags, fold order—that existing test suites rarely catch. The finding matters because it shows that agentic help for compiler work will need stronger semantic validation inside the agent loop, not just the ability to edit code and run checks.

Core claim

Across 19 real missed InstCombine optimizations, no evaluated coding agent simultaneously matches human-written patches on both behavioral validity and profitability. Sonnet 4.6 reaches human-level validity rates yet produces profitable rewrites for substantially fewer programs; Gemini 3 Flash matches human profitability at a modest validity cost. The dominant failure modes are under-generalization of the reported pattern and misuse of LLVM-specific mechanisms that conventional regression tests do not expose.

What carries the argument

PeepholeBench: a task suite that freezes each LLVM repository at the base commit of a merged pull request, supplies only pre-fix issue text, Alive2 examples, and InstCombine guidelines, then judges agent patches by mutation-generated tests, Alive2 refinement for correctness, and llvm-mca cost estimates for non-regressive profitability.

Load-bearing premise

The central comparison treats llvm-mca cycle and micro-operation counts on one x86 backend, plus Alive2 checks on generated mutants, as a faithful proxy for real correctness and performance of target-independent rewrites.

What would settle it

A re-evaluation of the same 19 tasks in which one agent simultaneously matches or exceeds the human patches on both measured validity and profitability rates under multi-architecture backend measurement or exhaustive expert review of every rewrite would falsify the claimed trade-off.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Passing existing LLVM regression tests is not sufficient evidence that an agent-written optimization is correct or profitable.
  • Agent loops for compiler work need integrated mutation testing, Alive2-style validation, and profitability checks rather than FileCheck alone.
  • Agents and human contributors can complement each other: agents can surface missed generalizations and flag human profitability regressions.
  • Future agents must handle LLVM-specific conventions (one-use guards, flags, fold interactions) that issue reports leave underspecified.
  • Benchmarks that score only test-suite passage will overestimate agent readiness for compiler optimization tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The observed validity–profitability trade-off suggests current models optimize different implicit objectives; multi-objective agent designs may be required to close both gaps at once.
  • Similar under-generalization failures are likely in other highly conventionalized codebases—kernel schedulers, database query optimizers—where local pattern-matching rules dominate.
  • Closing the loop with direct SMT feedback (Alive2-style) during editing could eliminate many of the correctness failures without waiting for human review.
  • Measuring real-world frequency of IR patterns, which the paper proposes for prioritization, would also let agents focus effort on high-impact rather than merely reported misses.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces PeepholeBench, a benchmark of 19 real LLVM InstCombine missed-optimization tasks drawn from post-Sept-2025 issues and merged PRs, and evaluates three state-of-the-art coding agents (Claude Code/Sonnet 4.6, Gemini CLI/Gemini 3 Flash, Codex CLI/GPT-5.4 and mini) on implementing the corresponding peephole rewrites. Agents receive only pre-fix issue context, Alive2 examples when available, and InstCombine contributor guidelines; patches are judged not only by buildability and llvm-lit seed tests but by behavioral validity (Alive2 refinement plus non-regressive llvm-mca cost on x86_64) and by comparison to human patches on a large mutation-generated suite. The central empirical claim is a validity–profitability trade-off: no agent simultaneously matches human-level behavioral validity and human-level profitability, with dominant failure modes of under-generalization and misuse of LLVM-specific mechanisms (one-use guards, flags, fold interactions) that seed tests rarely expose.

Significance. If the results hold, the paper makes a concrete and timely contribution to both agentic software engineering and compiler tooling. It supplies a realistic, leakage-aware benchmark for a domain where test-suite pass rates are known to be insufficient, pairs that benchmark with a behavioral evaluation pipeline (mutation generation + Alive2 + llvm-mca), and produces actionable failure themes with high inter-rater agreement (Fleiss κ=0.834). Strengths that should be credited explicitly include the post-cutoff PR selection, three independent runs with Pass@1/Pass@3, direct head-to-head comparison against accepted human patches on the same mutants, honest reporting that human patches themselves are not always non-regressive (Table VI), and the public release of PeepholeBench. These make the work useful both as a measurement artifact and as a roadmap for tighter agent–validator loops in optimizer development.

major comments (2)
  1. [§IV-B1, Table IV] §IV-B1 and Table IV: Metrics V and P are all-or-nothing over up to ~1,000 mutants per PR. A single failing mutant zeros an entire run, which is a defensible safety bar but makes Pass@3 hard to interpret. The paper should report, for runs that fail V or P, the fraction (and preferably the distribution) of mutants that fail, so readers can distinguish near-misses from broad regressions. Without this, the gap between seed-test passing and behavioral validity may overstate practical agent capability on the easier majority of cases.
  2. [§IV-C, Table V] §IV-C and Table V: Human profitability is only 16.4% of retained mutants, and most agents sit near 9%. The manuscript does not adequately explain whether this low base rate is expected (most mutants are precondition/edge variants that should leave cost unchanged) or an artifact of the mutation distribution. Because Finding 6 and the “matches human profitability” claim for Gemini 3 Flash rest on these percentages, the paper needs a short characterization of how many mutants are cost-neutral under a correct fold versus truly optimization-sensitive, and whether equal-weight averaging across PRs (some with very different seed/mutant mixes) is appropriate.
minor comments (6)
  1. [§IV-B1, §IV-C] Clarify earlier and more explicitly the two different profitability baselines: RQ1’s P is relative to the human-patch output, while RQ2’s profitability is relative to the unpatched base commit. A one-sentence callout in §IV-B1 and §IV-C would prevent misreading of Findings 3 vs. 6.
  2. [Table II, §IV-A1] Table II lists PR dates into 2026; ensure the camera-ready version states the exact mining cutoff and snapshot date so the post-knowledge-cutoff claim remains auditable as models evolve.
  3. [Fig. 1] Figure 1’s agent patch (1d) is a strong illustration of under-generalization; consider adding a one-line note in the caption that m_LShr vs. m_Shr and the missing low-bit-mask arm are the precise gaps later coded as the dominant theme in §IV-D1a.
  4. [§V] §V’s “agents produce larger patches (+27.1–43.9 net lines vs. +18.1)” is useful but appears without a supporting table or per-agent breakdown; a small appendix table would make the refactoring implication falsifiable.
  5. [§IV-D1c, §III-D] Minor wording: “truncto icmp” and similar missing spaces appear in §IV-D1c; also normalize “x86 64” vs. “x86_64” throughout.
  6. [§VI] Related work could more sharply distinguish PeepholeBench from LPO [14]: LPO discovers candidate rewrites; this work evaluates end-to-end agent implementation from issue reports under human-comparable constraints. One additional sentence would help.

Circularity Check

0 steps flagged

No significant circularity: purely empirical agent evaluation against external human patches, Alive2, and llvm-mca oracles.

full rationale

PeepholeBench constructs tasks from real post-cutoff LLVM issues/PRs, supplies only pre-fix issue context, and scores agent patches via independent external tools (Alive2 refinement for correctness; llvm-mca cycles/uops on x86_64 for non-regressive profitability) plus mutation-generated tests. Human patches serve as an external comparative baseline, not as a fitted parameter or definitional input. Behavioral validity (Def. III.2) and profitability (Def. III.1) are defined independently of agent outputs; inconclusive Alive2 cases are excluded rather than redefined. The sole self-citation (LPO [14]) appears only in Related Work as a discovery method contrast and is not load-bearing for any success metric, validity claim, or failure-mode taxonomy. No equation, metric, or central finding reduces by construction to its own inputs, fitted values, or author-only uniqueness theorems. The paper is self-contained empirical measurement.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The central empirical claims rest on standard LLVM semantics, the soundness of Alive2 refinement, and the usefulness of llvm-mca as a static profitability proxy, plus a small set of mutation hyper-parameters chosen by the authors. No free parameters are fitted to produce the headline trade-off; the invented entity is the benchmark itself.

free parameters (2)
  • mutation generation hyper-parameters (K=5 iterations, N=200 mutants/iteration, T=100·N attempts)
    Chosen by the authors to bound runtime while producing up to 1 000 mutants per PR; different values would change the exact validity/profitability percentages though not necessarily the qualitative trade-off.
  • Alive2 inconclusive threshold (>10 % exclusion)
    PRs with more than 10 % inconclusive Alive2 results are dropped; the cutoff is a design choice that affects the final set of 19 PRs.
axioms (4)
  • domain assumption Alive2 correctly decides LLVM IR refinement (or reports inconclusive) under the semantics used by InstCombine
    Used as the sole semantic oracle for behavioral validity (Definition III.2); paper acknowledges incompleteness and timeouts.
  • domain assumption llvm-mca cycle and uop counts on x86_64 are a sufficient proxy for the profitability of target-independent InstCombine rewrites
    Definition III.1 and all profitability comparisons rest on this; threats section notes possible architecture differences.
  • domain assumption The 19 selected post-2025-09 InstCombine PRs are representative of the class of missed peephole optimizations that agents would be asked to implement
    Selection criteria (labels, merge date, Alive2 coverage) define the population; generalization beyond this set is assumed.
  • domain assumption Standard pattern-matching and one-use conventions of LLVM InstCombine are the correct coding style against which agents are judged
    Agents are scored against human patches that follow the contributor guide; alternative but still correct styles may be penalized by FileCheck or fold-order interactions.
invented entities (1)
  • PeepholeBench no independent evidence
    purpose: Provide a fixed set of 19 real missed-optimization tasks with pre-fix repository state, mutation suite, and behavioral-validity oracle so that coding agents can be compared fairly to human patches.
    The benchmark is constructed by the authors; its value is demonstrated by the empirical results rather than by independent external validation at the time of writing.

pith-pipeline@v1.1.0-grok45 · 24842 in / 2843 out tokens · 30136 ms · 2026-07-12T07:43:27.926755+00:00 · methodology

0 comments
read the original abstract

Coding agents built on large language models are now capable of patching sizable real-world codebases, yet whether they can develop compiler optimizations remains an open question. To study this question, we introduce PeepholeBench, an evaluation framework whose tasks are constructed from real-world missed peephole optimizations reported against LLVM's InstCombine pass. Since missed peephole optimizations are typically fixed with small, localized patches, they offer a well-scoped but demanding testbed for coding agents: a correct fix demands rigorous reasoning about program semantics along with familiarity with optimizer-specific conventions. PeepholeBench derives its tasks from 21 resolved LLVM issues and 19 merged pull requests (PRs), supplies agents with only the issue context that existed before each fix, and assesses the resulting patches for both correctness and profitability. With PeepholeBench, we benchmark state-of-the-art coding agents on fixing missed peephole optimizations in LLVM's InstCombine pass, measuring their patches against the corresponding human-written fixes. We observe a tension between correctness and profitability, and no agent matches human developers on both dimensions at once. The dominant failure modes are overly narrow transformations and misuse of LLVM-specific mechanisms, errors that existing test suites rarely expose. Together, these results establish PeepholeBench as a realistic and challenging benchmark for coding agents, and suggest future directions for building agents that can more dependably assist compiler optimization development.

Figures

Figures reproduced from arXiv: 2607.02684 by Chengnian Sun, Chunhao Liao, Hongxu Xu, Xintong Zhou.

Figure 1
Figure 1. Figure 1: Missed InstCombine optimization addressed by LLVM PR #157030. The human patch handles both power-of-two and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the PeepholeBench workflow. To establish profitability, an optimization must yield a measurable performance gain. Even a semantically correct op￾timization may be rejected if it increases compiler complexity without making the generated code meaningfully faster [12]. This profitability can be estimated statically using tools like the LLVM Machine Code Analyzer (llvm-mca), which pre￾dicts the CP… view at source ↗
Figure 4
Figure 4. Figure 4: Seed tests in #158097 lack coverage of @use, masking a profitability regression where the agent introduces a redun￾dant fcmp instead of reusing the existing one. TABLE V: Percentage of test cases that are valid, profitable, and generalized-profitable relative to the human patch, aver￾aged over three runs per PR and then across PRs with equal weight. The values support relative comparison under the same gen… view at source ↗
Figure 5
Figure 5. Figure 5: A test case for #157030 where @llvm.fake.use creates a second use of %lshr. The fold fires but cannot eliminate %lshr, leaving it live alongside the new icmp eq and causing a profitability regression. b) Agent generalization beyond the human patch: We revisit #158097 from the generalization angle (see § IV-B3b for the test coverage perspective). The human patch imple￾ments the redundant-fcmp fold only for … view at source ↗
Figure 6
Figure 6. Figure 6: Canonicalization issue: mul with swapped operands; the non-canonical form (top) generates one extra mov. b) Canonicalization Issue: Sometimes, the IR produced by the agent patch has the same instruction sequence as the human patch’s output but in a different canonical form, such as using a different operand order or comparison predicate. These differences may trigger different lowering paths in LLVM’s back… view at source ↗
Figure 7
Figure 7. Figure 7: Fold interaction failure on #157030. define ptr @fail_source_gep_multiple_swap(i1 %x, ptr %y) { %a = select i1 %x, i64 21304, i64 55104 %g1 = getelementptr inbounds nuw i8, ptr %y, i64 %a call void @use(ptr %g1) %g2 = getelementptr i8, ptr %g1, i64 8148 ret ptr %g2 } [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 9 linked inside Pith

  1. [1]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inThe Twelfth International Conference on Learning Representations, 2023

    2023

  2. [2]

    Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    2024

  3. [3]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1592–1604

    2024

  4. [4]

    Openhands: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singhet al., “Openhands: An open platform for ai software developers as generalist agents,” inThe Thirteenth International Conference on Learning Representations, 2024

    2024

  5. [5]

    Gemini CLI documentation — geminicli.com,

    “Gemini CLI documentation — geminicli.com,” https://geminicli.com/ docs/, [n. d.], [Accessed 01-05-2026]

    2026

  6. [6]

    Claude Code overview - Claude Code Docs — code.claude.com,

    “Claude Code overview - Claude Code Docs — code.claude.com,” https: //code.claude.com/docs/en/overview, [n. d.], [Accessed 01-05-2026]

    2026

  7. [7]

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces,

    M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, L. Shi, J. Y . Shin, T. Walshe, E. K. Buchanan, G. Ye, H. Lin, J. Poulos, M. Wang, D. Lu, O. M. Mastromichalakis, Z. Xu, Z. Chen, Y . Liu, R. Zhang, L. L. Chen, A. Kashyap, J.-L. Uslu, J. Li, J. Wu, M. Yan, S. Bian, V . Sharma, K. Sun, A. Anand, A. Lanpouthakoun, B. Koopah, C. Hu, E. Guha, G. H. S. Dre...

    2026

  8. [8]

    Featurebench: Benchmarking agentic coding for complex feature development,

    Q. Zhou, J. Zhang, H. Wang, R. Hao, J. Wang, M. Han, Y . Yang, S. Wu, F. Pan, L. Fanet al., “Featurebench: Benchmarking agentic coding for complex feature development,” inThe Fourteenth International Conference on Learning Representations, 2026

    2026

  9. [9]

    Souper: A synthesizing superoptimizer,

    R. Sasnauskas, Y . Chen, P. Collingbourne, J. Ketema, G. Lup, J. Taneja, and J. Regehr, “Souper: A synthesizing superoptimizer,”arXiv preprint arXiv:1711.04422, 2017

    Pith/arXiv arXiv 2017

  10. [10]

    Alive2: bounded translation validation for llvm,

    N. P. Lopes, J. Lee, C.-K. Hur, Z. Liu, and J. Regehr, “Alive2: bounded translation validation for llvm,” inProceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, 2021, pp. 65–79

    2021

  11. [11]

    Provably correct peephole optimizations with alive,

    N. P. Lopes, D. Menendez, S. Nagarakatte, and J. Regehr, “Provably correct peephole optimizations with alive,” inProceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, 2015, pp. 22–32

    2015

  12. [12]

    InstCombine contributor guide — LLVM 23.0.0git documen- tation — llvm.org,

    “InstCombine contributor guide — LLVM 23.0.0git documen- tation — llvm.org,” https://llvm.org/docs/InstCombineContributorGuide. html, [n. d.], [Accessed 01-05-2026]

    2026

  13. [13]

    Optimization-directed compiler fuzzing for continuous translation validation,

    J. Kwon, B. Jang, J. Lee, and K. Heo, “Optimization-directed compiler fuzzing for continuous translation validation,”Proceedings of the ACM on Programming Languages, vol. 9, no. PLDI, pp. 627–650, 2025

    2025

  14. [14]

    Lpo: Discovering missed peephole optimizations with large language models,

    Z. Xu, H. Xu, Y . Tian, X. Zhou, and C. Sun, “Lpo: Discovering missed peephole optimizations with large language models,” inProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 2026, pp. 1136–1150

    2026

  15. [15]

    How to contribute to LLVM — Red Hat Developer — web.archive.org,

    N. Popov, “How to contribute to LLVM — Red Hat Developer — web.archive.org,” https://web.archive.org/web/20260313105502/https:// developers.redhat.com/articles/2022/12/20/how-contribute-llvm, [n. d.], [Accessed 01-05-2026]

    arXiv 2022

  16. [16]

    CLI – Codex — OpenAI Developers — developers.openai.com,

    “CLI – Codex — OpenAI Developers — developers.openai.com,” https: //developers.openai.com/codex/cli, [n. d.], [Accessed 01-05-2026]

    2026

  17. [17]

    llvm-mca - LLVM Machine Code Analyzer — LLVM 23.0.0git documentation — llvm.org,

    “llvm-mca - LLVM Machine Code Analyzer — LLVM 23.0.0git documentation — llvm.org,” https://llvm.org/docs/CommandGuide/ llvm-mca.html, [n. d.], [Accessed 02-05-2026]

    2026

  18. [18]

    Llvm: A compilation framework for lifelong program analysis & transformation,

    C. Lattner and V . Adve, “Llvm: A compilation framework for lifelong program analysis & transformation,” inInternational symposium on code generation and optimization, 2004. CGO 2004.IEEE, 2004, pp. 75–86

    2004

  19. [19]

    Peephole optimization,

    W. M. McKeeman, “Peephole optimization,”Communications of the ACM, vol. 8, no. 7, p. 443–444, 1965

    1965

  20. [20]

    C. N. Fischer and R. J. LeBlanc Jr,Crafting a Compiler with C. Benjamin-Cummings Publishing Co., Inc., 1991

    1991

  21. [21]

    Hydra: Generalizing peephole optimiza- tions with program synthesis,

    M. Mukherjee and J. Regehr, “Hydra: Generalizing peephole optimiza- tions with program synthesis,”Proceedings of the ACM on Programming Languages, vol. 8, no. OOPSLA1, p. 725–753, Apr. 2024

    2024

  22. [22]

    lit - LLVM Integrated Tester — LLVM 23.0.0git documen- tation — llvm.org,

    “lit - LLVM Integrated Tester — LLVM 23.0.0git documen- tation — llvm.org,” https://llvm.org/docs/CommandGuide/lit.html, [Ac- cessed 09-05-2026]

    2026

  23. [23]

    FileCheck - Flexible pattern matching file verifier — LLVM 23.0.0git documentation — llvm.org,

    “FileCheck - Flexible pattern matching file verifier — LLVM 23.0.0git documentation — llvm.org,” https://llvm.org/docs/{C} ommand{G}uide/{F}ile{C}heck.html, [Accessed 14-05-2026]

    2026

  24. [24]

    LLVM Language Reference Manual — LLVM 23.0.0git doc- umentation — llvm.org,

    “LLVM Language Reference Manual — LLVM 23.0.0git doc- umentation — llvm.org,” https://llvm.org/docs/{L}ang{R}ef.html, [Ac- cessed 21-06-2026]

    2026

  25. [25]

    LLVM IR Undefined Behavior (UB) Manual — LLVM 23.0.0git documentation — llvm.org,

    “LLVM IR Undefined Behavior (UB) Manual — LLVM 23.0.0git documentation — llvm.org,” https://llvm.org/docs/{U} ndefined{B}ehavior.html, [Accessed 21-06-2026]

    2026

  26. [26]

    Taming undefined behavior in LLVM,

    J. Lee, Y . Kim, Y . Song, C.-K. Hur, S. Das, D. Majnemer, J. Regehr, and N. P. Lopes, “Taming undefined behavior in LLVM,”ACM SIGPLAN Notices, vol. 52, no. 6, pp. 633–647, 2017

    2017

  27. [27]

    Using thematic analysis in psychology,

    V . Braun and V . Clarke, “Using thematic analysis in psychology,” Qualitative research in psychology, vol. 3, no. 2, pp. 77–101, 2006

    2006

  28. [28]

    Llm-in-the-loop: Leveraging large language model for thematic analysis,

    S.-C. Dai, A. Xiong, and L.-W. Ku, “Llm-in-the-loop: Leveraging large language model for thematic analysis,” inFindings of the association for computational linguistics: EMNLP 2023, 2023, pp. 9993–10 001

    2023

  29. [29]

    Thematic-lm: a llm-based multi-agent system for large-scale thematic analysis,

    T. Qiao, C. Walker, C. Cunningham, and Y . S. Koh, “Thematic-lm: a llm-based multi-agent system for large-scale thematic analysis,” in Proceedings of the ACM on Web Conference 2025, 2025, pp. 649–658

    2025

  30. [30]

    Measuring nominal scale agreement among many raters

    J. L. Fleiss, “Measuring nominal scale agreement among many raters.” Psychological bulletin, vol. 76, no. 5, p. 378, 1971

    1971

  31. [31]

    Models overview — platform.claude.com,

    “Models overview — platform.claude.com,” https://platform.claude. com/docs/en/about-claude/models/overview, [Accessed 11-05-2026]

    2026

  32. [32]

    Models — Gemini API — Google AI for Developers — ai.google.dev,

    “Models — Gemini API — Google AI for Developers — ai.google.dev,” https://ai.google.dev/gemini-api/docs/models, [Accessed 11-05-2026]

    2026

  33. [33]

    Models — OpenAI API — developers.openai.com,

    “Models — OpenAI API — developers.openai.com,” https://developers. openai.com/api/docs/models, [Accessed 11-05-2026]

    2026

  34. [34]

    The measurement of observer agreement for cate- gorical data,

    G. Landis JRKoch, “The measurement of observer agreement for cate- gorical data,”Biometrics, vol. 33, no. 1, p. 159174, 1977

    1977

  35. [35]

    The Often Misunderstood GEP Instruction — LLVM 23.0.0git documentation — llvm.org,

    “The Often Misunderstood GEP Instruction — LLVM 23.0.0git documentation — llvm.org,” https://llvm.org/docs/{G}et{E}lement{P} tr.html, [Accessed 28-06-2026]

    2026

  36. [36]

    Evaluating large 11 language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large 11 language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  37. [37]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...

    2021

  38. [38]

    Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering?

    S. Miserendino, M. Wang, T. Patwardhan, and J. Heidecke, “Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering?”arXiv preprint arXiv:2502.12115, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Agentic harness for real- world compilers,

    Y . Zheng, C. Li, S. Li, Y . Zhang, and Z. Su, “Agentic harness for real- world compilers,”arXiv preprint arXiv:2603.20075, 2026

    arXiv 2026

  40. [40]

    Rapgen: An ap- proach for fixing code inefficiencies in zero-shot,

    S. Garg, R. Z. Moghaddam, and N. Sundaresan, “Rapgen: An ap- proach for fixing code inefficiencies in zero-shot,”arXiv preprint arXiv:2306.17077, 2023

    Pith/arXiv arXiv 2023

  41. [41]

    Search-based llms for code optimization,

    S. Gao, C. Gao, W. Gu, and M. Lyu, “Search-based llms for code optimization,”arXiv preprint arXiv:2408.12159, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Large language models for compiler optimization,

    C. Cummins, V . Seeker, D. Grubisic, M. Elhoushi, Y . Liang, B. Roziere, J. Gehring, F. Gloeckle, K. Hazelwood, G. Synnaeve, and H. Leather, “Large language models for compiler optimization,” no. arXiv:2309.07062, Sep. 2023, arXiv:2309.07062 [cs]. [Online]. Available: http://arxiv.org/abs/2309.07062

    Pith/arXiv arXiv 2023

  43. [43]

    Com- piler generated feedback for large language models,

    D. Grubisic, C. Cummins, V . Seeker, and H. Leather, “Com- piler generated feedback for large language models,”arXiv preprint arXiv:2403.14714, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Towards llm-based optimization compilers. can llms learn how to apply a single peephole optimization? reasoning is all llms need!

    X. Fang and L. Mukhanov, “Towards llm-based optimization compilers. can llms learn how to apply a single peephole optimization? reasoning is all llms need!” no. arXiv:2412.12163, Dec. 2024, arXiv:2412.12163 [cs]. [Online]. Available: http://arxiv.org/abs/2412.12163

    Pith/arXiv arXiv 2024

  45. [45]

    Finding missed code size optimizations in compilers using llms,

    D. Italiano and C. Cummins, “Finding missed code size optimizations in compilers using llms,”arXiv preprint arXiv:2501.00655, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Ir-optset: An optimization-sensitive dataset for advancing llm- based ir optimizer,

    Z. Yang, L. Qiu, F. Lyu, M. Zhong, Z. Chai, H. Zhou, H. Cui, and X. Feng, “Ir-optset: An optimization-sensitive dataset for advancing llm- based ir optimizer,”Advances in Neural Information Processing Systems, vol. 38, 2026

    2026

  47. [47]

    Beyond pass- by-pass optimization: Intent-driven ir optimization with large language models,

    L. Qiu, Z. Yang, F. Lyu, M. Zhong, H. Cui, and X. Feng, “Beyond pass- by-pass optimization: Intent-driven ir optimization with large language models,”arXiv preprint arXiv:2602.18511, 2026. 12

    arXiv 2026