Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

The paper claims that a compiler-specific agent harness lifts LLM resolution of LLVM middle-end bugs by about 22%, while showing that its own test oracle accepts two to three times as many patches as an expert reviewer.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 02:29 UTC pith:YGMKGKGJ

load-bearing objection A genuinely useful first harness and benchmark for LLM compiler-bug repair, with an honest expert review that undercuts its own headline numbers — but the abstract overstates and the corrected rates rest on a single reviewer. the 3 major comments →

arxiv 2603.20075 v2 pith:YGMKGKGJ submitted 2026-03-20 cs.SE cs.AI

Agentic Harness for Real-World Compilers

Yingwei Zheng , Cong Li , Shaohua Li , Yuqun Zhang , Zhendong Su This is my paper
classification cs.SE cs.AI
keywords LLVMcompiler bug repairLLM agentsagentic harnessbenchmarkmiscompilationpatch validationexpert review
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Fixing bugs in LLVM's optimization passes is currently beyond what general software-repair agents achieve, so the paper builds a compiler-specific harness—tools for reproducing, debugging, and validating LLVM middle-end bugs—along with a 334-bug benchmark and a minimal agent built on those tools. Evaluated on the recent subset with five frontier models, the harness's agent beats the generic agent by about 22% per model, and the best configuration reaches 51.5% "resolved" by the automated test oracle. The paper's own expert review is the second half of the claim: fewer than 42% of test-passing patches are correct, so the best genuinely-resolved rate is 20.1%. If right, this means compiler repair is a distinct agentic domain where specialized interfaces help, and that test-passing benchmarks for compilers should be read as upper bounds on true capability, not as repair rates.

Core claim

The paper's central object is the harness it builds—called llvm-autofix in the body, llvm-harness in the abstract—a set of agent-friendly wrappers around LLVM's middle end: build, reproduce, cause extraction (stack trace or Alive2 counterexample), gdb debugging, editing, and llvm-lit/FileCheck validation. Around it sits llvm-bench, a 334-issue benchmark of crashes and miscompilations, and llvm-autofix-mini, a minimal four-stage agent. The claim is that this combination beats a generic software-repair agent by roughly 22% per model, and that the best frontier model resolves 51.5% of the recent 229-issue subset under the automated oracle. The expert-review part of the claim is that the same or

What carries the argument

The load-bearing artifact is the harness itself: agent-accessible tools wrapping LLVM's middle end. For crash bugs it pins down the stack trace; for miscompilations it uses Alive2 to produce a counterexample; it launches LLVM under gdb at a breakpoint placed just before the first transformation; and it validates patches through llvm-lit/FileCheck. The four-stage agent—Setup, Reason, Generate, Validate—spends its budget on root-cause analysis before editing, and the benchmark's difficulty splits (easy, medium, hard) make the oracle's leniency visible across model and agent combinations.

Load-bearing premise

The load-bearing premise is that passing the reproducer and LLVM's regression tests means a patch fixes the bug; the paper itself shows this oracle accepts incorrect patches, so if that premise fails the headline percentages are upper bounds, not actual repair rates.

What would settle it

Have two or more independent LLVM maintainers review the same accepted patches and measure label agreement. If agreement on 'correct' falls below roughly 90%, the paper's expert-verified rates are not stable; alternatively, run the best configuration's accepted patches through a differential-testing campaign against random IR programs and check whether semantic mismatches appear—if they do, the regression-test oracle is indeed too lenient.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the roughly 22% agent-level gain is real, compiler-specific interfaces are a necessary ingredient for LLM-based compiler repair, and generic agents are a weaker baseline than previously assumed.
  • Because fewer than 42% of test-passing patches are correct, any future 'resolved' rate on LLVM-style benchmarks should be accompanied by expert validation; test-passing alone overstates true fixes by a factor of roughly 1.5 to 3.
  • Hard-split issues are effectively unsolved—only one correct patch across all configurations—so multi-file compiler fixes are beyond current LLM agent capability.
  • The systematic failure modes (assertion weakening, wrong localization, narrow wrong fixes) imply that benchmarks should explicitly detect assertion edits and reward generalization beyond reproducers.
  • Model-specific failures, such as one frontier model's inability to follow the harness's tool-call format, mean harness design must trade rich tooling against instruction-following reliability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit: if the same inflation factor applies to other test-only compiler benchmarks, the field's reported LLM repair rates are systematically optimistic, and cross-model comparisons are confounded by how much each model 'cheats' the oracle.
  • The single-expert review is the only correctness gate; a testable extension would be to have several independent reviewers rate the same accepted patches. If inter-rater agreement is low, the 20.1% figure should be treated as a point estimate, not a firm bound.
  • A cheap ablation the paper does not run: disable the gdb/debug stage and remeasure. If the drop is large, dynamic state inspection—not just the tool wrappers—is the active ingredient, which would transfer to other systems-level repair tasks.
  • Because the live benchmark deliberately uses only the most recent year's issues, the data-leakage defense is temporal rather than causal; a stronger control would randomly hold out issues and check whether the post-cutoff advantage persists.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents llvm-autofix, an agentic harness for LLM-based repair of LLVM middle-end bugs, consisting of agent-friendly tools (build, reproduce, debug, edit, test), a benchmark llvm-bench of 334 reproducible bugs (with a live subset of 229 recent issues), and a minimal agent llvm-autofix-mini. The authors evaluate five LLMs with mini-SWE-agent and llvm-autofix-mini. They report that resolution rates on llvm-bench live drop by about 62% relative to SWE-bench Verified, that llvm-autofix-mini outperforms the mini-SWE-agent baseline by an aggregate factor of 1.22×, and that after expert review fewer than 42% of accepted patches are correct, putting the best genuine resolution rate at 20.1%. The paper also discusses common LLM failure modes and open challenges for compiler patch validation.

Significance. If the results hold, this is a substantial contribution. The paper provides a reproducible benchmark and harness for an under-served, high-impact domain (compiler bug repair), and it offers a rare expert-reviewed estimate of genuine capability, showing that LLVM's regression-test oracle overstates resolution rates by a factor of 1.5–3×. The benchmark construction uses external ground truth (upstream LLVM commits, alive2 counterexamples) and the evaluation protocol is documented in unusual detail (prompts, budgets, costs, failure taxonomy). The main comparative claim — that llvm-autofix-mini beats the mini-SWE-agent baseline for four of five models — appears robust even after expert correction. However, the abstract contains internally inconsistent and unsupported numbers, and the single-reviewer expert ground truth needs validation before the 'genuine capability' statistics can be taken at face value.

major comments (3)
  1. [Abstract; §4.1, Table 2; §4.2, Table 3] The abstract states that 'their performance improves by 62%' with the harness, but the only 62% figure in the body is the average reduction in resolution rate from SWE-bench Verified to llvm-bench live (Table 2). The agent improvement over mini-SWE-agent is reported as an aggregate '1.22×' (Table 3), not 62%. This is internally contradictory. Additionally, the abstract calls the tool 'llvm-harness' while the body uses 'llvm-autofix', and the GitHub URLs differ. The abstract must be corrected and aligned with the body.
  2. [§4.3, Table 4; §C.4] The 'genuine' resolution rates rest entirely on a single expert's binary labels, with no rubric, no inter-rater reliability check, and no confidence interval. Since §4.4 concedes that LLVM regression tests are inadequate for validating agent-generated patches, the expert label is the sole ground truth for the corrected rates. The analogy in §C.4 to standard LLVM code review is not a substitute for benchmark-label reliability. I request a second reviewer, a publicly documented rubric, or at least uncertainty bounds; without these, the claim that 'true capability consistently remains below 22%' is not robustly supported.
  3. [Abstract; §3] The abstract and introduction describe mini-SWE-agent as 'state-of-the-art' and claim that llvm-autofix-mini outperforms it by 22%. However, §3 deliberately excludes production-oriented agents such as Gemini CLI and Codex and positions mini-SWE-agent as a 'canonical baseline.' The 'state-of-the-art' claim is therefore unsupported by the experiments. Please rephrase to 'outperforms the mini-SWE-agent baseline' and avoid implying a comparison with stronger production agents that were not tested.
minor comments (5)
  1. [§2.2] The sentence 'These issues directly affect 64 out of 184 middle-end components in LLVM, with 176 affected indirectly' is numerically confusing (64 + 176 > 184). Please clarify whether 176 is the total pipeline count or the number additionally affected by the 64 directly affected components.
  2. [Appendix E.2, E.3] The string '<empth>' appears in the opt_stderr placeholder and should be '<empty>'.
  3. [§4.2] The sentence 'The bug repair capability of the four frontier models is generally better—improved by around 16% when executed with llvm-autofix-mini, compared with GPT 4o' is unclear. Specify what is being compared and how the 16% is computed.
  4. [§C.1] The McNemar tests are one-sided and multiple comparisons (five tests) are not adjusted. Report effect sizes and adjusted p-values, or explicitly state that these are exploratory.
  5. [§C.4, Table 8] The post-release data-leakage analysis uses very small subsets (14 and 21 issues for GPT-5 and Gemini). The higher post-release rates for llvm-autofix-mini may be due to small-sample variance; please add a caveat.

Circularity Check

0 steps flagged

No significant circularity; benchmark ground truth and correctness labels are external to the harness.

full rationale

The paper's derivation chain is self-contained with respect to circularity. llvm-bench is constructed from externally fixed upstream LLVM issues with golden patches taken from developer commits (A.1), and Stage II/III validation uses opt, alive2, and LLVM regression tests—external tools rather than the agents' outputs. Agent performance is measured as patches passing those same external tests, and the subsequent 'genuine capability' metric depends on a human LLVM expert's independent semantic review (§4.3), not on any quantity fitted inside the paper. No parameter is fit to a subset of data and then 'predicted' on a closely related quantity; all reported rates are direct measurements. The only self-citation (Li et al. 2024) is background support for the claim that certain components are bug-prone and is not load-bearing. Concerns about the abstract's '62%' phrasing, the single-expert review reliability, and the weakness of LLVM's regression tests admitted in §4.4 are correctness/validity risks, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

This is an empirical systems paper, so its 'free parameters' are design and budget choices that shape measured outcomes rather than fitted constants. The load-bearing choices are: the acceptance oracle (§3), the one-year live window, the split definitions, and the agent's breakpoint heuristic. The axioms are the unverified premises of the measurement chain (oracle validity, golden-patch validity, single expert, leaderboard baselines, model stability). No new physical or mathematical entities are postulated; the harness, tools, benchmark, and mini-agent are software artifacts whose evidence status depends on the public repo, which is referenced under two different names/handles.

free parameters (4)
  • Agent budget hyperparameters (token limit, edit limit, test limit, chat limit, context window) = 5M tokens; 25 edit calls; 25 online test calls; 500 chat rounds; 64K-token context (§3)
    Hand-set limits directly bound how many trajectories succeed; Fig. 7 shows resolution rises with token limit for llvm-autofix-mini, so the headline gap between agents is partly a function of these chosen limits.
  • llvm-bench-live time window = Issues from the most recent year: 229 issues as of Aug 26, 2025 (§2.2)
    The 'continually updated' one-year window is a hand-picked compromise between leakage avoidance and currency; it fixes which 229 issues constitute every headline rate.
  • Difficulty split labels (easy/medium/hard) = easy 76.3% (1 function), medium 13.2% (multi-function, 1 file), hard 10.5% (multi-file) (§A.2)
    Splits are defined by the golden patch's edit footprint, so the finding 'performance degrades on harder splits' is partly true by construction rather than an empirical discovery about model capability.
  • llvm-autofix-mini breakpoint placement = Pause before the crashing function (crashes) or before the first transformation (miscompilations) (§2.3)
    A heuristic 'based on our experience in fixing LLVM bugs'; the agent's 22-145% advantage over mini-SWE-agent depends on this placement being informative for typical reproducers.
axioms (6)
  • domain assumption An accepted patch (passing reproducers + component regression tests via llvm-lit/FileCheck and alive2) correctly resolves the bug.
    Stated in §2.1 Test & Validate and §3 ('a patch is considered an accepted patch if it passes all applied online and offline tests'). §4.3/Table 4 refutes this for correctness: 21-42% of accepted patches are correct. The paper knowingly keeps this as the automated metric.
  • domain assumption Upstream LLVM fixing commits are valid golden patches and their parent commits reproduce the bug.
    Benchmark construction Stages I-III (§A.1) filters on fixing commits and validates golden patches; no independent semantic check is applied to the golden patches themselves beyond the same test oracle.
  • domain assumption A single LLVM-maintainer expert review of every accepted patch is ground truth for genuine capability.
    §4.3 and §C.4; the paper argues this matches LLVM's merge-review standard, but genuine rates would shift if a second reviewer disagreed (no inter-rater agreement measured).
  • standard math alive2 counterexamples correctly certify miscompilations.
    Used in Reproduce & Cause (§2.1) to produce the 'direct cause' given to both agents; alive2 is bounded translation validation, sound for the counterexamples it reports within its bounds.
  • domain assumption SWE-bench Verified leaderboard numbers are valid comparison baselines without rerunning in the same harness environment.
    §4.1/Table 2 copy SWEV numbers 'verbatim from the leaderboard'; §C.4 concedes the cross-benchmark comparison 'should be interpreted qualitatively'. The 62%-decline claim rests on this.
  • domain assumption Model API behavior is stable over the evaluation window, or identifiable when it shifts.
    Footnote 3 documents a silent DeepSeek V3.2 update mid-experiment that contaminated results and forced a rerun; the same failure mode could affect other models undetected.

pith-pipeline@v1.3.0-alltime-deepseek · 29178 in / 22021 out tokens · 209093 ms · 2026-08-03T02:29:23.150245+00:00 · methodology

0 comments
read the original abstract

Compilers are critical to modern computing, yet fixing compiler bugs is difficult. While recent large language model (LLM) advancements enable automated bug repair, compiler bugs pose unique challenges due to their complexity, deep cross-domain expertise requirements, and sparse, non-descriptive bug reports, necessitating compiler-specific harnesses. To bridge the gap, we introduce llvm-harness, the first harness designed to assist LLM agents in understanding and fixing compiler bugs. Our current focus is on the middle end of LLVM, one of the most widely used compiler infrastructures. Central to llvm-harness are agent-friendly LLVM tools, a benchmark llvm-bench of 334 reproducible LLVM middle-end bugs, and a tailored mini agent llvm-autofix-mini for fixing LLVM middle-end bugs automatically. We evaluate five frontier models and find that they exhibit a performance decline when tackling compiler bugs with the state-of-the-art agent. With llvm-harness' enhancement, their performance improves by 62%. Our specialized mini agent llvm-autofix-mini further outperforms the llvm-harness-enhanced state-of-the-art by 22%. This emphasizes the necessity for specialized harnesses like ours to assist LLMs in compiler engineering tasks. Despite promising results, our expert review also reveals several open challenges that remain when applying LLMs for compiler engineering tasks. GitHub: https://github.com/dtcxzyw/llvm-harness

Figures

Figures reproduced from arXiv: 2603.20075 by Cong Li, Shaohua Li, Yingwei Zheng, Yuqun Zhang, Zhendong Su.

Figure 1
Figure 1. Figure 1: Compiler issues are challenging to diagnose and repair, in the absence of descriptive information. This is a comparison between LLVM issues (crash and miscompilation) and common software issues from Django; all three issues are simplified for brevity. powering languages from C/C++ and Rust to modern AI languages like Triton and Mojo. At present, we target its middle-end, which transforms an LLVM IR program… view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of Affected Components in llvm-bench [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: As splits increase in difficulty, frontier models tend to struggle or fail. G4, G5, GM, QW, and DS are short for GPT 4o, GPT 5, Gemini 2.5 Pro, Qwen 3 Max, and DeepSeek V3.2, respectively. 4.2. Baseline Comparison and Common Failures To investigate the practicality of llvm-autofix and the usefulness of llvm-autofix-mini, we compare llvm-autofix-mini against mini-SWE-agent [PITH_FULL_IMAGE:figures/full_fig… view at source ↗
Figure 4
Figure 4. Figure 4: Failure distribution of unresolved issues. G4, G5, GM, QW, and DS are short for GPT 4o, GPT 5, Gemini 2.5 Pro, Qwen 3 Max, and DeepSeek V3.2, respectively [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Miscompilations are more challenging than crashes. The Y-axis is “% Resolved” before expert review. G4, G5, GM, QW, and DS are short for GPT 4o, GPT 5, Gemini 2.5 Pro, Qwen 3 Max, and DeepSeek V3.2, respectively. 0 1 2 3 4 5M 0 15 30 45 60 mini-SWE-agent 0 1 2 3 4 5M llvm-autofix-mini G4 G5 GM QW DS [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Tool Call Distrib. (Per Issue) [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Understanding Agent-Based Patching of Compiler Missed Optimizations

    cs.SE 2026-07 unverdicted novelty 6.0

    A benchmark of real LLVM missed optimizations shows agent patches frequently have partial or incomplete scope overlap with developer patches, improved by retrieval and distillation of historical pull requests.

  2. Understanding Agent-Based Patching of Compiler Missed Optimizations

    cs.SE 2026-07 conditional novelty 6.0

    Agents often fix reported LLVM missed-optimization cases yet misalign with developer-intended scope; historical PR retrieval and distillation improve alignment and real-world IR hits.

Reference graph

Works this paper leans on

20 extracted references · 2 linked inside Pith · cited by 1 Pith paper

  1. [1]

    **Understanding the Issue**: Explain what the problem is and why it manifests

  2. [2]

    **Analyzing`opt`'s Log**: Highlight any relevant transformations, analysis results, or unexpected behavior observed in the`opt`output

  3. [3]

    **Root Cause Analysis**: Connect the observations to the underlying cause in the code

  4. [4]

    Assume all assertions are correct and investigate preceding code or conditions

    **Proposed Edit Point(s)** + Each edit point should be at least 1 lines long + NOTICE (on assertion failure): Assertion failures typically indicate earlier errors in execution. Assume all assertions are correct and investigate preceding code or conditions. Edit points can contain but cannot be limited to assertion statements

  5. [5]

    **Conclusion**: Summarize the fix and its expected effect. E.2. System Prompt Example:llvm-autofix-mini’s Generate Stage You are an expert LLVM developer. I'm encountering an LLVM bug that I haven't been able to fix. Your goal is to generate a patch that fixes the LLVM bug based on the information below. ## Bug Information ## ### Reproducer ### ```llvm de...

  6. [6]

    **Submit**: Provide the final, clean patch for review. E.3. System Prompt Example:mini-SWE-agent You are an expert LLVM developer. Please solve this LLVM issue: ------ BEGIN ISSUE ------ Type: miscompilation Reproducer (LLVM IR):```bash cat /tmp/test.ll define half @fabs_select_fabs(half noundef %x) { entry: %abs1 = call half @llvm.fabs.f16(half %x) %cmp ...

  7. [8]

    You may use provided tools to explore the codebase for more context

    **Analyze**: Analyze the provided information to fully understand the bug's cause and effect. You may use provided tools to explore the codebase for more context

  8. [9]

    Explain your reasoning and the specific changes you intend to make

    **Propose a Fix**: Outline your proposed solution. Explain your reasoning and the specific changes you intend to make. 3. **Implement**: Use the`edit`tool to apply your proposed changes to the code

  9. [10]

    + Is valid and does not modify any assertions in the code

    **Verify**: Use the`test`tool to confirm that your patch: + Is syntactically correct and does not introduce new syntax errors. + Is valid and does not modify any assertions in the code. + Fixes the original issue shown in the reproducer. + Does not introduce any new regressions

  10. [11]

    **Iterate**: If verification fails, analyze the failure, revise your proposal, and repeat the implementation and verification steps until all tests pass

  11. [13]

    Analyze the codebase by finding and reading relevant files

  12. [14]

    Edit the source code to resolve the issue

  13. [15]

    Verify your fix works by running the`submit-patch`command

  14. [16]

    Based on the output of`submit-patch`, repeat steps 1-3 as necessary until the issue is resolved ## Important Rules

  15. [17]

    Every response must contain exactly one action 26 Agentic Harness for Real-World Compilers

  16. [18]

    The action must be enclosed in triple backticks

  17. [19]

    Every action is executed in a new subshell

    Directory or environment variable changes are not persistent. Every action is executed in a new subshell. However, you can prefix any action with`MY_ENV_VAR=MY_VALUE cd /path/to/working/dir && ...`or write/load environment variables from files

  18. [20]

    Solving this issue does not require deleting any files or accessing the internet, so you are not permitted to use tools such as: rm, curl, wget, git, ssh, scp, ftp, telnet, ping, traceroute, nslookup, dig, nmap, apt, apt-get, yum, dnf, pacman, brew. <system_information> Linux 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 </system_...

    2025

  19. [2024]

    % Resolved

    URL https://openai.com/index/ introducing-swe-bench-verified/ . Accessed: Jan 17th 2026. OpenAI. Codex, 2025. URL https://github.com/ openai/codex. Accessed: Jan 17th 2026. Ou, X., Li, C., Jiang, Y ., and Xu, C. The mutators reloaded: Fuzzing compilers with large language model generated mutation operators. InProceedings of the 2024 29th ACM International...

    Pith/arXiv arXiv 2026

  20. [2025]

    9 Agentic Harness for Real-World Compilers Munley, C., Jarmusch, A., and Chandrasekaran, S

    URLhttps://arxiv.org/abs/2504.20412. 9 Agentic Harness for Real-World Compilers Munley, C., Jarmusch, A., and Chandrasekaran, S. LLM4VV: Developing LLM-driven testsuite for com- piler validation.Future Gener. Comput. Syst., 2024. Ni, Y . and Li, S. Interleaving large language models for com- piler testing.Proc. ACM Program. Lang., (OOPSLA2), 2025. OpenAI....

    Pith/arXiv arXiv 2024