REVIEW 17 cited by
Learning Performance-Improving Code Edits
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
With the decline of Moore's law, optimizing program performance has become a major focus of software research. However, high-level optimizations such as API and algorithm changes remain elusive due to the difficulty of understanding the semantics of code. Simultaneously, pretrained large language models (LLMs) have demonstrated strong capabilities at solving a wide range of programming tasks. To that end, we introduce a framework for adapting LLMs to high-level program optimization. First, we curate a dataset of performance-improving edits made by human programmers of over 77,000 competitive C++ programming submission pairs, accompanied by extensive unit tests. A major challenge is the significant variability of measuring performance on commodity hardware, which can lead to spurious "improvements." To isolate and reliably evaluate the impact of program optimizations, we design an environment based on the gem5 full system simulator, the de facto simulator used in academia and industry. Next, we propose a broad range of adaptation strategies for code optimization; for prompting, these include retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include performance-conditioned generation and synthetic data augmentation based on self-play. A combination of these techniques achieves a mean speedup of 6.86 with eight generations, higher than average optimizations from individual programmers (3.66). Using our model's fastest generations, we set a new upper limit on the fastest speedup possible for our dataset at 9.64 compared to using the fastest human submissions available (9.56).
Forward citations
Cited by 17 Pith papers
-
Multi-Source and Cross-Scenario Strategy-Guided Code Optimization
MoST improves LLM-guided code optimization by clustering optimization strategies from heterogeneous knowledge sources and transferring them across programming languages.
-
JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java
JETO-Mine is a reusable three-phase pipeline that mines 1.8 million Java commits to produce JETO-Bench containing 91 verified executable ETIPs, on which OpenHands succeeds at 14.3%.
-
RLPF: Reinforcement Learning from Performance Feedback for Code Generation
RLPF's staged performance reward lifts Qwen3-32B on PerfCodeBench from 11.1% to 54.6% correct-and-runnable and from 8.1% to 38.6% relative efficiency.
-
PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization
A profiler-guided, verifier-in-the-loop workflow more than doubles the rate at which an off-the-shelf LLM agent matches human-expert speedups on two repository-level code-optimization benchmarks.
-
SemOpt: LLM-Driven Code Optimization via Rule-Based Analysis
SemOpt generates Semgrep static-analysis rules from LLM-summarized optimization commits and uses them to locate and apply optimization strategies, outperforming retrieval-based baselines on C/C++ code.
-
TRPrompt: Bootstrapping Query-Aware Prompt Optimization from Textual Rewards
TRPrompt trains an 8B prompt model directly on natural-language textual rewards and reports the highest accuracies on GSMHard and MATH among the compared methods.
-
Rethinking LLM-Based RTL Code Optimization Via Timing Logic Metamorphosis
LLM-based RTL optimizers degrade on timing-heavy mutants, but the study's own data and methods do not fully support the headline claim.
-
SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
All 18 evaluated LLMs pass fewer SIMD-intrinsic code-generation tests than scalar-code tests on the new SimdBench benchmark, with the largest drops on SVE and RVV.
-
Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization
Reinforcement learning with execution feedback enables a code model to iteratively improve the efficiency of its own generated code, surpassing supervised and preference-based training methods.
-
AuPair: Golden Example Pairs for Code Repair
Providing a different submodularly selected (guess, fix) example on each repair call improves LLM code repair over best-of-N and self-repair.
-
ACECODER: Acing Coder RL via Automated Test-Case Synthesis
ACECODER shows that automatically synthesized test cases can serve as scalable reward signals for code RL, yielding consistent gains on HumanEval, MBPP, BigCodeBench, and LiveCodeBench.
-
CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
CWEval is a new benchmark that simultaneously checks functional correctness and security of AI-generated code with dynamic test oracles, exposing a large correct-but-insecure gap in current LLMs.
-
Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs
The abstract claims a new local search framework for code generation, but the manuscript body is a different mathematics paper.
-
Optimizing Code Runtime Performance through Context-Aware Retrieval-Augmented Generation
An LLM code optimizer using control-flow-graph differences and retrieved examples reports 7.3% average runtime reduction on 116 C++ programs versus zero-shot GPT-4o.
-
Distilling Desired Comments for Enhanced Code Review with Large Language Models
Desiview identifies desired review comments from code review datasets using the perplexity difference of the actual fix with and without each comment, and the distilled data improves LLaMA-based code review models.
-
ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Models
Fine-tuning code LLMs with PPO and an execution-based reward that combines test results with runtime relative to a reference solution improves both correctness and efficiency on EffiBench+.
-
Exploring the Potential of Llama Models in Automated Code Refinement: A Replication Study
A replication study finds that a 7-billion-parameter open-source CodeLlama model, tuned with low temperature and specific prompts, can match ChatGPT on a code-refinement similarity metric while keeping code on local hardware.
Discussion (0). Continue with ORCID to comment.