REVIEW 19 cited by
Learning Performance-Improving Code Edits
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
With the decline of Moore's law, optimizing program performance has become a major focus of software research. However, high-level optimizations such as API and algorithm changes remain elusive due to the difficulty of understanding the semantics of code. Simultaneously, pretrained large language models (LLMs) have demonstrated strong capabilities at solving a wide range of programming tasks. To that end, we introduce a framework for adapting LLMs to high-level program optimization. First, we curate a dataset of performance-improving edits made by human programmers of over 77,000 competitive C++ programming submission pairs, accompanied by extensive unit tests. A major challenge is the significant variability of measuring performance on commodity hardware, which can lead to spurious "improvements." To isolate and reliably evaluate the impact of program optimizations, we design an environment based on the gem5 full system simulator, the de facto simulator used in academia and industry. Next, we propose a broad range of adaptation strategies for code optimization; for prompting, these include retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include performance-conditioned generation and synthetic data augmentation based on self-play. A combination of these techniques achieves a mean speedup of 6.86 with eight generations, higher than average optimizations from individual programmers (3.66). Using our model's fastest generations, we set a new upper limit on the fastest speedup possible for our dataset at 9.64 compared to using the fastest human submissions available (9.56).
Forward citations
Cited by 19 Pith papers
-
Multi-Source and Cross-Scenario Strategy-Guided Code Optimization
MoST improves LLM-guided code optimization by clustering optimization strategies from heterogeneous knowledge sources and transferring them across programming languages.
-
JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java
JETO-Mine and JETO-Bench provide a configurable, Docker-based pipeline and dataset of 91 verified Java execution-time improvement patches, on which OpenHands fixes 13 of 91 issues (14.3%).
-
CppPerf: An Automated Pipeline and Dataset for Performance-Improving C++ Commits
CppPerf-Mine produces CppPerf-DB, a benchmark of 347 real-world performance-improving C++ patches (39% multi-file) from 42 repositories to evaluate repository-level repair tools.
-
Watts This Smell: A Comprehensive Taxonomy of Software Energy Smells
The paper creates and empirically validates a language-agnostic taxonomy of 12 primary software energy smells and 65 root causes using literature review and energy profiling of 21,000 Python code pairs.
-
PerfCoder: Large Language Models for Interpretable Code Performance Optimization
PerfCoder is a family of LLMs trained on optimization trajectories with human annotations and runtime-based preference alignment that achieves higher runtime speedups and optimization rates on the PIE benchmark than p...
-
RLPF: Reinforcement Learning from Performance Feedback for Code Generation
RLPF's staged performance reward lifts Qwen3-32B on PerfCodeBench from 11.1% to 54.6% correct-and-runnable and from 8.1% to 38.6% relative efficiency.
-
PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization
A profiler-guided, verifier-in-the-loop workflow more than doubles the rate at which an off-the-shelf LLM agent matches human-expert speedups on two repository-level code-optimization benchmarks.
-
JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java
JETO-Mine is a reusable three-phase pipeline that mines 1.8 million Java commits to produce JETO-Bench containing 91 verified executable ETIPs, on which OpenHands succeeds at 14.3%.
-
Chiseling Out Efficiency: Structured Skeleton Supervision for Efficient Code Generation
EffiSkel improves LLM-generated code efficiency by supervising on extracted structural efficiency skeletons via multi-task learning of code generation and skeleton prediction.
-
Optimas: An Intelligent Analytics-Informed Generative AI Framework for Performance Optimization
Optimas deploys a multi-agent LLM workflow to convert performance diagnostics into correct code transformations, delivering 100% valid code and performance gains in 98.82% of 3,410 experiments across benchmarks and HP...
-
Watts This Smell: A Comprehensive Taxonomy of Software Energy Smells
A literature-derived taxonomy of 12 energy smells and 65 root causes is mapped to measured energy differences in thousands of equivalent Python code pairs.
-
SemOpt: LLM-Driven Code Optimization via Rule-Based Analysis
SemOpt generates Semgrep static-analysis rules from LLM-summarized optimization commits and uses them to locate and apply optimization strategies, outperforming retrieval-based baselines on C/C++ code.
-
TRPrompt: Bootstrapping Query-Aware Prompt Optimization from Textual Rewards
TRPrompt trains an 8B prompt model directly on natural-language textual rewards and reports the highest accuracies on GSMHard and MATH among the compared methods.
-
Rethinking LLM-Based RTL Code Optimization Via Timing Logic Metamorphosis
LLM-based RTL optimizers degrade on timing-heavy mutants, but the study's own data and methods do not fully support the headline claim.
-
SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
All 18 evaluated LLMs pass fewer SIMD-intrinsic code-generation tests than scalar-code tests on the new SimdBench benchmark, with the largest drops on SVE and RVV.
-
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
LiveCodeBench collects 400 recent contest problems to create a contamination-free benchmark evaluating LLMs on code generation and related capabilities like self-repair and execution.
-
Teaching Large Language Models to Self-Debug
Self-Debugging teaches LLMs to identify and fix their own code errors through rubber-duck-style natural language explanations and execution feedback, delivering 2-12% gains over baselines on Spider, TransCoder, and MBPP.
-
Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs
The abstract claims a new local search framework for code generation, but the manuscript body is a different mathematics paper.
-
Self-Refine: Iterative Refinement with Self-Feedback
Self-Refine boosts LLM outputs by ~20% on average across seven tasks by having the same model iteratively generate, critique, and refine its own responses.
Discussion (0). Sign in to comment.