Pith. sign in

REVIEW 14 cited by

How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.06647 v4 pith:OHB5QLO3 submitted 2024-06-10 cs.SE cs.AIcs.LG

Ruizhong Qiu , Weiliang Will Zeng , James Ezick , Christopher Lott , Hanghang Tong This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords codeefficiencyllmsbenchmarkefficientevaluationalgorithmsenamel
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The emergence of large language models (LLMs) has significantly pushed the frontiers of program synthesis. Advancement of LLM-based program synthesis calls for a thorough evaluation of LLM-generated code. Most evaluation frameworks focus on the (functional) correctness of generated code; efficiency, as an important measure of code quality, has been overlooked in existing evaluations. In this work, we develop ENAMEL (EfficeNcy AutoMatic EvaLuator), a rigorous and high-standard benchmark for evaluating the capability of LLMs in generating efficient code. Firstly, we propose a new efficiency metric called eff@k, which generalizes the pass@k metric from correctness to efficiency and appropriately handles right-censored execution time. Furthermore, we derive an unbiased and variance-reduced estimator of eff@k via Rao--Blackwellization; we also provide a numerically stable implementation for the new estimator. Secondly, to set a high-standard for efficiency evaluation, we employ a human expert to design best algorithms and implementations as our reference solutions of efficiency, many of which are much more efficient than existing canonical solutions in HumanEval and HumanEval+. Moreover, to ensure a rigorous evaluation, we employ a human expert to curate strong test case generators to filter out wrong code and differentiate suboptimal algorithms. An extensive study across 30 popular LLMs using our benchmark ENAMEL shows that LLMs still fall short of generating expert-level efficient code. Using two subsets of our problem set, we demonstrate that such deficiency is because current LLMs struggle in designing advanced algorithms and are barely aware of implementation optimization. Our benchmark is publicly available at https://github.com/q-rz/enamel .

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reinforcement Learning for Code Optimization

    cs.LG 2026-07 conditional novelty 7.0 of 10

    On the new DMC-Optim benchmark, a staged correctness-plus-speed RL recipe lifts top-50% pass@1 from 18.0% to 31.3% (Qwen 2.5 7B) and 30.7% to 50.4% (CWM 32B) while keeping pure-correctness pass@1 roughly stable.

  2. EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A new six-language benchmark for code efficiency shows that LLMs produce correct but measurably slower code than expert baselines, with the best model at about 62 percent of human runtime efficiency.

  3. PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A profiler-guided, verifier-in-the-loop workflow more than doubles the rate at which an off-the-shelf LLM agent matches human-expert speedups on two repository-level code-optimization benchmarks.

  4. Correct but Slow: An Empirical Study of the GPU Kernel Evaluation Gap in Modern Domain-Specific Languages

    cs.SE 2026-07 unverdicted novelty 6.0 of 10

    Correctness-only evaluation of Triton and TileLang kernels admits slowdowns over 300× versus library baselines; library-relative efficiency and roofline utilization flag all such failures in a 22-kernel suite.

  5. Combining TSL and LLM to Automate REST API Testing: A Comparative Study

    cs.SE 2025-09 conditional novelty 6.0 of 10

    RestTSLLM combines Test Specification Language with few-shot prompting to generate REST API integration tests, and a comparative study finds Claude 3.5 Sonnet the best among eight LLMs.

  6. Evaluating Efficiency and Novelty of LLM-Generated Code for Graph Analysis

    cs.SE 2025-07 conditional novelty 6.0 of 10

    In a benchmark of LLM-generated C code for graph analysis, Claude Sonnet 4 Extended produced the most correct, fastest, and most memory-efficient implementations, beating human baselines on triangle counting.

  7. Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization

    cs.SE 2025-05 conditional novelty 6.0 of 10

    Reinforcement learning with execution feedback enables a code model to iteratively improve the efficiency of its own generated code, surpassing supervised and preference-based training methods.

  8. Directed Greybox Fuzzing via Large Language Model

    cs.CR 2025-05 reject novelty 6.0 of 10

    HGFuzzer uses an LLM to generate harnesses, reachable seeds, and custom mutators for directed greybox fuzzing, and reports triggering 17/20 known vulnerabilities plus finding 9 new ones with CVE IDs.

  9. SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence

    cs.CL 2025-02 conditional novelty 6.0 of 10

    SelfElicit uses deep-layer attention to automatically highlight relevant evidence sentences in the input context, yielding consistent QA accuracy gains across six instruction-tuned LLMs.

  10. CKGFuzzer: LLM-Based Fuzz Driver Generation Enhanced By Code Knowledge Graph

    cs.SE 2024-11 conditional novelty 6.0 of 10

    CKGFuzzer uses a code knowledge graph to guide LLM agents in generating, repairing, and mutating fuzz drivers, reporting a pooled 8.73% relative coverage gain over PromptFuzz on eight libraries and 9 new bugs.

  11. Breaking Silos: Adaptive Model Fusion Unlocks Better Time Series Forecasting

    cs.LG 2025-05 conditional novelty 5.0 of 10

    A learnable fusor that reads meta-features of an input time series and weights 13 pre-trained forecasters per sample outperforms each individual model on most benchmark samples, including zero-shot settings.

  12. Optimizing Code Runtime Performance through Context-Aware Retrieval-Augmented Generation

    cs.SE 2025-01 conditional novelty 5.0 of 10

    An LLM code optimizer using control-flow-graph differences and retrieved examples reports 7.3% average runtime reduction on 116 C++ programs versus zero-shot GPT-4o.

  13. Energy-Aware Code Generation with LLMs: Benchmarking Small vs. Large Language Models for Sustainable AI Programming

    cs.SE 2025-08 reject novelty 4.0 of 10

    On 150 LeetCode problems, GPT-4.0 and DeepSeek-Reasoner beat three 3B-parameter models on correctness and speed; the 52% energy-efficiency claim counts any of three SLMs on correct outputs, not a per-model advantage.

  14. CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation

    cs.SE 2025-04 reject novelty 4.0 of 10

    CoCo-Bench is a new multi-language benchmark of 705 manually reviewed samples that evaluates code LLMs across understanding, generation, modification, and review tasks, with difficulty-weighted scoring.

Pith tools