{"paper":{"title":"PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Current LLMs produce code that is functionally correct but far from expert-optimized on system-level performance tasks.","cross_cats":["cs.CL","cs.PL"],"primary_cat":"cs.SE","authors_text":"Hanyu Yang, Haochen Shi, Haoran Li, Huihao Jing, Shaojin Chen, Sirui Zhang, Wenbin Hu, Yangqiu Song","submitted_at":"2026-05-13T08:10:26Z","abstract_excerpt":"Large language models (LLMs) can often generate functionally correct code, but their ability to produce efficient implementations for performance-critical systems tasks remains limited. Existing code benchmarks mainly emphasize correctness or algorithmic problem solving, while realistic systems-level optimization is still underexplored. To address this gap, we introduce PerfCodeBench, an executable benchmark for evaluating LLMs on high-performance code optimization. The tasks require system-level implementation choices, hardware-aware optimization, and careful handling of performance bottlenec"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Our evaluation on a broad set of state-of-the-art LLMs shows a clear gap between model-generated code and expert-optimized implementations. The gap is especially large on tasks involving parallelism and GPU operations.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The selected tasks accurately capture realistic system-level implementation choices, hardware-aware optimizations, and performance bottlenecks that matter in practice.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"PerfCodeBench reveals that state-of-the-art LLMs produce functionally correct but significantly slower code than expert-optimized versions on system-level tasks, especially those involving parallelism and GPUs.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Current LLMs produce code that is functionally correct but far from expert-optimized on system-level performance tasks.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"ce9f3414af28d8893fc1a2f1c87a7f262d6036acd0ada58b77999e174696a3b1"},"source":{"id":"2605.15222","kind":"arxiv","version":1},"verdict":{"id":"56bb79b6-e790-4b12-8c5b-d5dd3b17467a","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-19T17:58:50.521493Z","strongest_claim":"Our evaluation on a broad set of state-of-the-art LLMs shows a clear gap between model-generated code and expert-optimized implementations. The gap is especially large on tasks involving parallelism and GPU operations.","one_line_summary":"PerfCodeBench reveals that state-of-the-art LLMs produce functionally correct but significantly slower code than expert-optimized versions on system-level tasks, especially those involving parallelism and GPUs.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The selected tasks accurately capture realistic system-level implementation choices, hardware-aware optimizations, and performance bottlenecks that matter in practice.","pith_extraction_headline":"Current LLMs produce code that is functionally correct but far from expert-optimized on system-level performance tasks."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2605.15222/integrity.json","findings":[],"available":true,"detectors_run":[{"name":"claim_evidence","ran_at":"2026-05-19T20:21:57.158243Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"doi_title_agreement","ran_at":"2026-05-19T18:31:18.819737Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"doi_compliance","ran_at":"2026-05-19T18:10:16.387371Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"ai_meta_artifact","ran_at":"2026-05-19T13:33:22.835500Z","status":"skipped","version":"1.0.0","findings_count":0}],"snapshot_sha256":"17bae781a0bec8d0c469624434c6193720fefc03dae918aa25262292e1530d19"},"references":{"count":52,"sample":[{"doi":"","year":2025,"title":"Introducing Claude Opus 4.5","work_id":"785c69cd-ef52-4c1c-80f5-3c8645ec6dbc","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2026,"title":"Claude Code Overview","work_id":"dcfdde82-d743-457f-b8f4-1f23552dc365","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2026,"title":"Claude model overview","work_id":"114f4f5b-3359-458c-ba85-d8e90a92db5e","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"Understanding software engineering agents: A study of thought-action-result trajectories","work_id":"ddc723de-4578-4bab-a626-5da26d2d16f7","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2026,"title":"ByteDance Seed. Seed2.0.https://seed.bytedance.com/en/seed2, 2026","work_id":"d73a7341-aa04-4bbf-8412-53ceacca5313","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":52,"snapshot_sha256":"948508292d948b51f13bfef70f9292e6f97b56534db1d54f33a298acfb604e6c","internal_anchors":4},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}